Cheng's Blog

<img src='image.png'>

The Markdown syntax won't work in this case, you have to use raw html like above.

Reference:
http://stackoverflow.com/questions/10628262/inserting-image-into-ipython-notebook-markdown

To my suprise, it is actually super easy to install MongoDB if you have brew installed:

brew install mongodb

Then, you have to set a path for MongoDB to store its data:

mkdir -p ~/data/db          # <- create a folder at any location
mongod --dbpath ~/data/db   # <- let mongodb know where you created the folder
mongod                      # <- run MongoDB server, use Ctrl + C to terminate it

To connect to the MongoDB server, open another terminal window:

mongo

That's it!

To import a JSON file into MongoDB:

mongoimport --db *database_name* --collection *collection_name* --type json --file *file.json*

For example:

mongoimport --db test --collection beijing --type json --file beijing_map.json

Mike Bostock (the author of D3.js) gave a talk about 'Design is a Search Problem'. Here is the distill of the talk:

Prototypes should emphasize speed over polish

It needn't look good, or even have labels. Make just enough to evaluate the idea. Then decide whether to go straight or turn. Identify the intent of the prototype. What hypothesis are you testing?

Transition from exploring to refining near deadline.

Reduce the amount of explorations as deadline approaches and start to focus on what's working.

Delete code as you go. Be ruthless.

Delete branches as you go (Mike uses git branches to manage his work).

Make your process reproducible.

A build system provides machine-readable documentation. Accelerate the reuse of parts from past projects. Here is the blog link he is refering to.

Try bad ideas deliberately.

You can't evaluate a visualization absent the data.Don't get too attached to your current favorite. Don't get stuck at local maximum; go down to go up. Design is chaotic.

I am using uikit for a web app that I am creating for my team. I really like the Stage demo's search field and want to mimic it. I extracted the CSS from the demo:

.uk-search:before {
  content: "\f002";
  position: absolute;
  top: 0;
  left: 0;
  width: 30px;
  line-height: 40px;
  text-align: center;
  font-family: FontAwesome;
  font-size: 14px;
  color: #444;
}

.uk-search-field {
  -moz-box-sizing: border-box;
  box-sizing: border-box;
  margin: 0;
  border-radius: 0;
  font: inherit;
  color: #777;
  -webkit-appearance: none;
  width: 30px;
  height: 40px;
  padding: 0 0 0 30px;
  border: 1px solid rgba(0, 0, 0, 0);
  background: rgba(0, 0, 0, 0);
  -webkit-transition: all linear 0.2s;
  transition: all linear 0.2s;
  vertical-align: middle;
  position: relative;
  padding: 0;
  cursor: pointer;
  z-index: 1;
}

.uk-search-field:focus{
    outline:0;
    width:140px;
    padding: 0 30px;
    cursor: text;
}

That's it. Now the search field's textbox can be hidden when it is not focused and slide out when it is focused.

Linking to the original post

Summary

1. I don't know anything
   Don't limit your thinking to dogma's and the conventional thinking. Judge yourself if other people's opinion is a valuable experience to learn from or simply a view limited by his/her own experience. Quotes directly from Paul:

   If you believe yourself to be a rational person, then you're in the trap.

   Things didn't work in the past doesn't mean it won't work in the future. Likewise, what worked before may not work again.

2. Kill all daemon process
   Computers run faster after restart (because the elimination of long running daemon processes in the background), so does human mind. Purposely creating silent moments everyday to eliminate the voices of "doubt, anger and others' opinions" that drag down our spirite.

3. Yes, and thank you.
   When dealing with setbacks, either setbacks from life or negative feedbacks from others, say "Yes, and thank you." It maintains the "forward flow of life". (Don't be stalled for long)

4. Choose the more interesting path
   This resonates with "The joy is in creating, not maintaining". For things that you can already see the outcome is not interesting. For things that are completely random is not interesting. Interesting things have to be genuinely interesting to you and by doing it you can discover something new.

5. Love what you do
   Real work usually comes with unpleasant or dull moments. Love what you do allows you to sustain and work with them.

6. Maintain a healthy disregard for the impossible
   Encourage feel thinking and doing the impossible.

In this part of the post, I would like to document the steps needed to run an existing Django project on gunicorn as the backend server and nginx as the reverse proxy server. (Please refer to Part 1 of this post)

Disclaimer: this post is based on this site but I added a lot of details that wasn't explained.

Setup nginx

Before starting the nginx server, we want modify its config file. I installed nginx via brew on my machine, and the conf file is located here:

/usr/local/etc/nginx/nginx.conf

We can modify this file directly to add configurations for our site, but this is not a good idea. Because you may have multiple web apps running behind this nginx server. Each web app may need its own configuration. To do this, let's create another folder for storing these site-wise configuration files first:

cd /usr/local/etc/nginx
mkdir sites-enabled

We can store our site specific config here but wouldn't it be better if we store the config file along with our project files together? Now, let's navigate to our project folder. (I named my project testproject and stored it under /Users/webapp/Apps/testproject)

cd /Users/webapp/Apps/testproject
touch nginx.conf

Here is my config file:

server {
    listen 80;
    server_name your_server_ip;

    access_log /Users/webapp/logs/access.log;     # <- make sure to create the logs directory 
    error_log /Users/webapp/logs/error.log;       # <- you will need this file for debugging

    location / {
        proxy_pass http://127.0.0.1:9000;         # <- let nginx pass traffic to the gunicorn server
    }
    
    location /static {
        root /Users/webapp/Apps/testproject/vis;  # <- let nginx serves the static contents
    }
}

Let me elaborate on the '/static' part. This part means that any traffic to 'your_server_ip/static' will be forwarded to '/Users/webapp/Apps/testproject/vis/static'. You might ask why doesn't it forward to '/Users/webapp/Apps/testproject/vis'?(without '/static' in the end) Because when using 'root' in the config, it will append the '/static' part after it. So be aware! You can fix this by using alias instead of root and append /static to the end of the path:

location /static {
    alias /Users/webapp/Apps/testproject/vis/static; 
}

Here is the folder structure of my project :

/Users/webapp/Apps/testproject/
    manage.py           <- the manage.py file generated by Django
    nginx.conf          <- the nginx config file for your project
    gunicorn.conf.py    <- the gunicorn config file that we will create later, just keep on reading
    testproject/        <- automatically generated by Django
        settings.py
        urls.py
        wsgi.py         <- automatically generated by Django and used by gunicorn later
    vis/                <- the webapp that I wrote
        admin.py
        models.py
        test.py
        urls.py
        template/
            vis/
                index.html
        static/        <- the place where I stored all of the static files for my project
            vis/
                css/
                images/
                js/

All of the static files are in the /testproject/vis/static folder, so that's where nginx should be looking. You might ask that the static files live in their own folders rather than right under the /static/ path. How does nginx know where to fetch them? Well, this is not nginx's problem to solve. It is your responsibility to code the right path in your template. This is what I wrote in my template/vis/index.html page:

href="{% static 'vis/css/general.css' %}

It is likely that you won't get the path right the first time. But that's ok. Just open up Chrome's developer tools and look at the error messages in the console to see which part of the path is messed up. Then, either fix your nginx config file or your template.

To let nginx read our newly create config file:

cd /usr/local/etc/nginx/
nano nginx.conf

Find the ' http { ' header, add this line under it:

http{
    include /usr/local/etc/nginx/sites-enabled/*;

This line tells nginx to look for config files under the 'sites-enabled' folder. Instead of copy our project's nginx.conf into the 'sites-enabled' folder, we can simply create a soft link instead:

cd /usr/local/etc/nginx/site-enabled
ln -s /full_path/to/your/django_project a_name

# in my case, this is what my link command looks like:
# ln -s /Users/webapp/Apps/testproject/nginx.conf testproject
# this would create a soft link named testproject which points to the real config file

Once this is done, you can finally start up the nginx server:

To start nginx, use

sudo nginx

To stop it, use

sudo nginx -s stop

To reload the config file without shutting down the server:

sudo nginx -s reload

Please refer to this page for a quick overview of the commands.

Setup gunicorn

Setting up gunicorn is more straight forward (without considering optimization). First, let's write a config file for gunicorn. Navigate to the directory which contains your manage.py file, for me, this is what I did:

cd /Users/webapp/Apps/testproject
touch gunicorn.conf.py  # yep, the config file is a python script 

This is what I put in the config file:

bind = "127.0.0.1:9000"                   # Don't use port 80 becaue nginx occupied it already. 
errorlog = '/Users/webapp/logs/gunicorn-error.log'  # Make sure you have the log folder create
accesslog = '/Users/webapp/logs/gunicorn-access.log'
loglevel = 'debug'
workers = 1     # the number of recommended workers is '2 * number of CPUs + 1' 

Save the file and to start gunicorn, make sure you are at the same directory level as where the manage.py file is and do:

gunicorn -c gunicorn.conf.py testproject.wsgi

The ' -c ' option means read from a config file. The testproject.wsgi part is actually referring to the wsgi.py file in a child folder. (Please refer to my directory structure above)

Just in case if you need to shutdown gunicorn, you can either use Ctrl + c at the console or if you lost connection to the console, use [1]:

kill -9 `ps aux | grep gunicorn | awk '{print $2}'`

Actually, a better way to run and shutdown gunicorn is to make it a daemon process so that the server will still run even if you log out of the machine. To do that, use the following command:

gunicorn -c gunicorn.conf.py testproject.wsgi --pid ~/logs/gunicorn.pid --daemon

This command will do three things:

1. run gunicorn with the configuration file named gunicorn.conf.py
2. save the process id of the gunicorn process to a specific file ('~/logs/gunicorn.pid' in this case)
3. run gunicorn in daemon mode so that it won't die even if we log off

To shutdown this daemon process, open '~/logs/gunicorn.pid' to find the pid and use (assuming 12345 is what stored in '~/logs/gunicorn.pid'):

kill -9 12345 

to kill the server.

This is it! Enter 127.0.0.1 in your browser and see if your page loads up. It is likely that it won't load up due to errors that you are not aware of. That's ok. Just look at the error logs:

tail -f error.log

Determine if it is an nginx, gunicorn or your Django project issue. It is very likely that you don't have proper access permission or had a typo in the config files. Just going through the logs and you will find out which part is causing the issue. Depending on how you setup your Django's logging config, you can either read debug messages in the same console which you started gunicorn, or read it form a file. Here is what I have in my Django's settings.py file:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            'datefmt' : "%d/%b/%Y %H:%M:%S"
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'null': {
            'level': 'DEBUG',
            'class': 'logging.NullHandler',
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'verbose'
        },
        'logfile':{
            'level':'DEBUG',
            'class':'logging.handlers.WatchedFileHandler',
            'filename': "/Users/webapp/gunicorn_log/vis.log",
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'django.request': {
            'handlers': ['logfile'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'django': {
            'handlers': ['logfile'],
            'propagate': True,
            'level': 'DEBUG',
        },
        'vis': {
            'handlers': ['console', 'logfile'],
            'level': 'DEBUG',
            'propagate': False,
        },
    }
}

Bad Request 400

Just when you thought everything is ready, and you want the world to see what you have built...BAM! Bad Request 400. Why??? Because when you turn DEBUG = False in the settings.py file, you have to specify the ALLOWED_HOSTS attribute:

DEBUG = False
ALLOWED_HOSTS = [
    '127.0.0.1',
]

Just don't put your port number here, only the ip part. Read more about this issue here: http://stackoverflow.com/questions/19875789/django-gives-bad-request-400-when-debug-false

Conclusion

Setting up nginx, gunicorn and your Django project can be a very tedious process if you are not familiar with any of them like me. I document my approach here to hopefully help anybody who has encountered the same issue.

The goal of this post is to summarize the steps needed to deploy a Django project running on nginx and gunicorn on Mac OSX. I am going to document a list of problems encountered and provide solutions below. (If you are running on Linux, I imagine the process will be very similar)

My setup (the installation order of the following packages matters, just keep on reading. But you don't have to have the exact same versions listed below):

 Python             2.7.9
 pip                6.0.6
 Virtualenv         12.0.4
 VirtualenvWrapper  4.3.1
 Postgres.app       9.4.0.1
 psycopg2           2.5.4
 Django             1.7.3
 gunicorn           19.1.1
 brew               0.9.5
 nginx              1.6.2

First thing first, install Python and pip. This should be the easiest part as the official website provides a installer. So I am not going to elaborate on this part.

Once Python and pip are installed, I recommend to install virtualenv and virtualenvwrapper next.

pip install virtualenv 
pip install virtualenvwrapper

Upon completion, let's create a virtual environment for our later setups:

cd ~                    <- cd to your user directory
mkdir .virtualenvs      <- create an empty folder called .virtualenvs
nano .bash_profile      <- open the .bash_profile file

Copy the following lines into your .bash_profile and save.

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh

Once the file is saved, make sure to reload it:

. .bash_profile         <- Note the space between the two dots

Now we can create a virtual environment to install the rest by:

mkvirtualenv webapp

Now your bash prompt should have (webapp) in front of it, meaning you are currently in the 'webapp' environment.

Next, you should install the Postgres.app. The Postgres.app encapsulate PostgreSQL into a nice installer and makes the installation process extremely easy. Once completed, install psycopg2 (Note: you have to install Postgres.app first before installing psycopg2 otherwise you will (run into an error)[http://cheng.logdown.com/posts/2015/01/17/install-postgresql-and-psycopg2-on-mac-osx])

pip install psycopg2

Then, we can install Django and gunicorn:

pip install django
pip install gunicorn

In the end, we need to rely on brew to install nginx. To install brew, copy and paste this line into your terminal:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Once completed, install nginx:

brew install nginx

Summary of Installation

Let me explain how each aforementioned package fits within the big picture. To make everything easier to understand for novice users like myself, I simplified the functionalities of the pieces below, so please go to their websites for a more accurate description.

1. Virtualenv and Virtualenvwrapper
Virtualenv allows you to create a 'sandbox' that contains all of the Python packages you need for deployment without messing up the system's python environment. Virtualenvwrapper as its name suggests, wraps around virtualenv to provide a list of 'shortcut' commands to reduce the amount of efforts needed to create/switch between/delete virtual environments. The sole purpose of these two pieces is to isolate your deployment environment from other Python environments on the same machine.

To illustrate this issue with an example, I develop and deploy Django projects on the same machine for learning purpose. When I work on a Django project, I use the latest version of Django. But when I deploy it, I use an older version. Without virtualenv, I have to install the latest Django for development and delete it and install an older version for deployment. They can never co-exist at the same time. Virtualenv allows me to create two separate environments with different Django version in each and co-exist nicely.

2. Postgres.app and Psycopg2
The Postgres.app installs the PostgreSQL server on your machine with one click. It also provides a shortcut for you to open up a Terminal as a client that connects to the database server. Psycopg2 is used by Django or any Python program to access the PostgreSQL database programmatically.

3. nginx and gunicorn
Why do we need two servers instead of one？Well, by using gunicorn alone, it is enough to get your Django project up and running. The purpose of adding nginx in front of it is for:

1. Load Balancing - The more servers you have the more users you can serve concurrently. But who should be the one that directs traffic to different servers? Well, the nginx server in this case. (I am aware that I have everything running on a single Mac...)
2. Acceleration - Not all of the contents on your web page are dynamically generated. For instance, CSS style sheets, the logo of your website, javascript libraries like jQuery. We can have a server dedicated for serving these static contents so that our gunicorn server can focus on serving dynamic contents.
3. Security - Users never see the gunicorn server. The only server exposed to the outside world is the nginx server. The gunicorn server will never accept traffics other than the ones forwarded by nginx.

Please read this link to learn more.

The choice of which server to use is arbitrary in this case. You can replace nginx with Apache and gunicorn with (uWSGI)[https://uwsgi-docs.readthedocs.org/en/latest/]. You should Google to find out more about them and see which one fits better with you.

This concludes the first part of this post. Here is the second part but I suggest you taking a break as the second portion is rather long.

By Vince Lombardi:

The joy is in creating, not maintaining

最近需要使用Django开发个Web App. Django官方推荐使用Postgresql作为数据库，所以这里记录一下在Mac OSX上安装的步骤和碰到的问题。

1. 最简单的方式是安装Postgres.App. 这个应用里自带了最新版本的PostgreSQL而且不需要学习数据库服务器启动和关闭的命令。程序安好后（别忘了拖拽到Application的文件夹里），会自动在端口5432开启一个PostgreSQL的服务器。在程序界面里还有一个很贴心的按钮 'Open psql'，可以自动为你在命令行里打开一个客户端与服务器进行连接。而且它会使用你当前的Mac用户名在服务器上为你注册成为一个superuser，让你立刻就可以在数据库进行任何的修改和操作。

2. 通过Python与PostgreSQL建立连接和操作需要安装psycopg2这个库。可以通过一下指令来安装：

$ pip install psycopg2

这个时候会出现一个错误：

Error: pg_config executable not found.

Please add the directory containing pg_config to the PATH

再次求助StackOverflow找到了答案，psycopg2在安装的时候需要pg_config这个程序。这个程序其实已经随着Postgres.app安装到了硬盘上，但是还没有被添加到系统的PATH里。以下是添加方式：

$ cd ~
$ nano .bash_profile

然后在bash_profile里添加, 你可能需要修改版本号码（9.4）：

export PATH=$PATH:/Applications/Postgres.app/Contents/Versions/9.4/bin

保存文件后重新加载bash_profile，注意两个点之间的空格：

$ . .bash_profile

确认pg_config确实可以被系统找到：

$ which pg_config

/Applications/Postgres.app/Contents/Versions/9.4/bin/pg_config

然后再次安装psycopg2就好了

$ pip install psycopg2

以上。

用Python 2.x会经常碰到一个错误:

UnicodeEncodeError: 'ascii' codec can't encode character ... : ordinal not in range(128)

搞清这个问题之前要先理解三个知识点:

1. UTF-8 vs Unicode
2. Encoding vs Decoding
3. Python 2.7里的 str 和 unicode

1. UTF-8 vs Unicode

这一点已经在[之前的博文]里解释过了(http://cheng.logdown.com/posts/2015/01/14/utf-8-vs-unicode:)，这里我来总结一下

• Unicode 只是一个文字与数字之间的映射。比如，'汉' 这个字在Unicode里的代码是 ‘6C49’，想对应的数字就是 27721。地球上每种语言里的每一个文字都有这样一个相对应的数字标识。这个文字与数字的映射表就是 Unicode。
• 当我们把这个映射后的数字存储在计算机上时，需要把它转换成 0 和 1. 我们可以简单的把 27721 转换成二进制代码 ’01101100 01001001‘ 来存储。但问题是计算机怎么能够知道这个两个字节是代表一个文字而不是两个文字? 这个时候就需要再有一种编码形式来告诉计算机将这个字节视为一个文字。这个编码就是UTF-8 (当然，UTF-8只是众多编码中的一种)

可以用这个顺序来理解：

屏幕上看到的文字     Unicode代码     根据UTF-8规范存在计算机内存或者硬盘里的模样
       汉      ->     6C49     ->       11100110 10110001 10001001

2. Encoding vs Decoding

在Python中把一个Unicode类转化为 0 和 1 的过程叫做Encoding。 把 0 和 1 反转为Unicode类的过程叫做Decoding。

在Python 2.7版本里，ASCII是默认的Encoding和Decoding的方法。

3. Python 2.7里的 str 和 unicode

当你把一个字符(不管该字符是英文字母还是ASCII里不能包含的字符)赋值给一个变量时:

han = '汉'

这个变量的类型都会是str:

In [113]:han = '汉'
         type(han)
Out[113]:str

但这里有很重要的一点需要理解：

In [124]:han = '汉'
         bin_han = '\xe6\xb1\x89'
         han == bin_han # 虽然在界面里我们看到的是'汉'这个字，但其实它是一堆字符，并不是我们看到的文字

Out[124]:True

str这个类的本质其实就是原始字符数据（raw byte data）。它并不是我们所看到的'汉'！

那么Unicode类也是这样吗？

In [114]:uni_han= u'汉'
         type(uni_han)
Out[114]:unicode

In [131]:uni_han= u'汉'
         u_han = u'\u6c49'
         uni_han == u_han # 在Unicode中存储的是u'\u6c49'而不是你所看到的u'汉'

Out[131]:True

理解了以上三个知识点，我们就可以很容易的解释 'ascii' codec can't encode character 这个错误的缘由了。

用示例来解释 'ascii' codec can't encode character

In [117]:han = '汉'
         print type(han)
         print len(han)
         str(han) 
         
Out[117]:<type 'str'>
         3              <- '汉'的长度是3，明明是一个字，为什么长度是3?
         '\xe6\xb1\x89' <- 答案在这里

当'汉'这个字被存储在内存中时，它会被转为三个字符'\xe6\xb1\x89'。所以len()给出的长度是3，而不是1. 那么为了让'汉'变成一个真正的字，我们就需要对它进行Decoding。（参看2. 把 0 和 1 转换为Unicode的过程叫Decoding）

In [125]:str.decode(han)

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-125-3ff96a3a19da> in <module>()
----> 1 str.encode(han)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 0: ordinal not in range(128)

这里Python抛出了异常。因为默认的ASCII编码无法Decode这个文字。因为这个文字的数值已经超过了0 - 127这个范围。所以我们需要使用UTF-8编码来Decode：

In [127]:str.decode(han, 'utf8')
Out[127]:u'\u6c49'

这里han这个变量被成功Decode为 u'\u6c49

In [142]:uni_han = u'\u6c49'
         len(uni_han)
Out[142]:1              <- 长度变为了正确的1

再来个示例作为结尾

猜猜这段代码的输出是什么：

uni_han = u'\u6c49'
print '\'{0}\'的长度是{1}'.format(uni_han, len(uni_han))

结果是:

---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-143-9bec6fa25583> in <module>()
      1 uni_han = u'\u6c49'
----> 2 print '\'{0}\'的长度是{1}'.format(uni_han, len(uni_han))

UnicodeEncodeError: 'ascii' codec can't encode character u'\u6c49' in position 0: ordinal not in range(128)

好伤心啊，本以为再也不会碰到这个问题了。那么问题出在哪呢？这部分代码'\'{0}\'的长度是{1}'是str，也就是原始的字符数据。我们想把一个Unicode （uni_han）混到它们里一起打印。这时，Python信心满满的用了默认的ASCII编码来Encode uni_han。结果可想而知，又是再次超出0 - 127的范围，无法Encode。这时，我们就需要告诉Python放弃ASCII吧，请使用UTF-8：

In [145]:uni_han = u'\u6c49'
         print '\'{0}\'的长度是{1}'.format(unicode.encode(uni_han,'utf8'), len(uni_han))
         '汉'的长度是1

另一种方法是让前半部分的str变为Unicode：

In [150]:uni_han = u'\u6c49'
         print u'\'{0}\'的长度是{1}'.format(uni_han, len(uni_han))
         '汉'的长度是1

总结

在Python 2.x里str就是原始的010101， Unicode是Unicode，这两个东西不能混着用。当一个文字被写到硬盘上时，或者打印到屏幕上时，需要使用正确的Encoding编码。Python默认使用ASCII，但其实应该用UTF-8。这个问题以后还会经常碰到。关键是要理解ASCII，UTF8，Unicode, Encoding和Decoding的定义和关系。