Cheng's Blog

The goal of this post is to summarize the steps needed to deploy a Django project running on nginx and gunicorn on Mac OSX. I am going to document a list of problems encountered and provide solutions below. (If you are running on Linux, I imagine the process will be very similar)

My setup (the installation order of the following packages matters, just keep on reading. But you don't have to have the exact same versions listed below):

 Python             2.7.9
 pip                6.0.6
 Virtualenv         12.0.4
 VirtualenvWrapper  4.3.1
 Postgres.app       9.4.0.1
 psycopg2           2.5.4
 Django             1.7.3
 gunicorn           19.1.1
 brew               0.9.5
 nginx              1.6.2

First thing first, install Python and pip. This should be the easiest part as the official website provides a installer. So I am not going to elaborate on this part.

Once Python and pip are installed, I recommend to install virtualenv and virtualenvwrapper next.

pip install virtualenv 
pip install virtualenvwrapper

Upon completion, let's create a virtual environment for our later setups:

cd ~                    <- cd to your user directory
mkdir .virtualenvs      <- create an empty folder called .virtualenvs
nano .bash_profile      <- open the .bash_profile file

Copy the following lines into your .bash_profile and save.

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh

Once the file is saved, make sure to reload it:

. .bash_profile         <- Note the space between the two dots

Now we can create a virtual environment to install the rest by:

mkvirtualenv webapp

Now your bash prompt should have (webapp) in front of it, meaning you are currently in the 'webapp' environment.

Next, you should install the Postgres.app. The Postgres.app encapsulate PostgreSQL into a nice installer and makes the installation process extremely easy. Once completed, install psycopg2 (Note: you have to install Postgres.app first before installing psycopg2 otherwise you will (run into an error)[http://cheng.logdown.com/posts/2015/01/17/install-postgresql-and-psycopg2-on-mac-osx])

pip install psycopg2

Then, we can install Django and gunicorn:

pip install django
pip install gunicorn

In the end, we need to rely on brew to install nginx. To install brew, copy and paste this line into your terminal:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Once completed, install nginx:

brew install nginx

Summary of Installation

Let me explain how each aforementioned package fits within the big picture. To make everything easier to understand for novice users like myself, I simplified the functionalities of the pieces below, so please go to their websites for a more accurate description.

1. Virtualenv and Virtualenvwrapper
Virtualenv allows you to create a 'sandbox' that contains all of the Python packages you need for deployment without messing up the system's python environment. Virtualenvwrapper as its name suggests, wraps around virtualenv to provide a list of 'shortcut' commands to reduce the amount of efforts needed to create/switch between/delete virtual environments. The sole purpose of these two pieces is to isolate your deployment environment from other Python environments on the same machine.

To illustrate this issue with an example, I develop and deploy Django projects on the same machine for learning purpose. When I work on a Django project, I use the latest version of Django. But when I deploy it, I use an older version. Without virtualenv, I have to install the latest Django for development and delete it and install an older version for deployment. They can never co-exist at the same time. Virtualenv allows me to create two separate environments with different Django version in each and co-exist nicely.

2. Postgres.app and Psycopg2
The Postgres.app installs the PostgreSQL server on your machine with one click. It also provides a shortcut for you to open up a Terminal as a client that connects to the database server. Psycopg2 is used by Django or any Python program to access the PostgreSQL database programmatically.

3. nginx and gunicorn
Why do we need two servers instead of one？Well, by using gunicorn alone, it is enough to get your Django project up and running. The purpose of adding nginx in front of it is for:

1. Load Balancing - The more servers you have the more users you can serve concurrently. But who should be the one that directs traffic to different servers? Well, the nginx server in this case. (I am aware that I have everything running on a single Mac...)
2. Acceleration - Not all of the contents on your web page are dynamically generated. For instance, CSS style sheets, the logo of your website, javascript libraries like jQuery. We can have a server dedicated for serving these static contents so that our gunicorn server can focus on serving dynamic contents.
3. Security - Users never see the gunicorn server. The only server exposed to the outside world is the nginx server. The gunicorn server will never accept traffics other than the ones forwarded by nginx.

Please read this link to learn more.

The choice of which server to use is arbitrary in this case. You can replace nginx with Apache and gunicorn with (uWSGI)[https://uwsgi-docs.readthedocs.org/en/latest/]. You should Google to find out more about them and see which one fits better with you.

This concludes the first part of this post. Here is the second part but I suggest you taking a break as the second portion is rather long.

In this part of the post, I would like to document the steps needed to run an existing Django project on gunicorn as the backend server and nginx as the reverse proxy server. (Please refer to Part 1 of this post)

Disclaimer: this post is based on this site but I added a lot of details that wasn't explained.

Setup nginx

Before starting the nginx server, we want modify its config file. I installed nginx via brew on my machine, and the conf file is located here:

/usr/local/etc/nginx/nginx.conf

We can modify this file directly to add configurations for our site, but this is not a good idea. Because you may have multiple web apps running behind this nginx server. Each web app may need its own configuration. To do this, let's create another folder for storing these site-wise configuration files first:

cd /usr/local/etc/nginx
mkdir sites-enabled

We can store our site specific config here but wouldn't it be better if we store the config file along with our project files together? Now, let's navigate to our project folder. (I named my project testproject and stored it under /Users/webapp/Apps/testproject)

cd /Users/webapp/Apps/testproject
touch nginx.conf

Here is my config file:

server {
    listen 80;
    server_name your_server_ip;

    access_log /Users/webapp/logs/access.log;     # <- make sure to create the logs directory 
    error_log /Users/webapp/logs/error.log;       # <- you will need this file for debugging

    location / {
        proxy_pass http://127.0.0.1:9000;         # <- let nginx pass traffic to the gunicorn server
    }
    
    location /static {
        root /Users/webapp/Apps/testproject/vis;  # <- let nginx serves the static contents
    }
}

Let me elaborate on the '/static' part. This part means that any traffic to 'your_server_ip/static' will be forwarded to '/Users/webapp/Apps/testproject/vis/static'. You might ask why doesn't it forward to '/Users/webapp/Apps/testproject/vis'?(without '/static' in the end) Because when using 'root' in the config, it will append the '/static' part after it. So be aware! You can fix this by using alias instead of root and append /static to the end of the path:

location /static {
    alias /Users/webapp/Apps/testproject/vis/static; 
}

Here is the folder structure of my project :

/Users/webapp/Apps/testproject/
    manage.py           <- the manage.py file generated by Django
    nginx.conf          <- the nginx config file for your project
    gunicorn.conf.py    <- the gunicorn config file that we will create later, just keep on reading
    testproject/        <- automatically generated by Django
        settings.py
        urls.py
        wsgi.py         <- automatically generated by Django and used by gunicorn later
    vis/                <- the webapp that I wrote
        admin.py
        models.py
        test.py
        urls.py
        template/
            vis/
                index.html
        static/        <- the place where I stored all of the static files for my project
            vis/
                css/
                images/
                js/

All of the static files are in the /testproject/vis/static folder, so that's where nginx should be looking. You might ask that the static files live in their own folders rather than right under the /static/ path. How does nginx know where to fetch them? Well, this is not nginx's problem to solve. It is your responsibility to code the right path in your template. This is what I wrote in my template/vis/index.html page:

href="{% static 'vis/css/general.css' %}

It is likely that you won't get the path right the first time. But that's ok. Just open up Chrome's developer tools and look at the error messages in the console to see which part of the path is messed up. Then, either fix your nginx config file or your template.

To let nginx read our newly create config file:

cd /usr/local/etc/nginx/
nano nginx.conf

Find the ' http { ' header, add this line under it:

http{
    include /usr/local/etc/nginx/sites-enabled/*;

This line tells nginx to look for config files under the 'sites-enabled' folder. Instead of copy our project's nginx.conf into the 'sites-enabled' folder, we can simply create a soft link instead:

cd /usr/local/etc/nginx/site-enabled
ln -s /full_path/to/your/django_project a_name

# in my case, this is what my link command looks like:
# ln -s /Users/webapp/Apps/testproject/nginx.conf testproject
# this would create a soft link named testproject which points to the real config file

Once this is done, you can finally start up the nginx server:

To start nginx, use

sudo nginx

To stop it, use

sudo nginx -s stop

To reload the config file without shutting down the server:

sudo nginx -s reload

Please refer to this page for a quick overview of the commands.

Setup gunicorn

Setting up gunicorn is more straight forward (without considering optimization). First, let's write a config file for gunicorn. Navigate to the directory which contains your manage.py file, for me, this is what I did:

cd /Users/webapp/Apps/testproject
touch gunicorn.conf.py  # yep, the config file is a python script 

This is what I put in the config file:

bind = "127.0.0.1:9000"                   # Don't use port 80 becaue nginx occupied it already. 
errorlog = '/Users/webapp/logs/gunicorn-error.log'  # Make sure you have the log folder create
accesslog = '/Users/webapp/logs/gunicorn-access.log'
loglevel = 'debug'
workers = 1     # the number of recommended workers is '2 * number of CPUs + 1' 

Save the file and to start gunicorn, make sure you are at the same directory level as where the manage.py file is and do:

gunicorn -c gunicorn.conf.py testproject.wsgi

The ' -c ' option means read from a config file. The testproject.wsgi part is actually referring to the wsgi.py file in a child folder. (Please refer to my directory structure above)

Just in case if you need to shutdown gunicorn, you can either use Ctrl + c at the console or if you lost connection to the console, use [1]:

kill -9 `ps aux | grep gunicorn | awk '{print $2}'`

Actually, a better way to run and shutdown gunicorn is to make it a daemon process so that the server will still run even if you log out of the machine. To do that, use the following command:

gunicorn -c gunicorn.conf.py testproject.wsgi --pid ~/logs/gunicorn.pid --daemon

This command will do three things:

1. run gunicorn with the configuration file named gunicorn.conf.py
2. save the process id of the gunicorn process to a specific file ('~/logs/gunicorn.pid' in this case)
3. run gunicorn in daemon mode so that it won't die even if we log off

To shutdown this daemon process, open '~/logs/gunicorn.pid' to find the pid and use (assuming 12345 is what stored in '~/logs/gunicorn.pid'):

kill -9 12345 

to kill the server.

This is it! Enter 127.0.0.1 in your browser and see if your page loads up. It is likely that it won't load up due to errors that you are not aware of. That's ok. Just look at the error logs:

tail -f error.log

Determine if it is an nginx, gunicorn or your Django project issue. It is very likely that you don't have proper access permission or had a typo in the config files. Just going through the logs and you will find out which part is causing the issue. Depending on how you setup your Django's logging config, you can either read debug messages in the same console which you started gunicorn, or read it form a file. Here is what I have in my Django's settings.py file:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            'datefmt' : "%d/%b/%Y %H:%M:%S"
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'null': {
            'level': 'DEBUG',
            'class': 'logging.NullHandler',
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'verbose'
        },
        'logfile':{
            'level':'DEBUG',
            'class':'logging.handlers.WatchedFileHandler',
            'filename': "/Users/webapp/gunicorn_log/vis.log",
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'django.request': {
            'handlers': ['logfile'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'django': {
            'handlers': ['logfile'],
            'propagate': True,
            'level': 'DEBUG',
        },
        'vis': {
            'handlers': ['console', 'logfile'],
            'level': 'DEBUG',
            'propagate': False,
        },
    }
}

Bad Request 400

Just when you thought everything is ready, and you want the world to see what you have built...BAM! Bad Request 400. Why??? Because when you turn DEBUG = False in the settings.py file, you have to specify the ALLOWED_HOSTS attribute:

DEBUG = False
ALLOWED_HOSTS = [
    '127.0.0.1',
]

Just don't put your port number here, only the ip part. Read more about this issue here: http://stackoverflow.com/questions/19875789/django-gives-bad-request-400-when-debug-false

Conclusion

Setting up nginx, gunicorn and your Django project can be a very tedious process if you are not familiar with any of them like me. I document my approach here to hopefully help anybody who has encountered the same issue.

Just found a great tool for debugging Django applications, the Django debug toolbar provides a toolbar to your webpage when DEBUG=True. It looks like this:

https://github.com/django-debug-toolbar/django-debug-toolbar

The toolbar will provides a ton of information, such as:

• The number of database queries made while loading the apge
• The amount of time it took to load the page (similar to Chrome dev tools' Timeline feature)
• The content of your Settings file
• Content of the request and response headers
• The name and page of each static files loaded along with the current page
• Current page's template name and path
• If Caching is used, it shows the content of the cached objects and the time it took to load them
• Singals
• Logging messages
• Redirects

To install it:

pip install django-debug-toolbar

Then, in your settings.py:

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',  # <- automatically added by Django, make sure it is not missing
    'debug_toolbar',                             # <- add this
    'myapp',                                           # <- your app 
)

# static url has to be defined
STATIC_URL = '/static/'

# pick and choose the panels you want to see
DEBUG_TOOLBAR_PANELS = [
    'debug_toolbar.panels.versions.VersionsPanel',
    'debug_toolbar.panels.timer.TimerPanel',
    'debug_toolbar.panels.settings.SettingsPanel',
    'debug_toolbar.panels.headers.HeadersPanel',
    'debug_toolbar.panels.request.RequestPanel',
    'debug_toolbar.panels.sql.SQLPanel',
    'debug_toolbar.panels.staticfiles.StaticFilesPanel',
    'debug_toolbar.panels.templates.TemplatesPanel',
    'debug_toolbar.panels.cache.CachePanel',
    'debug_toolbar.panels.signals.SignalsPanel',
    'debug_toolbar.panels.logging.LoggingPanel',
    'debug_toolbar.panels.redirects.RedirectsPanel',
]

That's it. Start your server by

python manage.py runser

Load up your page in the web browser, you should see a black vertical toolbar appearing on the right side of your page.

用select_related来提取数据和不用select_related来提取数据的差别。

我有一个Model（用户每天会填写一些问题，这个Model用来收集用户每天填写的问题的答案）:

class Answer(models.Model):
    day_id = models.IntegerField()
    set_id = models.IntegerField()
    question_id = models.IntegerField()
    answer = models.TextField(default='')
    user = models.ForeignKey(User)

    def __unicode__(self):
        result = u'{4}:{0}.{1}.{2}: {3}'.format(self.day_id, self.set_id, self.question_id, self.answer, self.user.id)
        return result

这个Model除了自带的Field外还有一个外键 user，用于引用Django自带的User表。

我需要将所有用户的每天的答案都输出到一个文件里, 所以首先要提取出所有的答案:

answers = Answer.objects.all()

for answer in answers:
    print answer

运行这个指令会对数据库进行981次请求。数据库里有980行数据，Answer.objects.all()算是一次请求。那么剩下的980次请求是在干嘛呢？问题出在:

result = u'{4}:{0}.{1}.{2}: {3}'.format(self.day_id, self.set_id, self.question_id, self.answer, self.user.id)

当打印answer这个Object的时候，除了会打印出自身的Field外，还会打印出user.id. 由于Answer这个表中没有user.id这个Field，所以要单独请求User的表给出相应的id。这个就是造成980次请求的原因。每次打印一个answer，都要请求一遍User表来获取相对应的用户id。

这种提取数据的方式明显是不合理的，为了能一次完成任务，可以使用select_related:

answers = Answer.objects.all().select_related('user')

for answer in answers:
    print answer

这样，数据库请求只有一次。select_related()的作用其实是把Answer和User这两张表给合并了:

SELECT ... FROM "answer" INNER JOIN "auth_user" ON ( "answer"."user_id" = "auth_user"."id" )

这样页面的载入时间就会被大大减少。

在一个文字框里加入灰色的注释文字帮助用户理解应该输入什么信息是常见的做法。在Django里有两种方式可以为输入框添加文字注释。假设我们有一个Model:

from django.db import models

class User(models.Model):
    user_name = models.CharField()

如果需要提示用户名应该是Email地址的话，有两种方法.

方法一：

import django.forms as forms
from models import User

class Login(forms.ModelForm):
    user_name = forms.CharField(widget=forms.TextInput(attrs={'placeholder': u'输入Email地址'}))
    
    class Meta:
        model = User

方法二：

import django.forms as forms
from models import User

class Login(forms.ModelForm):
  
    class Meta:
        model = User
        widgets = {
             'user_name': forms.TextInput(attrs={'placeholder': u'输入Email地址'}),
        }

When writing a user registration form in Django, you are likely to encounter this error message:

A user with that Username already exists.

This happens when a new user wants to register with a name that is already stored in the database. The message itself is self explainatory but what I need is to display this message in Chinese. According to Django's documentation, I should be able to do this:

class RegistrationForm(ModelForm):
    class Meta:
        model = User
        error_messages = {
            'unique': 'my custom error message',
        }

But this didn't work. It turns out that Django's CharField only accepts the following error message keys:

Error message keys: required, max_length, min_length

Thanks to this StackOverflow post, here is how Django developers solved this problem in the UserCreationForm, we can adopt their solution to this situation:

class RegistrationForm(ModelForm):
    # create your own error message key & value

    error_messages = {
        'duplicate_username': 'my custom error message'
    }
    
    class Meta:
        model = User
    
    # override the clean_<fieldname> method to validate the field yourself

    def clean_username(self):
        username = self.cleaned_data["username"]
       
        try:
            User._default_manager.get(username=username)
            #if the user exists, then let's raise an error message

            raise forms.ValidationError( 
              self.error_messages['duplicate_username'],  #user my customized error message

              code='duplicate_username',   #set the error message key

                )
        except User.DoesNotExist:
            return username # great, this user does not exist so we can continue the registration process

Now when you try to enter a duplicate username, you will see the custom error message being shown instead of the default one :)

I didn't expect creating a radio input in a form using Bootstrap3's style could have some many pitfalls.

Here is a model I have:

class UserProfile(models.Model):
    GENDER_CHOICES = (('M', 'Male',), ('F', 'Female',))
    fullname = models.CharField(u'Name', max_length=10)
    sex = models.CharField(u'Gender', max_length=1, choices=GENDER_CHOICES)
    # ... and 11 other fields

Here is the corresponding ModelForm:

from django.forms.widgets import RadioSelect

class UserProfileForm(ModelForm):
    class Meta:
        model = UserProfile
        widgets = {
            'sex': forms.RadioSelect()
        }

I have a sex field which let users to pick their gender. When running the code above, I get three choices instead of two:

• '--------' # django automatically insert this 'default' option
• 'Male'
• 'Female'

This is of course not what we want. There should only be two options, 'Male' and 'Female'. So to correct this issue, I had to add TWO extra parameters to the Model:

sex = models.CharField(u'Gender', max_length=1, choices=GENDER_CHOICES, blank=False, default='M')

Note that I added blank=False AND default='M'. If you only include blank=False, you will still see the '--------' option. It is important to have both of them set.

Now, I want to style my forms using Bootstrap3. To do this, I recommend using crispy-forms

from crispy_forms.helper import FormHelper
from crispy_forms.layout import Submit
from crispy_forms.bootstrap import InlineRadios

class UserProfileForm(forms.ModelForm):
    def __init__(self, *args, **kwargs):
        super(UserProfileForm, self).__init__(*args, **kwargs)
        self.helper = FormHelper(self)
        self.helper.method = 'post'
        self.helper.form_action = 'index'
        self.helper.layout.append(Submit('save', 'Save'))

    class Meta:
        model = UserProfile

Crispy-form creates a Bootstrap3 style input field for every single field defined in the model when this line runs:

self.helper = FormHelper(self)

The UserProfile model has 13 fields. It is nice that I don't have to specify each one of the 13 fields. But here comes the problem, when the radio input is been rendered. The male and female options are listed vertically. I want them to be listed horizontally. Crispy-form does provide an inline option for rendering radio inputs horizontally. This is nice, but how can I replace the default Layout of the gender field with the suggested inline Layout?

self.helper[1] = InlineRadios('sex')

Crispy-forms allows you to access the Layout of each field by using index.

With the tweaks above, now I have a wonderful radio input for gender selection :) So many small things need to be taken care of for something seems so trivial.

I had the following code:

from django.core.urlresolvers import reverse

class UserProfileView(FormView):
    template_name = 'profile.html'
    form_class = UserProfileForm
    success_url = reverse('index')

When the code above runs, an error is thrown:

django.core.exceptions.ImproperlyConfigured: The included urlconf 'config.urls' does not appear to have any patterns in it. If you see valid patterns in the file then the issue is probably caused by a circular import.

There are two solutions to this problem, solution one:

from django.core.urlresolvers import reverse_lazy 

class UserProfileView(FormView):
    template_name = 'profile.html'
    form_class = UserProfileForm
    success_url = reverse_lazy('index') # use reverse_lazy instead of reverse

Solution 2:

from django.core.urlresolvers import reverse

class UserProfileView(FormView):
    template_name = 'profile.html'
    form_class = UserProfileForm
    
    def get_success_url(self): # override this function if you want to use reverse

        return reverse('index')

According to Django's document, reverse_lazy should be used instead of reverse when your project's URLConf is not loaded. The documentation specifically points out that reverse_lazy should be used in the following situation:

• providing a reversed URL as the url attribute of a generic class-based view. (this is the situation I encountered)

• providing a reversed URL to a decorator (such as the login_url argument for the django.contrib.auth.decorators.permission_required() decorator).

• providing a reversed URL as a default value for a parameter in a function’s signature.

It is unclear when URLConf is loaded. At least I cannot find the documentation on this topic. So if the above error occurs again, try reverse_lazy

When a user request a page view from a website (powered by Django), a cookie is returned along with the requested page. Inside this cookie, a key/value pair is presented:

Cookie on the user's computer

Key          Value
---          -----
sessionid    gilg56nsdelont4740onjyto48sv2h7l

This id is used to uniquely identify who's who by the server. User A's id is different from User B's etc. This id is not only stored in the cookie on the user's computer, it is also stored in the database on the server (assuming you are using the default session engine). By default, after running ./manage.py migrate, a table named django_session is created in the database. It has three columns:

django_session table in database

session_key        session_data        expire_date
---------------------------------------------------
y5j0jy3l4v3        ZTJlMmZiMGYw        2015-05-08 15:13:28.226903

The value stored in the session_key column matches the value stored in the cookie received by the user.

Let's say this user decides to login to the web service. Upon successfully logged into the system, a new sessionid is assigned to him/her and a different session_data is stored in the database:

Before logging in:

session_key        session_data        expire_date
---------------------------------------------------
437383928373        anonymous           2015-05-08 15:13:28.226903

After logging in:

session_key        session_data        expire_date
---------------------------------------------------
218374758493        John               2015-05-08 15:13:28.226903

*I made up this example to use numbers and usernames instead of hash strings. For security reasons, these are all hash strings in reality.

As we can see here, a new session_key has been assigned to this user and we now know that this user is 'John'. Form now on, John's session_key will not change even if he closes the browser and visit this server again. Thus, when John comes back the next day, he does not need to login again.

Django provides a setting to let developers to specify this behaviour, in settings.py, a variable named SESSION_SAVE_EVERY_REQUEST can be set:

SESSION_EXPIRE_AT_BROWSER_CLOSE = False # this is the default value. the session_id will not expire until SESSION_COOKIE_AGE has reached.

If this is set to True, then John is forced to login everytime he visits this website.

Since saving and retrieving session data from the database can be slow, we can store session data in memory by:

#Assuming memcached is installed and set as the default cache engine
SESSION_ENGINE = 'django.contrib.sessions.backends.cache'

The advantage of this approach is that session store/retrival will be faster. But the downside is if the server crashes, all session data is lost.

A mix of cache & database storage is:

SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'

According to django's documentation:

every write to the cache will also be written to the database. Session reads only use the database if the data is not already in the cache.

This approach is slower than a pure cache solution but faster than a purse db solution.

Django's offical document did warn to not use local-memory cache as it doesn't retain data long enough to be a good choice.

By default the session data for a logged in user lasts two weeks in Django, users have to log back in after the session expires. This time period can be adjusted by setting the SESSION_COOKIE_AGE variable.

There are times when you are not worried about user authentication but still want to have each user sees only his/her stuff. Then you need a way to login a user without password, here is the solution posted on stackoverflow

Normally, when logging a user with password, authenticate() has to be called before login():

username = request.POST['username']
password = request.POST['password']
user = authenticate(username=username, password=password)
login(request, user)

What authenticate does is to add an attribute called backend with the value of the authentication backend, take a look at the line of the source code. The solution is to add this attribute yourself:

#assuming the user name is passed in via get
username = self.request.GET.get('username')
user = User.objects.get(username=username)

#manually set the backend attribute
user.backend = 'django.contrib.auth.backends.ModelBackend'
login(request, user)

That's it!

I was running a Django server on Ubuntu 12.04 and saw a lot of errors logged in gunicorn's error log file:

error: [Errno 111] Connection refused

One line of the error messages caught my eye, send_mail. Gunicorn is trying to send my the error message via the send_mail() function but failed. I realized that I didn't setup the email settings in settings.py.

I searched online for a solution and found two:

1. send email via Gmail's smtp
2. setup your mail server

Option 1 seems like a quick and dirty way to get things done, but it has a few drawbacks:

1. Sending email via Gmail's smtp means the FROM field will be your Gmail address rather than your company or whatever email address you want it to be.
2. To access Gmail, you need to provide your username and password. This means you have to store your gmail password either in the settings.py file or to be more discreet in an environment variable.
3. You have to allow less secured app to access your gmail. This is a setting in your gmail's account.

I decided to setup my own mail server because I don't want to use my personal email for contacting clients. To do that, I googled and found that there are two mail servers that I can use:

• postfix
• sendmail

Since postfix is newer and easier to config, I decided to use it.

1. Install postfix and Find main.cf

Note: main.cf is the config for postfix mail server

sudo apt-get install postfix

postfix is already the newest version.
postfix set to manually installed.

Great, it is already installed. Then, I went to /etc/postfix/ to find main.cf and it is not there! Weird, so I tried to reinstall postfix:

sudo apt-get install --reinstall postfix

After installation, I saw a message:

Postfix was not set up.  Start with
  cp /usr/share/postfix/main.cf.debian /etc/postfix/main.cf
.  If you need to make changes, edit
/etc/postfix/main.cf (and others) as needed.  To view Postfix configuration
values, see postconf(1).

After modifying main.cf, be sure to run '/etc/init.d/postfix reload'.

I see. So, I followed the instruction and copied the main.cf file to /etc/postfix/:

cp /usr/share/postfix/main.cf.debian /etc/postfix/main.cf

Add the following lines to main.cf:

mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mydestination = localhost

Then, reload this config file:

/etc/init.d/postfix reload

Now, let's test to see if we can send an email to our mailbox via telnet:

telnet localhost 25

Once connected, enter the following line by line:

mail from: whatever@whatever.com
rcpt to: your_real_email_addr@blah.com
data (press enter)
type whatever content you feel like to type
. (put an extra period on the last line and then press enter again)

If everything works out, you sould see a comfirmation message resemables this:

250 2.0.0 Ok: queued as CC732427AE

It is guaranteed that this email will end up in the spam box if you use Gmail. So take a look at your spam inbox to see if you received the test mail (it may take a minute to show up).

If you recevied the test email, then it means postfix is working properly. Now, let's config Django to send email via postfix.

First, I added the following line to my settings.py file:

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'localhost'
EMAIL_PORT = 25
EMAIL_HOST_USER = ''
EMAIL_HOST_PASSWORD = ''
EMAIL_USE_TLS = False
DEFAULT_FROM_EMAIL = 'Server <server@whatever.com>'

Then, I opened up Django shell to test it:

./manage.py shell

>>> from django.core.mail import send_mail
>>> send_mail('Subject here', 'Here is the message.', 'from@example.com',
    ['to@example.com'], fail_silently=False)

Again, check your spam inbox. If you received this mail, then it means Django can send email via postfix, DONE!

references:
http://stackoverflow.com/questions/5802189/django-errno-111-connection-refused
http://stackoverflow.com/questions/4798772/postfix-its-installed-but-how-do-i-test
http://stackoverflow.com/questions/26333009/django-postfix

When trying to access either the MEDIA_URL or STATIC_URL in the template, it is important to add the following to the template context processor in settings.py:

TEMPLATE_CONTEXT_PROCESSORS += (
    "django.core.context_processors.media",
    "django.core.context_processors.static",
)

Setting the MEDIA_URL or the STATIC_URL in settings.py is not enough.

When you have large number of models, it is eaiser to understand their relationships by looking at a graph, Django extensions has a handy command to convert these relationships into an image file:

1. Install Django extension by:

pip install django-extensions

 2. Enable Django extensions in your settings.py:

  INSTALLED_APPS = (
    ...
    'django_extensions',
  )

 3. Install graph packages that Django extensions relies on for drawing:

pip install pyparsing==1.5.7 pydot

 4. Use this command to draw:

./manage.py graph_models -a -g -o my_project_visualized.png

For more drawing options, please refer to the offical doc.

Use the --log-file=- option to send error messages to console:

gunicorn --log-file=-

After debugging is complete, remove this option and output the error message to a log file instead.

I was given a task to randomly generate usernames and passwords for 80 users in Django. Here is how I did it:

Thanks to this excellent StackOverflow post, generating random characters become very easy:

import string
import random

def generate(size=5, numbers_only=False):
    base =  string.digits if numbers_only else string.ascii_lowercase + string.digits
    return ''.join(random.SystemRandom().choice(base) for _ in range(size))

I want to use lowercase characters for usernames and digits only for passwords. Thus, the optional parameter numbers_only is used to specific which format I want.

Then, open up the Django shell:

./manage.py shell

and Enter the following to the interactive shell to generate a user:

from django.contrib.auth.models import User
from note import utils

User.objects.create_user(utils.generate(), password=utils.generate(6, True))

I saved the generate() inside utils.py which is located inside a project named note. Modify from note import utils to suit your needs.

While setting the timezone for a Django server, I made a stupid mistake in the settings.py file:

TIME_ZONE = 'Asia/Shanghai'
USE_TZ = True

There is nothing wrong with the parameter names and syntax. After loading up the server, Django keeps using UTC time despite the fact that I have set TIME_ZONE = 'Asia/Shanghai'

The problem is caused by placing TIME_ZONE above USE_TZ. The order of these two lines matter here. The right way to do is:

USE_TZ = True
TIME_ZONE = 'Asia/Shanghai'

Julien Phalip gave a great talk at DjangoCon 2015, he introduced three ways to hydrate (i.e. load initial data into React component on first page load) React app with data, I am going to quote him directly from the slides and add my own comments.

1. Conventional Method: Client fetches data via Ajax after initial page load.

   Load the data via ajax after initial page load.

2. Less conventional method: Server serializes data as global Javascript variable in the inital HTML payload (see Instagram).

   Let the server load the data first, then put it into a global Javascript variable which is embed inside the returned HTML page.

   In the linked Github project (which is a Django project), the server fetches from the database to get all of the data. Then, it converts the data from Python object to json format and embed the data in the returned HTML page.

3. Server side rendering: Let the server render the output HTML with data and send it to the client.

   This one is a little bit involved. Django would pass the data from DB along with the React component to python-react for rendering.

   python-react is a piece of code that runs on a simple Node HTTP server and what it does is receiving the react component along with data via a POST request and returns the rendered HTML back. (The pyton-react server runs on the same server as your Django project.)

So which method to use then?

We can use the number of round trips and the rendering speed as metrics for the judgement.

Method 1

Round trips: The inital page request is one round trip, and the following ajax request is another one. So two round trips.

Rendering time: Server rendering is usually faster then client rendering, but if the amount of data gets rendered is not a lot then this time can be considered negligible. In this case, the rednering happens on the client side. Let's assume the amount of data need to be rendered is small and doesn't impact the user experience, then it is negligible.

Method 2

Round trips: Only one round trip

Rendering time: Negligible as aforementioned.

Method 3

Round trips: Only one round trip

Rendering time: If it is negligible on the client side then it is probably negligible on the server side.

It seems that Method 2 & 3 are equally fast. The differences between them are:

• Method 2 renders on the client side and Method 3 renders on the server side.
• Method 3 requires extra setup during development and deployment. Also, the more moving pieces there are, the more likely it breaks and takes longer to debug.

Conclusion

Without hard data to prove, here is just my speculation: Use Method 2 most of the time and only Method 3 if you think rendering on the client side is going to be slow and impact the user experience.

There are some pitfalls when you need to create and login users manually in Django. Let's create a user first:

def view_handler(request):
    username = request.POST.get('username', None)
    password = request.POST.get('username', None)

Note that request.POST.get('username', None) should be used instead of request.POST['username']. If the later is used, you will get this error:

MultiValueDictKeyError

Once the username and password are extracted, let's create the user

User.objects.create(username=username, password=password, email=email) # DON'T DO THIS

The above code is wrong. Because when create is used instead of create_user, the user's password is not hashed. You will see the user's password is stored in clear text in the database, which is not the right thing to do.

So you should use the following instead:

User.objects.create_user(username=username, password=password, email=None)

What if you want to test if the user you are about to create has already existed:

user, created = User.objects.get_or_create(username=username, email=None)
if created:
    user.set_password(password) # This line will hash the password

    user.save() #DO NOT FORGET THIS LINE

get_or_create will get the existing user or create a new one. Two values are returned, an user object and a boolean flag created indicating whether if the user created is a new one (i.e. created = True) or an existing one (i.e. created = False)

It is import to not forget including user.save() in the end. Because set_password does NOT save the password to the database.

Login

Now a user has been created successfully, the next step is to login.

user = authenticate(username=email, password=password)
login(request, user)

authenticate() only sets user.backend to whatever authentication backend Django uses. So the code above is equivlent to:

user.backend = 'django.contrib.auth.backends.ModelBackend'
login(request, user)

Django's documentation recommends the first way of doing it. However, there is an use case for the second approach. When you want to login an user without a password:

username = self.request.GET.get('username')
user = User.objects.get(username=username)
user.backend = 'django.contrib.auth.backends.ModelBackend'
login(request, user)

The is used when security isn't an issue but you still want to distinguish between who's who on your site.

So to sum up the code above, here is the view_handler that manually create and login an user:

def view_handler(request, *args, **kwargs):
    email = request.POST.get('email', None)
    password = request.POST.get('password', None)
    if email and password:
        user, created = User.objects.get_or_create(username=email,
                                                   email=email)
        if created:
            user.set_password(password)
            user.save()

        user = authenticate(username=email, password=password)
        login(request, user)
        return HttpResponseRedirect('where_ever_should_be_redirect_to')
    else:
        # return error or redirect to login page again

When writing a Django project, it happens often that mulitple apps will be included. Let me use an example:

Project
    - Account
    - Journal

In this example, I created a Django project that contains two apps. The Account app handles user registration and login. The Journal app allows users to write journals and save it to the database. Here is the what the urls look like:

#ROOT_URLCONF


urlpatterns = [
    url(r'^account/', include('Account.urls', namespace='account')),
    url(r'^journal/', include('Journal.urls', namespace='journal')), #This namespace name is used later, so just remember we have given everything under journal/ a name

]

This above file is what the ROOT_URLCONF points to. Inside the Note app, the urls look like this:

urlpatterns = [
    url(r'^(?P<id>[0-9]{4})/$', FormView.as_view(), name = 'detail'),
]

So each journal has a 4 digit id. When a journal is access, it's url may look like this: www.mynote.com/note/1231/

Let's say user John bookmarked a journal written by another person. He wants to comment on it. When John tries to access that journal www.mynote.com/note/1231/, he is redirected to the login page. In the login page's view handler, a redirect should be made to Journal ID 1231 once authentication is passed:

def view_handler(request):
    # authentication passed

    return redirect(reverse('detail', kwargs={'id', '1231'}))

The reverse(...) statement is not going to work in this case. Because the view_handler belongs to the Account app. It does not know about the urls inside the Journal app. To be able to redirect to the detail page of the Journal app:

reverse('journal:detail', kwargs={'id', '1231'})

So the format for reversing urls that belong to other apps is:

reverse('namespace:name', args, kwargs)