The goal of this post is to summarize the steps needed to deploy a Django project running on nginx and gunicorn on Mac OSX. I am going to document a list of problems encountered and provide solutions below. (If you are running on Linux, I imagine the process will be very similar)
My setup (the installation order of the following packages matters, just keep on reading. But you don't have to have the exact same versions listed below):
Python 2.7.9
pip 6.0.6
Virtualenv 12.0.4
VirtualenvWrapper 4.3.1
Postgres.app 9.4.0.1
psycopg2 2.5.4
Django 1.7.3
gunicorn 19.1.1
brew 0.9.5
nginx 1.6.2
First thing first, install Python and pip. This should be the easiest part as the official website provides a installer. So I am not going to elaborate on this part.
Once Python and pip are installed, I recommend to install virtualenv and virtualenvwrapper next.
pip install virtualenv
pip install virtualenvwrapper
Upon completion, let's create a virtual environment for our later setups:
cd ~ <- cd to your user directory
mkdir .virtualenvs <- create an empty folder called .virtualenvs
nano .bash_profile <- open the .bash_profile file
Copy the following lines into your .bash_profile and save.
export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
Once the file is saved, make sure to reload it:
. .bash_profile <- Note the space between the two dots
Now we can create a virtual environment to install the rest by:
mkvirtualenv webapp
Now your bash prompt should have (webapp) in front of it, meaning you are currently in the 'webapp' environment.
Next, you should install the Postgres.app. The Postgres.app encapsulate PostgreSQL into a nice installer and makes the installation process extremely easy. Once completed, install psycopg2 (Note: you have to install Postgres.app first before installing psycopg2 otherwise you will (run into an error)[http://cheng.logdown.com/posts/2015/01/17/install-postgresql-and-psycopg2-on-mac-osx])
pip install psycopg2
Then, we can install Django and gunicorn:
pip install django
pip install gunicorn
In the end, we need to rely on brew to install nginx. To install brew, copy and paste this line into your terminal:
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Once completed, install nginx:
brew install nginx
Summary of Installation
Let me explain how each aforementioned package fits within the big picture. To make everything easier to understand for novice users like myself, I simplified the functionalities of the pieces below, so please go to their websites for a more accurate description.
1. Virtualenv and Virtualenvwrapper
Virtualenv allows you to create a 'sandbox' that contains all of the Python packages you need for deployment without messing up the system's python environment. Virtualenvwrapper as its name suggests, wraps around virtualenv to provide a list of 'shortcut' commands to reduce the amount of efforts needed to create/switch between/delete virtual environments. The sole purpose of these two pieces is to isolate your deployment environment from other Python environments on the same machine.
To illustrate this issue with an example, I develop and deploy Django projects on the same machine for learning purpose. When I work on a Django project, I use the latest version of Django. But when I deploy it, I use an older version. Without virtualenv, I have to install the latest Django for development and delete it and install an older version for deployment. They can never co-exist at the same time. Virtualenv allows me to create two separate environments with different Django version in each and co-exist nicely.
2. Postgres.app and Psycopg2
The Postgres.app installs the PostgreSQL server on your machine with one click. It also provides a shortcut for you to open up a Terminal as a client that connects to the database server. Psycopg2 is used by Django or any Python program to access the PostgreSQL database programmatically.
3. nginx and gunicorn
Why do we need two servers instead of one？Well, by using gunicorn alone, it is enough to get your Django project up and running. The purpose of adding nginx in front of it is for:
- Load Balancing - The more servers you have the more users you can serve concurrently. But who should be the one that directs traffic to different servers? Well, the nginx server in this case. (I am aware that I have everything running on a single Mac...)
- Acceleration - Not all of the contents on your web page are dynamically generated. For instance, CSS style sheets, the logo of your website, javascript libraries like jQuery. We can have a server dedicated for serving these static contents so that our gunicorn server can focus on serving dynamic contents.
- Security - Users never see the gunicorn server. The only server exposed to the outside world is the nginx server. The gunicorn server will never accept traffics other than the ones forwarded by nginx.
Please read this link to learn more.
The choice of which server to use is arbitrary in this case. You can replace nginx with Apache and gunicorn with (uWSGI)[https://uwsgi-docs.readthedocs.org/en/latest/]. You should Google to find out more about them and see which one fits better with you.
This concludes the first part of this post. Here is the second part but I suggest you taking a break as the second portion is rather long.
In this part of the post, I would like to document the steps needed to run an existing Django project on gunicorn as the backend server and nginx as the reverse proxy server. (Please refer to Part 1 of this post)
Disclaimer: this post is based on this site but I added a lot of details that wasn't explained.
Setup nginx
Before starting the nginx server, we want modify its config file. I installed nginx via brew on my machine, and the conf file is located here:
/usr/local/etc/nginx/nginx.conf
We can modify this file directly to add configurations for our site, but this is not a good idea. Because you may have multiple web apps running behind this nginx server. Each web app may need its own configuration. To do this, let's create another folder for storing these site-wise configuration files first:
cd /usr/local/etc/nginx
mkdir sites-enabled
We can store our site specific config here but wouldn't it be better if we store the config file along with our project files together? Now, let's navigate to our project folder. (I named my project testproject and stored it under /Users/webapp/Apps/testproject)
cd /Users/webapp/Apps/testproject
touch nginx.conf
Here is my config file:
server {
listen 80;
server_name your_server_ip;
access_log /Users/webapp/logs/access.log; # <- make sure to create the logs directory
error_log /Users/webapp/logs/error.log; # <- you will need this file for debugging
location / {
proxy_pass http://127.0.0.1:9000; # <- let nginx pass traffic to the gunicorn server
}
location /static {
root /Users/webapp/Apps/testproject/vis; # <- let nginx serves the static contents
}
}
Let me elaborate on the '/static' part. This part means that any traffic to 'your_server_ip/static' will be forwarded to '/Users/webapp/Apps/testproject/vis/static'. You might ask why doesn't it forward to '/Users/webapp/Apps/testproject/vis'?(without '/static' in the end) Because when using 'root' in the config, it will append the '/static' part after it. So be aware! You can fix this by using alias instead of root and append /static to the end of the path:
location /static {
alias /Users/webapp/Apps/testproject/vis/static;
}
Here is the folder structure of my project :
/Users/webapp/Apps/testproject/
manage.py <- the manage.py file generated by Django
nginx.conf <- the nginx config file for your project
gunicorn.conf.py <- the gunicorn config file that we will create later, just keep on reading
testproject/ <- automatically generated by Django
settings.py
urls.py
wsgi.py <- automatically generated by Django and used by gunicorn later
vis/ <- the webapp that I wrote
admin.py
models.py
test.py
urls.py
template/
vis/
index.html
static/ <- the place where I stored all of the static files for my project
vis/
css/
images/
js/
All of the static files are in the /testproject/vis/static folder, so that's where nginx should be looking. You might ask that the static files live in their own folders rather than right under the /static/ path. How does nginx know where to fetch them? Well, this is not nginx's problem to solve. It is your responsibility to code the right path in your template. This is what I wrote in my template/vis/index.html page:
href="{% static 'vis/css/general.css' %}
It is likely that you won't get the path right the first time. But that's ok. Just open up Chrome's developer tools and look at the error messages in the console to see which part of the path is messed up. Then, either fix your nginx config file or your template.
To let nginx read our newly create config file:
cd /usr/local/etc/nginx/
nano nginx.conf
Find the ' http { ' header, add this line under it:
http{
include /usr/local/etc/nginx/sites-enabled/*;
This line tells nginx to look for config files under the 'sites-enabled' folder. Instead of copy our project's nginx.conf into the 'sites-enabled' folder, we can simply create a soft link instead:
cd /usr/local/etc/nginx/site-enabled
ln -s /full_path/to/your/django_project a_name
# in my case, this is what my link command looks like:
# ln -s /Users/webapp/Apps/testproject/nginx.conf testproject
# this would create a soft link named testproject which points to the real config file
Once this is done, you can finally start up the nginx server:
To start nginx, use
sudo nginx
To stop it, use
sudo nginx -s stop
To reload the config file without shutting down the server:
sudo nginx -s reload
Please refer to this page for a quick overview of the commands.
Setup gunicorn
Setting up gunicorn is more straight forward (without considering optimization). First, let's write a config file for gunicorn. Navigate to the directory which contains your manage.py file, for me, this is what I did:
cd /Users/webapp/Apps/testproject
touch gunicorn.conf.py # yep, the config file is a python script
This is what I put in the config file:
bind = "127.0.0.1:9000" # Don't use port 80 becaue nginx occupied it already.
errorlog = '/Users/webapp/logs/gunicorn-error.log' # Make sure you have the log folder create
accesslog = '/Users/webapp/logs/gunicorn-access.log'
loglevel = 'debug'
workers = 1 # the number of recommended workers is '2 * number of CPUs + 1'
Save the file and to start gunicorn, make sure you are at the same directory level as where the manage.py file is and do:
gunicorn -c gunicorn.conf.py testproject.wsgi
The ' -c ' option means read from a config file. The testproject.wsgi part is actually referring to the wsgi.py file in a child folder. (Please refer to my directory structure above)
Just in case if you need to shutdown gunicorn, you can either use Ctrl + c at the console or if you lost connection to the console, use [1]:
kill -9 `ps aux | grep gunicorn | awk '{print $2}'`
Actually, a better way to run and shutdown gunicorn is to make it a daemon process so that the server will still run even if you log out of the machine. To do that, use the following command:
gunicorn -c gunicorn.conf.py testproject.wsgi --pid ~/logs/gunicorn.pid --daemon
This command will do three things:
- run gunicorn with the configuration file named gunicorn.conf.py
- save the process id of the gunicorn process to a specific file ('~/logs/gunicorn.pid' in this case)
- run gunicorn in daemon mode so that it won't die even if we log off
To shutdown this daemon process, open '~/logs/gunicorn.pid' to find the pid and use (assuming 12345 is what stored in '~/logs/gunicorn.pid'):
kill -9 12345
to kill the server.
This is it! Enter 127.0.0.1 in your browser and see if your page loads up. It is likely that it won't load up due to errors that you are not aware of. That's ok. Just look at the error logs:
tail -f error.log
Determine if it is an nginx, gunicorn or your Django project issue. It is very likely that you don't have proper access permission or had a typo in the config files. Just going through the logs and you will find out which part is causing the issue. Depending on how you setup your Django's logging config, you can either read debug messages in the same console which you started gunicorn, or read it form a file. Here is what I have in my Django's settings.py file:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
'datefmt' : "%d/%b/%Y %H:%M:%S"
},
'simple': {
'format': '%(levelname)s %(message)s'
},
},
'handlers': {
'null': {
'level': 'DEBUG',
'class': 'logging.NullHandler',
},
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'verbose'
},
'logfile':{
'level':'DEBUG',
'class':'logging.handlers.WatchedFileHandler',
'filename': "/Users/webapp/gunicorn_log/vis.log",
'formatter': 'verbose',
},
},
'loggers': {
'django.request': {
'handlers': ['logfile'],
'level': 'DEBUG',
'propagate': True,
},
'django': {
'handlers': ['logfile'],
'propagate': True,
'level': 'DEBUG',
},
'vis': {
'handlers': ['console', 'logfile'],
'level': 'DEBUG',
'propagate': False,
},
}
}
Bad Request 400
Just when you thought everything is ready, and you want the world to see what you have built...BAM! Bad Request 400. Why??? Because when you turn DEBUG = False in the settings.py file, you have to specify the ALLOWED_HOSTS attribute:
DEBUG = False
ALLOWED_HOSTS = [
'127.0.0.1',
]
Just don't put your port number here, only the ip part. Read more about this issue here: http://stackoverflow.com/questions/19875789/django-gives-bad-request-400-when-debug-false
Conclusion
Setting up nginx, gunicorn and your Django project can be a very tedious process if you are not familiar with any of them like me. I document my approach here to hopefully help anybody who has encountered the same issue.
I wrote two simple scripts for starting and stopping a gunicorn server that hosts a Django project.
Create a run.sh:
#!/bin/bash
source `which virtualenvwrapper.sh` #assuming you use virtualenv
workon your_env_name # if you do not use virtualenv, remove this line the line above
gunicorn -c gunicorn.conf.py your_project_name.wsgi --daemon
Create a stop.sh:
#!/bin/bash
kill -9 `ps aux |grep gunicorn |grep your_app_name | awk '{ print $2 }'` # will kill all of the workers
Once these scripts are created, don't forget to set the permission:
chmod 755 run.sh
chmod 755 stop.sh
To run them:
./run.sh
./stop.sh
