The Big Picture
We can divide the request life cycle of the django application into three layers.
1. Browser
2. Server
3. Django
Browser
Browser(client) is responsible for sending data to the server and also responsible for receiving the response back.
When you type a web address into your browser:
- The browser goes to the DNS server, and finds the real address of the server that the website lives on (you find the address of the shop).
- The browser sends an HTTP request message to the server, asking it to send a copy of the website to the client (you go to the shop and order your goods). This message, and all other data sent between the client and the server, is sent across your internet connection using TCP/IP.
- If the server approves the client's request, the server sends the client a "200 OK" message, which means "Of course you can look at that website! Here it is", and then starts sending the website's files to the browser as a series of small chunks called data packets (the shop gives you your goods, and you bring them back to your house).
- The browser assembles the small chunks into a complete website and displays it to you (the goods arrive at your door — new shiny stuff, awesome!).
Server
- It receives the request from the browser, based on the request it gives response back. If we take an example of NGINX server. It can handle the 10,000 requests in a second based on the resources of the server(RAM, Processor).
- If it receives more than 10,000 requests in a second it creates another process to handle it.
- We can divide request resources in two types.
- 1. static resource
- 2. dynamic resource(It has to process the data based on request to provide resource)
- If browser request for the static resources like images/css/javascript/etc, then NGINX serves the request without sending it to the uWSGI server.
- If browser requests for a dynamic request then NGINX passes it to the uWSGI to process the request.
- At this stage NGINX acts like a reverse proxy server. A reverse proxy server is a type of proxy server that typically sits behind the firewall in a private network and directs browser/client requests to the appropriate back-end server(uWSGI).
- Advantages of reverse proxy server are Load balancing, Web acceleration, Security and anonymity.
Django
- Django layer comes into the picture when request passed from nGINX to the uWSGI it takes the request middlewares from the settings and applies on the request to modify request.
- After applying the request middlewares it sends the request to url dispatcher. Url dispatcher is responsible for dispatching the request to a view based on the url pattern
- Here we implement the business logic. We access the database resources by writing Django Queries.
- The query passes to the ORM(Object Relation Mapper). ORM converts the django query into SQL(Structured Query Language) query and hits the database (MySQL/Postgres,etc). Database returns the query results in a relational table. ORM again converts these relational data into django queryset and returns back to the view.
- View passes the context(data that's retrieved from the database) to the template.
- Template renders the content with context data and forms the Response(HTML/XML/JSON/etc.)
- Again response middlewares which are defined in settings will apply and modifies the request and sends the response to the Browser(Client).
Let's discuss more about django's part.
Django Middlewares and the Request/Response Cycle
When setting up a new Django project, one of the first things you’ll do is wire up your URLconfs and set up some views. But what’s actually happening under the hood here? How does Django route traffic to the view, and what part do middlewares play in this cycle?
WSGI
WSGI is a tool created to solve a basic problem: connecting a web server to a web framework. WSGI has two sides: the ‘server’ side and the ‘application’ side. To handle a WSGI response, the server executes the application and provides a callback function to the application side. The application processes the request and returns the response to the server using the provided callback. Essentially, the WSGI handler acts as the gatekeeper between your web server (Apache, NGINX, etc) and your Django project.
Between the server and the application lie the middlewares. You can think of middlewares as a series of bidirectional filters: they can alter (or short-circuit) the data flowing back and forth between the network and your Django application.
The Big Picture — Data Flow
When the user makes a request of your application, a WSGI handler is instantiated, which:
- imports your
settings.py
file and Django’s exception classes. - loads all the middleware classes it finds in the
MIDDLEWARE_CLASSES
orMIDDLEWARES
(depending on Django version) tuple located insettings.py
- builds four lists of methods which handle processing of request, view, response, and exception.
- loops through the request methods, running them in order
- resolves the requested URL
- loops through each of the view processing methods
- calls the view function (usually rendering a template)
- processes any exception methods
- loops through each of the response methods, (from the inside out, reverse order from request middlewares)
- finally builds a return value and calls the callback function to the web server
Let’s get started.
Middlewares
Middlewares are employed in a number of key pieces of functionality in a Django project: you use CSRF middlewares to prevent cross-site request forgery attacks, for example. They’re used to handle session data. Authentication and authorization is accomplished with the use of middlewares. You can write your own middleware classes to shape (or short-circuit) the flow of data through your application.
process_request
Django middlewares must have at least one of the following methods: process_request
, process_response
, process_view
, and process_exception
. These are the methods which will be collected by the WSGI Handler and then called in the order they are listed. Let’s take a quick look at django.contrib.auth.middleware.AuthenticationMiddleware
, one of the middlewares which are installed by default when you run django-admin.py startproject
:
def get_user(request):
if not hasattr(request, '_cached_user'):
request._cached_user = auth.get_user(request)
return request._cached_user
class AuthenticationMiddleware(MiddlewareMixin):
def process_request(self, request):
assert hasattr(request, 'session'), (
"The Django authentication middleware requires session middleware "
"to be installed. Edit your MIDDLEWARE%s setting to insert "
"'django.contrib.sessions.middleware.SessionMiddleware' before "
"'django.contrib.auth.middleware.AuthenticationMiddleware'."
) % ("_CLASSES" if settings.MIDDLEWARE is None else "")
request.user = SimpleLazyObject(lambda: get_user(request))
As you can see, this middleware only works on the ‘request’ step of the data flow to and from your Django application. This middleware first verifies that the session
middleware is in use and has been called already, then it sets the user
by calling the get_user
helper function. As the WSGI Handler iterates through the list of process_request
methods, it’s building up this request
object which will eventually be passed into the view, and you’ll be able to reference request.user
. Some of the middlewares in settings.py
won’t have process_request
methods. No big deal; those just get skipped during this stage.
process_request
should either return None
(as in this example), or alternately it can return an HttpResponse
object. In the former case, the WSGI Handler will continue processing the process_request
methods, the latter will “short-circuit” the process and begin the process_response
cycle.
Resolve the URL
Now that the process_request
methods have each been called, we now have a request
object which will be passed to the view. Before that can happen, Django must resolve the URL and determine which view function to call. This is simply done by regular expression matching. Your settings.py
will have a key called ROOT_URLCONF
which indicates the ‘root’ urls.py
file, from which you’ll include the urls.py
files for each of your apps. URL routing is covered pretty extensively in the Django tutorials so there’s no need to go into it here.
A view has three requirements:
- It must be callable. It can be a function-based view, or a class-based view which inherits from
View
theas_view()
method to make it callable depending on the HTTP verb (GET, POST, etc) - It must accept an
HttpRequest
object as the first positional argument. This is the result of calling all theprocess_request
andprocess_view
middleware methods. - It must return an
HttpResponse
object, or raise an exception. It’s this response object which is used to kick off the WSGI Handler’sprocess_view
cycle.
process_view
Now that the WSGI Handler knows which view function to call, it once again loops through its list of middleware methods. The process_view
method for any Django middleware is declared like this:
process_view(request, view_function, view_args, view_kwargs)
Just like with process_request
, the process_view
function must return either None
or an HttpResponse
object (or raise an exception), allowing the WSGI Handler to either continue processing views, or “short-circuiting” and returning a response. Take a look at the source code for CSRF Middleware to see an example of process_view
in action. If a CSRF cookie is present, the process_view
method returns None
and the execution of the view occurs. If not, the request is rejected and the process is short-circuited, resulting in a failure message.
process_exception
If the view function raises an exception, the Handler will loop through its list of process_exception
methods. These methods are executed in reverse order, from the last middleware listed in settings.py
to the first. If an exception is raised, the process will short-circuit and no other process_exception
middlewares will be called. Usually we rely on the exception handlers provided by Django’s BaseHandler, but you can certainly implement your own exception handling when you write your own custom middleware class.
process_response
At this point, we’ll have an HttpResponse
object, either returned by the view or by the list of process_view
methods built by the WSGI handler, and it’s time to loop through the response middlewares in turn. This is the last chance any middleware has to modify the data, and it’s executed from the inner layer outward (think of an onion, with the view at the center). Take a look at the cache middleware source code for an example of process_response
in action: depending on different conditions in your app (i.e. whether caching is turned off, if we’re dealing with a stream, etc), we’ll want the response stored in the cache or not.
Note: One difference between pre-1.10 Django and later versions: in the older style MIDDLEWARE_CLASSES
, every middleware will always have its process_response
method called, even if an earlier middleware short-circuited the process. In the new MIDDLEWARES
style, only that middleware and the ones which executed before it will have their process_response
methods called. Consult the documentation for more details on the differences between MIDDLEWARES
and MIDDLEWARE_CLASSES
.
All Done!
Finally, Django’s WSGI Handler builds a return value from the HttpResponse
object and executes the callback function to send that data to the web server and out to the user.
So, two key takeaways:
- Now we know how the view function is matched to a URLconf and what actually calls it (the WSGI Handler).
- There are four key points you can hook into the request/response cycle through your own custom middleware:
process_request
’,process_response
,process_view
, andprocess_exception
. Think of an onion: request middlewares are executed from the outside-in, hit the view at the center, and return through response middlewares back to the surface.
One Last Thing! Testing
Writing tests is a good way to get a handle on how middlewares and the request/response cycle work. You can use Django’s RequestFactory
to mock up a request. It might look something like this:
from django.test import RequestFactory
from .views import some_view_to_test
from somewhere import MiddlewareToTest
factory = RequestFactory()
request = self.factory.get('/some/url')
middleware_modified_request = MiddlewareToTest.process_request(request)
response = some_view_to_test(middleware_modified_request)
middleware_modified_resposne = MiddlewareToTest.process_response(response)
Now you can examine your modified request and response objects and ensure that your middleware is behaving in the way you expect.
* Partial content appeared here in the blog post by Adam King