Monitor your django methods

So . . . it occurred to me that there wasn’t a real way to monitor the internals of a django app/project. Sure you can run munin (or any flavor of monitoring app) and watch apache response times, db hits, network lag and so on. All great, right? Well it turns out that if you are in the market for something “out of the box” to profile the actual methods inside your django shiz, then you are “out of luck”.

So, I browsed the interwebs and found some old 0.96 compliant code that showed promise inside the django docs. Using this inside your django projects will display hotshot results and let you know just what python modules/files/groups are taking what time to respond. Not sure about you, but this is amazing to me. Made minor adjustments to make the code 1.2 compliant. Thanks to the original 0.96 dev for this awesome code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
import sys
import os
import re
import hotshot, hotshot.stats
import tempfile
import StringIO
 
from django.conf import settings
 
 
words_re = re.compile( r'\s+' )
 
group_prefix_re = [
    re.compile( "^.*/django/[^/]+" ),
    re.compile( "^(.*)/[^/]+$" ),
    re.compile( ".*" ),
]
 
class ProfileMiddleware(object):
    def process_request(self, request):
        if (settings.DEBUG or request.user.is_superuser) and 'lookie' in request.GET:
            self.tmpfile = tempfile.mktemp()
            self.prof = hotshot.Profile(self.tmpfile)
 
    def process_view(self, request, callback, callback_args, callback_kwargs):
        if (settings.DEBUG or request.user.is_superuser) and 'lookie' in request.GET:
            return self.prof.runcall(callback, request, *callback_args, **callback_kwargs)
 
    def get_group(self, file):
        for g in group_prefix_re:
            name = g.findall( file )
            if name:
                return name[0]
 
    def get_summary(self, results_dict, sum):
        list = [ (item[1], item[0]) for item in results_dict.items() ]
        list.sort( reverse = True )
        list = list[:40]
 
        res = "      tottime\n"
        for item in list:
            res += "%4.1f%% %7.3f %s\n" % ( 100*item[0]/sum if sum else 0, item[0], item[1] )
 
        return res
 
    def summary_for_files(self, stats_str):
        stats_str = stats_str.split("\n")[5:]
 
        mystats = {}
        mygroups = {}
 
        sum = 0
 
        for s in stats_str:
            fields = words_re.split(s);
            if len(fields) == 7:
                time = float(fields[2])
                sum += time
                file = fields[6].split(":")[0]
 
                if not file in mystats:
                    mystats[file] = 0
                mystats[file] += time
 
                group = self.get_group(file)
                if not group in mygroups:
                    mygroups[ group ] = 0
                mygroups[ group ] += time
 
        return "<pre>" + \
               " ---- By file ----\n\n" + self.get_summary(mystats,sum) + "\n" + \
               " ---- By group ---\n\n" + self.get_summary(mygroups,sum) + \
               "</pre>"
 
    def process_response(self, request, response):
        if (settings.DEBUG or request.user.is_superuser) and 'lookie' in request.GET:
            self.prof.close()
 
            out = StringIO.StringIO()
            old_stdout = sys.stdout
            sys.stdout = out
 
            stats = hotshot.stats.load(self.tmpfile)
            stats.sort_stats('time', 'calls')
            stats.print_stats()
 
            sys.stdout = old_stdout
            stats_str = out.getvalue()
 
            if response and response.content and stats_str:
                response.content = "<pre>" + stats_str + "</pre>"
 
            response.content = "\n".join(response.content.split("\n")[:40])
 
            response.content += self.summary_for_files(stats_str)
 
            os.unlink(self.tmpfile)
 
        return response

To use this middleware you will first have to install it inside of your django project and add it to the MIDDLEWARE_CLASSES tuple. Once this is in place all requests will travel through the middleware like so .

Now that every request is traveling through the newly created middleware, we want to see results right? To see a profile of any route inside your project simply type ?lookie at the end of the route:
Lookie
Lookie

Once you do this your browser will be filled with lines of hotshot profiles on each method that was called to make your magic request . . . magical:

Notice that it even profiles the methods that make django work ;) . Hotshot is magical.

I wouldn’t reccomend using this on anything live for too long as the middleware overheard might be too much for high traffic sites. You can always leave the code and just not include it inside the MIDDLEWARE_CLASSES tuple. Put it in place when things seem slow or you just want to test it out. Have fun and be safe people of the interweb.



8 Comments

  1. Jonas Patel wrote:

    This is a great piece of code. I’ve just put it in place and a lagging method inside of a model. So this makes me extremely happy as I would have had no way to know this with real data. Testing only takes one so far, but this pushes the boundaries for finding live problems. Many thanks for sharing.

  2. Shinkar wrote:

    Hi, I apologize for asking this question here, but I am not ableto find a contact form or something so I assumed I could I leave my request here. I am running a django instance with version 1.2 and want to switch over the ORM to be 100% MongoDB based. How can I do this without recoding?

  3. ptownsend wrote:

    Super site, and nice little django trick with hot shot too!

  4. Zane Mckaskle wrote:

    Nice blog, bookmarked!

  5. Alfred wrote:

    @shinkar: basically you can’t, well . . . not without coding a little. It’s not too hard to port the ORM to something that maps classes to documents and it is equally not as hard to replace things like auth/sessions/forms. . . . my suggestion would be to checkout MongoKit. I have invested a lot of time in the core and find it a suitable path for ORM replacement.

  6. Carrie Lowe wrote:

    Hi, I apologize for asking this question here, but I am not ableto find a contact form or something so I assumed I could I leave my request here. I am running a django instance with version 1.2 and want to switch over the ORM to be 100% MongoDB based. How can I do this without recoding?

    • Alfred wrote:

      You will have to recode if you are looking for 100% mongodb. I have created some basic and some advanced ORM’ish representations of mongodb data. I am dreadfully behind on blogging, but will try to get these posted soon.

    • Alfred wrote:

      In the absence of my blog post (that I will prob never write :/) go and check out MongoEngine and MongoKit. Both of these are really neat. MongoEngine is prob the closest to the django orm at the moment.

      Hope this helps Carrie.