Perspective Unspoken

My perspective on Git, Docker, Python, Django, PHP and other stuff

Using Munin to graph stats from Django

What’s Munin?

Munin is a great tool for graphing any kind of statistics for your projects. The awesome thing about Munin is that the format it required is pretty straight forward. All we have to do to provide data for Munin is point it to a script that prints out data in a simple format.

I’ll admit though that as someone who was new to Munin, I had a hard time finding a nice tutorial that explained clearly how to view a graph that Munin had created. There are tons of tutorials out there that explain how to create a graph and the different parameters and so on, but how do I view that graph? I spent alot of time trying to find a tutorial that either discussed where the images were stored, how the images were generated, if you could generate them on the fly… etc.

So, in answer to that question. It would seem the easiest way to view graphs you’ve created is to take advantage of the fact that Munin comes configured with a nicely configured Apache site. That site is able to give you a window into all the graphs that Munin creates.

munin

Installing Munin

So let’s get started by installing Munin and configuring the Apache site.

Before we activate apache.conf, we need to edit some of the directives to make them a little more accepting of us. Have a look at the highlighted lines. Ensure that you’ve commented out the sections that are commented out and added the sections not commented.

Finally, activate the site.

Getting data into Munin

For this example, we’re going to log the number of requests every 5 seconds hitting our server. In order to accomplish this, we’ll do a middleware to increment the number of requests then we’ll use a management command to output the information for Munin to use.

I used Redis to get this done.

And then we can access the Redis instance using the code:

Let’s have a look at the approach for the middleware.

We break the logging apart into 5 minute periods by logging the time when we first log a request in that period. That is stored in Redis directly. So we fetch the LOG_START_TIME, if there’s none, that means we’ve either never logged a request. So we quickly set the time to the current time in that case. Next, we check if the more than 5 minutes have expired. If it is, we reset the count to zero. Then we increment the REQUEST_COUNT in Redis.

The reason I’m not using cache methods get and set is that it doesn’t scale well. When there are several requests per second there are serious race conditions. The increment operation is atomic and that’s exactly what we need.

Next, we need to build a script that will output information for Munin. In order to do this, I built a management command. Simple stuff.. have a look:

Munin will call the script directly or it will call it with the “config” argument. When it asks for a config, we print out some information that looks like this:

We tell Munin, the graph title, axis labels and information on a field called “requests” we want it to to place on the graph. One very important line is the graph category. The Graph Category tells you where to look in the Munin interface for your graph. Don’t forget I said that.

Then calling it without arguments it prints out a one line for each field (in our case, just one field) with the value.

Indicating that so far, I’ve had 1705 requests in the last five minutes.

The script is simple though, if “config” print out the MUNIN_CONFIG we constructed outside the class. If not, then just print out the value we get back from the cache that we had set.

Viewing results in Munin

So, it turns out that the Cron job for Munin runs every 5 minutes, hence why I chose that time period to work with for this tutorial. The official documentation says it can’t be changed, yet there’s a ticket that seems to suggest that this has been changed in the latest build. I’ll have to check that out later.

Anyway, now that we have a script that Munin can read from we need to create what’s called a Munin plugin. In order to do that we need to add a script to the /etc/munin/plugins/ folder or symlink to the script from that folder. I prefer the latter. So I made a scripts/request_count.sh file that just calls my management command. You’ll need to to use the python executable in your Virtual Environment (if you have one). In my case I’m using vagrant and virtualenv so the virtual environments are stored in ~/.virtualenvs.

Next, do the symlink to your script, wherever it is.

Finally, tell Munin which user to run the script as by editing /etc/munin/plugin-conf.d/munin-node:

Restart Munin.

In about five or so minutes, you should be able to see your graph in the Munin Interface under the Graph Category you specified. Mine was “MuninSandbox”. See screen shot below:

munin2

 

Once that shows up, you should be able to click on that to see your graphs:

munin3

 

As you can see though, my graph isn’t very exciting because I found out some time into playing around with Munin that it only does 5 minute logs… soo.. yeah.

Faking request counts

As a bonus, I wrote a script to generate fake requests using Apache Benchmark.

It uses the $RANDOM function provided by Bash to generate a random number between 1 and 50 requests to hit the server with, then it sleeps for a second then repeats.

jaywhy13 • January 17, 2016


Previous Post

Next Post

000webhost logo