Lean, simple server monitoring with Monit

Monit is a comparitively lightweight server monitor. I’d say its biggest sellig point is th ease ans sped of setup compared to the more enterprise-grade monnitoring services like Nagios, Icinga, Munnin, Zabbix, etc. Those are likely overkill unless in an enterprise situation.

Although I say comparitively lightweight you’ll find as you dig into Monit it has a great depth of features. I came to use it recently because a problem I’ve recently run into with PHP-FPM dying for no apparent reason leaving me with a bad gateway timeout for nginx.

Monit to the rescue which will detect the dead service, restart it, log it and continue monitoring.

Install

This assumes you’re running Ubuntu/Debian in all the following but you should be able to adapt it for your own distro.

1
2
sudo apt-get update
sudo apt-get install monit

After Monit is installed go ahead and use your fave editor (Vim obviously) and edit monits config file:

Basic Setup

1
sudo vim /etc/monit/monitrc

There’s a few key things you’ll want to change in here and a few that are optional but the config file is great with loads of explanation on each variabe you can change.

/etc/monit/monitrc
1
2
3
4
5
6
7
8
9
10

# Checks pocesses every 2 minutes (120 secs) for up state
set daemon 120

# Use the localhost as the mailserver for sending alerts (change as appropriate)
set mailserver localhost

# The email address to send alerts of services being dwn and restarted
set alert [email protected]

Service Monitoring

Service monitoring is just one of the many core facets of Monit but its the one I’m focussing on in this blog. Even within basic service monitoring there’s a lot extra options you can use but I’m keeping it simple for now. If you want to go into more detail such as file checking, space, network monitoring, etc check out the full manual at monit HQ.

Heres a very basic example for checking 3 services on yoru box:

/etc/monit/monitrc
1
2
3
4
5
6
7
8
9
10
11
12
13

###############################################################################
## Services
###############################################################################
check process php5-fpm with pidfile /var/run/php5-fpm.pid
start program = "/etc/init.d/php5-fpm start"
stop program = "/etc/init.d/php5-fpm stop"
check process nginx with pidfile /var/run/nginx.pid
start program = "/etc/init.d/nginx start"
stop program = "/etc/init.d/nginx stop"
check process mysql with pidfile /var/run/mysqld/mysqld.pid
start program = "/etc/init.d/mysql start"
stop program = "/etc/init.d/mysql stop"

Afer doing any changes to the monitrc file you must run:

1
sudo monit reload

Web Interface

If receiving notifications and allowing Mnit to do its thing isn’t enough then you can satisfy your inner voyeur by setting up the Monit httpd service. In the monitc file find the commented out section below:

/etc/monit/monitrc
1
2
3
4
5
6
7
8

# set httpd port 2812 and
# use address localhost # only accept connection from localhost
# allow localhost # allow localhost to connect to the server and
# allow admin:monit # require user 'admin' with password 'monit'
# allow @monit # allow users of group 'monit' to connect (rw)
# allow @users readonly # allow users of group 'users' to connect readonly

The comments make it pretty self explanatory but to give you an idea I’ve included a simple example here:

/etc/monit/monitrc
1
2
3
4
5
6
set httpd port 2500 and         # The port to allow access from
use address 101.102.103.104 # Your servers IP address to bind to
allow 89.90.91.92 # Your IP address
allow username:password
allow @monit
allow @users readonly

With those settings you can visit http://101.102.103.104:2500 and after putting in the username and password above you will be presented with something like this:

Going Further

This really only scractes the surface of what Monit can do as you may have noticed with the huge number of examples and sub-rules to checks in the monitrc file. I’d strongly recommend reading through the config file and checking out the manual too for some of Monits more powerful features.

Troubleshooting

Problem 1 - Missing PID files

If you find your MySQL server doesn’t have a pid file (one of my servers didn’t) then its as simple as adding it to your config file (in MySQL’s case: /etc/mysql/my.cnf) like so:

1
pid-file    = /var/run/mysqld/mysqld.pid

Then set the location of that pid file in your server monitoring configs as above.

Problem 2 - Alert emails not working

This is pretty common especially if you’re using the localhost. Its most likely they are going to spam or your server is simply picking them up due to a misconfigured Postfix install. Check your spam folder and try a different alert email on a different domain to narrow down the issue.

Currently listening to…

Old Ubuntu/Debian apt updates and those 404's

If like me you have had annoying 404’s pop up during apt-get update and no amount of fiddling with the /etc/apt/source.list then it means your Ubuntu has reached End of Life. AKA: it’s an OAP.

It needs a bus pass.

So to help it out simply change all instances of archive.ubuntu.com and security.ubuntu.com to old-releases. Like so:

/etc/apt/source.list
1
2
3
4
5
6
7
8
9
10
11
12
13
# From this:
#deb http://archive.ubuntu.com/ubuntu quantal main
#deb http://archive.ubuntu.com/ubuntu quantal-updates main
#deb http://security.ubuntu.com/ubuntu quantal-security main
#deb http://archive.ubuntu.com/ubuntu quantal universe
#deb http://archive.ubuntu.com/ubuntu quantal-updates universe

# To this:
deb http://old-releases.ubuntu.com/ubuntu quantal main
deb http://old-releases.ubuntu.com/ubuntu quantal-updates main
deb http://old-releases.ubuntu.com/ubuntu quantal-security main
deb http://old-releases.ubuntu.com/ubuntu quantal universe
deb http://old-releases.ubuntu.com/ubuntu quantal-updates universe

Then run sudo apt-get update to get the ‘fresh’ list of packages you’re used to But its highly recommended you upgrade your distro to a more up-to-date version by following the steps at Ubuntu’s community EOL page.

5 Cool Corporate Open Source Contributors You Never Thought Of

I had the idea for this post after accidentally stumbling across KrakenJS and realising its a Paypal open source project.

OK so its pretty obvious companies like Google, RedHat, Oracle, etc all contribute to the open source community and even stand on its shoulders to a great extent. But after finding Paypal’s contributions I did some digging to find some lesser known coroporate open source contributors. And some cool projects popped up.

There was a time when I’d do a lot of coding using Google API’s and open source libraries but over the years I’d become dissillusioned with them and as a result subconciously steered clear of any future corporately sponsored projects.

Quick KrakenJS intro before the main list

Not only is KrakenJS a much beefed up version of the Node framework ExpressJS with lads of extras including:

  • Application Security
  • NPM Proxy
  • Dust I18N
  • Dust Context

It’s also a way to structure your project. It is like Sinatra’s Padrino of the Ruby world.

1. Paypal

Paypal Logo

As I’ve already mentioned Paypal’s greatest contribution (IMHO) KrakenJS here’s Paypals GitHub page with all their projects:

2. Netflix

First it was a major banking organisation. Now we’ve the darling of media streaming services to the public and paraiah to US ISPs.

Netflix Logo

The Netflix open source contribution is no less impressive than PayPals - maybe even more so which is why its criminal its not more widely known. So here’s a heads up on their:

Some of their notable projects:

  • Asgard - Web interface for application deployments and cloud management in Amazon Web Services (AWS)
  • Scumblr - cool custom search engine. Check the page for full info.
  • SimianArmy - Tools for keeping your cloud operating in top form.

3. Spotify

Some of their projects:

Check out the developer blog for all their API’s and SDKs.

4. UK Government

HM Government Portcullis

What? That stuffy old wood and leather palace with its yays and nays? Yep - ineed they have such a prolific open source collection on Github (19 pages of repos!) It was hard to dig out the gems. Its worth checking them all and there’s some that are criminally under-appreciated. Here’s a few:

  • Magna Charter - Accessible, useful, beautiful barcharts from HTML tables.
  • Gov.UK Styleguides - Good to see the government have style guides in place.
  • Router - Just, erm. Wow. Check the readme - maybe you’ll make more sense of it than me. Seems built on Go and handles routing for the main Gov.uk site.

One thing I have noticed is the UK Government does seem to like it’s Ruby, esp. Ruby on Rails.

Check out the full 19 page monster list at AlphaGov.

5. The Guardian

The UK newspaper currently has a list of repos to rival the Government’s but the potential for cool finds does seem a bit higher. Here’s a few I managed to dig out:

Of course the Guardian wouldn’t be much of a newspaper without its own developer blog now would it:

If you think I’ve missed off any great projects let me know: doug (at) tintophat (dot) com

Why Hexo?

It’s taken a lot of umming and aring to come to this decision but it was finally made when I decided I’d had enough of said umming and arring.

Hexo Logo

My basic requirement

A site hosted on Github generated using a static site generator with posts written in Markdown.

Simple?
Well I’d narrowed it down to Jekyll and Pelican initially.

Why not Jekyll

Jekyll is hugely popular and if you check out my links below it is by far the biggest static site generator out there. However, I was slightly biased against precisely because of its popularity. Sometimes I prefer the underdog and Jekyll was too popular, too obvious.

Another reason was I wanted to avoid muddying the waters of my Python knowledge by taking on a Ruby driven generator. I’d worked with Ruby liberally in the past with one main site built using Rails and an API endpoint in Padrino about 4or so years ago. Even turned my old blog into Mephisto temporarily at ne point. I wanted to try something new-ish in a way.
I know I can avoid the Ruby side of it and simply use as is but I wanted to eventually get my hands dirty in ehatever I was using. That ruled out any Ruby-based generator - including the next most popular Octopress, which again didn’t interest me much.

Why not Pelican?

Apart from the ugly as hell project blog which doesn’t really sell itself?

Pelican Blog Screnshot

I do use Pelican for another site and its been gathering dust. Its not intuitive to use although slowly improving over time it was messy to deploy and setup. You could either use Fabric (great, I’ve used Fabric extensively in the past) or the Makefile that it generated. It turns out the Makefile method often had better results. Though generating posts was still slower and messier than I’d have liked. Nothing seemed polished and for a static site gen 5 years old I expected better.

I went through several iterations that never saw the light of day. First it required I set up a virtualenv, install Pelican, then its dependencies, etc, etc, etc. I was fiddling around trying to make a nice custom theme because all the existing themes weren’t to my liking. This was completly different to the days of my first tech blog where I just chose whatever Worpress theme looked nice at the time and I changed with the changing of the seasons. Pelican had no nice default themes and was a pain to use.

I simply wanted somewhere to write - I didn’t want to fight to write.

So on I went considering even hosted blogs like Tumblr, Wordpress, Blogger. I started looking at other dev blogs to see what they used and quickly noticed many of them had the same feelings I did.

No custom themeing (or very little). Often 3rd party hosted on 3rd party platforms.

I found the following two sites invaluable and visited them repeatedly:

It was after some time I noticed a new(-ish) monster on the horizon by the name of Hexo. Cool name, clean site, oh - what this. Holy shit deployment to die for. So simple - I Just add “git” to the config file? Seriously?!

All using custom terminal and based off JS/Node?

Its packages and plugins easily available via npm. For something so relatively young it offered a hell of a lot and in a very slick package.

And so here’s to a hopefully bright future with Hexo.

TL;DR

  • Clean, simple CLI.
  • Comprehensive.
  • Just different enough.
  • Nice site.
  • Chance to dabble in EJS, Node, Stylus, etc.

As mentioned in a previous post I’ll be converting my other blog to Hexo in the not too distant future.

Hello, World

Seems only right to start a new blog with this. It’s far from my first having started a tech blog (remaining nameless) around about 2005/6. Not long after leaving uni. It lasted for quite a while had a lot of visitors but I got bored and so killed it off, dropped the domain, etc.

I started my previous blog when the tech world seemed to be in a state of flux. Rails hit the scene not long after Agile Development became a cool word. Django came not long after and I blogged liberally during the whole Ruby vs Python malarky. A hea;lthy dose of youthful exuberance and arrogance also added to my desire to start the blog back then.

Now I’ve decided to start one…well why not? It isn’t the only one. I’ve got plans to re-launch another personal blog in the not too distant future. Very unrelated to this.

In the meantime if you wanted to know who I am read the blogs title or read the About page.