Remote monitoring a web server job queue

I use some software called Huginn to do various automated web searching and scraping. One use case is checking the status of all the institutions where The Great Wave is currently on view, or not. I also have it do automatic auction searches for various items I’m looking for, that would otherwise take up a bunch of my time. And much more.

Huginn, we have a problem

Huginn is installed on one of my web servers and does its thing on hourly, every day of the week without fail. Well, that’s not quite true. Sometimes, something happens outside of its control and the job queue gets jammed. It should be able to cope with this event, but for whatever reason it doesn’t. There’s no easy way to tell when this has happened, and the way I usually notice is by my RSS feed appearing less busy than usual. So, the most recent time this happened I promised myself I’d find a solution to monitoring the job queue remotely.

Help from the community

Another Huginn user mentioned that I could send a regular request to healthchecks.io, which would raise a notification if the requests stopped coming in (a method known as the Dead man’s switch) which is exactly what would happen when the job queue is jammed. This was a good idea! Whilst I could be notfications by email or various push notification services, I’d prefer to get my notifications the same way as my server notifications - through uptimerobot.com - so I added that to the process, with the help of Pēteris Caune the creator of healthchecks.io.

Here is my final recipe:

1. healthchecks.io to get a Ping URL
2. create a new Huginn Scenario with a Website Agent to request the above Ping URL every hour
3. healthchecks.io to get a Status Badge URL (JSON format)
4. uptimerobot.com Keyword monitor watching the contents of the Status Badge URL (check for keyword “up”, and start incident when keyword does not exist)

And that’s it! Next time the job queue jams, the Uptime Robot app will send me a push notification. I know it works because I tested it—a very important step!

Future features?

Maybe this setup will help me spot a pattern in the queue jams? Or perhaps I can take this further and have the Huginn restart when it gets jammed? As with all these sorts of things, I take it step by step and make one change at a time. Let’s see how it goes.