Projects: JobMonitor

This application allows you to automate the monitoring of jobs. Particularly useful for SysAdmins who have to trawl through a large number of emails daily to ensure everything is running correctly.

This application was written in response to one of the SysAdmin's biggest nightmares - keeping track of a large number of emails that were sending job logs on a daily basis. If you're receiving one email a day, it's easy enough to check it, but if you have a large number of emails that come in to confirm that a different number of processes have completed as expected, it can quickly become a nightmare.

Say, for instance, you have 20 emails per day that are sent with the log files of ETL processes, database space reports, Firewall logs, Exchange Server disk usage. They could be coming from different servers on different environments. How do you make sure you catch all the errors on these. Set up email filters? Ok, well what happens if your email client is down or you have move to a new machine. Or what happens when an email doesn't come in at all? Or what happens if you're away and someone else needs to check instead? Or if you're filtering isn't intelligent enough to deal with all the possible error messages? You get the idea.

This is where the Job Monitoring application comes in. Let me describe the process of how it works, so you can see the potential benefits.

Set Up

1. Define a job name - e.g. "Exchange Server Space Report"
2. Define an email subject line that will denote that job name. This has a one-to-many relationship with the job name, so you may have many individual subject lines that relate to the same job name (such as checking a number of different Exchange Servers, each one having the name of the server in the subject line). You can define the first portion of the subject line and any text after that will be allowed as well - e.g. you define"Exchange Server Database Usage Report" and it will still find "Exchange Server Database Usage Report: 05/05/05"
3. Define error messages for this email. You can set up system wide defaults, and then override those for specific emails as necessary. You can also define error messages (regular expressions that if found will flag an error), false positives (which will be excluded from errors found), and required values (which must be found, or an error will be flagged). An example of the use of a false positive is that you have an error defined as the word "error", but you have a false positive of "0 errors".
4. Optionally, you can define a schedule that you expect to receive this email on. This takes a cron format, so for instance, you can say you expect a certain email to come every Monday at 10am as follows: "00 10 * * 1"
5. For each email, you also define what email address you want to send the notification to, and optionally a message to attach to this email (for example, explaining the possible reasons, what this job's function is, etc.).

Okay, so now you're all set up, let's walk through the actual process of what the program does. It's a Python application that uses a database backend (any database supported by Django), and has a web-based GUI for configuration and status viewing written in Python using the Django web application framework:

1. Connects to a defined POP3 mailbox and checks for any emails.
2. Downloads any new emails into a PostgreSQL database and classifies them as "Unknown". Deletes all successfully downloaded emails from the POP3 mailbox.
3. Looks in its configuration for any matches of subject line of the new emails. If found, it classifies the emails according to the subject line. If not, they will remain "Unknown" and this will be made visible in the web-based GUI.
4. It will then check the emails for any errors, false positives, and required values as defined in the configuration. If any are found, it will be flagged as an error, and an email will be sent out.
5. It will then check the schedule, and if a match is found for the current time for any email that should have arrived, it will check this, and if a message is not found it will also send out an error email as above.

All options and configuration are controlled through the web-based GUI and stored in the PostgreSQL database. It requires Django, which can be run as a standalone web server, or can be used as part of an Apache web server using mod_python, for instance.

So there you have it. Interested in getting Job Monitor for your business? Browse the source code or contact us !