24 October 2009
When you forget to prune your logging data

The last 8 months I have worked on and off on a municipal IT project, which have had many different subcontractors before me. The system pretty much does what it has to do, but the code is pretty smelly and could use a good deal of refactoring.
It's a learning experience to read such code. Once in a while you find little gold nuggets and intelligent solutions to problems, that you can really learn from. You also learn from the code smells and the truly "criminal" stuff that you find. It has made me think more on what constitutes a production ready system. More on that in a coming post.
What I want to share, was a pending bug that I found, after one of the servers the system runs on started to act strange and finally went dead. I found the following code:
static void Main(string[] args)At first glance this looks fine (if you approve of a global exception handler like this), but once you've made the decision to log data of any kind, you must remember to prune your log, so that it does not grow out of proportion - which would be, when it takes up all hard disk space.
{
try
{
Application app = new Application();
app.Start();
}
catch(Exception ex)
{
Logger.Log(ex);
}
}
Once this happens, the OS will throw an IOException every time a write to disk is attempted. You see the trouble now?
- An exception occur somewhere in the system
- The exception is caught by the catch statement
- The logger tries to write the details of the exception to a log
- The hard disk is full and the OS throws an IOException
- Repeat from 2
I guess the programmer who wrote the Logger class did not ever consider, that his implementation would have the potential to crash an entire server, because it lacked pruning of the logs. I'll not lie. Only a few years back I would not have thought of it either.
So here's a rule of thumb you might try to remember:
If your code adds bytes to any persistent media, it is almost certain, that it should also remove bytes with some regularity.
Labels: Anti-pattern, Warstory
Subscribe to Posts [Atom]

