Automatic testing FTW

16 Jul 2014

When you’re building something made of software (!), it’s widely-accepted best practice to write tests for it. Nothing new there. Attending an excellent event in Cambridge last week, a superb presentation by Tim Perry from Softwire made me realise something else though.

Writing tests makes your application more secure. This sounds trivial, but I mean tests like “does the widget do what it’s supposed to do” tests, not “does the app refuse log in if the password is wrong”.

Tim’s talk drove it home to me: the tools everybody relies on are flawed. Programming languages, web servers, databases and frameworks will all have vulnerabilities. Unit, functional and integration tests make your app more secure because you can update Python, or nginx, or Rails, run your tests and confidently push out the update. Without fearing that you’ve broken something, you’re able to upgrade, patch the vulnerabilities and move on.

Without tests, the temptation to “leave well alone, upgrading might break stuff” is too strong.

I’m glad that, at Owlstone, for instance, even though we just run a well-known CMS (it could almost entirely be a pure static site), I have a suite of tests I can run after every module upgrade. Hosted on heroku and comprehensively tested, stepwize routinely achieves 100% monthly uptime and very few runtime exceptions. I update it (specifically its Ruby gems regularly). In both these situations, I’ve been able to invest the time and effort to write and maintain tests.

Not all the applications I work on are in the same state, of course. I recently answered an administrator’s request: “hey, why’s this bit of our app stopped working?”. The underlying components were reasonably up-to-date, but nobody had used (or tested) one part of the web app for months. Turned out that Django’s upgrade to version 1.6 changed the behaviour of its BooleanField class, which the code relied upon. Thorough reading of Django’s (extensive) release notes might not have been enough for the maintainers to realise the impact, but the most basic functional test would’ve flagged it up.

In other applications, the underlying components are out of date, although they are patched against security loopholes.

Which is better, stability or having the latest libraries? With a well-oiled set of tests you get the best of both worlds, but creating and maintaining tests is expensive, because they require their own development and maintenance. Hopefully as time goes on and the world becomes ever more “web literate”, the time and expense required to create and maintain tests will become more acceptable. Newer test frameworks certainly reduce the necessary developer effort.

Tags: webmaster, coding

|