>100 in-testsuite+ bugs in MailNews Core & Thunderbird
March 23rd, 2009 Posted in Mozilla, SeaMonkey, ThunderbirdA discussion at our status meeting last week, prompted me to take a look at how many bugs MailNews Core and Thunderbird have marked as in-testsuite+. I was pleased to see it was just reaching 100.
(For those that don’t know, we mark a bug as in-testsuite+ when we have an automated test that covers that bug).
I committed the first in-testsuite+ bug for MailNews in November 2007, this covered some very basic address book functions. Since then we’ve grown the tests to cover address book (more extensively), POP3, IMAP, SMTP, NNTP, bayesian spam filter, aspects of message composition and sending, gloda (the new search database), text import of address books, searching messages, folder management, and a few other things.
Although our actual code coverage of these areas is not very big, we do cover a lot of different areas of MailNews with these tests. Our major protocols are covered (to a limited extent) with the use of fake servers - as a result of the password manager switch, we also have basic authentication tests for these protocols. So if someone breaks something significant, we’re much more likely to know about it before it even hits the nightly builds.
We’ve never (yet) had a policy of all patches requiring unit tests, because for Thunderbird we haven’t got the same testsuite possibilities as Firefox - mainly due to the lack of a browser interface. Various solutions have been proposed and partially investigated, I expect we’ll be looking at the chrome/UI area of testing more after we release Thunderbird 3.
We’re going to be looking at how to encourage writing more unit tests, and what we can do to make writing them easier, so keep watching.
I’d like to say thank you to all those that have written unit tests for MailNews so far. Keep going, unit tests have, and will make Thunderbird & SeaMonkey better products.
One Response to “>100 in-testsuite+ bugs in MailNews Core & Thunderbird”
By Kent James on Mar 23, 2009
Of the 100+ in-test-suite+ bugs, over half were done by you. I checked my personal count, and I’m a (distant) second with 16 in-testsuite+ bugs (after I added 4 that I believe you missed.) Somehow I bet that the vast majority of asuth’s bugs should also have in-testsuite+ set as well, which would probably make him a much more prolific test writer than I am.
I was inspired to learn to do automated testing after reading some of the material on test-driven development, which makes a lot of sense to me. After all, developers should be doing testing anyway, and automating it so that it can be repeated makes a lot of sense. There is an initial learning curve, and the development of a personal test writing style and infrastructure that works with your style and specialty. But at this point, for me it is very natural to write a test for any changes that I do to a .idl file. It would be great though if there was some way to see if the written tests exercised all of the changed code, as I know that many of my tests were pretty basic does-it-work-at-all tests rather than attempting to explore the edge cases.
At the same time, there are other cases where it would be pretty difficult to write tests for new bugs. That includes:
1) minor fixes in code where there currently is no test coverage at all.
2) stupid fixes (where we just prevent the immediate cause) of crashers
I hope that you can encourage unit tests to be the norm in the future, but at the same time not force them in cases where they are much more effort than rfixing the original bug.
And thanks for all of your work on the testing infrastructure. Generally it works very well now from my perspective.