I have no qualms with the basic premise of test-driven development. However, it doesn’t take long before one realizes that any test that is even moderately complicated can itself become a source of bugs. At some point, it feels like the tests themselves are going to need test suites which themselves will need test suites etc ad infinitum.
A case in point is a bug I stumbled upon in lib::filter. The purpose of this module is to allow its user “only allow some specified modules to be locateable/loadable”.
lib::filter accomplishes this by installing an @INC-hook: When perl starts searching @INC for a specific library to load, the hook checks whether the file being requested is allowed to be loaded.
If you look at the test reports for lib::filter, you see almost all green.
Except that a recent cpan-outdated | cpanm on my Windows test-bed pulled it in as a dependency, and it failed the most basic tests.
Basically, lib::filter was correctly refusing to load IPC::Cmd, but it was letting List::Util through.
This was really weird. All I could find on CPANTesters were passing tests except for another report from a Windows system.
This made me suspect something Windows specific was going on. I was rather confused. It soon became apparent that List::Util was being pulled in by Test::More on my system but not the others.
Keep in mind perldoc -f require:
Note that the file will not be included twice under the same specified name. (emphasis mine)
At least that clarified why only one of the two almost identical cases was passing. But, I kept thinking there must be something Windows specific.
Except that I remembered brian’s post about breaking changes in Test::More.
Yup, I had installed a development version of Test::More with this version of Perl on my system. And, this development version, apparently, is loading List::Util whereas earlier versions did not.
So, where is the bug here?
Is my test platform buggy because it includes a development version of Test::More?
Or, is the test buggy for assuming that because neither Test::More nor Test::Exception pulled in List::Util or IPC::Cmd at a particular time in history, they will never pull those modules in ever?
The test itself is buggy because it does not set the expected pre-conditions explicitly. A library like lib::filter cannot go back in time and prevent the loading of a library that happened before lib::filter’s hook was installed in @INC.
Because require will not look up a module in @INC locations once it has been loaded, the test must trick perl into thinking that modules used to check the disallow functionality have not yet been loaded — without affecting the testing framework or the rest of the tests in the script.
My pull request addresses the immediate issue. Clearly, if one is going to have a lot of tests following this pattern, one can write a harness in keeping with the DRY principle.
For now, replacing:
dies_ok { require IPC::Cmd };
dies_ok { require List::Util };with
dies_ok { local %INC = %INC; delete $INC{'IPC/Cmd.pm'}; require IPC::Cmd };
dies_ok { local %INC = %INC; delete $INC{'List/Util.pm'}; require List::Util };will suffice.
Another alternative is to instead ship a little lib directory with the distribution with some trivial module files, and use those to test this functionality, instead of referring to modules that are in widespread use, and, therefore, might end up being already loaded by the time the test is being run.
In this case, using a development version of Test::More allowed us to catch an invalid assumption in a test.
PS: You can discuss this post on /r/perl.