Django Testing Framework == Fail 9
I was reading on a plane recently, and was about halfway through the Django book, when I realized that I hadn't seen anything about how to test my application yet. I tend to be a test first developer and use lots of mocks to play with things, especially at the beginning of a project, so I flipped to the index, turned to the test page, and saw the following:
Testing was still under development when this book was being written, so to learn more you’ll need to read the documentation online at http://www.djangoproject.com/documentation/0.96/testing/
Not a whole lot of help when you're on a plane.
Still, I was willing to give them the benefit of the doubt. Maybe they could create a bolt-on testing framework that would really be something.
Unfortunately, it's just not very good.
Perhaps I'm spoiled by Rails and Rspec, but to me, the Django testing framework feels like something that they put in place just because someone told them that they needed a testing framework. It's not an integral part of the system.
Let's look at their test types.
First of all we have doctests. These emulate the output of the Python interactive interpreter exactly inside of your test code. All this does is to encourage the bad practice of writing some code, firing up your your interpreter, running a few cases, and then cutting and pasting the results into a file. There's no need to think through all of the cases or what you're really doing here. It's just an after the fact, feel-good, safety check.
Second, we have Unit tests. Not the most sophisticated testing framework in the world, but these look like a better choice, very much like any sort of xUnit tests.
The biggest problem I have with these, and the Django testing framework in general, is that the tests don't look to be compulsory. Why, in this day and age, don't we make the generation of a test automatic, and make the user decide to remove it if they really, really don't want to test, not the other way around?
Testing as an afterthought inevitably leads to poorly tested code.
I certainly hope that, as Django continues to grow and mature, testing take a more central role in the future.
<insert ruby v. python flamewar here>
I'll have to admit that your article left me a little confused.
That's not so different than the idea of running some code in an interpreter to do some basic testing is it? Later you say:
Why is running your code in an interpreter a bad practice? You seem to be contradicting yourself. Doctests offer one of many ways that one can test code. To be thorough it is probably a good idea to employ a number of different test vehicles in one's projects. I guess the fact that Django includes more than one test type is a problem for you.
I think you may also be missing the point of doctests. They provide a type of documentation and example use cases for the end user. If I do some initial testing in the interpreter and then save that session to a file I get to reuse what I've just done later as test cases. Seems pretty slick to me. I have not found anywhere in the documentation where the devs say that doctests are the "be all, end all" of testing. It's just a method that is convenient and is easily supported.
Keep in mind that the Django project is still fairly young. The developers and the surrounding community are filled with amazingly talented people. It's just that there are only so many hours in a day and these people are volunteering time to make this project happen. There are other compelling features that require time and effort and which will make Django more attractive to a wider audience as a development platform. To suggest that the Django team doesn't care about testing or test platforms implies that you are fairly ignorant when it comes to evaluating software and projects.
I'll give you the benefit of the doubt that you're aware that other software development methodologies exist besides just "test driven development". These other processes are often dictated by the environments and external requirements in which they're employed. There are still a lot of developers out there who either weren't trained, mentored in, have accepted, or even believe (gasp) that test-driven methodology is the holy grail of software process (at least in the strictest definition of it) I don't think you'll find too many people (if anyone) closely tied to the progress of Django that will tell you that testing or creating test suites is useless, or an afterthought.
It is common in the Django community that many people (most as a pure guess) run the trunk or very recent revisions of Django. It stays pretty stable. Now how do you suppose they pull that off?
Are compulsory tests really such a great idea? Many (most?) Rails projects I've seen have the automatically generated test stubs but never actually added any proper tests to them. Those aren't compulsory tests; they're just a false sense of security.
The Python community actually has a long and respectable history of testing, with doctests being just one example of the testing culture that has taken root there. Don't be too quick to dismiss them; many TDD proponents in the Python world swear by them.
Perhaps I've dismissed doctests too quickly, but they seem like to me like an a bit of a afterthought in the process. I would rather force the developer to think about writing code that's clear enough in the first place that another developer shouldn't need an inline example to see how it works.
I also wonder where people see all of these Rails projects that have no tests. I've worked on 3 major Rails projects and I've consulted on half a dozen others. I've never seen a project that didn't have a full test-suite.
Maybe "major" Rails projects are more likely to have test suites than smaller projects? I have to admit I don't spend much time in the Rails world so maybe my experience has not been representative.
Getting back to your original post, I do think it's unfair to make a grand statement about Django's testing framework being "Fail" and "not very good" but then going light on the details as to why you think that. Are there any critical testing tools you would like to see added?
As I said before, the Python community has an extremely mature testing culture and some excellent testing tools. Maybe we just need more documentation on how to use those with Django.
doctests are ideal for certain types of designs eg pipes and filters, but not always ideal for all types of testing eg integration testing, system testing etc etc.
unittest, doctest and whatever other test framework you find aren't necessarily competing approaches, but complementary. Some might not fit your particular needs, but many, many python developers find them extremely useful.
By the way, I find doctests wonderful for test-driven development.
@Simon: One of the critical tools that I would like added is a way to do integration testing.
When I'm working on a Rails project I use the built in integration tests or the RSpec story runner.
There doesn't seem to be anything about integration testing in the Django documentation, but perhaps I'm missing something.
I agree. Good to get the tests in early for security sake I think.
Django supports Rails-style integration tests using the Django Test Client, which allows you to simulate a user browsing a site and run assertions against both the generated HTML and the template and context that was used to create it.
http://www.djangoproject.com/documentation/testing/#the-test-client
I assume that's what you mean by "integration tests" - that term isn't used in Django's documentation (Rails is the first place I've seen to use that term to describe simulated web client testing).