Software Reality

DDT is Code (And Test) Complete

Actually the chapters were all wrapped up a couple of weeks ago but this is the first chance I've had to blog it. The twelfth and final chapter (Unit Testing Algorithms) has landed on the Alpha Books page, if you're interested in getting a sneak peek. At the moment we're just waiting for page proofs to arrive - always an exciting stage, as the manuscript suddenly resembles a "finished", print-ready book - which should happen in the next week or so.

 

I'm especially pleased with the Algorithms chapter, as it demonstrates that the design driven testing technique can be applied to other design diagrams besides the "core set" that we've based the book around  (that is: robustness diagrams for conceptual design, and sequence diagrams for detailed design; while customer-level tests are driven from structured use case scenarios and high-level business requirements).

DDT: Automated integration testing

Another chapter in the bag, thankfully. Just one more to go (about algorithm testing)... But in the meantime, here's how the integration testing chapter starts:

Automated integration testing is a pig, but it’s an important pig. And it’s important for precisely the reasons why it’s so difficult.

The wider the scope of your automated test – i.e. the wider the chasm that your integration testing pig has to fly across (rather like the one in the above photo) – the more problems you’ll encounter; but these are also “real-world” problems that will be faced by your code, both during development and when it’s released. So an integration test that breaks – and they will break frequently, e.g. when a dependent system has changed unexpectedly – isn’t just a drag, it’s providing an early warning that the system is about to break “for real”. Think of an integration test as your network canary.

Integration tests are also important, and worth the pain of setting them up and keeping them working, because without them, you’ve just got unit tests. By themselves, unit tests are too myopic; they don’t assert that a complete operation works, from the point when a user clicks “Go” through the complete operation, to the results displayed on the user’s screen. An end-to-end test confirms that all the pieces fit together as expected. Integration is potentially problematic in any project – which is why it’s so important to test.

====

One of the themes in this chapter is that you can use a conceptual design (i.e. robustness diagram - a picture of a use case, showing controllers, entities and boundary objects) to identify what kinds of tests to write - e.g. a controller "talking" to an entity class suggests that you'd write a normal unit test; a controller talking to an external system interface suggests that you'd write a "controller-level" integration test; a controller talking to another controller suggests that you'd write an "algorithm-level" unit test; and so on.

It's kind of a systematic approach to driving tests from your design...

DDT alpha book - 2 new chapters available

Two new chapters fresh up on the Apress alpha books page:

Chapter 5 "Detailed Design and Unit Testing"

In DDT we make the distinction between unit testing - testing at the atomic level - and controller testing - testing at the molecular level. Chapter 5 shows how to drive "atomic-level" unit tests from a detailed UML design, identifying test cases and scenarios from key points in the design.

Chapter 6 "Conceptual Design and Controller Testing"

A conceptual design is the first foray into the design of a system, having produced use cases to describe the user/system interaction. The system behaviour is captured in "controllers" (logical software functions). Later, each controller will become one or more "real" functions/methods on your classes; but having identified these high-level functions at a conceptual level, why not write tests at this level as well? That's what controller testing is all about - it gives you more "bang for your buck", as you need to write fewer tests but they give you essentially the same feedback as micro-grained unit tests.

Whereas a unit test may be "concerned" with one software method, a controller test operates on a small group of closely related methods that together perform a useful function.

Requirements Testing 101 - Involve Your Team

So many projects suffer from "over the fence syndrome": that is, the business analysts put together a specification (if you're lucky) and throw it over the fence to the developers. The developers code up the system and throw it over the next fence along to the testers. The testers sign off on the developed system, maybe even referring back to the original requirements spec, and then the system gets released - thrown over the really big wall (with the barbed wire at the top) to the customer waiting outside. That's not the way to operate.

In theory, the sort of project I've just described ticks the boxes of "failsafe development": there are requirements specs, analysts, even a team of testers and a release process. So what could possibly go wrong?

The problem is that in this kind of setup, no-one talks to each other. Each department regards itself as a separate entity, with delineated delivery gateways between them. Throw a document into the portal, and it emerges at some other point in the universe, like an alien tablet covered in arcane inscriptions that will take thousands of years for linguistic experts to decipher. Naturally, requirements are going to be misinterpreted. The "sign-off" from the testers will be pretty meaningless, as they were never closely involved in the requirements process: so their understanding of the true business requirements will be superficial at best.

Really effective testers will question the requirements and spot loopholes in business logic. And you'll be eternally grateful to them when they prevent a project from being released into production with faulty requirements. But the testers don't stand a chance of doing this if they're not closely involved with developing the requirement tests.

Let the testers "own" the requirement tests (i.e. be responsible for their development and maintenance). The customer and your business analysts build the requirements up; the testers write tests to knock them down again.

Another way to think of the business tests suite is that it's an independent audit created by a testing organization to show that the system created by the developers does what the customer intended. If the developers are left to manage the testing themselves, then you end up with the Green Bar of Shangri-La syndrome: the system tells us there's a green bar, so everything's okay man... This is why it's vital to have a separate team - the QA testers, who are answerable to the business, not to the developers - verifying the software. And if the testers are involved early enough, they will also verify the requirements themselves.

DDT Chapter 8 (Requirements Testing) is now up on the Apress Alpha Books page...

Code Bubbles and Porky Visual Studio

Here's my latest article at The Register:

The latest incarnation of Visual Studio had a difficult gestation, with critics pointing to the slowness and instability of the beta release...

... This isn't just a criticism leveled solely at Visual Studio; it's against all the modern IDEs out there - VS in day-to-day use is not so different from what we were doing with the first version of Visual Basic 19 years ago. Somehow, Microsoft managed to fatten Visual Studio up to porcine levels and yet using the product feels like you're trapped in the 1990s.

Java IDEs suffer from the same problem, to an extent. IDE evolution has become a game of feature creep, each new release touting bold new features, drawing in concepts well outside an IDE's core competency - half-baked UML support springs to mind - and packing it all into a single frame. The notion of "doing one thing, and doing it well" has been laughed out the window.

And the reason for this stagnation of innovation in the IDE space? The industry leaders have become portly, comfortable, and middle-aged. Eclipse is smoking its logical representation of a pipe, content to let plug-in developers pad out its ecosystem. NetBeans plays perpetual catch-up while its fate is decided for it. IntelliJ no longer sells itself on productivity, having fallen into the "feature checklist" trap.

The time is ripe for a brand-new contender to stroll in with something fresh and steal developer mindshare from beneath the leaders' pipe-yellowed moustaches - just as a young Eclipse once did to NetBeans.

The bubblicious Code Bubbles IDE could be just that new contender. It puts a leash around Eclipse to do the donkey work for it, but invents an entirely new front-end so that programming becomes more like playing with Matey in the bath. A prototype from Brown University in New England and video about Code Bubbles has passed the frothy developer test, with viral emails flying around the workplace.

>> Read more at The Register

DDT Acceptance Testing chapter now fully baked

Counting backwards (as we appear to be writing the book in reverse), the next chapter - Chapter 8, Acceptance Testing - is now up on the Apress Alpha Books page.

We've divided acceptance testing into two areas - business requirements (the subject of Chapter 8), and behavioral requirements (the subject of Chapter 7, which we've just completed and should be up on the Alpha Books page soon).

So Chapter 8 is really Acceptance Testing Part II (or Part I, depending which way around you look at it...)

We think QA testing teams should be very happy with Chapter 8, as it demonstrates an automated method of turning business requirements into test reports and checklists. The chapter shows how to generate customer-facing (and tester-facing) business requirement tests from the requirements in your analysis model, how to organise those tests, and how to generate a test plan document from them. This process supports an independent Quality Assurance organisation, providing a level of assurance that the customer's requirements have been met, beyond the development team asserting that "all our unit tests light up green."

User Story != Use Case - but User Story != Story, Either

A user story is not a use case.

That might seem as obvious as saying "A door is not a cat", but there's enough scope for confusion - not just in the similar names - that user stories and use cases are often used interchangeably (when describing what to do next in a software project) when they probably shouldn't be.

Alistair Cockburn sums it up nicely: A user story is to a use case as a gazelle is to a gazebo.

But also:

A user story is not a story.

In other words - this is not a user story:

"The user types in a story and clicks Submit. The system publishes it to the website so that other users can read it."

It's really a use case. But it also doesn't help that there are many definitions of "use case", and many opinions on how much detail should go into one. But generally, the one consistent aspect of a use case is that it's a narrative: a description of a user's journey through the UI, and the system's responses as he clicks and types his way along: "The user does this or that; the system displays XYZ data in response. Next, the user does this... etc." In other words, the text of a use case is really a story.

A user story, meanwhile, is anything but a narrative. And this, I think, is the main source of the confusion. The name "user story" implies a story - a narrative. And it really doesn't help matters that the name is often shortened to "story". But a user story is anything but a story - at least in the narrative sense. I could see people starting to call narrative-based requirements "stories", but it's a false etymology, as they say.

'Design Driven Testing' now available as an alpha book

"Design Driven Testing", which I'm co-authoring with Doug Rosenberg, is due for release by Apress later this year.

It's a little unsettling to see chapters available so soon after writing them. But definitely a good thing.

Three chapters are available in Apress' Alpha Books programme so far... we've just submitted another two chapters for technical review, so these should land on the Alpha Book page pretty soon.

We've changed the chapters around a bit in the last few weeks. Here's the current line-up:

Part I - DDT vs TDD

1. Somebody Has it Backwards
2. Hello World Using TDD (implementing a "Login" story with TDD)
3. Hello World Using DDT (implement a "Login" story/use case with DDT)

Part II - A Walk Backwards Through the Mapplet 2.0 Project

(In this part of the book, we walk backwards through a real-life DDT project being run at ESRI Professional Services in Redlands, California. Our story begins with the completed product - code, tests, UI - and ends at the beginning, with the initial set of requirements. We demonstrate how tests were used to provide feedback at each stage of the project. The project's actually going on right now, and we're documenting its progress - meaning we're effectively writing this set of chapters backwards...)

4. Code and Finished Product
5. Detailed Design and Unit Testing
6. Preliminary Design and Behavioural Testing
7. Use Cases and Acceptance Testing
8. Test Cases and Requirements

Part III - Advanced DDT

9. Unit Testing Don'ts (Testing antipatterns)
10. Unit Testing Design "TODOs" (how to write "test-conscious" code)
11. How to write Enterprise Architect Test Case transform templates
12. When Designing for Testability is Taken Too Far

The book's also now visible on Amazon - which makes me think that I should probably get on and finish it now...

Programmer Archaeology - The perils of joining a mature project

The majority of software projects are ongoing efforts - never ending cut-and-release marathons. New screens and functions are released on (or around) specific agreed dates, but the project itself keeps on going. So it’s far more common for developers to join a project mid-flow, with its own dusty corners that no-one dares to disturb, its homegrown O-R mapping frameworks, crusted together with the foibles and wrinkles of a dozen long-since-departed self-styled code gurus.

By contrast, starting a brand new project from scratch is a rare joy: the opportunity to set the standards, consider the architecture, evaluate the available technologies given the business requirements; and to produce clean, maintainable code that’s easily unit-tested.

Ah, that last one. There’s nothing quite like the pain of joining a “mature” project and discovering that unit tests were never even a glimmer in the eyes of the project’s founding coders; the forefathers who laid the treacle-like foundations for their successors to follow.

Like a Spanish explorer in a South American jungle, you’re unlikely to find a lost city of gold; instead, prepare to discover the overgrown remnants of one long-lost programmer’s vain attempt to add unit tests - a strained nod toward provable code quality, quickly abandoned when the poor programmer was beaten back by the tangle of dependencies, the onslaught of complex constructors and static initializer blocks like Aztec ghosts sending the programmer scrambling to the far end of the jungle.

Test Smarter, Not Harder

A culture of 100% test coverage has grown up around TDD. Code quality “measuring” tools such as Clover are used as a pugil stick to beat developers over the head with, if their code doesn’t measure up to an arbitrary set of metrics: and one of these is a measure of how much product code is covered by unit tests.

(Clover does have some pretty awesome features by the way, such as test optimization; but institutionalising test coverage as a measure of code quality isn’t one of them).

TDD itself can become a highly elaborate, labour-intensive way of creating code. It’s not uncommon to end up with far more test code (which the customer never gets to see, and which arguably doesn’t really “do” anything) than product code, which is what the customer is really paying for. Yes, the customer is also paying to have product code of a sufficient quality, and that’s what TDD addresses. But there has to be a better way of improving code quality with tests: a way that doesn’t soak up developers’ time and turn the tests into the main software deliverable, in the developers’ minds.

There are three main shortcomings to TDD:

1. It’s too time-consuming
2. You end up with too much test code, with code paths duplicated by different tests
3. The final amount of test code and product code that you see is disproportionate to the amount of refactoring/evolutionary design that went on to get there

To sum these three issues up: TDD is Too Damn Difficult; it takes too much effort to produce a bit of “production code” result.

Not using TDD – just writing tests after you’ve written some code – also has problems, in particular the aimlessness of this approach. It’s difficult to know when you’re done writing enough tests – in practice there won’t be enough time to hit 100% coverage, so the team must stop at some arbitrary timeboxed stage somewhere along the road. Most dissatisfying.

So here’s the idea. These shortcomings can all be addressed by first doing a small amount of structured up-front analysis and design, and identifying which parts of the code you’ll gain the most benefit from testing. We call this approach Design Driven Testing or DDT (the linked tutorial illustrates how to use an add-in for Enterprise Architect to automate much of DDT).

Take this sequence diagram, an overview of the code path for a hotel search (click the image for the full-size version):