Software Reality

Message Queue shootout

If you're looking for low latency messaging, here's an interesting comparison of some of the more popular MQ products: ZeroMQ, RabbitMQ, ActiveMQ and MSMQ. As the author noted, ZeroMQ is a different beast to the others, as it has a broker-less, peer-to-peer architecture - meaning that it doesn't require a standalone, single-point-of-failure server process.

The benchmark results are startling to say the least: ZeroMQ stands tall over the others, it's like placing the Burj Khalifa next to Big Ben.

However the results are easily debunked. (Debunked is probably the wrong word, as the author does state that ZeroMQ shouldn't really be part of the test). Without doubt ZeroMQ is a quality product, an extremely fast client; but it doesn't provide non-durable queues, meaning it isn't failsafe, and you'll need to provide your own auditing and recovery. An in-memory process is always going to outperform an equivalent process that regularly huffs its data out to disk.

But needs vary, and if you need "very" low-latency messaging (say 10,000 messages/second with latencies under 200ms), a non-broker-based architecture like ZeroMQ or 29West is needed.

There are plenty of other high-performance messaging solutions out there, of course: Kafka, RavenMQ, HornetQ, the Spread Toolkit, 29West (or Informatica Ultra Messaging as it is now) and of course Tibco EMS.

TfL's Countdown II embraces API-hungry developer masses

As part of an occasional series on "cool" IT projects, I wrote this piece on TfL's Countdown II project, for The Register. The project updates the old saying about three buses coming along at once, to "You'll wait 10 minutes and then 3 buses will arrive within 30 seconds of each other":

As winter sets in, commuters in London stand huddled at bus stops, hoping the shelters' electronic signs report an accurate "due" arrival time rather than a crappy guess.

Those lucky enough to have Transport for London's Countdown system of signs installed at their stops can judge by the minutes flickering on the display whether to brave the chill or nip into the warm caff for a coffee.

The problem is, the signs are not at every stop so some passengers are out in the cold figuratively as well as literally. The equipment is also susceptible to vandalism, and the timetable information isn't being pushed to the phone in your pocket.

But TfL plans to change this with the launch of Countdown II, which went live on Monday after three years in the making. ...

Read the full article

Acceptance Testing - Expanding Use Cases using Structured Scenarios

Chapter 7 of Design Driven Testing just went onto the Sparx Community website as a free download. (Sparx are the creators of Enterprise Architect, nowadays the *only* real choice for enterprise analysis and design modeling).

Somewhat uniquely, the book features a real production system as its teaching example. A worldwide interactive hotel mapping (GIS) application, designed with the ICONIX Process, built using Java, Flex, and the ESRI ArcGIS Server mapping software, that we call "Mapplet 2.0", which is in production use on the VResorts.com travel website.

Here's how the chapter is introduced on the Sparx site:

This chapter focuses on Acceptance Testing, and leverages EA's Structured Scenario editor heavily to accomplish something we call "use case thread expansion" where all of the sunny day / rainy day permutations of a use case are expanded out into a complete set of tests.

In test driven approaches to development, unit testing often gets most of the attention.  However, unit testing is generally most useful in discovering "errors of Commission" (more poetically, "whoops, I coded that wrong").  Unit testing is of much less help in discovering "errors of Omission" (more poetically, "whoops, I didn't think of that").  In general errors of Omission are much trickier to detect, and there is very little automated support for detecting them.  We worked very closely with the development team at Sparx as they developed the "use case thread expander", and it brings a very unique and useful capability to the industry.

As you read this chapter, make sure you don't miss the discussion at the end of the chapter called "And the moral of the story is..." where we describe some actual "errors of Omission" that were caught and fixed before the release of our mapping software using these acceptance testing techniques, and how fixing these errors improved the user experience.

Here's the book excerpt over at Sparx.

In case you're still undecided about whether to buy a copy of DDT (or like me, if you just like reading free chapters!), Chapter 4 is also available for download. This chapter presents the full ICONIX Process design of the Mapplet project, starting from functional requirements and use cases, all the way down to reverse engineered class diagrams from the final code.   One of the unique virtues of using a production example as a teaching example in a book like DDT is that it's possible for readers to look at the use cases in the attached chapter, and then compare them to the released software as deployed on VResorts.com.

Petabyte-chomping big sky telescope sucks down baby code

My latest article at The Register:

Robert Heinlein was right to be worried. What if there really is a planet of giant, psychic, human-hating bugs out there, getting ready to hurl planet-busting rocks in our general direction? Surely we would want to know?

Luckily, big science projects such as the Large Synoptic Survey Telescope (LSST), which (when it's fully operational in 2016) will photograph the entire night sky repeatedly for 10 years, will be able to spot such genocidal asteroids - although asteroid-spotting is just one small part of the LSST's overall mission.

Two years ago we spoke to Jeff Kantor, LSST data management project manager, who described the project as "a proposed ground-based 6.7 meter effective diameter (8.4 meter primary mirror), 10 square-degree-field telescope that will provide digital imaging of faint astronomical objects across the entire sky, night after night."

I caught up with Jeff again a couple of weeks ago, and asked him how this highly ambitious project is progressing.

...

They now have a prototype system of about 200,000 lines of C++ and Python representing most of the capability needed to run an astronomical survey of the magnitude typically done today. Next, they have to scale this up to support LSST volumes. According to Jeff: "We hope to have all of that functioning at about 20 per cent of LSST scale of the end of our R&D phase. We then have six years of construction and commissioning to 'bullet-proof' and improve it, and to test it out with the real telescope and camera."

Read the full article here.

Design Driven Testing - Test Smarter, Not Harder

It's finally done, and available to buy on Amazon!

This news item was just posted on the Sparx Systems website:

The groundbreaking book Design Driven Testing brings sanity back to the software development process by flipping around the concept of Test Driven Development (TDD)—restoring the concept of using testing to verify a design instead of pretending that unit tests are a replacement for design. Anyone who feels that TDD is “Too Damn Difficult” will appreciate this book. Design Driven Testing shows that, by combining a forward-thinking development process with cutting-edge automation, testing can be a finely targeted, business-driven, rewarding effort. In other words, you’ll learn how to test smarter, not harder.

• Applies a feedback-driven approach to each stage of the project lifecycle.
• Illustrates a lightweight and effective approach using a core subset of UML.
• Explains how to use Enterprise Architect to generate automatically acceptance test cases from structured scenarios
• Explains how to use Enterprise Architect and the Agile/ICONIX add-in to generate JUnit and FlexUnit test code
• Follows a real-life example project using Java and Flex/ActionScript.
• Presents bonus chapters for advanced DDTers covering unit-test antipatterns (and their opposite, “test-conscious” design patterns), and showing how to create your own test transformation templates in Enterprise Architect.

What you’ll learn

• Create unit and behavioral tests using JUnit, NUnit, FlexUnit.
• Generate acceptance tests for all usage paths through use case thread expansion.
• Generate requirement tests for functional requirements.
• Run complex acceptance tests across the enterprise.
• Isolate individual control points for self-contained unit/behavioral tests.
• Apply Behavior Driven Development frameworks like JBehave and NBehave

Design Driven Testing should appeal to developers, project managers, testers, business analysts, architects… in fact anyone who builds software that needs to be tested. While equally applicable on both large and small projects, Design Driven Testing is especially helpful to those developers who need to verify their software against formal requirements. Such developers will benefit greatly from the rational and disciplined approach espoused by the authors.

DDT is Code (And Test) Complete

Actually the chapters were all wrapped up a couple of weeks ago but this is the first chance I've had to blog it. The twelfth and final chapter (Unit Testing Algorithms) has landed on the Alpha Books page, if you're interested in getting a sneak peek. At the moment we're just waiting for page proofs to arrive - always an exciting stage, as the manuscript suddenly resembles a "finished", print-ready book - which should happen in the next week or so.

 

I'm especially pleased with the Algorithms chapter, as it demonstrates that the design driven testing technique can be applied to other design diagrams besides the "core set" that we've based the book around  (that is: robustness diagrams for conceptual design, and sequence diagrams for detailed design; while customer-level tests are driven from structured use case scenarios and high-level business requirements).

DDT: Automated integration testing

Another chapter in the bag, thankfully. Just one more to go (about algorithm testing)... But in the meantime, here's how the integration testing chapter starts:

Automated integration testing is a pig, but it’s an important pig. And it’s important for precisely the reasons why it’s so difficult.

The wider the scope of your automated test – i.e. the wider the chasm that your integration testing pig has to fly across (rather like the one in the above photo) – the more problems you’ll encounter; but these are also “real-world” problems that will be faced by your code, both during development and when it’s released. So an integration test that breaks – and they will break frequently, e.g. when a dependent system has changed unexpectedly – isn’t just a drag, it’s providing an early warning that the system is about to break “for real”. Think of an integration test as your network canary.

Integration tests are also important, and worth the pain of setting them up and keeping them working, because without them, you’ve just got unit tests. By themselves, unit tests are too myopic; they don’t assert that a complete operation works, from the point when a user clicks “Go” through the complete operation, to the results displayed on the user’s screen. An end-to-end test confirms that all the pieces fit together as expected. Integration is potentially problematic in any project – which is why it’s so important to test.

====

One of the themes in this chapter is that you can use a conceptual design (i.e. robustness diagram - a picture of a use case, showing controllers, entities and boundary objects) to identify what kinds of tests to write - e.g. a controller "talking" to an entity class suggests that you'd write a normal unit test; a controller talking to an external system interface suggests that you'd write a "controller-level" integration test; a controller talking to another controller suggests that you'd write an "algorithm-level" unit test; and so on.

It's kind of a systematic approach to driving tests from your design...

DDT alpha book - 2 new chapters available

Two new chapters fresh up on the Apress alpha books page:

Chapter 5 "Detailed Design and Unit Testing"

In DDT we make the distinction between unit testing - testing at the atomic level - and controller testing - testing at the molecular level. Chapter 5 shows how to drive "atomic-level" unit tests from a detailed UML design, identifying test cases and scenarios from key points in the design.

Chapter 6 "Conceptual Design and Controller Testing"

A conceptual design is the first foray into the design of a system, having produced use cases to describe the user/system interaction. The system behaviour is captured in "controllers" (logical software functions). Later, each controller will become one or more "real" functions/methods on your classes; but having identified these high-level functions at a conceptual level, why not write tests at this level as well? That's what controller testing is all about - it gives you more "bang for your buck", as you need to write fewer tests but they give you essentially the same feedback as micro-grained unit tests.

Whereas a unit test may be "concerned" with one software method, a controller test operates on a small group of closely related methods that together perform a useful function.

Requirements Testing 101 - Involve Your Team

So many projects suffer from "over the fence syndrome": that is, the business analysts put together a specification (if you're lucky) and throw it over the fence to the developers. The developers code up the system and throw it over the next fence along to the testers. The testers sign off on the developed system, maybe even referring back to the original requirements spec, and then the system gets released - thrown over the really big wall (with the barbed wire at the top) to the customer waiting outside. That's not the way to operate.

In theory, the sort of project I've just described ticks the boxes of "failsafe development": there are requirements specs, analysts, even a team of testers and a release process. So what could possibly go wrong?

The problem is that in this kind of setup, no-one talks to each other. Each department regards itself as a separate entity, with delineated delivery gateways between them. Throw a document into the portal, and it emerges at some other point in the universe, like an alien tablet covered in arcane inscriptions that will take thousands of years for linguistic experts to decipher. Naturally, requirements are going to be misinterpreted. The "sign-off" from the testers will be pretty meaningless, as they were never closely involved in the requirements process: so their understanding of the true business requirements will be superficial at best.

Really effective testers will question the requirements and spot loopholes in business logic. And you'll be eternally grateful to them when they prevent a project from being released into production with faulty requirements. But the testers don't stand a chance of doing this if they're not closely involved with developing the requirement tests.

Let the testers "own" the requirement tests (i.e. be responsible for their development and maintenance). The customer and your business analysts build the requirements up; the testers write tests to knock them down again.

Another way to think of the business tests suite is that it's an independent audit created by a testing organization to show that the system created by the developers does what the customer intended. If the developers are left to manage the testing themselves, then you end up with the Green Bar of Shangri-La syndrome: the system tells us there's a green bar, so everything's okay man... This is why it's vital to have a separate team - the QA testers, who are answerable to the business, not to the developers - verifying the software. And if the testers are involved early enough, they will also verify the requirements themselves.

DDT Chapter 8 (Requirements Testing) is now up on the Apress Alpha Books page...

DDT Acceptance Testing chapter now fully baked

Counting backwards (as we appear to be writing the book in reverse), the next chapter - Chapter 8, Acceptance Testing - is now up on the Apress Alpha Books page.

We've divided acceptance testing into two areas - business requirements (the subject of Chapter 8), and behavioral requirements (the subject of Chapter 7, which we've just completed and should be up on the Alpha Books page soon).

So Chapter 8 is really Acceptance Testing Part II (or Part I, depending which way around you look at it...)

We think QA testing teams should be very happy with Chapter 8, as it demonstrates an automated method of turning business requirements into test reports and checklists. The chapter shows how to generate customer-facing (and tester-facing) business requirement tests from the requirements in your analysis model, how to organise those tests, and how to generate a test plan document from them. This process supports an independent Quality Assurance organisation, providing a level of assurance that the customer's requirements have been met, beyond the development team asserting that "all our unit tests light up green."