Let’s make testing Agile, they said. Uh, what did they mean by that?

People talk about adopting more Agile practices and using a DevOps-centric culture, but development teams regularly say that testing is a bottleneck.

The Agile development methodology has taken firm root in development houses around the world. Agile replaces the old programming technique of building an entire application before testing it and deploying it – commonly referred to as the “waterfall” method. Instead, with Agile, teams build just a small piece of the application, test it rigorously, then add another small piece and test that rigorously. Then they iterate the process. It has been hugely successful.

The problem is that the development team can get far ahead of the test team. By its nature, testing is a slower, more deliberative process than software development. So a programming team can find itself in a situation like this: The developers add Feature X, which they send off to be tested. Meanwhile, the developers get busy on Feature Y, while the testers work on testing Feature X. By the time the test team finishes the Feature X test process, the developers are working on Z, and then they have to go back to the earlier code to fix defects. 

The easiest and most obvious solution might be to slow down the development process, but who wants to kneecap the developers? The last thing we want to do is make it harder or slower to serve end users.

How about one team?

Traditionally, application development uses separate development and QA teams. But in Agile, there shouldn't be a “dev team” and a “test team,” says Isaac Sacolick, president of digital transformation consultancy StarCIO and author of the book Driving Digital.

“There should be one Agile team responsible for delivering a high quality, business impactful software release that reduces technical debt,” Sacolick says. “Agile user stories and requirements should include acceptance criteria that structure what testing is required. The Agile team of developers, testers, and other roles are then responsible not just for delivering the functionality, but also for ensuring that acceptance criteria are met and that there are defined and executed tests validating the criteria.” 

For teams following DevOps practices, these tests should be automated and many of them instrumented in their continuous integration (CI/CD) pipelines, Sacolick explains. “The team ‘keeps up’ because Agile teams commit to the volume and quality of work as a team. In practice, this is done by teams collaborating on getting a small number of stories done, including testing, before starting new stories.” 

Software engineering lead Brian Okken agrees with Sacolick’s single-team view. “QA can only keep up by being part of development. I'm quite surprised why any developer or organization would want to separate development of production code and development of test code to different people,” Okken says.

Set expectations and get involved early

You may not have the option of changing the traditional team structure, perhaps for corporate reasons. When that’s the case, the approach should start with setting expectations of the QA team, says Anne Hungate, head of infrastructure and operations at RTI International, a research non-profit firm. “When you are putting together the team, decide what a successful QA team looks like. Have this conversation at the start,” she says.

Don’t let the QA team hesitate. “Normally, QA will wait until dev work is done and then run tests. My question to them is, ‘Why are you waiting for code? Why aren’t you looking at stories to see if they are testable?’ Start testing early in the lifecycle. There are things that can be tested without executing code. Those are where you find logic issues to keep your teams going forward,” Hungate says. 

Ask what more QA can do beyond testing. “Can QA simplify processes? Can it sit in between paired programmers and be a third eye? Can it start to write and automate tests so dev doesn’t have to?” Hungate asks. “All this is early work. When QA teams come in late in the lifecycle and you insist on end-to-end testing, you’re going to slow down the dev team.”

Involved developers in testing

The common project process is that the programmers write code, the testers test, and they don’t do each other’s job. But Okken thinks that is a wrong way to approach the task. “Companies that desire to speeding up software testing need to invest in training their software developers in testing techniques,” he says

Some people fear that developers don’t know how to do QA properly, says Okken. “That may be true. But developers can be taught, and there are ways to get around the skill gap in the meantime.” 

In one potential scenario, test coaches are embedded in the development teams. The testing staff maintains the regression suite, the report system, and so on; they may be involved as well in code reviews, exploratory testing. The QA professionals can teach, mentor, and assist developers with writing tests. But fundamentally, the developers write almost all of the automated tests. 

“Again, the tests are developed iteratively with the software under test, so both will be complete at the same time,” Okken notes. 

Another potential scenario is for developers to begin testing each feature and then test engineers take over. Although this is the least effective, Okken suggests, it can be done successfully. “Have software developers do unit tests around critical functional algorithmic code, and skeleton or tracer bullet system level tests for features they are working on. They will develop tests while developing code under test,” he says. 

The tests may not be complete when the developer hands it off to a test engineer, but that test engineer can fill out the test cases where necessary. If done with test code review or pair programming, this model can lead to the first scenario described above. 

For these teams, Sacolick recommends that the dev teams take on most of the functionality testing and select tools that enable automation. “The test teams can then focus on testing that requires a completed build like performance testing, static code analysis, and penetration testing. This approach will slow down delivery and likely create bottlenecks, but at least dev and test teams understand their responsibilities,” he adds.

It’s not just code

Hungate enumerates four project components to consider from the start:

• Personas describe what the application’s user is like, using anything from language, eyesight, performance, response time, and more.
• Project assumptions, or what the developers take for granted about the project and its users
• Workflows, or the iterative means of delivering updates to the code; and
• User stories, which capture and describe a software feature from an end-user perspective. It describes the type of user, what they want, and why.

Too many firms skip these tasks or leave them to the end. In one past project, Hungate says, a week was spent just on persona definitions. Then they worked on workflow. “We asked: ‘Does every persona need to access each workflow?’ Where we said ‘no’ we had less work to do because we didn’t have to test that criteria,” Hungate says.

Automate, automate, automate

Automated software testing is a fundamental part of Agile software development, even though it is not included in the manifesto. Automated testing helps in many ways, say Okken. But in general, a robust test suite helps ensure working software, increases a team’s ability to refactor and extend a software system, and respects individuals by automating the generally boring task of manual regression testing.

“Automated tests also speed up development, further respecting the time of software developers, and allowing faster and more frequent deliveries to end users,” Okken says. “The development of automated tests during production code development helps developers understand the problem domain, the API, the problem at hand better, and help them in turn develop better software. Why would anyone want that learning to go to a separate team and not to the development team?” 

In adopting DevOps, you are discarding the traditional method of development, commonly called “waterfall,” for the more iterative process of building a small amount and testing rigorously we know as Agile. 

It stands to reason, then, that if you are changing the method of development, you should change your method of testing as well. That might mean merging test and dev, as advocated above, or simply a tighter integration between the two. It comes down to finding balance and experimentation, since every coding environment is different, to avoid having the testers or developers waiting for each other.

So, who’s in charge? Recent trends—and the activities of a number of far-thinking companies—suggest that companies may benefit from adding a Chief Quality Officer to their C-suite.

by Andy Patrizio

Andy Patrizio has been covering the Silicon Valley (and everywhere else tech) for more than 25 years, having written for a range of publications that includes InformationWeek, Network World, Dr. Dobbs Journal, and Ars Technica. He is currently based in Orange County, California.

‍