From the Agile Transformation Trenches

How agility drives product development - an inside view

Greetings. In my last post - part two of a three-part series on Agile Testing - I talked about the importance of test automation in Agile (Agile Testing: what does it take? Pt. 2). Before finishing that thread, I would like to expand on this topic a bit more.  You see, today is a milestone for me.  We are releasing a significant update to my product line - a suite of test automation products - and I think it is a great moment to share with you what makes me so excited about the work I do. First of all, I consider myself fortunate to work with a great team. Collectively, we are responsible for defining the strategy of Borland's Lifecycle Quality Management (LQM) products. In the last couple of years the general direction of these products has been greatly influenced by Borland's own transition from a traditional, waterfall-based organization to one that lives and breathes Agile.

During our transition we not only realized that we needed to change many of the long- established processes and cultural norms, we also realized that we needed to enhance our products in order to better support Agile development scenarios in addition to continuing to improve our support for the core needs of all QA teams. Also, as an ISV we obviously have the same need for quality as any of our customers and, consequently, we not only use our own products to test our software but leverage our many years of experience in the testing domain to continuously improve our products.

As you may have seen, Borland today announced the latest versions of the Silk products which represent a significant step in supporting 21st century testing. The new releases not only provide true test automation for state-of-the-art applications, but they now offer some really exciting enhancements to better support Agile teams. Now I can already hear many of you moan about how I am trying to sell the Borland products. But that part of the post is done. The link above will let you  take a closer look at the products if you are interested. My real intention for this post is to share the experiences that were the genesis for these products and through that sharing, hopefully, show you why am so excited about them.

There is no hiding in Agile

As we adopted more and more Agile practices in the Linz office, we noticed some issues surfacing and then becoming more and more pressing. Amongst these was our need for better support of Agile workflows in our Silk products. For example, we quickly learned that Agile teams work in a pull model as opposed to the traditional method of assigning work items. This was especially relevant from a test management perspective. As a consequence of our own teams' experiences, we realized we needed to provide much more collaborative views that allowed the team members to see what was going on so that they could select testing-related tasks from a list of available work items that had not yet been picked up by another team member, or have easy and quick access to related information. These requirements led to a number of enhancements in SilkCentral Test Manager such as Agile project templates, a new Web-based manual testing interface and an integration to Version One to support additional Agile test planning tools. These enhancements quickly proved to be an enormous help in our teams' daily work process by supporting self-organization, providing the necessary visibility into the many activities, and making collaboration much easier and efficient.

Another important need we quickly identified was the ability to leverage existing tests as well as the ability to easily integrate new tests, e.g. unit tests. It quickly became clear that it was a major hurdle for the Agile teams to have to maintain test plans both externally (for example as part of a unit testing framework) as well as in the test management tool. This gave us the idea for external test package support, which now allows users to just point to the external test plan and execute all the tests that are part of it. Gone are the days where we needed to manually create test stubs in the test management tool and add the necessary details in a subsequent manual step which, of course, is cumbersome when dealing with a large number of tests. Now it is as simple as pointing to a suite of existing unit or FitNesse test plans and executing them directly from SilkCentral Test Manager. And voila! All your results get automatically pulled in!

Finally, probably the most important requirement we uncovered, was the need to automate increasingly more tests. Anyone who has built test automation frameworks in the past knows that this is a very challenging task. The brittleness of test scripts combined with the potentially fast- changing GUI of an application are just two of the factors that have made the creation of a robust test automation framework extremely difficult. Our teams quickly realized they needed to do something about it.

Addressing the challenges

We immediately identified two closely related areas of improvement that we would have to address. First of all, from the perspective of a tester we looked for ways to improve the testability of our applications. Introducing unique identifiers for each of the objects in the products we develop was a significant step to help improve testability of the applications as well as the stability of test scripts. This also quickly lead to an improved understanding from the product development point of view about how we needed to enhance our test tools' capabilities to best leverage these kinds of testability enhancements of an application. In order to be able to build solid test scripts we needed to have stable and consistent object recognition. So the first obvious improvement we set out to make to the Silk suite was to ensure that the products provided a way to consistently find simple locators which were unique for each control - the same for each call, the same for each build, and the same for each instantiation of the application. Ideally, they would be readable and follow standard naming conventions well. With that in mind we enhanced all of our products so that they would intelligently optimize their object recognition strategies accordingly for testability.

Another major challenge our teams faced was around testing AJAX and other Web 2.0 applications. Due to their asynchronous nature they cannot be tested reliably with a traditional serialized test logic approach. It became obvious that synchronization needed to be done in the test tool rather than at the script level. Consequently, we enhanced SilkTest with everything that was needed to remove the necessity for the tester to try and mimic the (nondeterministic) nature of asynchronous applications allowing for much leaner test script code.

Finally, with all of our teams moving to two week iterations, speed became of paramount importance. The teams needed to be able to run their ever-increasing regression sets in much smaller time frames. There was an increased need to test more - and more often. We quickly realized that we needed to directly leverage RIA technologies to speed up replay of test scripts instead of leveraging an on-the-surface approach like using Javascript engine-injection for browsers. This approach, used by many other testing tools, just didn't work for our teams. The security restrictions, incredibly slow replay speed, and inability to handle actions that happen outside the browser's DOM (like Windows message boxes that come up) are just some of the problems we ran into with other tools. So what do you do if you are a test tool vendor? You build what you need!

These are just a couple of the challenges that our teams experienced that influenced the direction of the product line. One of the key advantages you have when you get immediate feedback through internal usage, and then have the opportunity to and refine and improve products on a sprint-by-sprint basis is that customers (and we are one of our customers) reap the benefits very quickly.

Bye for now.

Agile Testing: what does it take? Pt. 2

Hello again. This is the second in a three-part series on the most critical issues in Agile testing.  In my first post, I talked about how people are affected by a change to the Agile paradigm focusing on team integration and the required skill sets (Agile Testing: what does it take? Pt. 1). Now let's look at the role of testing tools, and more specifically the importance of test automation.

Striving for a high percentage of test automation is indeed another important aspect of Agile testing. In order to keep up - or accelerate -- the velocity of Agile teams, test automation becomes an absolute must. I'd even say it's impossible to keep up with quality in an Agile environment without test automation.

In Agile environments there is a serious need for speed! The best way to accelerate the Code-and-Test Process is with fast, automated test scripts. Automated testing enables you to achieve "Simplicity - the art of maximizing the amount of work not done," which is considered essential in an Agile environment. 

Once tests no longer have to be performed manually, teams can test much more rapidly and achieve more in less time. Automation also enables testers to create simple, reusable scripts, which they can deploy to save time and increase the consistency of testing across similar user stories, or requirements within and across projects. And, finally, automated testing allows "sponsors, developers, and users … to maintain a constant pace, indefinitely." It significantly lightens the workload of testers and eliminates the need for late-night and weekend testing marathons that can that burn teams out. 

In addition to automating the more common areas of unit testing and functional testing, automated performance testing gives teams the ability to address fundamental issues in the architecture of their application early on when it is easy to make changes.  

So, I've covered two key concepts:  integrating and automating. In the last post in this series, I will expand on this in the context of the "test early and often principle," which not only works well in Agile environments but is a key contributing factor to keeping quality levels high. 

Bye for now

Agile Testing: what does it take? Pt. 1

Greetings. My name is Joachim Herschmann, and I am a Director of Product Management here at Borland located in Borland’s Linz office. I have over 15 years of experience in the software development and testing disciplines, and I am a certified ScrumMaster. I joined Borland in April 2006 through the acquisition of Segue, a leading testing solutions provider, and I am now responsible for Borland’s Lifecycle Quality Management (LQM) (aka testing) products.

When you are developing testing products you are in an interesting position. Not only are you building and evolving the products, but you use them internally on a daily basis as part of your development work! With Borland being in the midst of an Agile transformation, adapting testing to the Agile paradigm has become very important to us. So in my first few posts, I’d love to talk about some of the issues I find interesting and critical to evolving testing and QA for an Agile environment. After all it is quite a change coming from a traditional testing background.

This is the first post in a 3 part series on this topic, and I would like to kick it off by sharing a few observations about how people are affected by Agile's paradigm of team integration (QA and dev team members collaborating as one team) and the implications this has for skill sets. In future posts I’ll touch on  testing tools, the increasing importance of test automation in Agile, and the process side of things -- highlighting the necessity for testing early and often.

So let’s talk about team integration.By that I mean the tight interlocking of testers and developers that the Agile approach requires –- e.g. pairing of developer/testing focused team members. I feel like this is quite possibly the most important issue related to testing in Agile, because it is a fundamental change in the way things have always been done. It requires a new, shared responsibility for quality by all team members and not just a dedicated (and separate) QA team.

Without a doubt there are a number of benefits that come with the integration of testers into a development team. Here in Linz, since we have integrated the members of what used to be a centralized “QA team” into our Scrum teams we have seen several of these benefits first-hand.  Improved communication is an obvious one, since now our functional test scripts and code are developed in parallel based on the same requirements and changes.

Also, a combined team has an improved ability to meet quality objectives as everyone is now proactively responsible for quality. Our own teams have seen improved test coverage through test pairing and by sharing libraries of unit tests (test scripts created by developers) and functional tests (test scripts created by testers). It also takes us less effort to develop automated scripts and guarantee code-and-test coverage. Because our combined teams are able to isolate defects and quality issues earlier, we have also seen improved velocity since we integrated them.

But all of this doesn’t come for free. To make it work, it is necessary to expand the tester skill set. While the core responsibilities of QA -- identifying the most appropriate implementation approach, implementing, setting up and executing individual tests, logging outcomes, verifying test execution, analyzing and recovering from execution errors –- remain, there is also a need to adapt the traditional testing role to accommodate Agile development practices

Among other things, this means everyone needs to become a member of “the team” and the Dev/Test barrier gets removed. And, removing this barrier is much more than just changing organizational structure and seating arrangements. It’s most definitely a cultural change –- which means an investment in time and training.

To make it all work, it also becomes necessary for testers to develop programming skills –- that is part of the evolution of the tester’s role in an Agile environment. Testers need to get involved in test conception from the beginning. Many roles in software development have evolved over the years and Agile will certainly require testers to become more skilled as well. There are a number of ways this can be achieved and not all involve formal training or classes. For example, allowing the team members to invest a certain percentage of their time for self education and leveraging Agile best practices like pairing will provide team members who don't have a strong development background an opportunity to ramp up their skills.

In my next post, I will explore a more tool-focused topic and talk about the increased importance of test automation in Agile environments. Bye for now.

Agile enterprise-readiness: the 'one sprint behind' model

Many people confuse the agile values and principles with the practices that support those principles. Scrum, for instance, consists of a set of practices that have proven themselves to support the agile values and practices. This confusion often leads to judgments that one group or another is 'not really doing agile' because they are not following all the prescribed practices or have modified them in some way. (I've written about my irritation about such judgments before: here and here, for instance).

My policy is: as long as your team understands the agile principle(s) that a practice supports and also understands and accepts the impacts of your deviation from the standard practice, then you're still agile.

In many cases, teams have to adapt agile practices in order to deal with institutional constraints that are outside their sphere of influence. One such constraint at Borland is that all of our enterprise testing (e.g., performance, scalability, integration, localization) is done by our Enterprise Quality Center (EQC) in Singapore--a long way from our scrum teams in Europe and the US.

When I started on Borland's first agile team (in Austin, TX), we had to figure out how to get the enterprise testing done effectively, and done in as agile a fashion as possible. I call the model that we came up with the 'no-more-than-one-sprint-behind' model.

Here's how our product development teams deliver code. Our scrum teams practice 2 or 3-week sprints, with two or three releases each year. Many of our scrum teams deliver code to customers at the end of each sprint--but always just for acceptance testing, not for enterprise deployment. Only releases are designated for enterprise deployment. Also, the last sprint of each release cycle is a wrap-up sprint with little or no new code development beyond bug fixes.

Given these practices, we decided to have two definitions of 'done'. At the end of each sprint, each story is accepted as functionally complete--coded, documented and functionally tested. Enterprise readiness testing is then performed the following sprint. So, at the end of the following sprint--at the latest--the stories from the previous sprint are accepted as enterprise-ready. And since no new stories are completed in the last sprint before a release, this gives the enterprise test team time to catch up.

Some of our scrum teams have gone well beyond (or below?) the one-sprint-behind model. They hand off new stories for enterprise testing as soon as they're done and get the enterprise testing completed well before the end of the following sprint.

The institutional constraint of having enterprise testing done on the other side of the world requires significant compromises to several agile principles, most notably, The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. But our scrum teams continually try to keep the principles in mind and to understand the cost of their compromises. Given the constraint, most of our scrum teams--and the company as a whole--have been satisfied with the results of the enterprise readiness testing.

In another post, I'll share some of the details of how we manage the enterprise testing work in our sprints.

I was close, gettin' closer, just this far away; come to find out close only counts in horseshoes and hand grenades

Who is responsible for quality? The facile answer is QA, but that doesn't work very well in reality. Quality is the responsibility of the team as a whole, developers included. You can't test quality into a product – if everyone isn't concerned with it, it isn't going to happen. Without quality, then your product is merely okay at best, close to the goal but not quite there. In an agile world, quality is largely determined by acceptance, and stories can't get accepted until they are both coded and tested. A common question is,
Photo by LarimdaME "If QA needs to test what dev creates within the same iteration, what do the developers do at the end of the sprint?" Well, how about... testing?

Unit tests can always be improved, even when practicing test-driven development. And just because QA creates a test case doesn't mean a developer can't execute or automate it. Something our team does periodically is a multi-user test, a "pokeathon" if you will. We get the extended team (support, sales, etc. are invited as well as the engineers) together for an hour, create a conference bridge so we can all talk to one another, and start banging away on a single install of our webapp. This unscientific and undisciplined method of testing has uncovered a wealth of issues for us, improved our overall quality, and turned out to be a bit of fun at times, too. I highly recommend trying it out.

Post title is a lyric from Horseshoes & Hand Grenades by Trent Summar and the New Road Mob.

More on QA specialists vs. testers

In my previous post, QA specialists vs testers, I mused that agile is good for the careers of QA engineers:

Nobody is relegated to the role of tester--a subset of QA work that's generally considered lower status. Instead, every QA engineer on an agile team must function more like a lead QA engineer: planning testing strategies, advising the team on what testing is necessary, and making sure that the testing work is accomplished by various team members.

The post garnered several comments. In this post, I'd like to address some of the thoughts and questions in the comments.

First, Andrew and Jorge question my use of terminology. Andrew writes:

To me, the role of Quality Assurance on a project is to make sure deliverables (test scripts, code, functional designs) live up to a certain standard, which you inact at the beginning of a project. The QA is seen as a subject matter expert for this area. To me, a tester is someone who actually physically tests the application.

I agree 100% with that definition. But Jorge asks:

Does the author try to say QA Specialist = QA Engineer? If that's the case, there is no reason to use different naming for the same meaning (QA best practice should avoid that)

In general, I would say, yes, the two terms are interchangeable. The point I was trying (unsuccessfully, I see) to make in using the term "QA Specialist" was specific to agile: a scrum team consists of members with various specializations, which makes the QA engineer the team's QA specialist. But I agree with Jorge, it's probably an unnecessary distinction to make. "QA Engineer" has all the same connotations.

Mark Johnson points out that this change of focus for QA engineers has also been good for the company and products overall:

As the manager of the QA team, I had been pushing for them to do less testing and more QA for several years.

Agile legitimised this and made it clear that the developers role is to develop working features, not throw half-working software over the wall to testing.

I have changed the role of the senior QA staff to include the negotiation with the stakeholders of the acceptance criteria before design and coding are started; this has given them higher profile within our organisation and our clients and raised their status.

I don't really have any reply to that except to congratulate Mark. That's the outcome that I would expect from a successful agile implementation.

Finally,Sean Wilson asks about how I undertook the mentoring that I described in my earlier post.

The QA engineer whom I described was originally assigned with me to the same scrum team. Our mentoring relationship started with some credibility issues with one of the developers. To over-simplify the situation, one of the developers on the team felt that the QA engineer was not sufficiently trying to solve problems or find information herself before involving him and he felt his time was being wasted.

The other QA engineer and I discussed the problem, and we agreed that I would become her 'safety.' If she couldn't figure out a problem or find information, she should come to me first, and I would help her find other methods to address the issue at hand before resorting to the developer in question. We also explicitly agreed that she could ask me anything; no question was dumb with me. And finally, if she did have to take an issue to the developer, I mentored her on how to present it, e.g., describe the issue as methodically as possible, list the steps she'd already taken to resolve it herself, etc.

Due to this agreement, we formed a relationship of trust, and it eventually led to broader mentoring. I have to say, though, that the success of the relationship lay squarely with the other QA engineer: she recognized her weaknesses, sought out a mentor and actively worked to better herself professionally.

I eventually left that team and this QA engineer assumed a more active role in leading quality efforts. Over the course of a few months, I was pleased to see that she was flourishing in this role.

QA specialists vs testers

Back in July, I wrote on my agile testing blog:

I'm becoming increasingly concerned about the suitability of conventional QA engineers in agile environments. Since agile methodologies stress that the entire team is responsible for testing and that as much testing as possible be automated, the QA specialist has limited usefulness on an agile team.

When I re-read that entry today, it felt like I wrote it years ago, not a few months. I still hold the same general belief--that distributing testing among scrum team members reduces the role traditionally played by the QA engineer--but my take on the situation is much more positive now.

While agile adoption could indeed lead to a total reduction in the number of QA engineers needed in the industry, it can be a good opportunity for individual QA engineers. Nobody is relegated to the role of tester--a subset of QA work that's generally considered lower status. Instead, every QA engineer on an agile team must function more like a lead QA engineer: planning testing strategies, advising the team on what testing is necessary, and making sure that the testing work is accomplished by various team members.

I have seen first-hand the positive effects of scrum on the professional lives of individual engineers. I mentored a QA engineer on one of our teams here who first struggled with the challenges of her role on her scrum team. But I'm happy to report that she has since flourished and really come into her own as a 'QA specialist' on the team. She has gained the respect of the entire team and raised the overall quality of the software that her team develops.

I'm interested to know if others have witnessed the same types of effects.