From the Agile Transformation Trenches

Agile Scope: A picture and a few words

A couple of weeks ago, I was peer reviewing some work by a colleague. I was struggling to communicate what was and what was not covered by the popular agile processes  - what I call the scope of the process. For example, Scrum and eXtreme Programming essentially focus on a single small, cross-functional, development team. At a fundamental level, they do not cover larger teams or projects or organizations consisting of many small dependent teams. They say little about how customers or product owners decide on the business value and priority of the user stories or backlog items they create. Neither do they say much about working with a formal testing organization.

While there are plenty of suggestions and some popular approaches to extending Scrum and eXtreme Programming to cover multiple dependent teams, they are just that; extensions to the original core set of rules or practices. 

Anyway rather than try and explain this by waving my hands around I resorted to drawing a picture to communicate what was in my head. Although very simplistic, it seemed to help so I thought I'd share it. If I have done it correctly, you should be able to click on the image below to see it full-size (legibly in other words) in a separate, pop-up window.

The circles represent perpetual activities all running concurrently taking input off lists (or queues, backlogs, or whatever) and generating results on subsequent lists/queues/backlogs.

The green dashed box represents the scope of Scrum/XP and supporting tools like Borland TeamFocus today; sets of small independent teams working from their own backlogs.

The red dashed line represents the space where more and more organizations that have been using agile successfully for a while seem to be focusing. In these organizations individual development teams are working well. The result is that inefficiencies in working practices around the individual teams are starting to show up. This enlarged scope is closer to that covered by an approach like Feature-Driven Development (FDD) with broader demand management aspects informing the individual team product backlogs and collaboration across teams around an overall domain and technical architecture where appropriate. It is the space I see the combination of tools like Borland Team Demand and Team Focus being applicable, for example. Of course, we can't forget Borland Togetherfor recording and communicating the overall architecture as it emerges from collaborative design sessions.

Outside of the red dashed line are the activities that trigger new projects or changes to existing systems, and my colleague's interesting contention that this should all be inspired from a business process improvement perspective. He asserts that all IT development work should be about improving one or more steps in a process somewhere and if we understand our processes or our customer's business processes well enough we can use that knowledge to identify and prioritize development work. For example, Borland as a software tools vendor needs to understand the end-to-end processes involved with developing and delivering software. Identifying pain points, bottlenecks, opportunities for increased efficiency, etc,  in those processes enables us to deliver genuinely useful and easy-to-use products.

What the picture does not show explicitly is the current interest in more sophisticated agile testing with tools like the Borland Silk suite. The idea is included implicitly in the Design, Build and VERIFY activities.

Also not shown are the measurement and analytic aspects that I view as another dimension to the picture entirely.

All suggestions for a better picture would be gratefully received.

He planed and worked that wood until he made silver from the yellow pine

Craftsmanship has always been widely admired and appreciated. Anyone can pick up a paintbrush, but few compare to Monet; a wooden chair from Walmart will provide a place to sit, but the beauty of handcrafted custom furniture is undeniable. There is a movement in software to try and capture some of this magic, but with the large amount of truly terrible programs and outright project failures this crusade clearly hasn't yet taken hold. Given that, is there anything more satisfying that completing a project? Whether it is woodworking or software, that feeling of a job well done is beyond compare.

One of my teams recently got to experience this euphoria: Borland TeamDefine 2009 was just released to the market for the first time. TeamDefine is simulation software that helps people flesh out requirements early in the cycle, eliminating time-wasting rework. Agile is all about short feedback loops, so a with tool like this which can deliver both rapid prototypes and gather comments from a wide variety of sources, those feedback loops can get even shorter. Most agile stories are a collection of words with an occasional screen shot; if a picture is worth a thousand words, a simulation is worth a thousand pictures! The earlier a team knows it is building what the customer really wants, the more likely they are to build well-crafted software.

Our team is now using our own product to explore stories, which is very cool. I've been on a lot of different software troupes, but the number that actually got to use what they built as a part of their job is very small. We have had a lot of fun building TeamDefine, and it has developed into a product of which we are unabashedly proud (and are having a lot of fun using). Check it out for yourself by watching the (funny) video below and downloading a free trial. Play with it and let me know what you think!

Post title is a lyric from Walnut Street by Walt Wilkins.

Roadmaps? Agile don't need no stinkin' roadmaps!

.... or so some people may have thought. 

A couple weeks ago I worked with Luke Hohmann from Enthiosys to facilitate  the Agile Requirements Workshop here at the Borland Software headquarters in Austin, TX.  We invited a local group of about 25 agilists in Austin from various companies.  Some were executive level and some were team level.  Some had small teams at small companies, while some hailed from very large companies with distributed teams. Overall, it was a very good cross-section of participants.  Our team of about 8 people from Borland and Enthiosys worked as facilitators and observers. Our topic was aptly, "Agile Requirements". 

What we hoped to understand more fully is the requirements process for Agile teams.  Specifically, how requirements are identified, expressed, prioritized, validated, and organized in Agile teams? Our task was not to review the project management process... as sometimes we fell into.  Rather, it was requirements we were interested in.  Whatever you call them in your own shop: requirements; stories; use cases; user stories; or goals; we wanted to create a shared learning environment to discuss our experiences and then publish the findings for others.  

One of the questions I personally had going into this gig was whether or not a high-performing agile development team would even need a traditional "roadmap".   I wondered if these teams or companies execute so well that it lessens the need for a longer term vision into their future.  Do they communicate so well across teams that the coordination of releases is not a problem?  In short, do they need these stinkin' roadmaps?"

I asked myself this because traditional waterfall roadmaps can be a highly political artifact in some organizations, with many choosing to ignore the roadmaps altogether since they are so high level (stratospheric?) in nature. Is the same true for Agile?  Have roadmaps been tossed into the dump heap of useless documentation along with the Functional Requirements Specification (FRS) as items that simply do not help teams perform better?

Which brings me back to our workshop observations.  We found that the team and the leaders really wanted, and needed, to see farther down the road.  In fact, not only is a roadmap important in Agile... it is probably more important than it ever was in waterfall.  

The observations we made at our Agile Requirements Workshop were centered around the need to better find and prioritize the right market problems for the best profit.  This is a roadmap issue.  We heard comments like "I want to see further than just over the hood" which led us to the realization that roadmaps are needed in all organizations, and particularly Agile ones.  Luke Hohmann wrote a good initial post about our observations, and there will be more coming as we put flesh on the experience report.

More about our experience at Borland

What we've done at Borland, with help from Enthiosys, is to implement and customize an Agile Roadmapping framework that helps us build more effective roadmaps using collaborative techniques with cross-functional teams.  I presented a webinar with Pragmatic Marketing on this topic in November, along with Scott Gilbert from Enthiosys, that describes it pretty well.  I'll expand on this topic later, but I will say that the implementation of this technique went extremely well and the results were outstanding.  The legally blessed and approved "roadmap" document is then abstracted out of this exercise for sharing beyond the internal teams.  We do this every quarter with every suite.

H ere is a picture of the process... one of our Product Managers in deep conversation with members from a cross section of Borland.  (Sales, Services, Support, Marketing, and of course, R&D)

David at 37signals wrote that he believed roadmaps are not needed. David brings up great points about the misuse of roadmaps and how it may lead to dysfunctional behavior.  But there is a big difference between an artifact called "the roadmap," and an activity called "roadmapping."  Both will result in a roadmap.  But doing the former without the latter (especially in an Agile environment) will lead to poor results, apathy or mutiny in the team, and a lot of silo'ed organizations.

What we REALLY need are roadmaps in Agile development teams!  They should be built using Agile techniques... collaboratively, but focused on the business needs.  Roadmaps should be refined regularly as a part of the product process so that you end up with a more effective, relevant, and cohesive product strategy in your organization. 

Stay tuned for a future post that details Borland's experiences (good, bad, and ugly) as we transitioned to a more Agile approach to roadmapping and roadmap creation.  In the meantime, I'd be curious to get your feedback.  How do you build your roadmap and share it with your team(s)?  What have you learned?

Using a Repository for Agile Requirements Collaboration

In a post on Agile Requirements Modeling, Scott Ambler wrote:

"It is the responsibility of project stakeholders to provide, clarify, specify, and prioritize requirements. Furthermore, it is the right of project stakeholders that developers invest the time to identify and understand those requirements."

Agile helps motivate a development team with shared goals. But, how does the customer proxy (those in charge of ensuring softare meets customer and business needs) extend shared goals beyond the development team to stakeholders? I have seen many development teams invest but I see fewer customer proxies making a similar transformation in this area. The role of the customer proxy is to elicit requirements from stakeholders, integrate input from various sources, and seek endorsement of the aggregated view. It is their job to insure the business needs are coherent and directed, not divergent.

So how do they accomplish this? Writing user stories on index cards as "a placeholder for a future conversation" is great when the stakeholders can be asked for their input on demand. Unfortunately, the existence of a customer proxy generally indicates that stakeholders are not available in this way. Without help from the customer proxy, the development team may not even know who to involve in the conversation.

In the past, I have relied on the familiar tools of MS Office: Word, Excel, and Visio to help solve this challenge. Yet, even for a small set of stakeholders, a number of problems emerge from such a document-centric approach.

• Error prone: Manual or document-position identification schemes can cause confusion when changes are made by multiple people at the same time.
• Extensive manual effort: Inputs from stakeholders come in the form of document edits which are painful to merge.
• Divergent interactions: Inputs from stakeholders come at different times, often complicating the merge problem as some feedback is against outdated information.

As a result of these problems, the distribute-review-update-reply cycle for requirements can take weeks or even months to complete. This is where the Agile idea of enabling, not controlling, change starts to apply to requirements.

One solution is to have stakeholders practice "continuous integration" for requirements. That is to say, they commit comments or changes to requirements to one repository that automates the tedious aspects of change management. A requirements management repository, such as CaliberRM, can automatically identify every requirement uniquely, can protect users from confusion of parallel development of requirements, and can make sure all stakeholders are looking at the same set and versions of requirements. It enables a single point of access for edit cycles so everyone stays on the same page. This shifts my role from "doing" requirements management as if it were contract negotiation, to "supervising" requirements management as a collaboration among stakeholders. For me, this is working smarter, not harder. For my stakeholders, this increases visibility and transparancy.

A repository approach also scales better, enabling members of the development team to get involved earlier. Whenever possible, try to bring members of the development team to business discussions. This helps establish trust between the development team and customer proxy. As they see the complicated sea of changing requirements, they can appreciate the judgment of the customer proxy and understand why their ability to accept changing requirements is so valued. A requirements repository also keeps the development team in the loop when they cannot be available in-person. Developers can be notified when a requirement is ready for estimation and QA can be invited to participate in on-line discussions to ensure the testability of requirements.

A requirements repository can make a big difference in collaborating with remote stakeholders in order to keep them happy and engaged, even before software is delivered.

Key Questions for a Customer Proxy

Over a year ago, Roman Pichler wrote an important post on Being an Effective Product Owner. In it he writes:

Proactive stakeholder management is particularly important in larger organizations, where the stakeholders include not only customers but also internal functions, such as production support, service or sales, as well. Taking into account and prioritizing the various interests early on and involving stakeholders regularly, for instance in form of user story writing workshops and sprint reviews, is crucial to ensuring that the software can successfully work in its target environment.

While frequent, iterative, and incremental deliveries help course correct based on working code, this is not an excuse to "just code it and see". Getting good at Agile means getting good at managing stakeholder expectations. As a Product Manager, when I deal with a changing requirement, there are 4 essential questions I need to answer over its lifecycle:

• Is everyone on-board? - I need to ensure the business itself is setting common goals.
• Should we make this change? - I need to enable changing goals. Ironically, this means saying no to some changes.
• Is everything in place? - I need to know that code, tests, documentation, or any other part is done.
• What did we get? - I need to express the value of what was delivered to all stakeholders.

It is not that asking the questions requires much effort. Rather, it is the delay between asking the question and getting the answer that determines how much change I can accommodate. If it takes 2 weeks to get stakeholders to agree to the meaning of a change and another 2 weeks to get estimates from the development team, then I will be unable to take lessons learned in one 4-week iteration into the plan of the next. Even if the effort is measured in man-minutes, the latency of the process impedes my capacity to change requirements. In order to become change adept with regard to requirements, a customer proxy needs to become very efficient in answering these questions.

I would be very eager to know how other people are answering these questions. What practices help make the answers readily available? What tools make it easier to get the answers on demand?

Why is the Work of a Customer Proxy so Hard?

Earlier this year, Machiel Groenveld wrote a post titled, Scrum: The Mythical Product Owner Role. In it he wrote:

"In my experience the Product Owner role is by far the most confusing and hardest role to 'get right' and provokes the most discussion. Even the mere definition of what this person should do is under debate amongst most Scrum practitioners I've talked to."

When I was a Product Manager, I played the role of Product Owner for a number of products at Borland. As Director of Product Management, I now do more coaching and enablement for my Product Managers. While Groenveld's comment applies to Scrum, I think it also applies in a more general sense to anyone who plays the role of "customer proxy". I first heard the term customer proxy from an XP team who needed to accept a compromise to the principle of Onsite Customer. They were developing software for on off-site customer and decided a member of the team would "channel" the customer needs for the rest of the team. In this case, the role rotated through the team. Although my first experience was back in 1999, I have seen many Agile teams accept similar compromise due to the modern reality of distributed business.

A customer proxy's main responsibility is to represent customer needs in planning, implementation, and testing. As the team has questions about what is best for the customer, the proxy provides answers and is accountable to the customer for those decisions.

A common reason for a customer proxy is the mismatch between short, iterative planning cycles for the development team and long, inflexible planning cycles for stakeholders. Scrum and XP push the customer proxy to slice requirements into smaller units of work that can fit within short development iterations. For many customer proxies, this requires new skills in elicitation, analysis, and validation. For elicitation, the customer proxy needs techniques to get smaller requirements from stakeholders at the outset. For analysis, the customer proxy needs streamlined ways to associate value, risk, and effort with requirements. For validation, the customer proxy needs to obtain agreement and commitment on granular requirements without losing sight of larger business goals. Like many other practices in Agile development, the discipline required for these practices is not less but more. All too often, the move to Agile exposes the fact that not enough of these things were happening before the transition.

I have met many people in the role of customer proxy who have previously played a similar role in more traditional development, whether they were requirements analysts or project managers. Unfortunately, many of them did not grasp the essential difference between Agile and the single-threaded waterfall style of development -- the shift in mindset around change. In Agile development, the customer proxy stops resisting scope creep but instead makes the consequences visible and apparent.

This was a big change for me when I first became a Product Manager and started applying Agile principles. The need to shift focus from controlling requirements change to getting good at enabling requirements change is a big one. But it is a necessary one when it comes to coaching coaching stakeholders on their new rights to add requirements, change their minds, and reprioritize.

In my experience, FDD offers much more guidance than Scrum or XP for a customer proxy struggling with these issues. In particular, FDD describes several specific requirements elicitation and planning techniques that bring domain experts and the business into the process itself, while also providing guidance on how to slice things into parts that can fit into iterations. On one hand, the FDD techniques may seem like traditional waterfall in the focus on up-front requirements elicitation and design. On the other hand, the techniques are still performed with frequent iteration. Such up-front activities assure business and developer collaboration, thus reducing the dependence on a customer proxy.

In the end, I'm left wondering about the role of a customer proxy when we say we value "customer collaboration over contract negotiation." How much does dependence on a customer proxy undermine this Agile principle?

"You don't have to change. Survival is not compulsory"

In my role as a Director of Product Management at Borland, I have to deal with a lot of change so that quote from W. Edwards Deming really hits home. Let me explain.  

Marketing discovers a new feature that is suddenly a "must have". A customer decides their top 10 defects are no longer as important as their new top 10 list. Senior management acquires a company with an existing product line and needs new integrations. Engineering decides yesterday's technology is passé and wants to start using a new technology. Sales expands to a new country and discovers a new set of localization requirements. Finance wants to cut license costs so they demand removal of all commercial components. And sometimes it is just as much my fault –- I do not always know what I really want until I can see some working software. At times, I just want to say, "Stop!"

Despite my wishes, I know the changes will not stop. Hence, I know from practical experience that change never stops, especially when it comes to requirements. Freezing requirements just means refusing to meet new needs. The success of my products depends on my ability to embrace change. Of course, this is easier said than done in my role as Product Manager.

The truth is that I have been coaching people about embracing change for a long time. In 2001 and 2002, I worked as a Mentor at TogetherSoft where I helped coach customers on adopting Feature Driven Development as a way to deal with change. I was also fortunate to observe some early adoption of Extreme Programming (XP). After Borland acquired TogetherSoft at the end of 2002, I saw more agile practices during customer engagements and started to observe Scrum and XP implemented on a large, enterprise scale. In 2005, I left field consulting behind to join the Borland engineering team as a Product Manager. It is in this role that I have been working as part of a broader Borland initiative to drive Agile practices, such as Scrum, into common use across the organization and my product team specifically.

Having been a part of Borland for nearly 5 years, I have experience in the application of many tools and techniques. I have taken a turn with many different techniques such as business process modeling, use case analysis, OOA/D with UML, and lately user stories and personas. I have also taken a turn with many different types of tools for requirements, modeling, change management, version control, and development. I comfortably consider myself a "tools person". I like using tools and, unlike some folks I have met in the Agile community, I think they play important role in software development -- just not more important than individuals and collaboration.

Through all of these experiences, I know one thing -- for all that experience, I still have a lot to learn. This is why I am so excited to start blogging. I look forward to the sharing of problems encountered and solutions tried. I hope you gentle readers will join me in exploring the challenges of being Agile in the modern business world. Together we just might find the path to survival.