Thursday, December 21, 2017

What is wrong with Scrum Product Backlog?

2017 SCRUM GUIDE UPDATE – Sprint Backlog change: “To ensure continuous improvement, it includes at least one high priority process improvement identified in the previous Retrospective meeting.”. My first question is: OK… but how could be represented one process improvement as a Product Backlog Item? 

BREAKING SCRUM RULES – one of the most common and ugliest forms it is to consider the Product Backlog as being formed from user stories. It this your current approach? Is that what your coach told you? Well, that it is deeply wrong also according to Scrum creators.  

Here it is what Jeff Sutherland said (see the reference below about user stories): <<Backlog items include everything the team needs to do in one ordered set of activities >> and <<Wherever possible, backlog items should deliver complete vertical slices of functionality across work layers>>, where this “vertical slice of functionality” could be very well represented by a user story. 

According to the Scrum Guide, the Product Backlog is composed of Product Backlog Items. What are these Product Backlog Items, if there is such confusion among the Scrum practitioners (coaches and “masters” included)?

Back to the Scrum Guide (2017), here are some of its references to the PBIs:

  • <<The Product Backlog is an ordered list of everything that is known to be needed in the product. It is the single source of requirements for any changes to be made to the product>>
  • <<The Product Backlog lists all features, functions, requirements, enhancements, and fixes that constitute the changes to be made to the product in future releases. >>
  • <<The Sprint Backlog is the set of Product Backlog items selected for the Sprint >>
  • <<The Sprint Backlog makes visible all the work that the Development Team identifies as necessary to meet the Sprint Goal >>
  • <<The Increment is the sum of all the Product Backlog items completed during a Sprint and the value of the increments of all previous Sprints.>>

The PBIs are part of Product Backlog, Sprint Backlog, and (Product) Increment. So, having only the Scrum Guide as a guideline we can believe that the Backlog and PBIs are a kind of “Product Breakdown”, a list of deliverables directly related to the product. The “twin” of the product breakdown is the work breakdown, that organizes the work in association with related deliverables, not necessarily directly related to the product. So, considering the Scrum Guide PB is rather a “product breakdown”, but according to above-mentioned quotes from Jeff Sutherland is a “work breakdown” ~ “everything the team needs to do”. Why this confusion?

SIMPLIFICATION - The Scrum Guide contains some simplifications that could not offer enough guidelines even for Scrum creators point of view. The deliberate approach to make Scrum Guide and Scrum to seems very simple could be misleading. For example, the Product Backlog was intended to be a “work breakdown” that “whenever is possible” should deliver something directly related to the product “vertical slices of functionality across work layers”. Whenever is possible this work breakdown should be oriented to the value stream and value delivery, but it is a work breakdown not something else.

CONFUSION AND PROBLEMS - The work breakdown it is confused (by some Scrum practitioners) with a list of requirements or with the product breakdown. Here are some examples of problems induced by this confusion:

  • Poor estimations without improvement - Estimations are made directly on requirements, without appropriate envisioning over the solution (… and that mean a much higher risk for inaccuracy). In fact, it is even worse: estimation skills are not developed with time because are not correctly exercised     
  • Forgotten work - The work needed for the release, and not directly related to the product change could be forgotten. That work will be discovered too late and that is a risk for the plans and for the results.
  • Life-cycle problems – we could have various agile and lean life-cycles, and all will express the work breakdown. We could have severe problems of understanding of the life-cycle if the team will make the mentioned confusion.  For example, the release envisioning will be poorly made because will stopped at the level of requirements envisioning. In other cases, the transition to production is almost forgotten: the team and their collaborators will discover very late in the release significant risks and “unexpected” remaining work.  
  • Throwing away the alternatives - the Product Backlog has a specific way to model the work (a serialized, ordered list). There are also other ways to model the work in agile or lean manner, as the Work Item Pool (see Agile Modeling references)    

ESTIMATION – estimation should be always associated with a piece of work (for some requirement, fixes, or others), and not directly with requirements (user stories). We can implement the same requirement in an unlimited variant of solutions, that could have very different needs for effort and resources. When you “estimate” a user story, you are estimating, in fact, the work related to that user story, already having an envisioned solution approach. In fact, you cannot decompose the functionality into independent realizable parts if you will not have the “product breakdown” needed to get the mentioned “vertical slices”.  

WORK ITEM - The Product Backlog Item (“everything the team needs to do”) represent, in fact, a piece of work (not a piece of the product, and not a piece of functionality), but with the recommendation to be whenever is possible a direct change of the product, a direct contribution to the value stream and value delivery. Using engineering term and not vague terms, that is “working software” from Agile Manifesto.

If the Product Backlog Item it is in fact, a work item, there is no problem to conform to the 2017 Scrum Guide recommendations to “includes at least one high priority process improvement” in the Sprint Backlog, and there is no problem to “include everything the team needs to do”.

GUIDELINE – If you need a clearer guideline, you can the Work Item List from Disciplined Agile. it is having the same purpose as Scrum Product Backlog, but without over-simplification: “we know that we do more than just implement new requirements each iteration, we often do non-requirement related work such as take training, review products of other teams, address defects (I believe that defects are simply another type of requirement) and so on.    

More, with Disciplined Agile you have options/alternatives to the Work Item List as the Work Item Pool (when working in Lean variants of the life-cycle). In the same time, the vague concept of “backlog refining” that is strictly related to existent of the Backlog it is replaced by more generic Look Ahead Modeling.

FINAL THOUGHTS  – It is absolutely OK to breakdown you work considering the value stream and the product breakdown as a Product Backlog, but you should also remember that:
  • work breakdown is not a list of requirements, and not even a product breakdown (but a subsequent aspect)
  • you can always use an Work Item List instead of Product Backlog, but sometime you can use alternatives as Work Item Pool (see Disciplined Agile/Agile Modeling references bellow)
  • do not confuse simplifications with the real things and always ask for guidelines about what is behind anything that is simplified by convention    

Bibliography and References

Monday, September 25, 2017

Modeling the process: Rolling Wave and Practices

WATERFALL MODEL - One-dimensional: use disciplines for both content and time in a waterfall-like sequence.

ITERATIVE MODEL - Two-dimensional: content and time are two distinct dimensions. We can work in one iteration in “parallel” for all disciplines.

AGILE MODEL (SIMPLIFIED) -  Compatible with the Iterative model, but introduce practices as basic “brick” for the content part. Practices also have a “parallel” distribution in time, many of them are repeating for each iteration. 

SCRUM MODEL  Content: only a small subset of Agile practices. Time: iteration only. Release it is rather out of scope (... and there is a gap between iteration and product backlog). See [B2].

XP MODEL - Content: contain a robust set of core agile practices & others (also recommends to address the scaling factors). Time: iteration and release milestones. Inside iteration: day, week. Beyond release: Quarter, Product Life-Cycle. See [B1]

DISCIPLINED AGILE MODEL - Extends the XP/Scrum model to a more generic approach: it recommends some Core Practices but is not prescriptive. In fact, the core of the DA guidelines it is about how to build your process: select the life-cycle, distribute/select practices in the life-cycle. These decisions are related to a concrete context: we decide about practices needed to realize generic process goals in context; life-cycle decision it is also related to the context. DA also recommend the Rolling Wave for planning as a pragmatic and effective approach. 
Life-cycles example: Continuous Delivery suppose a very specific information magement / Rolling Wave. Everything must be quick and good. Often feedback and quick validation are mandatory. In this case Disciplined Agile guidance recommend adoption of some Core Agile Practices for testing, design and collaborative work.   
Why do I expect from Disciplined Agile? To make more explicit the fact that use the Rolling Wave (with practices) model to describe the process.

“ESSENCE” MODEL – In some parts is conceptually similar to Unified Process (Jacobson, one of the  “Essence” creators, was also one of the UP creators): it describes the process starting from types of activities. In also defines the practices as something that “provides guidance to deal with some dimensions of software development” [B3 – Chapter 17], and, for example, associates practices with an activity as “A requirements elicitation practice” [B3 – Chapter 17]. This model will not be so effective for the Core Agile case, where each practice will contribute to more dimensions of development (and to more “activities” from traditional process description).
Another problem of the “Essence” is the definition of the method as a “composition of practices” [B3 – Chapter 18] (also Jacobson write/tweet about freeing the practices from methods prison). This definition it is incomplete for all the methods and for some of them it is inaccurate. Incomplete: all the methods are mentioning first some principles and only then the practices. Also, some methods have explicit descriptions for development life-cycles. So, we have at least principles, practices and possible a life-cycle. More, in the case of Disciplined Agile, we cannot describe that method as a composition of practices, because beyond the recommendations of some Core Agile Practices, DA it is mainly a guidance to compose a kind of custom method in a specific context by selecting from the industry practices.  

SPECIFIC AGILE CONTENT: CORE PRACTICES - Core Agile Practices have a specific Lean trait: one practice has multiple benefits, in various content aspects. These practices cannot be classified into one discipline: design, implementation, testing, etc. Modeling the content by encapsulating one core practice inside one discipline is a deep misunderstanding of Agile.

INFORMATION RELATED WORK: ROLLING WAVE - In software development, the work it is mostly related to information: we gather information, build information, we distribute, validate and get feedback about information. The progress of this work could be modeled as a Rolling Wave. The wave could address high granularity aspects as product roadmaps and release, medium granularity aspects as iterations and days, or low granularity aspects as TDD or Pair Programming. We cannot separate these levels because are influencing each. 

ROLLING WAVE -  Rolling Wave is not only about planning or milestones, buy about how the information related to the work and its results evolve or could evolve:
  • The problem - Understand and clarifying the problem to solve
  • The solution – prove your solution iteratively
  • Risks – resolve the associated risks in decreased impact order
  • Generic development aspects: knowledge, complexity, incertitude
It is a wave of information that could be gathered, created, distributed and validated (via feedback). Our process it is effective and efficient if we apply the “right wave” model in our context.  Waterfall propose an unrealistic model about the wave: gathering and building up front and precisely a lot of information. That it is mathematically impossible: complex problems can be solved only via repetitive feedback, and these kinds of problems have rather a short range of prediction.  Iterative model and Agile have introduced many feedback based practices such as small releases, iterations, retrospectives and continuous integration.   
Important: any model process propose a kind of rolling wave (effective or not), but only in Disciplined Agile, this model it is explicitly used, where also the process guidance follow this model.


The standard definition for "rolling wave planning" is <<plan things that are near in time to you in detail and things that are distant in time at a higher level>>. Practically, we will always have a rolling wave: we will always know less about the distant time things, no matter what it is our wishful thinking or not. 
Extending the definition, we will always have a rolling wave that it is also applied to the execution of the plan. The real model of a process supposes continuously enhancing the information and knowledge.  
In case of approaches like Waterfall, the intended plan does not match too well with the concrete realization and execution and the real rolling wave is ineffective.  


Core Agile Practices have a fundamental contribution to rolling waving the work, where their multiple benefits (per practice) will address many aspects related to the main domain issues: complexity, incertitude, and knowledge. Each of these practices has multiple contributions in information-based work to: create, gather, distribute, slice, validate (and get feedback) information.  That means we can have a feasible, effective rolling wave/work.      
You can find more information about their contribution in the post Core Agile Practices and Rolling Wave process model. Some of the Core Agile Practices: Small Releases, Iteration, Retrospectives, Pair Programming, Envisioning, Look Ahead Modeling, Just in time modeling, Continuous Integration, Clean Code


We have two main drivers to build/adapt the right rolling wave:
  • Domain specifics - the problems that are specifics to software development domain.
  • Context-specific – the problems that are specific to our context
Domain specifics: Core Agile Practices will allow building an effective & efficient rolling wave.
Context-specific: We need guidance to select/adopt the proper practices and guidance in context   


The right process model:
  • Will be represented as a(n information) rolling wave, with practices distributed in a life-cycle 
  • Will consider the adapt to the context: will be a specific rolling wave with life-cycle and specific practices. We will need guidance to be effective/efficient in this adaptation.  
  • Will also consider the Core Agile Practices - whenever it is possible - in order to increase adaptability and leanness
A good example for distributing practices in the life-cycle is the usage of process goals and decision points used by Disciplined Agile.  

Bibliography and References

[B1] Extreme Programming Explained: Embrace Change, Second Edition
  • By Kent Beck, Cynthia Andres, Addison Wesley Professional, November 16, 2004
[B2] Scrum Guide
[B3] The Essence of Software Engineering: Applying the SEMAT Kernel, Addison Wesley 2013, by Ivar Jacobson, Pan-Wei Ng, Paul E. McMahon, Ian Spence, Svante Lidman
  • Chapter 17 – Zooming In to Provide Details
  • Chapter 18 - Reaching Out to Different Kinds of Development
[B4] Disciplined Agile –  Full Delivery Life Cycles (agile and lean)
[B5] Disciplined Agile –  Rolling Wave related articles
[B6] Disciplined Agile –  Process Goals