Technical debt and waste: check trends and act preventively

I will now discuss a concrete case (...I actually found it in several organizations) in which one of the main risks for a software development company is the high level of technical debt for the product and its documentation.
The final effects are the followings:

• Components difficult to modify and fix
• Economic inefficiency in developing and maintaining several product variants, as required by the market
• High risk when technologies need to be changed: components need to be completely rewritten, documentation does not reflect current functionality, but only a long series of change requests, time and cost high now, and impossible to bear in the future.
• The risk for the future, given that technologies change very often (even if you keep the same technology, it will change at a rapid rate) is infinite.
  

How did they get here? 

They somehow complied with the basic recommendations for technical debt and flow optimization:

• Technical debt was not a major impediment… until it became so
• Flow delays were not a big problem… until they became so

For a while, at any time, they could have told you “Our flow is good, we have no major delays and technical debt does not cause us much trouble"
Important - the technical debt accumulates, it's like in the metaphor with the Boiled Frog. You also need to consider measurements trends and future implications, not just current ones.
My advice about Technical Debt: measure the trends and act preventively.
Likewise, to avoid waste: it is not enough to measure the delays in the flow, this being only the final effect of many sources of waste. Find the various sources of waste, calculate the trend, and take preventive action.

We have more strategies to deal with the waste (read more here):

• correct
• correct early 
• avoid
  

Using Lean - From Solution Discovery to Problem Discovery

CONTEXT: IMPROVEMENT - the problem/solution terms are used here in the context of process improvement. "Problem": a weakness in the process that reduce effectiveness and efficiency. "Solution" - a change in the process that will improve the e&e factors

SOLUTION RECIPES. A typical  "process method" will provide us some solutions with the implicit or explicit guidance to use them as recipes, sometimes as universal recipes.

SOLUTION DISCOVERY. A a much better process improvement approach will provide solution discovery guidance. Your situation is unique, so you need a tailored solution. 

Typical agile methods do not even provide support for solution discovery

PROBLEM DISCOVERY. A robust approach should also provide problem discovery guidance. We need to find out what are the root causes that affect the effectiveness and the efficiency of our work. Optimizing the whole is a fundamental lean principle. Without problem discovery guidance we will not optimize the whole. This is also important because its is often difficult to identify the root cause of visible symptoms.   

Important observation: the two word guidance - "do retrospectives" or "inspect and adapt", is just a title, not a real guidance.   

DA FIRST IMPRESSION. A first impression about Disciplined Agile (DA Toolkit) might  tell us that DA focuses only on solution discovery (that is, anyway, much better that just offering recipes) but does not address the problem discovery aspect. Let's take a look.   

INITIAL DA. There are several aspects that are related to problem discovery even in the initial form of Disciplined Agile.

Capability Map & Guidance - DA organizes its guidance based on process capabilities (process blades and process goals). If our process is in trouble, it is much easy to find out  where the problem is, looking at process capabilities. Here are some examples from DA Delivery (solution development):  Improve quality, Accelerate value delivery, Grow Team Members, Coordinate Activities, Address Risks. All these concepts are very easy to correlate with potential issues and could help to discover where the real problems are. 

Adapt to context -  Letting the team (the people directly involved) decide and adapt to context are fundamental concepts from DA and is strongly linked to a better problem discovery. Enterprise Awareness - is another example of a principle that could help in this direction.

CURRENT DA. Disciplined Agile has continuously evolved and also improved its support for problem discovery

DA has now a generic, holistic, Lean-based guidance for problem discovery. Here are some examples:

• Optimize the whole - consider the whole value stream, not just teams local aspects
• Slice the work at business level, not just for development 
• Asses the process problems and try to find the root causes
  

There are still some problems to be solved

• The lean based guidance is based too much on detecting delays compared to detecting the other sources of waste and avoiding waste in general. Se my previous post for more info. 
  

A STEP FORWARD. My improve proposal is to provide more support for the problem-discovery dimension.

 

A guidance toolkit could be developed for this aspect as well. We can capitalize on the existent knowledge and experience about different types of waste and use this knowledge to more easily  detect our concrete problems in our concrete situation. The content of this guidance could refer:

• Waste source map. A map for known categories of problems. Description of different source of waste: what kind of actions and behaviors produces such waste and what are the usual symptoms. Here is an example.
  
• Lean patterns - see below section. 
  

LEAN PATTERNS - A pattern describes a category of solutions for a category of problems. There are already many lean patterns, but sometimes disguised as "principles". If that guidance will tell us more about the "how" part, it may not be a principle.

Here are some example of lean patterns

Source of waste: defects. Some categories of solutions that could address this waste: build-in quality, test-first approach, non-solo-work. We can see from the start that there are multiple strategies available to address the defects and multiple possible choices of practices inside each strategy. When an Agile method will tell you that they have one "best practice" for defects that is debatable from the start. 

Source of waste: big inventory that come from use of long releases. Symptoms: increased complexity of work, hard to manage the high number of change request that a longer "project" could accumulate. Category of solution: slice the work and deliver in smaller batches.  

Source of waste: big inventory that come from "parallel work", multi-tasking. Symptoms: lack of focus, increase complexity, delays, defects.  Solution category: serialize the work.  

 

AN INTEGRATED APPROACH FOR IMPROVEMENT

Set up the goal to reduce/avoid waste with whatever strategy works best in your case. 

Use holistic lean principles and guidance 

• e.g. optimize the whole value stream, try to find the root causes
  

Use existent knowledge about categories of problems - sources of waste and their symptoms to

• Identify waste/problems - check if we have concrete problems from these categories: check the symptoms and if any of these sources of  waste are present 
• Asses the impact of the waste -  to get an idea how much we want to solve that problem
  

Leverage lean patterns - Use known lean patterns to have an idea about the category of the solutions.

 

Select process capabilities to improve - Use the problem description, the ideas about the impact and possible solutions to select the process capabilities to address.

 

Leverage DA Toolkit guidance about solution discovery and select - for your context - concrete practices and, eventually, bring to life the solution ideas from the lean patterns.   For example you may find more practices that could realize build-in quality, test first approach or non solo work in your context.

Potential Solution! Kaizen. (update) "It's not really a solution until you've verified that it actually solves the problem."- Scott Ambler feedback. Yes, we will identify a solution and then experiment to see if it works. We can use the Kaizen approach - a loop of continuous improvement in small steps, in which each step is an experiment with a new identified "solution". We can even do better than that and use Guided Continuous Improvement with Disciplined Agile Toolkit.

The purpose of this post is encourage the further development of the DA Toolkit (or similar) to fill the gaps from the support for the Problem Discovery dimension.

Using Lean - Optimization strategies

There are several strategies to optimize our work and to reduce or avoid waste. We will discuss the advantages and disadvantages of each strategy and a possible holistic approach that includes them all.

Correct: assess final symptoms - Most important final symptoms are defects and delays. I use the term final, because all other sources of waste could end up causing defects and delays. After discovering the delays, for example, we can assess these effects and then try to find the root causes and correct them.
Advantages -  This is like the "Year-end accounting": if we have flaws in our approach, most likely it will translate in these kinds of waste.
Disadvantages - It may be too late. We discover the problems late and apply the corrections even later.

Correct early: assess intermediate symptoms - we should consider and discover all known significant sources of waste, including those with delayed effects such as lack of built-in quality (which creates defects and delays later). We should also take into account the trends of wastes. See more on the previous post.
Advantages -  The problems are corrected early. For example. we can improve the quality before inducing significant delays.
Disadvantages - Has a good accuracy, but we cannot evaluate exactly how much possible delays and defects are reduced.

Avoid: use past experience. If we know how to avoid the waste, we can do the right thing the first time.
Advantages - that should produce a maximum effect and best work optimization.
Disadvantages - it is not so clear how much waste has been avoided.

Question: When and how can we apply the three approaches?

A simple answer might be to use them in this sequence:

• Avoid, use past experience
• Correct early. What cannot be solved by experience should be discovered and resolved early 
• Correct, assess final symptoms. This should be the last line of defense, but it is also very important and must not be skipped.
  

 A better answer will have some additional comments:

• Make an initial assessment based on existent information and metrics. Apply last two steps (corrections) based on historical information.
  
• Always use conclusions from late steps to learn and shift to the left the optimization. 
  

We need to evolve from correction to prevention 

This approach is consistent with Disciplined Agile Guided Process Improvement logic: succeed early instead of failing fast. My recommendation is to use DA Toolkit or any similar guidance in all three steps. At the same time a good experience in using Lean to avoid waste will be very useful.

Use delays detection to improve the process

 

SOURCES OF WASTE - there are more significant sources of waste and a good approach to improvement should consider reducing and eliminating these various sources.

WHY DELAYS ARE IMPORTANT - The Delays (Waiting) waste plays an important role for several reasons 

• a great business/process will react in timely manner, so delays preclude a good process
• usually, the benefits of reducing delays are bigger that other improvements 
• we act to improve locally and we forgot the delays between different teams and business units  
• most others significant source of waste will have delays as an effect 
• any delay will produce further delays in cascade

It is obvious that we should discover the delays starting with the inter-teams workflow and we should act with high priority on reducing and eliminating them.  

IS DELAY REDUCTION A SILVER BULLET ? So, knowing how important it is, what would be the recommended course of action for improvement?

A GAME -  Let's play a game: what happens if we relied mainly on detecting and then eliminating delays for improvement?

Is rather reactive and not proactive – It is rather reactive and not proactive - we measure the last effect of different sources of waste and it could be too late: we are already compromising costs, time and quality in a significant way.

We need to be prepared to identify the root causes, not just to measure the effects. So we need to learn and practice by discovering the other significant sources of waste and their common cause.

A classic example: A Scrum team that relies solely on the practices of the Scrum Guide and does not take into account technical excellence and internal quality. It could go fast for a few months, maybe two years and only after a while they will become slower and they will discover the delays as an effect. It is too late! We already have a product with high technical debt and as a consequence: is rigid (difficult to change), fragile (any change will cause uncontrolled defects). The team could solve this debt, but it will cost money and cause other delays. Moreover, the team should start just now learning and practicing how to produce software with better internal quality.

Future hidden delays - delays are sometimes the accumulated effect of our work habits over a significant period of time. Measuring only the final effect - the delays - will be too late and, consequently, a source of more delays and waste.  Even worse - these other waste can instantly induce other waste - avalanche effect -  until we could measure some delays.  

A better approach - we should pay attention, discover, measure and avoid all known significant sources of waste, including those with delayed effects such as lack of built-in quality (which creates defects, modifies delays, corrects delays and more). We should also take into account the trends of wastes, so as not to react too late to these wastes and related causes.

Even better: Guided Process Improvement We measure, monitor and react to all known and discovered source of waste. in case of problems we will fail fast and maybe we will improve in due time. With Guided Process Improvement (which is recommended by Disciplined Agile) we can use knowledge bases - as DA toolkit -  to succeed early instead of failing fast.     

Assess versus Prevent -  Identifying delays is perhaps the power gun for assessing the current problems of the process. However, in order not to end up with significant problems - to prevent them for the first time - we need a much more sophisticated arsenal.  Remember - Lean is about Avoiding Waste.  

 

Whole team, dependencies and context

 Whole team it is an interesting agile concept that could bring more benefits

• enables organization around the product and services and usage of cross functional skills 
• reduces work dependencies between teams 

Reducing dependencies it is important because it will avoid some sources of waste such as some significant delays. 

However, you cannot do always all the work inside the team. Examples:

• work can exceed the team's capacity for a period of time
• you may need some special skills that are not usually required
  
  

Here are some example of options

• collaboration with Enterprise Architects
• collaboration with others teams 
• collaboration with members from other teams
  

Any external dependency could lead to unexpected delays and other subsequent waste. A good idea to reduce these dependencies is to include these collaborators in the team for a while. Replace external de dependencies with internal ones. Is it extraordinary or not?

Once upon a time (true story) ... a customer pays to "borrow" a team member instead of buying development services. At first glance, it was a much cheaper solution. Three moths later, the development company has to rewrite that part of the code. What happened? An independent team member could not provide the same quality of service as being part of a dedicated team for that product.

Let's study a few cases. 

Case 1 Collaborating teams. Two teams of Agile products (team A and team B) collaborate with each other. Each team works collaboratively: using non-solo work/pairing and collective ownership of all artifacts. The only major problem is the dependence between teams, which could cause delays / waste.

Case 2 New team member supplements the missing team capacity.  Team A will borrow a developer to team B for a period. The new team member cannot be 100% effective because need to learn and practice what is specific to the new team. The collective ownership in team A is broken for a while. 

Case 3 New team member provides services for changing product B. Team A no longer depends on Team B? What about the collaborative work and collective ownership of Team B? What happens to the quality of service when changing product B?

We cannot just move people between teams and pretends to reduce dependencies. We want to reduce some delays but we can introduce others delays and also other sources of waste as poor quality and knowledge gaps. 

A starting point is to consider the whole, not only inter-team dependencies, but also the effectiveness of each team after the change. There are no simple solutions. We need to talk with both teams and involve them in creating a collaboration scenario that work best in the context.