Technical debt and waste: check trends and act preventively

I will now discuss a concrete case (...I actually found it in several organizations) in which one of the main risks for a software development company is the high level of technical debt for the product and its documentation.
The final effects are the followings:

• Components difficult to modify and fix
• Economic inefficiency in developing and maintaining several product variants, as required by the market
• High risk when technologies need to be changed: components need to be completely rewritten, documentation does not reflect current functionality, but only a long series of change requests, time and cost high now, and impossible to bear in the future.
• The risk for the future, given that technologies change very often (even if you keep the same technology, it will change at a rapid rate) is infinite.
  

How did they get here? 

They somehow complied with the basic recommendations for technical debt and flow optimization:

• Technical debt was not a major impediment… until it became so
• Flow delays were not a big problem… until they became so

For a while, at any time, they could have told you “Our flow is good, we have no major delays and technical debt does not cause us much trouble"
Important - the technical debt accumulates, it's like in the metaphor with the Boiled Frog. You also need to consider measurements trends and future implications, not just current ones.
My advice about Technical Debt: measure the trends and act preventively.
Likewise, to avoid waste: it is not enough to measure the delays in the flow, this being only the final effect of many sources of waste. Find the various sources of waste, calculate the trend, and take preventive action.

We have more strategies to deal with the waste (read more here):

• correct
• correct early 
• avoid
  

Using Lean - From Solution Discovery to Problem Discovery

CONTEXT: IMPROVEMENT - the problem/solution terms are used here in the context of process improvement. "Problem": a weakness in the process that reduce effectiveness and efficiency. "Solution" - a change in the process that will improve the e&e factors

SOLUTION RECIPES. A typical  "process method" will provide us some solutions with the implicit or explicit guidance to use them as recipes, sometimes as universal recipes.

SOLUTION DISCOVERY. A a much better process improvement approach will provide solution discovery guidance. Your situation is unique, so you need a tailored solution. 

Typical agile methods do not even provide support for solution discovery

PROBLEM DISCOVERY. A robust approach should also provide problem discovery guidance. We need to find out what are the root causes that affect the effectiveness and the efficiency of our work. Optimizing the whole is a fundamental lean principle. Without problem discovery guidance we will not optimize the whole. This is also important because its is often difficult to identify the root cause of visible symptoms.   

Important observation: the two word guidance - "do retrospectives" or "inspect and adapt", is just a title, not a real guidance.   

DA FIRST IMPRESSION. A first impression about Disciplined Agile (DA Toolkit) might  tell us that DA focuses only on solution discovery (that is, anyway, much better that just offering recipes) but does not address the problem discovery aspect. Let's take a look.   

INITIAL DA. There are several aspects that are related to problem discovery even in the initial form of Disciplined Agile.

Capability Map & Guidance - DA organizes its guidance based on process capabilities (process blades and process goals). If our process is in trouble, it is much easy to find out  where the problem is, looking at process capabilities. Here are some examples from DA Delivery (solution development):  Improve quality, Accelerate value delivery, Grow Team Members, Coordinate Activities, Address Risks. All these concepts are very easy to correlate with potential issues and could help to discover where the real problems are. 

Adapt to context -  Letting the team (the people directly involved) decide and adapt to context are fundamental concepts from DA and is strongly linked to a better problem discovery. Enterprise Awareness - is another example of a principle that could help in this direction.

CURRENT DA. Disciplined Agile has continuously evolved and also improved its support for problem discovery

DA has now a generic, holistic, Lean-based guidance for problem discovery. Here are some examples:

• Optimize the whole - consider the whole value stream, not just teams local aspects
• Slice the work at business level, not just for development 
• Asses the process problems and try to find the root causes
  

There are still some problems to be solved

• The lean based guidance is based too much on detecting delays compared to detecting the other sources of waste and avoiding waste in general. Se my previous post for more info. 
  

A STEP FORWARD. My improve proposal is to provide more support for the problem-discovery dimension.

 

A guidance toolkit could be developed for this aspect as well. We can capitalize on the existent knowledge and experience about different types of waste and use this knowledge to more easily  detect our concrete problems in our concrete situation. The content of this guidance could refer:

• Waste source map. A map for known categories of problems. Description of different source of waste: what kind of actions and behaviors produces such waste and what are the usual symptoms. Here is an example.
  
• Lean patterns - see below section. 
  

LEAN PATTERNS - A pattern describes a category of solutions for a category of problems. There are already many lean patterns, but sometimes disguised as "principles". If that guidance will tell us more about the "how" part, it may not be a principle.

Here are some example of lean patterns

Source of waste: defects. Some categories of solutions that could address this waste: build-in quality, test-first approach, non-solo-work. We can see from the start that there are multiple strategies available to address the defects and multiple possible choices of practices inside each strategy. When an Agile method will tell you that they have one "best practice" for defects that is debatable from the start. 

Source of waste: big inventory that come from use of long releases. Symptoms: increased complexity of work, hard to manage the high number of change request that a longer "project" could accumulate. Category of solution: slice the work and deliver in smaller batches.  

Source of waste: big inventory that come from "parallel work", multi-tasking. Symptoms: lack of focus, increase complexity, delays, defects.  Solution category: serialize the work.  

 

AN INTEGRATED APPROACH FOR IMPROVEMENT

Set up the goal to reduce/avoid waste with whatever strategy works best in your case. 

Use holistic lean principles and guidance 

• e.g. optimize the whole value stream, try to find the root causes
  

Use existent knowledge about categories of problems - sources of waste and their symptoms to

• Identify waste/problems - check if we have concrete problems from these categories: check the symptoms and if any of these sources of  waste are present 
• Asses the impact of the waste -  to get an idea how much we want to solve that problem
  

Leverage lean patterns - Use known lean patterns to have an idea about the category of the solutions.

 

Select process capabilities to improve - Use the problem description, the ideas about the impact and possible solutions to select the process capabilities to address.

 

Leverage DA Toolkit guidance about solution discovery and select - for your context - concrete practices and, eventually, bring to life the solution ideas from the lean patterns.   For example you may find more practices that could realize build-in quality, test first approach or non solo work in your context.

Potential Solution! Kaizen. (update) "It's not really a solution until you've verified that it actually solves the problem."- Scott Ambler feedback. Yes, we will identify a solution and then experiment to see if it works. We can use the Kaizen approach - a loop of continuous improvement in small steps, in which each step is an experiment with a new identified "solution". We can even do better than that and use Guided Continuous Improvement with Disciplined Agile Toolkit.

The purpose of this post is encourage the further development of the DA Toolkit (or similar) to fill the gaps from the support for the Problem Discovery dimension.

Using Lean - Optimization strategies

There are several strategies to optimize our work and to reduce or avoid waste. We will discuss the advantages and disadvantages of each strategy and a possible holistic approach that includes them all.

Correct: assess final symptoms - Most important final symptoms are defects and delays. I use the term final, because all other sources of waste could end up causing defects and delays. After discovering the delays, for example, we can assess these effects and then try to find the root causes and correct them.
Advantages -  This is like the "Year-end accounting": if we have flaws in our approach, most likely it will translate in these kinds of waste.
Disadvantages - It may be too late. We discover the problems late and apply the corrections even later.

Correct early: assess intermediate symptoms - we should consider and discover all known significant sources of waste, including those with delayed effects such as lack of built-in quality (which creates defects and delays later). We should also take into account the trends of wastes. See more on the previous post.
Advantages -  The problems are corrected early. For example. we can improve the quality before inducing significant delays.
Disadvantages - Has a good accuracy, but we cannot evaluate exactly how much possible delays and defects are reduced.

Avoid: use past experience. If we know how to avoid the waste, we can do the right thing the first time.
Advantages - that should produce a maximum effect and best work optimization.
Disadvantages - it is not so clear how much waste has been avoided.

Question: When and how can we apply the three approaches?

A simple answer might be to use them in this sequence:

• Avoid, use past experience
• Correct early. What cannot be solved by experience should be discovered and resolved early 
• Correct, assess final symptoms. This should be the last line of defense, but it is also very important and must not be skipped.
  

 A better answer will have some additional comments:

• Make an initial assessment based on existent information and metrics. Apply last two steps (corrections) based on historical information.
  
• Always use conclusions from late steps to learn and shift to the left the optimization. 
  

We need to evolve from correction to prevention 

This approach is consistent with Disciplined Agile Guided Process Improvement logic: succeed early instead of failing fast. My recommendation is to use DA Toolkit or any similar guidance in all three steps. At the same time a good experience in using Lean to avoid waste will be very useful.

Use delays detection to improve the process

 

SOURCES OF WASTE - there are more significant sources of waste and a good approach to improvement should consider reducing and eliminating these various sources.

WHY DELAYS ARE IMPORTANT - The Delays (Waiting) waste plays an important role for several reasons 

• a great business/process will react in timely manner, so delays preclude a good process
• usually, the benefits of reducing delays are bigger that other improvements 
• we act to improve locally and we forgot the delays between different teams and business units  
• most others significant source of waste will have delays as an effect 
• any delay will produce further delays in cascade

It is obvious that we should discover the delays starting with the inter-teams workflow and we should act with high priority on reducing and eliminating them.  

IS DELAY REDUCTION A SILVER BULLET ? So, knowing how important it is, what would be the recommended course of action for improvement?

A GAME -  Let's play a game: what happens if we relied mainly on detecting and then eliminating delays for improvement?

Is rather reactive and not proactive – It is rather reactive and not proactive - we measure the last effect of different sources of waste and it could be too late: we are already compromising costs, time and quality in a significant way.

We need to be prepared to identify the root causes, not just to measure the effects. So we need to learn and practice by discovering the other significant sources of waste and their common cause.

A classic example: A Scrum team that relies solely on the practices of the Scrum Guide and does not take into account technical excellence and internal quality. It could go fast for a few months, maybe two years and only after a while they will become slower and they will discover the delays as an effect. It is too late! We already have a product with high technical debt and as a consequence: is rigid (difficult to change), fragile (any change will cause uncontrolled defects). The team could solve this debt, but it will cost money and cause other delays. Moreover, the team should start just now learning and practicing how to produce software with better internal quality.

Future hidden delays - delays are sometimes the accumulated effect of our work habits over a significant period of time. Measuring only the final effect - the delays - will be too late and, consequently, a source of more delays and waste.  Even worse - these other waste can instantly induce other waste - avalanche effect -  until we could measure some delays.  

A better approach - we should pay attention, discover, measure and avoid all known significant sources of waste, including those with delayed effects such as lack of built-in quality (which creates defects, modifies delays, corrects delays and more). We should also take into account the trends of wastes, so as not to react too late to these wastes and related causes.

Even better: Guided Process Improvement We measure, monitor and react to all known and discovered source of waste. in case of problems we will fail fast and maybe we will improve in due time. With Guided Process Improvement (which is recommended by Disciplined Agile) we can use knowledge bases - as DA toolkit -  to succeed early instead of failing fast.     

Assess versus Prevent -  Identifying delays is perhaps the power gun for assessing the current problems of the process. However, in order not to end up with significant problems - to prevent them for the first time - we need a much more sophisticated arsenal.  Remember - Lean is about Avoiding Waste.  

 

Work Optimization: Avoiding Waste with XP, DA and Scrum

Avoid Waste (!Waste)

Here are some of the most known work optimizing principles related to Avoiding Waste, where many of them have as source TPS philosophy to avoid sources of Muda/Waste (it is better to return to original ideas, beyond various further Lean evolutions; see biblio-TPS) The other two are the ones that bother me the most, and could be specific for knowledge work: Not Used Talents and Improvement Stalemate.  In our case, these sources of waste will be interpreted from software domain point of view.

Principles & practices. XP, Scrum and DA  

As in the case of Core Agile Capability we need have a traceability between principles, values and the practices that bring them to live. A such trace could be found at the end only from what is useful in the real work.

In the following parts, we will try to present how this realization is done by practices from original main Agile methods: XP-Extreme Programming and Scrum and how we can get a more robust approach using DA - Disciplined Agile. We mention only practices that are explicitly mentioned by the methods & their guidance ... and what we have found that works in real practice. Here it is summary, I will come back with details for each part.   

"What we need" - description of what we have find that it is working to manage that aspect of the waste.

Important: if Core Agile Capability – Responding to Changes - is not met, the software development is not optimized from the start: this core capability address all the optimizing work principles at their fundamentals.
.

No Inventory: reducing work in progress

What we need

·   First, we need to “make the work easy” (See biblio-Kent_Beck): a big work in progress (inventory) will exponentially increase the complexity & incertitude, will reduce reaction time (core required capability) and quality.

·      Reduce the work in progress (WIP) on all possible dimensions, from making the release smaller to avoid multi-tasking   

Avoid Defects

What we need

·    To prevent defects & catch them as early as possible ~ provide build-in quality; all the practice that “shift left” the quality risks will be useful  

·    First, to make the work easy: see Avoid Overproduction and Avoid Big Inventory    

Avoid Waiting

What we need

·  Work should be “Ready” – actionable (!) -  before start using main resources: we do not want to wait for dependencies related to information, artifacts and others; what is missing usually is the overall perspective, that a Rolling Wave approach could bring: product, release, iteration, inside iteration   

·    We want to avoid bottlenecks in work, such skills (specialization) bottlenecks   

Avoid Overproduction

What we need

·   To avoid “Big work” that bring many risks: it is hard to include & manage required changes, it is a high probability to deliver results that are no more needed       

·   To develop and be ready to deliver big value first; will be much easier drop not started “junk stories” on request

·  To not guess about production needs: mixing envisioning and JIT on requirements and solution, implement what and if is needed

Avoid Over-Processing

What we need

·   Avoid extra work, not required by customer

·   Avoid process ceremonial that does not bring value

·  See also dedicated sections for:  Avoiding Defects (Built-In Quality), Avoid “Transport & Motion”, Avoid Re-Work  

Avoid “Transport & Motion”

What we need

• If, in case of manufacturing, products could be damaged by transport & motions, we can found an equivalent for this waste, also for knowledge work/software development: unnecessary long chain of information handover & information duplicate in a long chain of artifacts.
• Successive handovers could be caused by lack team ownership over information and lack of cross-functional skills   

Avoid Not-used talents

What we need

·        Practices support to quick and effective learning & skills improvement

·        To develop cross-functional skills. Example: a programmer with requirements & testing skills will perform TDD and design better   

Continuous Improvement & no persistent impediments

What we need

·        To avoid blockage on process improvements 

·        A process that make the impediments visible & it is effective on removing them

Avoid rework

What we need

·        To avoid “Waterfall rework”: big work with too few feedbacks could increase rework to infinite  

·        To avoid “Scrum-Agile rework”: lack of look ahead on requirements & solution could induce need of rework 

·        To avoid late defects detection – see Avoid Defects

·        Finally, we need to balance envisioning and defer of the commitment (JIT) and also get as much feedback it is possible   

Final thoughts

Work optimization it is related to some main ideas:

·        Teams should choose the optimum practices and ways of work in context

·        A good range of skills and known practices maximize the also the range of available options  

Disciplined Agile principles are about choices and growing team’s members, offer a solid fundament and guidance for work optimizations, while including also XP and Scrum related practices.

Notes about practices

Specific for Agile is that with one practice we can get multiple benefits. If the same practice could address more sources of waste, we could consider that practice as more valuable for our process.

See specified bibliography for each method for more details about mentioned practices

  

Bibliography

Biblio-TSP

·        Toyota Production System: Beyond Large-Scale Production 1st Edition, by Taiichi Ohno, Productivity Press; 1 edition (March 1, 1988)

·        https://en.wikipedia.org/wiki/Toyota_Production_System

Biblio-Kent_Beck

·        https://www.facebook.com/notes/kent-beck/mastering-programming/1184427814923414/

Biblio-DA Disciplined Agile

·        http://www.disciplinedagiledelivery.com/

·        Agile Modeling Practices - http://www.agilemodeling.com/

·        “Disciplined Agile Delivery:  A Practioner’s Guide to Agile Software Delivery in the Enterprise”, by Scott Ambler and Mark Lines

Biblio-XP Extreme Programming

·        Extreme programming explained: embrace change, Kent Beck, Addison-Wesley Professional; US ed edition (October 5, 1999)  

Biblio-Scrum

·        Scrum Guide 2016, US edition - http://www.scrumguides.org/docs/scrumguide/v2016/2016-Scrum-Guide-US.pdf#zoom=100

·        Software in 30 Days: How Agile Managers Beat the Odds, Delight Their Customers, And Leave Competitors In the Dust 1st Edition, by Ken Schwaber , Jeff Sutherland