What is wrong with Agile?

 Alongside numerous works that popularize the agile way of working, occasionally we come across articles or presentations created by experienced professionals who express exasperation with Agile.

Why is this happening?

Recently, a friend sent me the link to this article, suggestively titled “The Age of Agile must end” [See B1-Age of Agile]. I could easily see myself spending an hour discussing the points I might disagree with in this article, and perhaps even another two hours exploring areas of potential agreement. It strikes me as an excellent example of these perspectives, where software professionals or other professionals express their dissatisfaction with Agile. In short, I can say that I resonated with certain problems described. Here are some aspects of how Agile is used that, from my perspective, could profoundly affect our work:

Agile Simplified as Scrum: A good part of the "Agile world" is centered around Scrum, or similar prescriptive methods, serving as the sole reference point.

Challenges with New Products: Developing a new product often demands a distinct life cycle compared to established products. The Scrum-Agile cycle is more suited to the latter. Therefore, a variety of development life cycles should be available for selection.

Adapting Lean Techniques: When integrating Lean into an Agile-lean approach, customization according to the situation and the current stage in the product's life cycle becomes crucial.

Unaddressed Technical Debt: The issue of quality, technical debt, and a deficiency in technical excellence remains a significant concern within the practical Agile landscape.

Technical Debt in New Products: Indeed, technical debt can swiftly accumulate and impede even the development of a freshly conceived product.

Compromising for Expediency: The tendency to cut corners often gives rise to the accumulation of technical debt.

Limited Focus on Short-Term Goals: Concentrating solely on short-term objectives and hoping for medium and long-term goals to somehow resolve themselves is a common practice.

To me, these are symptoms of "Incomplete Agile". Let's talk about them.

 

 

Agile simplified as Scrum

 

Reducing Agile to Scrum (and other prescriptive methods), along with the Agile manifesto, doesn't provide a robust foundation for developing Agile processes that can adapt to various situations. A significant portion of the "Agile world" relies heavily on these limited references. Many people criticize Agile for not aligning well with their specific contexts and the outcomes they want to achieve. In reality, this often points to this "Incomplete Agile" approach.

Definitely, Agile, as well as your process, has more than 10 practices.

 

New Products Problems

The essential tools required in this scenario cannot be found within the realm of Scrum. While the Scrum lifecycle could indeed prove valuable for a mature product, an alternative lifecycle better aligned with this situation exists: the Exploratory Lean Startup, one of the several Agile-Lean lifecycle options detailed in the DA Toolkit (see [B2 – DA Toolkit]). Instead of adhering to regular deliveries, the focus shifts towards conducting numerous small experiments to identify market needs and showcase the proposed solution approach.

A more detail discussion about the type of development life-cycle could be used in the different stage of the product life cycle can be found in one of my previous posts (See [B4 Delivery vs Product ]).

When you begin a new product, you are in an exploratory stage and should adapt your approach. 

 

Adapting Lean Technique  

Some distinct approaches are necessary for various stages of the product development lifecycle. For well-established teams and existing products, pinpointing delays (a significant result of diverse sources of waste) holds paramount importance. Conversely, for emerging products, it becomes even more critical to detect trends in the primary waste sources before substantial delays accumulate. Developing a new product requires drawing from the experience of waste prevention.

Also, we're not precisely employing the Lean manufacturing approach; rather, we're utilizing a distinct approach that is better tailored for knowledge work.

You should eliminate delays from your workflow, and even better, strive to avoid them.

 

Unaddressed Technical Debt

Scrum deliberately does not tackle this challenge explicitly, possibly because its creators known XP approach and aimed only to introduce additional elements. As a result, teams and products that intentionally embrace an exclusively Scrum-based approach, without addressing the "technical excellence" aspect emphasized in the Agile Manifesto, face the potential of encountering significant risks.

A key phrase in the mentioned article [See B1 Agile Age] is “Invest in foundational efforts like design systems and a clean tech stack so future efforts can be more efficient.”

The XP approach for this aspect might remain vague or challenging to duplicate. The team that embodies the XP style had a high level of skill. They didn't create a "mess" and used refactoring to align their evolving system design with new requirements

Contrastingly, a typical Agile team faces more risk in generating "mess" and accumulating technical debt. It's advisable to invest intentional effort into overall design and technology stack options decisions. Establishing a foundational base is necessary, followed by the opportunity to refine it as experience grows and new system requirements emerge. This approach indeed pursues a goal that extends beyond the immediate delivery.

 Software ages akin to milk, not maturing like fine wine.

 

Technical Debt in New Products

 

We have several competing goals concerning the economics of the first delivery and the feasibility of continuing the business post the initial delivery. These include swiftly determining if our product demanded by the market is viable and if our technical solution is feasible, preferably without incurring substantial expenses.

Establishing a robust internal design is essential, enabling modifications to the product without compromising its integrity or significantly increasing costs through the accumulation of technical debt.

Moreover, it's important to recognize that a product consistently has facets necessitating exploration and expertise. Therefore, opting for the appropriate life cycle to accommodate these needs is crucial.

Beware: you can create hard-to-change software from the start.

 

Compromising for Expediency

 

What is our objective? Is it solely the current delivery? Is it both the current delivery and future deliveries? Do we want to ensure the continuity of our business? Technical debt accumulates over time, leading to progressively worsening delays and escalating quality issues. There could be several possible reasons for not effectively addressing these issues: perhaps we don't fully comprehend them, or perhaps we do understand them but take a reckless approach.

Behind many technical disasters lies inadequate or reckless behavior.

Limited Focus on Short-Term Goals

 

We need a deliberate effort and dedicated actions to address also our medium- and long-term objectives. Here are some examples. We need to ensure that the internal design quality remains manageable and aligned with both current functional and non-functional requirements. This practice aids us in delivering high-quality products within designated timeframes and at minimized costs across short, medium, and long-term periods. To achieve this, as well as other equally valuable objectives, we should continuously enhance the knowledge and skills of our personnel. Additionally, it's crucial to monitor sources of waste to avert the repercussions stemming from their accumulation.

Long and medium-term goals encompass more than simply the aggregation of short-term goals.

 

How to address these problems

We needed to effectively address both our short-term and medium- to long-term objectives. This necessitates a deliberate and ongoing commitment to improvement. To achieve this, we should draw upon the collective industry experience, rather than relying solely on specific Agile methods. Our development cycle and practices should be chosen based on our unique circumstances, requiring a thorough understanding of available options. While tools like ChatGPT are trendy, their suggestions should complement our decision-making rather than dictate it. But are these insights structured and validated by industry experience? Success might be a matter of luck. If we seek access to industry-proven expertise concerning these options and guidelines on their contextual application, the Disciplined Agile Toolkit (see [B2 – DA Toolkit]) serves this purpose. If our aim is to overcome the challenges posed by an incomplete or ill-suited Agile approach, it's prudent to customize it ourselves within our specific context, leveraging the wealth of industry experience already available to us.

A good starting point is the online DA Browser knowledge base (See [B3 – DA Browser])

Do not try to solve any problem you have with the same set of 10 practices.Adapt Agile practices to the specific context of the product, the team, and the organization.

Utilizing multiple references for Agile & Lean is recommended, and you can leverage the organized structure of these references available in the DA Toolkit.

 

REFERENCES

[B1 Age Of Agile]  The age of Agile must end, by Michael Burnett, Feb 12 2023

https://uxdesign.cc/the-age-of-agile-must-end-bc89c0f084b7

[B2 – DA Toolkit] Disciplined Agile Toolkit

https://www.pmi.org/disciplined-agile/

[B3 – DA Browser] – Disciplined Agile Browser

https://www.pmi.org/disciplined-agile/da-browser

[B4 Delivery vs Product ] Delivery life-cycles versus product lifecycle

https://agileand.blogspot.com/2020/01/life-cycle-evolution-example.html

 

 

The math behind Agile and Lean... and the future of AI

USING FEEDBACK

 

Here is one of the very specific Agile & Lean approaches: developing and delivering in small batches. Commentary: there are many practices in agile and lean that use feedback, but I'm talking here about the fundamental one, related to the life cycle of deliveries.

Who recommended this, and with what support and arguments?

Agile manifesto: The Agile manifesto is clearly centered around the approach of small releases as evidenced by the principles outlined in the AM (quotes): 

“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”, and

 “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.” (See [B- AM] ).

XP makes an explicit recommendation for small releases (see [B-XPE1], [B-XPE2]) and builds its practices ecosystem on this hypothesis. Integration and test-first approaches are among the supporting practices.

Scrum, on the other hand, doesn't emphasize the importance of small releases explicitly. While they suggest that work should be potentially releasable after each Sprint, there is no explicit requirement for releases to be small. (See [B-SG])

Disciplined Agile - In contrast, Disciplined Agile has "Accelerate value delivery" as a key mindset and delivery process goal, with a variety of options, practices (including those from XP), and guidance to help achieve it (see [B-DA AVD]). They also embrace the Lean approach of delivering in small batches, supported by numerous arguments.

If we want to distinguish between the role of feedback in human work processes versus technological processes, it may be best to abandon this traditionalist approach, as we have seen how AI, a “technology”, is increasingly capable of replacing humans in various tasks.

The math behind this - There are some situations where feedback control systems cannot be avoided: complex systems with time-varying dynamics, where it is not possible to obtain an analytical model to predict or control the system's behavior. Feedback loops provide the necessary information to obtain a flexible and adaptive approach to respond to changing conditions and achieve desired outcomes in such contexts. In such cases, there is no upfront formula to solve your problem, and you should rely on feedback loops to manage complexity. Remember that Big Up From Requirements, Design and Planning was one of the defining trait of waterfall approach.

Back to software: What are some main traits of software development and other knowledge work-based domains? Uncertainty and complexity. Attempting to solve the complexity and uncertainty related to knowledge work without using a lot of feedback loops (through the agile and lean approach) seems to be mathematically infeasible.

 

USING CONTEXT KNOWLEDGE

 

Math: However, feedback-based approaches also have some limitations. These approaches rely on a model of the system's dynamics and behavior, and if this model is inaccurate, it can severely limit system performance.

For example, feedback may be given through testing, either manual or automatic. If we cannot accurately model the test cases, or if we don't understand that certain tests are related to specific dynamic race conditions or competition levels, we may not achieve the desired results. Similarly, if we try to copy feedback-based practices that work well for another team or product, they may not be suitable for our context and the specific problems we are trying to solve.

To overcome these limitations, it is essential to understand and adapt to our context, continuously updating our model of it, beyond relying on feedback alone.

How powerful was medicine when it also collected the symptoms, but did not understand the internal processes of the body? In the meantime, the means of observation and measurement improved, as well as the understanding of these internal processes.

Who recommended this, and with what support and arguments?

Agile Manifesto: The authors of the Agile manifesto acknowledged that they were only able to agree on values and principles, not specific practices, as each of them situation and work context is different.

XP recommends adapting the framework if it's not working, but there is no specific guidance on how to tailor it. (See [B-XPE2])

Scrum provides little guidance beyond its lightweight framework, as the intention is for teams and organizations to define the rest of the process in their specific context. (See [B-SG])

Disciplined Agile was developed to provide guidance and options for practices and life cycles (See [DA-LC]) in context, and the DA Toolkit is a free online knowledge base for this purpose (See [B-DATK]). Disciplined Agile also use System Thinking (system is a product of the interaction of its parts, not just the sum of its parts.) to be able to have a more correct model of the problems to be solved.

We must tailor the process according to our specific context and not copy the solution from another context

In conclusion, attempting to solve the complexity and uncertainty of knowledge work without considering context (e.g. for modeling our process and selecting practices) is inherently flawed.

 

USING EXPLORATION

 

There are various factors that can impede finding good solutions to complex problems. One such factor arises when developing a new product, where the requirements and solution spaces may not be well defined. In such cases, it may be necessary to undertake extensive exploration to determine the functionalities that potential customers require, as well as to identify the feasible and optimal technical solutions among numerous options.

To my knowledge, James Coplien argues that agile as "responsive iterative design" does not work well (See [B-JC] ), and I think he is referring to situations where just doing iterative, incremental work will not yield the best option from several possibilities.

While some argue that agile methods do not work well in such situations, we contend that feedback and small batches are still useful for solving complex problems and understanding the context model. However, relying solely on feedback and small batches, can lead to local optimization and to overlooking better solutions. By selecting a technical solution and gradually improving it, we may miss out on superior options.

Math Support - To address such issues, a possible mathematical approach is to use genetic algorithms,  which explore a broader range of solutions and options (and not just optimize, develop and improve the initial one (See [B-GA]). There are also many other mathematical options to avoid a "local" maximum (See [B-LO])

In terms of process approaches, one example is the product life cycle model proposed by Kent Beck - Explore. Expand, Extract - which emphasizes exploration in the initial part of this life-cycle. (See [B-Beck])

Disciplined Agile also provides a dedicated development and delivery life cycle variant for situations where multiple options need to be explored, called "Exploratory Lean Startup" (See [B-EXLC]). Essentially, this approach involves exploring multiple options while using the fail-fast idea, as there is often limited time and resources available to explore options in detail. Additionally, DA recommends leveraging existing sources of knowledge or experts, such as the DA toolkit for practice options, to move faster or avoid unnecessary fail-fast sequences.

 

Some Conclusions

Although Agile and Lean processes are not inherently based on math, there is a mathematical foundation underlying them. Unfortunately, this foundation is not widely known, and the various aspects of Agile and Lean are often discussed separately, without recognition of their interconnectedness. Disciplined Agile provides comprehensive support for the various dimensions of this mathematical logic, offering an integrated and robust approach to agile and lean methodologies. XP and agile principles provide support only for the part related to feedback use.

Trivia ... future AIs

These mathematical dimensions of solving complex problems should also be part of the capabilities of future AIs. An AI can learn, it can use feedback, it can learn "offline". But how well can it build a model of the context, how well can it explore more options (and not in just a narrow field)? For now, exaggerating, they seem to be just talking encyclopedias with computing power.

  

References

 [B- AM] Principles behind Agile Manifesto - https://agilemanifesto.org/principles.html

 [B-XPE1] – Extreme Programming Explained First Edition, By Kent Beck, Addison-Wesley, 1999

• About „small releases” or „Short Releases”, see Chapter 10 - A quick Overview, Chapter 11 - How could this work, Chapter 26 Xp At work

[B-XPE2] – Extreme Programming Explained: Embrace Change, Second Edition, By Kent Beck, Cynthia Andres, Addison-Wesley,  2004

• Chapter 15, Scaling XP
  

[B-SG] Scrum Guide - https://scrumguides.org/scrum-guide.html

[B-DA AVD]- Disciplined Agile, Accelerate Value Delivery Process Goal 

https://www.pmi.org/disciplined-agile/construction-goals/move-closer-to-deployable-release 

[DA-LC] - https://www.pmi.org/disciplined-agile/lifecycle

[B-LO] For local versus global optimization - see https://en.wikipedia.org/wiki/Global_optimization

[B-GA] For genetic algorithms – see https://en.wikipedia.org/wiki/Genetic_algorithm

[B-EXLC] Disciplined Agile, Exploratory Life Cycle: https://www.pmi.org/disciplined-agile/lifecycle/exploratory-lifecycle

[B-JC]  Why Responsive Iterative Design is Evil - James Coplien | Codecamp_Festival 2022 

 https://www.youtube.com/watch?v=tjXrNeolmhg

[B-Beck] – Product development triathlon  

https://www.facebook.com/notes/kent-beck/the-product-development-triathlon/1215075478525314/

https://medium.com/@kentbeck_7670/the-product-development-triathlon-6464e2763c46

[B-DATK] Disciplined Agile Toolkit - https://www.pmi.org/disciplined-agile/toolkit