USING FEEDBACK
Here is one of the very specific Agile & Lean approaches: developing and delivering in small batches. Commentary: there are many practices in agile and lean that use feedback, but I'm talking here about the fundamental one, related to the life cycle of deliveries.
Who recommended this, and with what support and arguments?
Agile manifesto: The Agile manifesto is clearly centered around the approach of small releases as evidenced by the principles outlined in the AM (quotes):
“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”, and
“Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.” (See [B- AM] ).
XP makes an explicit recommendation for small releases (see [B-XPE1], [B-XPE2]) and builds its practices ecosystem on this hypothesis. Integration and test-first approaches are among the supporting practices.
Scrum, on the other hand, doesn't emphasize the importance of small
releases explicitly. While they suggest that work should be potentially
releasable after each Sprint, there is no explicit requirement for releases to
be small. (See [B-SG])
Disciplined Agile - In contrast, Disciplined Agile has "Accelerate value delivery" as a key mindset and delivery process goal, with a variety of options, practices (including those from XP), and guidance to help achieve it (see [B-DA AVD]). They also embrace the Lean approach of delivering in small batches, supported by numerous arguments.
If we want to distinguish between the role of feedback in human work processes versus technological processes, it may be best to abandon this traditionalist approach, as we have seen how AI, a “technology”, is increasingly capable of replacing humans in various tasks.
The math behind this - There are some situations where feedback control systems cannot be avoided: complex systems with time-varying dynamics, where it is not possible to obtain an analytical model to predict or control the system's behavior. Feedback loops provide the necessary information to obtain a flexible and adaptive approach to respond to changing conditions and achieve desired outcomes in such contexts. In such cases, there is no upfront formula to solve your problem, and you should rely on feedback loops to manage complexity. Remember that Big Up From Requirements, Design and Planning was one of the defining trait of waterfall approach.
Back to software: What are some main traits of software development and other knowledge work-based domains? Uncertainty and complexity. Attempting to solve the complexity and uncertainty related to knowledge work without using a lot of feedback loops (through the agile and lean approach) seems to be mathematically infeasible.
USING CONTEXT KNOWLEDGE
Math: However, feedback-based approaches also have some limitations. These approaches rely on a model of the system's dynamics and behavior, and if this model is inaccurate, it can severely limit system performance.
For example, feedback may be given through testing, either manual or automatic. If we cannot accurately model the test cases, or if we don't understand that certain tests are related to specific dynamic race conditions or competition levels, we may not achieve the desired results. Similarly, if we try to copy feedback-based practices that work well for another team or product, they may not be suitable for our context and the specific problems we are trying to solve.
To overcome these limitations, it is essential to understand and adapt to our context, continuously updating our model of it, beyond relying on feedback alone.
How powerful was medicine when it also collected the symptoms, but did not understand the internal processes of the body? In the meantime, the means of observation and measurement improved, as well as the understanding of these internal processes.
Who recommended this, and with what support and arguments?
Agile Manifesto: The authors of the Agile manifesto acknowledged that they were only able to agree on values and principles, not specific practices, as each of them situation and work context is different.
XP recommends adapting the framework if it's not working, but
there is no specific guidance on how to tailor it. (See [B-XPE2])
Scrum provides little guidance beyond its lightweight framework, as the
intention is for teams and organizations to define the rest of the process in
their specific context. (See [B-SG])
Disciplined Agile was developed to provide guidance and options for practices and life cycles (See [DA-LC]) in context, and the DA Toolkit is a free online knowledge base for this purpose (See [B-DATK]). Disciplined Agile also use System Thinking (system is a product of the interaction of its parts, not just the sum of its parts.) to be able to have a more correct model of the problems to be solved.
 |
| We must tailor the process according to our specific context and not copy the solution from another context |
USING EXPLORATION
There are various factors that can impede finding good solutions to complex problems. One such factor arises when developing a new product, where the requirements and solution spaces may not be well defined. In such cases, it may be necessary to undertake extensive exploration to determine the functionalities that potential customers require, as well as to identify the feasible and optimal technical solutions among numerous options.
To my knowledge, James Coplien argues that agile as "responsive iterative design" does not work well (See [B-JC] ), and I think he is referring to situations where just doing iterative, incremental work will not yield the best option from several possibilities.
While some argue that agile methods do not work well in such situations, we contend that feedback and small batches are still useful for solving complex problems and understanding the context model. However, relying solely on feedback and small batches, can lead to local optimization and to overlooking better solutions. By selecting a technical solution and gradually improving it, we may miss out on superior options.
Math Support
- To address such issues, a possible mathematical approach is to use genetic
algorithms, which explore a broader range of solutions and options (and not
just optimize, develop and improve the initial one (See [B-GA]). There are also many other mathematical options to avoid a "local" maximum (See [B-LO])
In terms of process approaches, one example is the product
life cycle model proposed by Kent Beck - Explore. Expand, Extract - which emphasizes exploration in the initial part of this
life-cycle. (See [B-Beck])
Disciplined Agile also provides a dedicated development and delivery life cycle variant for situations where multiple options need to be explored, called "Exploratory Lean Startup" (See [B-EXLC]). Essentially, this approach involves exploring multiple options while using the fail-fast idea, as there is often limited time and resources available to explore options in detail. Additionally, DA recommends leveraging existing sources of knowledge or experts, such as the DA toolkit for practice options, to move faster or avoid unnecessary fail-fast sequences.
Some Conclusions
Although Agile and Lean processes are not inherently based on math, there is a mathematical foundation underlying them. Unfortunately, this foundation is not widely known, and the various aspects of Agile and Lean are often discussed separately, without recognition of their interconnectedness. Disciplined Agile provides comprehensive support for the various dimensions of this mathematical logic, offering an integrated and robust approach to agile and lean methodologies. XP and agile principles provide support only for the part related to feedback use.
Trivia ... future AIs
These mathematical dimensions of solving complex problems should also be part of the capabilities of future AIs. An AI can learn, it can use feedback, it can learn "offline". But how well can it build a model of the context, how well can it explore more options (and not in just a narrow field)? For now, exaggerating, they seem to be just talking encyclopedias with computing power.
References
[B- AM] Principles behind Agile Manifesto - https://agilemanifesto.org/principles.html
[B-XPE1] – Extreme Programming Explained First Edition, By Kent Beck, Addison-Wesley, 1999
- About „small releases” or „Short Releases”, see Chapter 10 - A quick Overview, Chapter 11 - How could this work, Chapter 26 Xp At work
[B-XPE2] – Extreme Programming Explained: Embrace Change, Second Edition, By Kent Beck, Cynthia Andres, Addison-Wesley, 2004
- Chapter 15, Scaling XP
[B-SG] Scrum Guide - https://scrumguides.org/scrum-guide.html
[B-DA AVD]- Disciplined Agile, Accelerate Value Delivery Process Goal
https://www.pmi.org/disciplined-agile/construction-goals/move-closer-to-deployable-release
[DA-LC] - https://www.pmi.org/disciplined-agile/lifecycle
[B-LO] For local versus global optimization - see https://en.wikipedia.org/wiki/Global_optimization
[B-GA] For genetic algorithms – see https://en.wikipedia.org/wiki/Genetic_algorithm
[B-EXLC] Disciplined Agile, Exploratory Life Cycle: https://www.pmi.org/disciplined-agile/lifecycle/exploratory-lifecycle
[B-JC] Why Responsive Iterative Design is Evil - James Coplien | Codecamp_Festival 2022
https://www.youtube.com/watch?v=tjXrNeolmhg
[B-Beck] – Product development triathlon
https://www.facebook.com/notes/kent-beck/the-product-development-triathlon/1215075478525314/
https://medium.com/@kentbeck_7670/the-product-development-triathlon-6464e2763c46
[B-DATK] Disciplined Agile Toolkit - https://www.pmi.org/disciplined-agile/toolkit
