Starting (Disciplined) Agile: a possible approach

 Agile stalemate: "The land that Scrum forgot" 

Current large adoption of Agile is mainly based on Scrum. The stalemate on realizing more process improvements it is also - in many cases - related to Scrum. The Scrum team organization practices are easy to adopt, but many teams have no hints how to go further with improvements and with Agile. The original main good option was to adopt XP or Scrum & XP.  What is missing in many Scrum based Agile adoptions was describe by Robert C. Martin in “The Land that Scrum Forgot”.

The more recent approaches like DAD, comes with some complementary ideas: make the Agile habits explicit (as Agile life-cycles), offer guidance support instead of prescriptions, the teams should own & adapt their process context and others. 

First steps problems. Scrum advantage

Even I can argue that DAD approach is better than Scrum-only or  than Scrum & XP (in fact DAD allow to have a as possible choices Scrum-style or XP-style approaches), Scrum could have, in practice, a significant advantage on making the first steps, and making this Agile start it is fundamental. Here are some reasons:

•  The prescribed practices from Scrum area easy to adopt and it is less likely to be in a deadlock on starting with Agile. On the other hand, many XP practices are not so easy to adopt. DA comes with more guidance and with choices, but a team that struggle with the first Agile steps will still have difficulties:  “How should I start? What are the first factors that should be subjects of my choices?”
•  The main advantage of Scrum practices is related more not to the direct benefits, but with removing of the negative side effects of a lot of legacy (bad) practices that are replaced. Example: if a team use the self-organizing power (enabled by Agile) to make poor process choices and disregard, for example, the practices for a collaborative work, they will have worse results that a Scrum-based team. Simplifying: I will have better results using 3 useful practices from a set of 3 than using just one from a set of 10.

No stalemate: not in the first steps or further

 

We finally need to have an approach offers this kind of support: 

• Allow an easy start and make more likely the first steps in Agile
• Quickly definitely replace some legacy habits with many negative side-effects
• Avoid the further stalemates that too often are encountered with Scrum-only approach

Here is a possible approach on getting these results using DAD. The starting goals are the followings:

        Containers for managing the generic process goals: life-cycle and iterations
        Start working in a collaborative manner: adopt non-solo work practices
        Start persistent improvement: adopting retrospectives

 

Start working in a collaborative manner: non-solo work practices

As was mentioned before, the simple and rather vague Scrum practices for team collaboration have the great advantage of consistently replacing the old habits of non-collaborative work.

We can easily go further and get better results by adopting Agile Modeling/Disciplined Agile non-solo work practices – work with others for:

•     Release Envisioning – Requirements and Architecture Envisioning
•     Ahead look on requirements or design – Look Ahead Modeling
•     Just in time requirements or design – Model Storming
•     Pair Programming (XP practices)
•     Others

The essential is to offer guidance to the team and to start practicing non-solo work as soon it is possible. Agile-based process improvements and Agile retrospective works only if the team work is based on continuous and actively collaboration.

Start improvement by adopting retrospectives

The improvement based on Agile and other sources must become continuous and permanent. If the team does not adopt retrospectives, this goals is difficult to st accomplish.

Containers for generic process goals: life-cycle and iterations

In order to get a proper process in context, a team should think to the best options and choices to fulfill these goals. Anyway, we need some containers to manage these goals. Scrum use the Sprint ~ the iteration as the main container. XP goes further and have an implicit life-cycle.

In DAD we can find a more robust approach: we have both iteration and more explicit agile and lean life-cycles (it is the team choice on selecting one).

Ok, but “How should I start”? 

For an “agile beginner” team here is an option for main process containers:

•      The iteration - is generic for not-very-advanced cases
•      The agile-basic life-cycle – fits with most of the cases and it is similar with XP approach

As soon it is possible - using proper guidance – the team should be aware of process goals and how these goals are distributed along the life-cycle and iteration.
Of course, after (or in the same time) familiarizing with the Agile basic-life cycle, the team could explicitly adopt other types of life-cycles depending on the context.

Next step: Just do it! 

Using this initial framework – process goals across life-cycle and iterations – the team should start working collaboratively (non-solo work practices) and use retrospectives and other instruments to decide how to improve and adapt their process. Make sure that this approach is permanent and the process will have further optimizations. 

In time

There is a good probability that some stakeholders, the product owner, someone in development team ... or even you, to become unexpectedly unavailable during your current development.

If you believe that your are in-time, you are in fact delayed.

If you are already delayed, you are in fact much more delayed.

If you want to be in time, you need to be ahead.

Look Ahead Modeling:  http://agilemodeling.com/essays/modelAhead.htm

 
(To be continued)

Software developers "curse"

It is more likely that mathematicians and software programers to be introverts. Maybe it is just a myth. Anyway, an observable symptom is this: too often they just want to work "peacefully" alone. Also too often, if you want to extract them from this status, you can have same kind of surprise as in the case you want to steal the food of a baby polar bear (an exaggerated metaphor, but somehow very suggestive).

The problems is this: the domain - software development - nature and difficulty mandatory require , beyond developers intellectual abilities, also a high degree of collaborative work. It is not enough to be passively cooperative, it is rather necessary to be actively collaborative.

The "curse" of software developers is that they need to educate themselves to become some very collaborative introverts. 

That is bad? No, that it is interesting!

Polydamas Syndrome

 

..Great strength but little sense

 

Polydamas was an panktration (panktration - almost no rules mix of boxing and wrestling) champion at 93rd Olympiad, 408 BC.  Beyond Olympic success, there are some legends about his strength and power, that could remember of mythological Herakles (Hercule): he kill a lion with bare hands, he stop a moving chariot and win a fight with more "immortals" of  Persian king Darius.

However, he died trying to held with this hands the roof of cave that is crashing down around him.  

"The death of Polydamas, the Thessalian, when he was crushed by the rocks, made clear to all men how precarious it is to have great strength but little sense."  - Diodorus Siculus, Historical Library,  9.14.2

The key concept is the feasibility of an endeavor: no matter the skills, strength and power you have , but only if all these abilities are matching with the difficulty of the task. Knowing what we cannot do, the limits of our abilities it is very important.

Of course, we want always to go further and test and improve our abilities, but the also the involved risks matter, as in the Polymadas case.  

Polydamas Syndrome in software development 

 

Software development it is an area where this syndrome it is manifesting rather very often. The high rate of projects failures is not only a problem of inadequate management and inadequate engineering but also a problem of overestimating the involved skills versus the difficulty of  the context problems. 

Waterfall approach, for example, it is sign of this syndrome: if the iterative development try to manage the complexity in smaller parts, waterfall claim that will "eat" all the complexity in one piece.

Software development is the activity where most often smart people repeatably fails due to wrong estimation of their abilities on facing complexity.

Here are other examples from real life:

• Reckless: We have no idea about how to solve what the customer want, but we commit to solve it (eventually with also a committed delivery date). Possible reasons: wants to gain appreciations from customer and management. The root cause could be a gambling disorder-like behavior or just an immense vanity  
• Inadvertent: We know that we have no idea about the solution, but we hope that we will find it. That is happening many times with individuals that want a higher role or position and just hope that they will succeed without some previously proved practice. 

There are similar terms that the ones used by Martin Fowler describing the Technical Debt Quadrant. The "Polydamas Syndrome"  instantly produce or induce Technical Debt, that sometime could be irreversible.

By their nature, people could be reckless, inadvertent,  "normal" or too cautions. We need an adequate work discipline in order to get from all of them a good enough estimation for the balance between abilities and difficulties and also of the resulted risks in case of significant imbalance.     

Agile development at least it is defined as the "art of possible" and any manifestation of "Polydamas Syndrome" is a sign of severe agility problems and in general severe problems in the software process that should not be tolerated.

Solution Spikes and Architectural Spikes from XP are an possible answer to feasibility problems in Agile approaches, but a more complete set of practices could be found in DAD - Disciplined Agile Delivery that also have feasibility related milestones and an explicit approach for such risks.

"Seduction"

A big problem is that people with "Polydamas Syndrome" are very seductive for management and for customers. They represent at the first sight the dream collaborators: they will quickly commit to any request. The consequences are visible rather on medium and long terms. The main problem is when the management of the development side does not control the result risks. It is rather not expected that the customer side to be aware of these risks and they will surprised when the effects will become visible.

Human evolution patterns: from beginning of history to software

Beginning of human history

 

Asia Minor, more than 11,500 years ago, just after the Ice Age and 7,000 years before Stonehenge or Cheops. No agriculture. No domestic animals. No metals. Just some groups of hunter-gatherers. But … they have settlements, deposit of foods and a community. This community start building some massive stone temples. Their culture influence other groups and communities in the same area. 1000 years later, in that region are founded the first evidences of the agriculture and the first cities. And that was the beginning of the world history.

The first engineer and the first architect have existed before the first farmer.

The key of the evolution - community

 

Making an 11,000 years’ time travel, I want to quote from “Manifesto for Software Craftsmanship”, initiated by Robert C. Martin: “Not only individuals and interactions, but also a community of professionals”. 

The key word for evolution is community. 

The discovery of the Gobekli-Tepe temples in Asia Minor changes the model of the civilization evolution from: farming-settlement- religion-temples-cities to this one: settlement, religion, temples, farming then cities. (Of course that religion exist before settlements, but it is about more structured forms of religion).

Evolution sequence: settlement, religion, temples, farming then cities. 

The base of civilization was the settlement, the moment when was crystallized a stable group that share common values: from common food deposits to a common culture and form a community. This community was able to build temples and to distribute their culture

Hunter-gatherers, settlement, religion, temples, farming then cities. We can translate that to: teams, practices, sharing values, community and culture, distributing the culture, transformation: new practices, new values and new communities.

The software new civilization: between dilution and evolution

 

The recent evolution of software development has follow a similar (cyclic) pattern: the late 80s and the 90s start with some new practices, new small communities with new values. That has been finalized with some cultural “settlements”: CMMI, UML, Unified Process, Agile Methods and Agile Manifesto. This culture will be distributed and new communities will be formed.

Unfortunately, the initial new software civilization was somehow diluted during its expansion: the only widely adopted method was Scrum (very lightweight method) and many values, especially engineering values were almost forgotten. (See Robert C. Martin comments here  https://www.scrumalliance.org/community/articles/2010/december/the-land-that-scrum-forgot ). 

There is a new wave that want to recover the forgotten values and go further: the Software Craftsmanship movement and the Disciplined Agile Delivery (DAD) method are the best examples.  Both are dealing with some biggest elephants in the Agile room.

DAD want to make explicit the Agile cultural values and practices and recognizing that context matter, this method offers guidance for adapting the process to different situations.

Foundations of Software Craftsmanship are older and I will remember here only some contributors of the Agile related foundations: Kent Beck, Martin Fowler, Scott W. Ambler and Robert C. Martin. The last one – Uncle Bob – try to push further this wave with Clean Code and Clean Architecture initiatives. Unfortunately this part is difficult enough to not see yet other valuable contributors or even valuable distributors. A symptom of this problem is that most of the “distributors” talk only about Martin SOLID principles, without an understanding of the bigger picture or of other aspects. It is just a copy-paste distribution.

There are two divergent evolution aspects now: a distribution that dilute values and new endeavors for enhancing and improving the original values.

Human Evolution patterns

 

The evolution patterns seems to be in this sequence:

• Prehistory
• Teams & practices
•  Small and slow transformations, many of them being lost because of limited distribution and sharing

• History
• Settlement: sharing common values, community, culture
• Distribution: sharing culture, new communities 
• Transformation: communities generate new practices and new values 

Evolution factors: communities enable values sharing and creating of new ones 

Involution factors and sequence:

• Success with diluted values
• Dissolution of communities
• Cultural regression 

 Involution factors: cutting the value distribution chains and the success of “non-valuable values”

The involution factors seems to be the same in history of human civilization and in the one of software development.  The human history has several regressions caused by “success of diluted values”: such as the Greek Dark Ages (1100 BC–800 BC), the Dark Ages of Middle Ages. The history does not necessary represent a continuous linear progress and same risks could affects the software development domain.  For example, the rising of new web technologies cause a dilution on design quality ("Architecture lost years" - Robert C. Martin), even that is only because of their continuous changes. Poor design is finally resource consuming and expensive and that cause regression or slowing of the overall evolution.

These patterns that seems to be the ones that have "shaped" the human evolution. I think that should be considered for any aspect of evolution and for different levels, from large communities and "populations" to organizations level.

Any invention of human genius is born dead if is not shared and embraced by a community that will make that invention as a valuable step in the evolution. On the other hand, a community that does not profits from its "inventors" it is not a community. We need to enable, recognize and share the real values in order to have a community.

See - http://ngm.nationalgeographic.com/2011/06/gobekli-tepe/mann-text

NEXT

• Principles for a community of professionals - I
• Principles for a community of professionals - II 

Discussing "Inventory" in Software Development - Classifying approaches

Interesting comments of Martin Fowler starting from this idea : "So as an example - one of the principles of lean manufacturing is the elimination of inventory.This leads to the question of whether there is an analogous item to inventory in software development."

http://martinfowler.com/bliki/MetaphoricQuestioning.html

Martin Fowler make a very interesting comment about a IMO less interesting idea "up front documentation is the equivalent of inventory": "Now I agree we need to substantially reduce this kind of speculative documentation; but the rationale for doing so must come from thinking about the software development process, not from purely reasoning by analogy."

Here is my comments on these subjects.

Manufacturing-Software Development Analogies

We should be careful on these analogies because these domains areas are pretty different. Examples: Just-In-Time approach in manufacturing must find solutions for a JIT Production (where the design was previously done "offline") and software development must find find solutions for a JIT Design ("Code is design").

Inventory in Software Development 

Manufacturing inventory mean (simplified) a reserve queue of (done) products, ready to be sold. A similar concept in software development IMO is not the massive up-front documentation (based on too many assumptions) , but rather in the motive of producing a such documentation: a massive set of requirements allocated to the same release/delivery/project. "No inventory" approach means as small is possible set of requirements per release. Indeed from this point of view, the classification is already done:

• Large set of requirements releases - non-agile, non-lean approach
• Small set of requirements releases - Agile approach, Lean approach
• Very small set Small set of requirements releases - Continuous Delivery approach

An explicit description of  lean and Agile life-cycle models could be found in DAD- Disciplined Agile Delivery: https://disciplinedagiledelivery.wordpress.com/lifecycle/ 

More: With an increased construction productivity and/or a higher envisioning productivity, a larger set of requirements could be delivered in the same time. Productivity suppose here here good results and quality 

The resulted formula of "Inventory" for software development:

(Set of Requirements "volume")/productivity.

Comment: the "formula" could be enhanced to:

 (Work volume")/productivity. 

But, in this case, it is too vague.  The main aspects that increase volume or complexity of the "work" must be specified. here are some of them:

• Requirements volume and complexity at the start
• Solution that must match the requirement 
• Team productivity 
• Process

All elements of the formula are "requirements" for "No Inventory" process in software development:

• small set of requirements
• good design
• no junk stories with associated phantom features
• high productivity (all contributing aspects)
• small releases 
• adapted/effective/efficient process 

Finally, seems that starting with this assumption about a "No Inventory" approach for software development we are very close to the description of a Lean/Agile approach.

See more here:

• JIT – Just in time and Software Development 
• Two dimensions: Just in time and Envisioning

Requirements, Business Needs, Value: all together

Traditional 

The so called "traditional" approach to materialize WHAT should be done (by a software system) it  is described by "requirements". It is specified also that requirements must be prioritized, but too often that failed because of lack of focus.

Unified Process 

A simplified extension it is provided by Unified Process that define a hierarchical pyramid-like model where Business Needs are in top of Requirements, and there is a trace between each business need and corresponding requirements.

Extended

An extended approach it is described by various sources related to software engineering (Unified Process included): the relationship between target business ans software system requirements are more detailed modeled: with flows (use cases) and data. The software systems use cases/flows are integrated in the overall business use cases/flows (and also the data/information).  

Agile

The most common additional concept associated with the requirements from the Agile viewpoint it is the value (provided to the business) encapsulated by a set of the requirements.

Comments

• Traditional: seems to be incomplete, because all others are adding valid extensions
• Extended - flows and data: it is a valid conceptual model about how the business and software system are integrated in details, but seems also to be incomplete on dimension that quantify priorities of the requirements (for other reason that detailed dependencies that are clearly defined here)
• Business Needs - could be very useful because specify "requirements of the requirements". Anyway, could not describe detailed relationship between business and requirements (as flows and data) but rather most important aspects and it is useful for strategic decisions related to the requirements.
• Value - it is somehow similar with the Business Needs, but add something more: we can address the business needs better of worse, that mean we could deliver more or less value for same business needs (example: by quality and supplementary non-functional requirements). It quantify that they customer will finally receive. 

An integrated approach 

All the above approaches are correct, but all are incomplete. Unfortunately, for the sake of promoting only one method or only one approach, there are advertised - "sold" -  only some of them. It is very common these days, when Agile it is very trendy, to talk about only about value and diminish the others. A coherent and professional software engineering approach should consider all of them and use them when and where it is appropriate.

Story

We have to define WHAT the software system must do for the target business. This "what" need a name and this name it is Requirements. Anyway, these requirements cannot be defined directly, we need to be aligned to some goals and use specific practices and concepts:

• Should be a traceability between target business objectives and the requirements: we can use here the Business Needs concept 
• We need to be more precise about what we shall deliver for business (related to this business needs), and how well are addressed. Here we can use the Value concept 
• The Value help us to fulfill another goal: prioritizing requirements according to what the target business evolution needs 
• Both Business Needs and Value provide instruments to strategically manage the requirements 
• For detailed relationship between business and requirements and their integration, we can use the flows & data model. We can ignore this aspect, but in reality business and system are interacting in a such manner and we need an explicit understanding about what it is really happening. If fact that it is what we will implement and we need to be aware of that.  How much detailed representation "on paper" it is necessary, that it is a another discussion  

Conclusion: there are more aspects and dimensions on the process of managing the requirements and could be very useful to use more concepts and practices (each having its role): Business Needs, Value, Requirements, Flows and Data.

Notes

• User stories are just particular form (less structured and minimalist) of flows & data approach
• "Features" concept it is less useful because it is too generic  ( such as "functionality")