50 Shades of Ceremony

This thing is the third instalment in a triptych about me being a software architect. Some time ago I started out by explaining what I consider a good software architect. Recently the second chapter explored why I think I’m a good software architect. This final part wraps it up and completes the overall picture, illustrating how I typically deal with software architecture. And it starts, of course, with a provocative, blunt statement:

Agile is Dead, Long Live Waterfall

Okay, that should have put at least some of you on edge. Don’t you just love clickbait? Yet, in this case, I also have a point. Bear with me.

So without further ado, gather around the 4K Campfire ↗ children, because grandpa Christophe is going to tell the tale of nature, nurture and nudges.

Don’t You Worry, It’s Our Nature

Whatever we do, we do it according to our nature, our nurture and some nudges. Some of our qualities we get through our DNA, which consists of some generic human qualities and more specific qualities we get from our parents. That’s what we get to start with, and give or take a few world record level physical advantages, we get more or less the same foundation. In the end what we do with it, is what makes the real difference. Still, by nature some of us seem to have a natural ability to excel.

Yet, even those that excel by nature, still have to work for it. The capacity to do what is needed, to do the reps, to put in the time, to go beyond limits, to handle stress,… is an important factor that is largely addressable to nurture. Nurture is that additional set of qualities that our parents, our family and basically everyone we encounter in our life, lay down upon us. In this category also fall life-changing experiences, both good and bad. Without nurture, our genetic qualities would be much like an unmaintained garden. Every garden needs attention, watering, fertilising, weeding. Good gardening can turn any piece of bare land into a beautiful garden, just as well as bad (or no) gardening will turn even that beautiful garden into a barren wasteland.

An interesting aspect of nurture, is that it doesn’t stop with everything around us. It can also become self-manifesting. When through nurture our natural qualities have become well maintained and have grown into great human properties, a self-reinforcing meta-property can appear and continue nurturing these qualities by itself. It’s the moment where we choose to invest in our qualities and enhance them even further. We no longer need external motivation, we can motivate ourselves. And so we start nurturing our own nature.

Sometimes, nature’s given qualities don’t flourish, even if they are nurtured properly. Sometimes we need… a nudge, a small (or larger) pat on the back, a push in the right direction, a forceful hand, a boss, a law, a police officer, a judge,…

If we were perfect beings - now there’s some real SciFi - we wouldn’t need these nudges, we would all adhere to common sense, and wouldn’t cut corners or worse, do things that (might) hurt other people. Alas, we’re human beings, and human beings need… nudges.

Enter Ceremony

Since none of us really likes being nudged - yet it seems to be the only way we really can be nurtured - our ancestors, since the dawn of the human race, have come up with clever ways to nudge us, without us really noticing it. They invented ceremonies, which we seem to accept much more easily.

Ceremonies implement nurturing nudges in a story, in a celebration, in a more pleasant routine, or simply in a commonly accepted pattern all need (or at least agree) to adhere to. And those ceremonies are all around us: from our earliest moments, we are bombarded with ceremonies to nurture us, to teach us how to nurture our nature’s given qualities.

In Kindergarten we are told to “stand in line” and “hold hands”. Well not really told, a song guides us in doing so, and we happily sing along. Other songs teach us “how to wash our hands” or “be silent”. For some reason, most children respond better to these songs than to simple commands like “stand in line”, “hold hands”, “wash your hands” or “be silent”. Somehow, it seems to be in our common nature to not respond well to commands, yet in the form of a ceremony we happily comply.

The most prevalent example of such a set of widespread ceremonies is religion. It doesn’t matter which flavour of religion, they all serve the same purpose: “Religion ideally serves several functions. It gives meaning and purpose to life, reinforces social unity and stability, serves as an agent of social control, promotes psychological and physical well-being, and may motivate people to work for positive social change.” (kindly copy/pasted from Sociological Perspectives on Religion from Howard Community College ↗). So basically, religion nudges and does so using several ceremonies.

Still, religion also had to learn how to do this effectively. Take Catholic theology, the ten commandments ↗. Moses did bring the ten commandments on tablets to the people. And those ten commandments are already a ceremony by itself, still that clearly wasn’t enough. Take for example the sixth commandment ↗: “You shall not commit adultery.” I bet that one, at that time, pretty much sounded like “please be silent” in Kindergarten today. Still, the underlying nurture to this nudge was to avoid spreading STDs and ensure a stable family for children to grow up in. To add weight to these words, religion invented the ceremony of marriage. Because everybody loves a good party, presents,… and most of the time, this seems to give enough incentive to “not commit adultery”. Most of the time. And then there are always lawyer nudges and prenup nudges.

The list of ceremonies goes on and on: the army uses ceremonies to tame the most untameable and make them willingly run into hostile fire, most of us still follow the 9 to 5 rat race ceremony,…

Ceremonies are good. We need them. They help implement the nurturing we need to turn our nature’s given qualities into positive and valued properties. Still it is important to see through them and know what they stand for. If not, we might turn something good into something very bad. We don’t have to look far for prime examples of this: despite the best intentions, literalist and fundamentalist interpretations of religious stories and ceremonies has led to some devastating historic events, and still does today.

It basically boils down to the fact that the ceremony can’t be the final answer, you must know the answer to the “why this ceremony?”-question and understand the nature of the nurturing nudges that the ceremony tries to uphold. But we’ll get back to that in a minute.

Meanwhile in 2001…

… Kent, Mike, Robert, Martin and 13 other friends came up with their own commandments. They wanted to nurture a growing batch of new programmers - many of whom were summoned recently due the Millenium Bug ↗ and could use some guidance. So they went up a hill (or rather slope) and came down with an Agile Manifesto ↗.

Agility ↗ or nimbleness is an ability to change the body’s position quickly and requires the integration of isolated movement skills using a combination of balance, coordination, speed, reflexes, strength, and endurance.

Although agility clearly finds its roots in the physical world, all six required skills are all perfectly applicable to the realm of software development. Lacking one of them, undermines agility, the ability to change the body’s position quickly, the ability to change direction quickly, the ability to change rapidly and adequately under changing circumstances. Now where did this focus on responsiveness come from?

Since the dawn of (business) software development, man has tried to harness its complexity by means of different project management flavours. Emerging in parallel to the growing software development possibilities, they also grew from simple patterns to mastodons. While software development became more flexible - moving from punch-card-based mainframes to personal computers and all the goodies we have today - these project management methodologies kept adhering to their old principles. Even when projects became larger, they kept doing things just like before.

The result was simple: evolve or perish. Once a critical mass was reached, projects began to fail due to their rigidity, incapacitated to respond to changing circumstances.

Nothing New Under the Sun

In contrast to popular believe, the Agile Manifesto wasn’t the beginning of the Agile Age of Software Development, it was rather the summarising conclusion. Those 17 Agile Apostles were in fact 17 proponents of several agile methodologies that had emerged over the preceding decades. They didn’t invent anything, they merely wanted to come up with a common message, touch common ground, reach a consensus, show that there is a unified Agile Approach, a foundation for defending a better way to create software.

For example, already in the 1960s, at NASA’s Project Mercury, test-driven development was applied.

If you thought Scrum ↗ was conceived as an implementation following the Agile Manifesto, think again. Its origins date back to the early 1990s.

By the end of that same decade, Extreme Programming ↗ saw the light of day, taking several of already established agile best practices and scaling them to the extreme.

Many other agile methods had also already emerged by 2001 and representatives from all - DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming,… - were present to jointly agree on something substantive. They did bring the accumulated experience and knowledge from all these methodologies and methods together in a clear and simple manifest. Know your history ↗!

We hope that our work together as the Agile Alliance helps others in our profession to think about software development, methodologies, and organisations, in new – more agile – ways. If so, we’ve accomplished our goals.

Jim Highsmith, for the Agile Alliance

History since then has proven them right. They absolutely reached that goal. The manifesto has given a common name to the rising tides: Agile. And people have thought (and talked) about it exuberantly.

Good Intentions Pave the Road…

History also tends to repeat itself, the pendulum swings both ways, people become overly excited and try to make profit on anything, even if what they sell is no longer in line with its origins.

Just like the early project management methodologies started off with the best intentions, and ended up being a big mess no one wanted anything to do with anymore, the Agile Hype also saw the rise of several Agile Methodologies to replace the old ways with new and fresh ways to tackle the complexities of software development. Everybody now had a common enemy: Waterfall. Since you all know this, I’ll skip explaining it and continue down the paved road to Agile Hell.

There is no silver bullet in software development and project management. Sticking the name Agile on a methodology, does’t make it agile. Take for example the most ludicrous of them all: SAFe ↗. The only thing safe to say is that it can’t be more the opposite of the intensions of the Agile Manifesto.

One of the most profound aspects isn’t applied: “Business people and developers must work together daily throughout the project.” SAFe tries to avoid this with old Waterfall-like practices and tries to reinstate big design upfront with e.g. big room planning sessions, resulting in long-lived plans that simply fail for the same old reasons. Although probably well intended, reality has proven that it’s all but agile.

In general, by trying hard to solve all software development and project management problems under the Agile Umbrella, many recent Agile Evolutions have fallen in the pitfall of over-complication and have ushered the dead of Agile in a way. I’m not alone in the world shouting that agile is dead ↗.

It is of course a statement that requires thoughtful interpretation, because it is in fact a very dense version of “the way certain methodologies have introduced a plethora of ceremonies under the umbrella of agility have buried the actual intensions of the Agile Manifesto, and with it its merits”. So…

Agile Ceremonies

… should we not use them then? Absolutely do use them! Yet, make sure you understand them and don’t start using them unless you need them. Basically, you should apply Agile Principles also when selecting your Agile Methodology and applying its ceremonies.

The Agile Manifesto wants to nurture the nature of software development. And just like many other nurture attempts, it also needs to have its nudges to make people move in the right direction. It needs ceremonies: daily stand-ups, backlog grooming, pair programming, sprint kick-offs and retrospectives,… Because that’s what all those are: ceremonies, songs and dances that guide you in implementing the intentions of the Agile Nurture.

To benefit from all these and more, while avoiding the stepping stones into Agile Hell, it is crucial to be aware what they are: ceremonies that help you implement the intentions of the Agile Nurture. Fail to understand that very aspect, start to see them as a thing in their own right, implement them in a dogmatic way and you will end up feeling as miserable as … when you were tumbling down the Waterfall.

On the other hand, when you do understand the underlying agile nurturing intentions and can liberate yourself from the literalist and fundamentalist interpretation of the ceremony, you will be able to benefit from these Agile Methodologies and will by (now nurtured) nature avoid the pitfalls.

Essential Agile

I clearly remember reading the Agile Manifesto ↗ back in 2001. It was during the very first years of my professional life. Probably because I hadn’t first hand experienced the perils of Waterfall, I never understood the “rivalry” as such.

About a year before I got a copy of Extreme Programming Explained ↗ and after reading it we applied its intended nurturing as what to this date has been my Essential Agile methodology, based on the simple motto “Don’t Bite Off More Than You Can Chew”.

Phrased with well-known current terminology it boils down to:

• keep track of requested business features in a backlog, assign and update priorities to these features in cooperation with business
• during a short time-boxed period, take features from the backlog & implement them, release the implemented features at the end of that period
• apply a test-driven development approach, based on the features’ specifications created together with business
• ensure that with each commit (back then it was CVS, so it was even more important) the build doesn’t fail

I dare to say that over the course of roughly 25 years, this Essential Agile methodology hasn’t failed me once and about every (other) Agile Methodology that I encountered with clients essentially didn’t bring anything new to the table. I believe the reason for that is that it simply implements (the essence of) the Agile Manifesto ↗:

• Focus on individuals and interactions
• Always have working software
• Customer collaboration is key
• Be able to respond to change at all times

If you live by the nurtured intentions, it doesn’t matter which ceremony you follow. Even more, when a ceremony no longer matches these intentions, you will notice it. When such deviation occurs, the same Agile Principles can help you put it back on track.

Personal Agile

No two persons are alike, nor are two teams, nor are two organisations. A methodology (or ceremony) that works for one person, one team, one organisation doesn’t necessarily work for any other person, team or organisation.

If Agile Methodologies are ceremonies, nudges, then being Agile is nurture, the abstract version, the goal, the vision. I prefer that abstract version, the pure nurture, pretty much like I prefer software architecture - more on that to come.

I don’t care which Agile Methodology you prefer to be able to implement the nurture, as long as it serves the nurture and doesn’t become a thing of its own, or worse, goes against the intended nurture.

In my 25 years of professionally focussing on software development, as a software architect I’ve always taken the same approach: look at the existing capabilities of the organisation of my client, define the current intersection with common sense best practices, map that to the envisaged goal and define steps to fill in the gaps to reach that goal. To do this, both the architecture and the methodology has always, without any exception, been a custom-tailored one - cherry-picking from of the vast amount of software development and project management methodologies, to create a well-grounded foundation. To apply these I try to lead by example and coach until the existing organisation has settled in this new way of working.

“New” might be a great overstatement. It most of the time means taking a good look at how people actually currently work, formalise that while fitting it to the closest existing architecture/methodology and present this personalised way of working as a first step. Only that simple first step often brings teams much closer together, resulting in a big gain, without any cost whatsoever. Just be explicit about your ceremony. It’s the foundation of communication and interaction. The rest will always follow.

Agile Architecture

With my approach to a more Personal Agile, I touched upon the two aspects that often travel along the same road when I join a new client: software architecture and project methodology. That I prefer an Agile (Project) Methodology must be clear by now. So what about architecture? How does software architecture fit in an Agile Mindset?

The 17 apostles of the Agile Alliance are real code monkeys. And I mean this in the best possible intention of that phrasing. They have made it their life’s work to study, contemplate, design, document the art of coding. Take any of their books, and code literally flows off the pages. Beautiful code, pragmatic code, code patterns,… Each of them have given me much inspiration over the past +25 years.

If you’ve read some of my other things, you might have encountered - or you’ve heard me tell - this anecdote about interactive agents. The bottom-line of this tale is that not every business process justifies a software solution. So not every business process should go on a development team’s backlog. Before it gets on any backlog, it should pass another round of “think before you act”. In this case it’s the architecture phase, where things like capabilities, domains, processes,… live.

Just like the iterative breakdown of a problem in “bites you can chew”, the architecture phase is also just-another-iteration where the even larger business need is analysed, broken down in manageable pieces, distributed over domains and defined as a solution. And that solution might require a piece of software and thus can be put on the backlog of an autonomous development team that applies all the Agile Goodness to the development of that feature, just like the architects have done before them.

At the architecture level code and specific technologies are taboo. Those are the responsibility of the development teams. Architecture deals with the segregation of capabilities and processes, defining the interactions between them, specifying the protocols that operate between them and maybe assign a development effort to some of them, all to ensure the realization of the dream of the client, according to best practices and good principles.

And, of course, every good architect will interact and collaborate with business and the development teams to support the focus on working software, making changes to the architecture where needed, to be able to respond to change at all times. Different levels of iterations take feedback from each other and keep on turning.

Long Live Waterfall

Don’t go chasin’ waterfalls
Please stick to the rivers and the lakes that you’re used to
I know that you’re gonna have it your way or nothing at all
But I think you’re moving too fast

I promised you I had a point when I started this thing with the blunt, provocative “Agile is Dead, Long Live Waterfall!” statement. Earlier, I skipped explaining Waterfall, since you all know it. Now I’ll try to explain why I believe you are misguided into thinking “Waterfall is bad”. Mkay!

Waterfall was an easy metaphor that enabled polarisation and the creation of a common enemy. And for the sake of creating a new movement, which indeed holds a better approach to handling large software projects, I won’t even blame anyone for doing so. Yet, in our software world, I do believe in evolution instead of revolution and I do believe that without changing the intentions of the new approach, it could as well have been called Waterfall 2.0 or maybe even more fashionably Waterfalls.

I said it before, I never understood the “rivalry” between Agile and Waterfall. Maybe because I always interpreted Waterfall as “Think before you act and verify afterwards” or in more codish “Analyse, Code, Test”. In a way, is there any other way to do anything? So if Waterfall - according to me - is nothing more than the three evident steps to reach anything, why do I agree that somewhere it went so wrong that whatever came next (spoiler: Agile) wanted to oppose itself so hard against it?

If you’ve read this far, you already have read my answer: “While software development became more flexible - moving from punch-card-based mainframes to personal computers and all the goodies we have today - these project management methodologies kept adhering to their old principles. Even when projects became larger, they kept doing things just like before.”

Possibilities in realm of Information Technology progress at an exponentially growing pace: new programming languages, faster hardware, new middleware, new programming paradigms,… With those possibilities projects also got bigger. Although I like to blame feature-creep, technology-for-technology and so on, still there is no denying that thanks to these possibilities foremost projects got new opportunities to automate processes that could not be automated before. And it holds a self-propelling and self-augmenting effect because due to the high degree of automation nowadays, life has also accelerated. And we want more.

So projects got bigger, yet we kept doing thing the same way, just bigger. The analysis phase got bigger, the coding phase got bigger, the testing phase got bigger. We simply applied the most evident approach to growing projects: scale linearly. We simply forgot to evolve. Until things blew up. Size does matter people!

In stead of scaling up our manageable, small Waterfall to an unpredictable, large Waterfall, we should have simply scaled out our single Waterfall to more Waterfalls. Preferable even smaller ones. Scale out, in stead of scale up.

How we missed this will forever be beyond my comprehension. Because, what do you typically do when workload is increasing? You hire more workers. You don’t replace the worker you have simply by a bigger worker.

So, please stop bashing Waterfall, since in doing so you only show that you don’t get the essence of Agile and Waterfall. We simply scaled it wrongfully. By bashing it, you are missing the underlying nurture that teaches “Think before you act and verify afterwards”, which is what every ceremony out there implements. In fact, Agile and Waterfall in a way have nothing in common. The former tells us “Don’t Bite Off More Than You Can Chew” while the latter advises “Think before you act and verify afterwards”. The two are brothers in arms, in no way rivals.

Can’t we reinstate Waterfall? It has been mistreated enough by now. I understand it’s difficult to simply go back to calling it Waterfall, the name is burned. We also can’t simply call it Agile. It’s bigger than that. So what should we call this Hybrid then?

Agile Waterfall Hybrid ↗ anyone?

The Closer

If by now you think you should feel offended, … Okay, great, you’ve made it up to here. So you at least you did read it. Or were you looking for a TL;DR? 😇

Read it again! You clearly missed some point(s). Every word has been carefully crafted, to add something to the overall picture I’m painting here. Remember I argue against summaries? Maybe you skimmed over some part? Take my word as the author for it: there is not a single word, reference or idea in there that points any finger at anyone, nor your preferred ceremonies.

So with Marvin Minsky’s words in mind, “If you understand something in only one way, then you really don’t understand it at all.” Read it again and see where you might have misinterpreted something and come back to this point to conclude that you at least had a good time, ruminating on some of the things we all share in our (professional) lives: the (almost) physical laws of nature, nurture and nudges, all linked together just like the laws of thermodynamics and shared through 50 shades of ceremony.

#howcanihelp