The Agile Manifesto
Since that famed group of programmers crafted the Agile manifesto back in 2001, the methodologies that sprung from it gained rapid and widespread popularity. It was a backlash against the micromanagement of every line of code produced. Project size used to be based on how many lines it took to build the software!
No one could argue it was time for a change. The values from the manifesto were revolutionary (though likely evolved in part due to RAD approaches):
Individuals and interactions over Processes and tools
Working software over Comprehensive documentation
Customer collaboration over Contract negotiation
Responding to change over Following a plan
We now have years of project data to draw on along with feedback from managers and companies, and unfortunately what we see across the board is a high failure rate and sub-par results from projects that use Agile methodologies.
Agile Principles in Practice
When someone says they develop using Agile, what they are really referring to is one of the management methods based on its principles, such as Scrum. They all use some form of an Iterative approach, meaning that development happens during repetitive cycles where work from the previous cycle is built upon and expanded.
So what happened? Let's examine each of the Agile principles and what occurs when they are used in reality
1. Customer satisfaction by rapid delivery of useful software
Customer satisfaction is not derived from software simply working or even being put in their hands quickly. Satisfaction stems from their problem being solved successfully: a solution that makes their life easier.
Typically, a Project Manager would be gathering voice of the customer through various means early in the project to fully understand the business case, but many Agile methodologies ditch a PM in favor of other communicator roles, like the Product Owner, and development begins with a loose set of goals.
The Product Owner is focusing on the Stakeholders at each iteration's review period. That will almost never include the actual end user, because imagine how long it would take to coordinate a focus group at the end of every iteration!
Change is therefore driven from within the organization to a large extent, which is probably why they had to come up with #2.
2. Welcome changing requirements, even late in development
Probably because the customer finally had a chance to look at it...
But the fact is the more code that exists, the more complicated changes become. Changing one feature in the software will have a domino effect on many other aspects. Anyone involved in software development or testing will recognize that late changes and re-factoring of code takes an enormous amount of time at the end as compared to changes made early on in development. The number of bugs found will multiply with the complexity of the software.
I know why this principle exists- no one wants to see a major flaw in the software right at the end and not be able to fix it because typical project constraints are too tight. Unfortunately, the other extreme means you are developing without a clear plan and will theoretically never finish your project well enough to actually use it.
And this happens far too often. Agile projects often fail because they can't reach their objectives before resources give out. Iterations towards the end of development produce less new work because more time is spent debugging massive amounts of code as a result of new changes.
3. Working software is delivered frequently (weeks rather than months)
Early on, you feel like great progress is being made because you have something tangible right away. It is a lot easier to manage Stakeholders if you have something new to show them in each conversation, as opposed to just giving them an estimate on how much programming has been completed during a typical project execution phase.
Because you can actually see the work, there is more clarity around what hasn't been done yet.
However, the same pitfall occurs here as did with #1. Focusing simply on working software, and not whether it is meeting the business case is an easy trap to fall into.
4. Close, daily cooperation between business people and developers
This is the utopia we always dreamed of... but it never happens. Most business people do not have time daily to devote to the project. Even more incomprehensible is that most of them don't want to be involved daily. This is one of those bullet points that has CYA written all over it.
Ambiguous Business Person: "Why didn't this online form have these 23 if/then scenarios built in?"
Smug Programmer: "You were invited to EVERY daily meeting. This is on you."
At the other end of this spectrum is the micro-manager that sits next to the developer watching every change and giving feedback. Occasionally this works well, but if not managed properly the behavior can be detrimental to overall productivity.
5. Projects are built around motivated individuals, who should be trusted
Excellent in theory, harder in practice. Sometimes you just have to work with who you have, and sometimes that means you end up with a stinker on the team, which will effect morale, which will effect productivity, and on it goes... Don't underestimate the need for a strong project leader.
6. Face-to-face conversation is the best form of communication (co-location)
True. 55% of communication is non-verbal! Rarely are teams located in one place anymore, though. With many companies opting to partially outsource, you are trying to coordinate teams across time zones, potentially cultures, and the actual distance. Unless you are using video conferencing at every meeting, a lot is getting lost.
By not keeping track of the expectations being set at each meeting, there will be miscommunication among the team members. Ever have one of those conversations? "Wait, I thought you were doing that task? We just talked about it yesterday." "No, I said I couldn't do it without full read/write access to the database so you said you would."
And because daily meetings are surprisingly hard to coordinate when the team isn't in the same location, that sort of issue may take a lot longer than a day to uncover.
7. Working software is the principal measure of progress
When it works and serves to meet a need, this can be an excellent approach.
8. Sustainable development, able to maintain a constant pace
As discussed in #2, a constant pace is not possible as the code base grows and more bugs are introduced with new changes. We know that adding people late to a project can actually cause further delays ("The Mythical Man-Month"). You could offset some of this with really good planning up front, but these methodologies shy away from detailed planning unless it is specific to the iteration at hand.
9. Continuous attention to technical excellence and good design
This is an aspiration that should be present in every project no matter the methodology. It does require a veteran programmer to conduct peer code reviews and UX specialists to weigh in as development happens. A/B testing with focus groups can refine design even further.
Without a strong project leader too coordinate these activities, they will often fall by the wayside. Quality control processes in more traditional project management exist for a reason.
10. Simplicity—the art of maximizing the amount of work not done—is essential
Now we herald Lean development practices. Focusing on the simplest solution to the problems outlined in the business case is no longer an option but a necessity in order to be successful. The problem here is knowing what doesn't need to be done rests heavily on knowing what does have to be done. Planning, people.
11. Self-organizing teams
Three words: college team projects. It doesn't work any better in the real world either.
12. Regular adaptation to changing circumstances
The larger the organization, the slower it will be to adapt to change. Though largely unintentional, the resistance to change becomes an expectation among employees and gets incorporated into the culture. Smaller companies have the advantage here, but in small teams having just one member be resistant to change can cause the same issues and breed negativity.
Agile Became The Ultimate Excuse for poor work
Aside from the challenges various methodologies have had in successfully applying Agile principles, perhaps one of the biggest reasons that Agile is beginning to lose its shine has to do with how loosely it is used to justify bad development behaviors. I have literally heard every one of these statements and so have many other business owners. And the end result of all of them is disorganization and poor customer satisfaction.
"We don't document our software or version releases because we are Agile."
"We don't do impact studies. We will just go in and change it again if it causes an issue."
"We don't do scheduled roll-outs, because we develop on the fly."
"The customers will use what we tell them to."
"We don't notify customers of upcoming changes. We don't know when we will finish them."
And many, many other statements that make me cringe. Thankfully hybrid methodologies are gaining popularity, striking a good balance between too much management and too little.