Agile Software Development - Avoiding the Waterfall Trap
Why does it matter?
Why Agile Often Devolves into Waterfall
For the last quarter century, Agile development has been hailed as a revolutionary methodology for making software development more flexible, collaborative, and responsive to change. Traditionally, teams relied on the Waterfall approach—mirroring other engineering disciplines where blueprints and permits precede construction. This desired predictability appeals to stakeholders. However, given the complexity of software systems, such predictability rarely materializes in practice. Unlike other engineering disciplines where upfront planning enables consistent production of physical items (from bridges to automobiles to electronics), software behaves more like an organic system, benefiting from continuous monitoring and adaptation—much more like gardening than might be expected in a field labeled as engineering.
The need to address Waterfall's shortcomings led to the Agile Manifesto. Published in 2001, the manifesto outlined core values and principles that emphasized adaptability, collaboration, and swift value delivery. Yet despite these ideals, many organizations find their Agile practices gradually transforming into rigid, bureaucratic processes reminiscent of the very Waterfall methodology they sought to escape. This raises a crucial question: why does Agile so often devolve into Waterfall? To answer this, we must first understand how Waterfall works and compares to Agile. From there, we can explore how to maintain an Agile footing and realize its full potential.
What is the Waterfall Methodology?
The Waterfall methodology is a traditional approach to software development that follows a linear, sequential process. It consists of distinct phases—requirements gathering, design, implementation, testing, deployment, and maintenance—with each phase requiring completion before the next can begin. This approach demands extensive upfront planning and detailed documentation, leaving minimal flexibility for changes once development starts. While Waterfall offers apparent clarity and predictability, it struggles to handle evolving requirements or unexpected challenges, making it poorly suited for dynamic environments.
The methodology's inflexibility comes from its "plan-driven" nature, measuring success through adherence to predefined timelines and deliverables. This structure typically results in extended development cycles and delayed feedback since working software isn't delivered until the project's end. These characteristics position Waterfall as the opposite of Agile, which emphasizes flexibility and iterative progress—traits that better align with creating software that meets customer needs.
Early in my career, I encountered the Waterfall approach in my first job at L-3 Communications. Not only was the Agile Manifesto barely only a year old and not widely disseminated, but the work we were doing was highly controlled as a Department of Defense contractor working on projects that involved secure cryptography and needed to be launched into space and run untouched for decades. This required careful requirements analysis and planning, but even then it was clear that Waterfall processes could do more harm than good as the software was being developed.
How does Agile Differ?
Agile emerged as a response to Waterfall projects that consistently missed deadlines and failed to satisfy customers. Waterfall's linear progression—from requirements gathering through system design to implementation and release—couldn't provide the flexibility needed as software became increasingly prevalent. This lead to a more obvious dichotomy between what the customer thought they wanted as opposed to what they truly wanted or needed in a finished product. As hardware became commoditized and software distribution grew more dynamic through the Internet, the ability to adapt plans for moving targets became crucial. The shift from mainframe computing to portable physical media (e.g., floppy disks and CDs) and later to cloud services and app stores enabled teams to deliver software incrementally through continuous updates. This is when the “softness” of software to truly shine as development teams found the ability to mold the solution to the customers’ needs.
The Agile Manifesto emphasized rapid delivery of working functionality and frequent iteration to achieve more predictable and desirable outcomes. While this flexibility is appealing in theory, implementing it effectively requires both diligence and intention—it demands a comfort with ambiguity and uncertainty that many find challenging. Teams must be able to see enough of the future to understand their general trajectory without getting caught up in the specifics of the final outcome.
My first experience with an Agile approach didn’t happen until a decade after the Manifesto was released. Working for a small team on a direct-to-consumer product, it was a revelation to be able to chunk up the work into consumable parts and deliver intermediary functionality. Even with a greenfield project, we focused on regular test and internal delivery cycles to ensure the effort was worth the investment and the original overall goal of the project was met with regular feedback loops to ensure we stayed on course.
The Spirit vs. The Letter of Agile
At the heart of this issue lies a fundamental misunderstanding of the Agile Manifesto. The manifesto emphasizes individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. These values are intentionally broad, giving teams flexibility to adapt them to their unique context. However, many organizations implement Agile practices superficially—following the "letter of the law"—without embracing the underlying philosophy. This leads to an overemphasis on rigid ceremonies, metrics, and processes that ironically mirror Waterfall development's inflexibility. This rigidity ultimately undermines Agile's intended goals. While the resulting linear process gives management and stakeholders a way to monitor progress, organizations often overlook the crucial tradeoff between flexibility and observability—and worse, some mistakenly believe they can maximize both. By trying to apply the facade of something that sounds “cool” like Agile while still holding onto the underlying tenets of Waterfall, teams end up with the worst parts and limitations of both, rather than the opposite as intended.
The Rise of "Agile Theater"
The desire to appear agile while maintaining predictable long-term plans often leads to "Agile Theater"—where teams merely go through the motions of agile practices without embracing their principles. Consider these examples:
Strict Ceremonies: Daily stand-ups, sprint planning, and retrospectives devolve into mandatory rituals with little purpose. Instead of fostering collaboration and transparency, these ceremonies become checkbox exercises. Stand-ups turn into rote status reports rather than meaningful team interactions. Retrospectives, meant to generate constructive feedback, either descend into complaint sessions or focus on superficial "vanity metrics" that serve only to document their own existence.
Story Pointing as a Performance Metric: The concept of “story points” originated as a concept of “ideal days” from the concept of “Extreme Programming”, and didn’t even come from the more common Scrum form of Agile. As the intent was never to get to the granularity of less than a day, it lead to confusion and tension with management. This lead to teams misusing story points to measure productivity instead of estimating effort and complexity. As a result, teams would inevitably end up gaming the system and pressuring developers to inflate estimates. When story points get converted to time estimates, they lack the precision needed for anything beyond rough approximations.
The Waterfall Mindset in Disguise
These practices often lead to the re-emergence of a waterfall mindset in an agile disguise. For example:
Fixed Scopes and Timelines: Organizations often insist on fixed scopes and deadlines, treating sprints as miniature waterfalls where changes are discouraged until the next planning cycle. Stakeholders attempt to divide predetermined roadmaps into two-week sprint increments—essentially just splitting up available time. This contradicts the core principle that teams should be able to adapt. While maintaining focus within a sprint helps avoid context switching, truly agile teams must be able to adjust at sprint boundaries, if not more frequently. This flexibility may disrupt longer-term roadmaps, but it's essential. Particularly in early-stage projects and companies, excessive advanced planning—a hallmark of Waterfall—limits the ability to respond to business needs.
These fixed timelines are especially prevalent in event-driven deliverables, such as working for a sports league such as Major League Baseball. When timelines can’t slip, as games will be played whether or not the MLB App is ready, it is important to focus on figuring out how to adjust scope. Additionally, those timelines are generally known well in advance, often over a year in the future, so while it is inherently Waterfall-esque to try to hit those future deadlines from far out, sticking to an Agile approach where adjustments are made often to ensure calibration with those timelines end up resulting in the best outcome for both the development team and the users.
Command-and-Control Management: Instead of empowering teams to self-organize, some managers use agile artifacts like burndown charts or velocity metrics to enforce top-down control. Teams can easily game metrics such as story points completed, pull requests merged, or lines of code contributed to meet arbitrary goals—satisfying management without delivering real value.
Water-Agile-Fall: While engineering teams experienced with Agile development may follow agile processes successfully, other product development functions like design and QA often fall into waterfall patterns. Design and requirements aren't delivered until fully complete, while testing begins only after all development is finished, creating burdensome testing cycles before production release.
I have seen this problem across companies both small (Diamond Kinetics) and large (Major League Baseball), where those other functions are treated as services that happen out of band with the software development process. This makes coordinated delivery difficult, as the communication across those boundaries are rarely in sync and therefore it becomes more linear like Waterfall than the iterative that Agile prescribes. If engineers aren’t able to review early designs and help shape and scope the effort, then they feel the need to implement the functionality to completion before handing it off to testing. This results in mini-waterfalls that break with the spirit of the manifesto and result in falling back to a Waterfall process under the hood.
A hidden watefall
How to Avoid the Trap
In order to successfully implement Agile processes, the team must be committed to the concepts and strive to “keep the faith” in the intent of the Manifesto, focusing on people over processes and early and often delivery. To prevent Agile from devolving into Waterfall, teams and organizations must actively resist the temptation to prioritize form over function. Here are some practical steps:
Revisit the Agile Manifesto Regularly: Make the manifesto a living document within your organization. Discuss its values and principles in team meetings to ensure everyone understands the "why" behind Agile practices. Use this understanding to tailor ceremonies and processes to your team's needs. If a stand-up feels unnecessary, skip it. If retrospectives aren't yielding actionable insights, try a different format. This approach enables following the spirit of the law without getting tied up in the letter of the law.
Focus on Outcomes, Not Outputs: Shift the emphasis from metrics like velocity and story points to the value delivered to customers. Encourage teams to measure success based on impact, not activity. A regular release schedule can help maintain focus on outcomes. Trust your teams to make decisions and adapt processes as needed. Avoid micromanagement and foster a culture of autonomy that prioritizes results.
Especially when working at a smaller and earlier stage startup, such as Diamond Kinetics, it was important to focus on what could be delivered with confidence. A “release train” where releases were possible on any given week provided the necessary guardrails to plan work against while accounting for the necessary steps of testing and submission for app review. Leveraging this approach effectively tripled the number of successful releases from one year to the next.
Additionally, this involves including all product development functions, from product to design to testing, in the outcome. Doing so provides better accuracy around true task estimation. When two or more people are involved across multiple disciplines, it is easier to understand why it is impossible to estimate durations at finer granularities of days if not weeks. While the programming effort for a small feature may take only a few hours, including the effort to define, design, implement, and test that functionality, and the necessary handoffs between disciplines results in any effort worth doing taking the better part of a working week in all but the most trivial examples.
Avoid Sprinting the Marathon: Most Agile teams settle on two-week sprints, yet meaningful software projects span far longer. They're more like marathons than sprints—and no one sprints a marathon. Cramming each two-week sprint with maximum work leads to team burnout. Build in flexibility by allowing sprints that tackle technical debt or enable exploration and learning. Teams need these occasional deviations from fixed schedules to remain adaptive and responsive to business needs.
Plan Roadmaps against Sprints: Rather than splitting a long-term roadmap into sprints, adopt a Now/Next/Future roadmap template with a sliding window:
Now - What is being worked on in the current sprint and possibly the next one
Next - What's up for the next month or two
Future - What's on the horizon for the rest of the quarter
This approach, with constant review during sprint planning, provides a bottom-up view of what the team can produce. While accurate prediction may be challenging initially, refining this approach will serve the team well through its growth. By focusing on what the team can produce and how that evolves over time, you'll achieve the best possible outcome.
Ensure Feedback Loops Exist: Multiple feedback loops are essential for maintaining agility and effective iteration. Internally, regular and productive retrospectives enable teams to provide candid feedback to leadership and stakeholders about their work environment and needs. Externally, user experience studies and A/B testing generate insights that help teams better serve their customers. Proper system observability and monitoring round out these feedback loops by ensuring teams can maintain and improve product health.
Is Agile Always the Answer?
While Agile is widely accepted as the best practice for modern software development, we shouldn't automatically assume it's always the right choice. As with our earlier discussion about the "letter of the law," teams should carefully consider whether an Agile approach truly fits their project. Some circumstances—like developing low-level firmware or creating software with strict security requirements (such as military or space applications)—may actually benefit from a Waterfall approach. These scenarios often demand extensive upfront planning to ensure all requirements are met, and their deliverables are less likely to change over time.
Nevertheless, incorporating some degree of agility can still prove valuable. Given today's complex software systems, it's best to adapt your process to suit both your team and project. Rather than viewing it as a strict choice between Agile and Waterfall, consider it a spectrum that skilled teams and leaders can navigate to their advantage.
Conclusion
Agile is a mindset, not a checklist. Specifically, that mindset is about flexibility in the name of delivery. When organizations lose sight of this fundamental truth, they risk falling into the same traps of rigidity and inefficiency that plague Waterfall development. By staying true to the Agile Manifesto's principles and prioritizing outcomes over processes, teams can avoid the pitfalls of "Agile Theater" and unlock agile development's full potential. This approach not only leads to better software but also creates an environment where creativity, collaboration, and adaptability thrive. While teams can adopt Waterfall practices when truly necessary, this should be done cautiously and selectively. Ultimately, leadership must determine the most effective approach based on their experience and context.
Well said!