Developing business-critical software is complex. Though promised outcomes may be transformative, the path there is filled with pitfalls: changing requirements, technical rework, budget overruns, missed deadlines, even outright cancellation after significant investment.
Industry data shows only 29% of IT projects succeed – the rest are challenged or fail completely. Of those unsuccessful, nearly 40% are due to inadequate project management and planning.
So getting your software methodology right matters greatly. Let‘s examine the two dominant yet opposing approaches:
Waterfall emphasizes rigor, governance, early locking down of requirements and upfront planning for predictability.
Agile focuses on rapid iterations, customer collaboration, embracing change and decentralizing decision making abilities to teams.
Philosophically, waterfall seeks to tame complexity via structures and controls. Agile accepts unpredictability as given and leverages it through feedback and adaptation.
As with most things, there are merits to both ways in the right circumstances. The key is recognizing their differences, strengths and shortcomings relative to your situation. Evaluate not just immediate development practicalities but also how each maps to your business objectives, timelines and risk appetite.
Let‘s explore across several key dimensions:
Handling Evolving Requirements
Software has two broad categories of requirements:
Functional capture capabilities, features and business logic. These define "what" the system should do.
Non-functional shape platform elements like its architecture, infrastructure, security and performance. These determine "how well" the system works.
Functional requirements especially tend to be slippery – changing frequently even mid-development as new market insights surface. Others can be pinned down upfront.
Agile is optimized for flux. Its iterative build-review cycles expect discoveries and changing needs as understanding grows. Priorities simply get rebalanced through the backlog. New work can get incorporated without derailing delivery cadences.
Waterfall however freezes requirements early, investing in detailed specifications based on initial knowledge. Change gets harder downstream as rework snowballs. Formal change requests must reopen planning,specs and testing post-signoff. This bureaucratic procedure frustrates business sponsors while blowing up budgets.
For a public workforce management system undergoing digitization, new labor regulations mandated late feature additions. The agile project took this in stride, merely adjusting scope and completion targets. In contrast, an immigration system overhaulsaw huge cost overruns and delays from requirements churn, requiring executive intervention.
Clearly for novel software initiatives with lots of unknowns, agile flexibility trumps rigid processes. Analyze which parts of your project warrant upfront definition versus exploratory discovery-based approaches. Strike the right balance.
Planning Horizons
Both methodologies plan intentionally but differ vastly in time horizons:
Waterfall maps out the entire project upfront in decreasing detail by sequential phase – from requirements to design, development, testing and finally deployment. Teams estimate effort, staffing needs and costs early on. Progress gets tracked against these milestones.
But this rests on illusion of predictability. For anything complex involving custom development, unforeseen issues abound. Change cascades, reopens previously "complete" tasks and creates knock-on delays.
Agile doesn‘t attempt to plan too far beyond what can actually be forecasted. It lays out a high level product vision and roadmap but plans in detail only near term sprints spanning 2-4 weeks. There are continuous checkpoints on working software rather than intermediate phase reviews. Scope gets added based on business priority and team bandwidth.
This "Just in time" planning lowers risk from unknowns, while ensuring continued alignment with sponsors. The key is not learning about defects late when too much has been built upon faulty assumptions. Fail fast, fail cheap.
Evaluate your culture‘s readiness for long term uncertainty versus need for detailed early project visibility, contingency funding and resource allocations typical in annual budgeting processes. Adopt planning horizons and tracking metrics suited to your risk preferences.
Team Structures
Developing software requires multiple skillsets – business analysis, system architecture, software engineering, user experience design, testing and more. Work must also transition from concept to implementation.
Methodologies take different approaches on how talent works together:
Waterfall relies on specialized separate phases executed by different groups in a relay race. Upstream analysts detail what end users need. This gets handed to architects and designers to specify technical solutions. Implementation and testing plans then get passed onto engineers before final deployment.
While straightforward in theory, handoff inefficiencies manifest through errors and miscommunications across phase boundaries. Discoveries mid-project also necessitate changes rippling back up already completed components. This bloats costs and timelines.
In contrast an agile team is a small cross-functional group that owns a product area end-to-end. One sprint they could be clarifying features with business partners, while next building and testing code. Team members dynamically balance strategy, design and delivery work each iteration.
This unity of purpose and whole-lifecycle ownership leads to greater shared understanding and less waste. According to Bizagi‘s study, self-organizing teams are 17% more productive on average.
Evaluate skills in your own organization and consider phasing agile adoption across workstreams where capabilities already exist versus nurturing T-shaped, collaborative agile teams longer term.
Tracking and Managing Risks
Given their differing attitudes on change, agile and waterfall surface and mitigate uncertainties differently:
Waterfall tries to identify then lock down requirements early limiting risk. But unforeseen issues accumulate, escaping detection under layer upon layer of specifications now expensive to rework. Late stage failures cascade severely.
Fixing defects post-development costs 5-10x more than handling in analysis or design. Integration surprises can further multiply time and cost overruns.
Agile expects uncertainties will emerge regularly. Its use of working software over intermediary documents ensures revelations happen earlier when less has been built on shaky foundations. There is greater surface area for inspection with builds every 2-4 weeks. Failures provide input to subsequent sprints rather than derailing end timeline targets.
Studies peg cost multipliers from late stage fixes in waterfall projects as high as 200x compared to agile! Though continuous uncertainty may be culturally discomforting, agile ultimately proves lower risk through greater transparency and maneuverability.
Assess appetite in your own context for upfront stability guarantees versus active embracing of ambiguity and course correction. Track software development risks accordingly.
Realizing Benefits
Given agile‘s multiple advantages on paper, its expansion within organizations should be natural. Yet according to McKinsey, 70% of attempts at agile transformations fall short of goals despite enthused kickoffs.
The impediments lie less with agile practices themselves but rather cultural inertia, conflicting legacy structures and inability to recognize flipping power dynamics they require:
Earlier we noted agile teams are cross-functional, autonomous and collaborative. But enabling this needs management relinquishing directives that worked previously with waterfall compartmentalization. They must instead grow coaching and servant leadership muscles.
Likewise, central planning units used to waterfall prescriptiveness end up bypassing confidence in team discretion and customer responsiveness. Finance struggles with funding through fast outcome-based iterations versus cost-center budgets.
For self-direction to work, executives must nurture agile talent, align supporting processes and upgrade their own mindsets around empowerment.
Therefore, freshly adopting agile requires deliberate change management and pilot testing before going full steam. Be patient. Let benefits accumulate steadily. Seed collaborative behaviors through the organization. With growing proof points, funding and staffing too can transform.
Key Takeaways
Waterfall and agile offer opposing models for orchestrating software innovation and technology change initiatives:
Apply waterfall for projects with clear fixed requirements, long timelines or little appetite for uncertainty given enterprise risk profile. Leverage its strength in predictability.
Start agile when dealing with ambiguity, needing faster user feedback or frequent priority shifts. Facilitate rapid inspection and adaptations.
Evolve carefully, assessing cultural readiness alongside talent and delivery processes. Change contexts before imposing practices. Customize hybrid approaches suitable for your environment.
I hope examining project management methodologies in business context helps pick what aligns best. Share your experiences or questions below!
