Scrum, Kanban, Lean, XP, or Waterfall: How to Choose Your Optimal Development Methodology?

Each project is unique in terms of time, costs, team, stakeholders, deliverables, and technologies. The software development methodology should be chosen in accordance with these specific features but not because some framework is popular.

I would like to help you decide which software development methodology to choose. Together with Keenethics Project Managers, I have compiled this ultimate project management guide. Here, you will find out how to choose the development methodology which would suit your needs, resources, and interests best.

---

So, here is the plan of what I am going to talk about:

1. Waterfall vs Agile
2. Scrum, Kanban, Lean, and XP as top software development methodologies based on Agile
3. Final remarks

For each software development methodology that I consider, I will outline:

1. Brief history overview
2. Methodology basics
3. Pros and cons
4. Typical usage
5. Example 

Waterfall vs Agile 

Waterfall and Agile are the two dominant but very different approaches to project management. Waterfall is an old-fashioned, traditional approach, and Agile is a more modern, flexible, popular approach.

Basically, the very first decision you have to make when choosing the project management methodology is whether you want to go with the old-but-gold rigid approach or the modern and flexible approach. Let’s have a deeper look at both before you answer this question for yourself. 

Waterfall

BRIEF HISTORY OVERVIEW

Waterfall is a linear software development approach, the first mention of which dates back to 1956 when Herbert D. Benington gave the speech at the Symposium on Advanced Programming Methods for Digital Computers. However, Benington did not use the name “Waterfall”, and nor did Winston W. Royce — the computer scientist who allegedly formalized the Waterfall approach for the first time in his 1970 article.

This is what the very first version of Waterfall looks like, according to Winston W. Royce. At least, this is how most people interpreted his article.

SOFTWARE DEVELOPMENT METHODOLOGY BASICS

In modern-day Waterfall, the software development process consists of 6 major stages:

• Collecting system requirements
• Analyzing requirements and creating models and schemas
• Designing software architecture
• Coding the software
• Testing and debugging software 
• Managing operations: installation, migration, maintenance, and support

Importantly, for a new stage to start, the previous one has to be fully completed. Otherwise, there is no chance to return to any of the finished levels or to run any two of them simultaneously. The exception to this rule is the Sashimi model — one of the approaches to Waterfall that allows for stages to overlap.

PROS AND CONS

In Waterfall, planning and designing processes are simpler and more straightforward since the customer and the development team reach an agreement on the project deliverables from the very beginning of the product development. The customer can stay away from development — there is no need for them to be constantly involved. The overall scope of work and all the expenses are known in advance. Respectively, the progress can be easily measured, and project costs can be predicted with good accuracy. Thus, the Fixed Price model suits Waterfall best.

 At the same time, it is very difficult, if not impossible, to collect and analyze all the project objectives and requirements and to formulate them in a manner that they will not be changed later throughout the project development process. However explicit the project requirements are, the final product may not be what customers expected unless they are involved at each stage of the project development, which is not the case with Waterfall.

TYPICAL USAGE

If you have a small project with clear deliverables that are not likely to change, you are limited in time, and your team consists of well-experienced developers resolved to work hard, Waterfall will fully meet your needs. However, if you are not sure about project requirements, redesign, redevelopment, and retesting will make your project costs rocket.

EXAMPLE

Agile

BRIEF HISTORY OVERVIEW

The history of Agile is not the usual one. It did not emerge as a philosophy from which different software development methodologies evolved. By contrast, a few like-minded PM methodologies appeared and united to advance the philosophy of Agile.

The first attempts to create a philosophy of lightweight and flexible software development date back to the 1960s, shortly after Waterfall emerged. Then, in 1995, Scrum was introduced. Extreme Programming (XP) and feature-driven development (FDD) appeared in 1996 and 1997 respectively. In 2001, the creators of Scrum, XP, FDD, RAD (rapid application development), UP (unified process), and DSDM (dynamic systems development method) came together and issued the Agile Manifesto.

In 2011, the Agile Alliance created the Guide to Agile Practices, which is now known as the Agile Glossary and Terminology. Nowadays, it is the ultimate source of knowledge about Agile philosophy.

SOFTWARE DEVELOPMENT METHODOLOGY BASICS

As the name itself suggests, Agile is a software development methodology that lets the team adjust to any changes in the working process quickly and easily. The software development environment may be rather turbulent while the ideas, resources, or plans may take an unexpected turn. So it is necessary to be flexible in order not to undermine or postpone project success. One of the twelve central principles of Agile is to harness change even in the late stages of the development process in order to ensure the customer’s competitive advantage.

Basically, any agile methodology consists of Waterfall-like iterations, which repeat until the product is satisfactory from the business logic, code quality, and user experience perspective. Also, it always leaves room for changes in plan, scope, resources, or deadlines. Iteration and flexibility are the two key competitive differences between Agile and Waterfall.

Besides that, Agile focuses on people and interactions rather than on tools and processes. A self-organized, self-disciplined, and cross-functional team, which can timely adjust a project development plan, is the core of this framework. Yet, it does not mean that there is no room for managers. Well… Maybe, there isn’t. Agile is based on leaders, not managers, who are able to create a productive working environment, to motivate and inspire their team, and to keep track of project progress without making their team members feel pressured or closely watched.

PROS AND CONS

In Agile, the customer is involved in each development process stage, so the requirements are timely adjusted, and the final product is fully satisfactory. If the project has strict time limits, Agile gives an opportunity to quickly develop and release the most basic version, which later on may be built upon. Also, having timely identified and fixed all the bugs, one may save a great deal of time and money.

Meanwhile, Agile will not work for the customers who do not want to be engaged throughout the process of project development but simply want to get the final product. Even though Agile does greatly optimize the process of making changes, additional time and cost expenses are unavoidable. Since the Agile methodology is iterative by its nature, frequent refactoring may be necessary. Otherwise, the quality is going to suffer badly.

Still, Agile is more effective that Waterfall — may Waterfall supporters forgive me for this statement. The software development process can be compared with a long drive from home — the idea — to your dream-city — the final product. Imagine that before leaving from home, you choose a route, plan all the stops, and claim that you will arrive at your destination at 8 PM. 

If you choose to follow the Waterfall approach, you cannot change the route or add any new stops. However, what if there are roadworks on your route? What if there is a traffic jam? Or what if you feel tired and need to take a break from driving but cannot do that because the next stop is planned in two hours? Even if you make it to your destination in time, you will be tired, angry, and frustrated. 

Meanwhile, if you choose to follow the Agile approach, you can adjust the route if you learn about roadworks or traffic jams. You can take a break when you want to and miss a stop if you have no need to stop. There are no guarantees either that you will make it to your dream city in time, but at least your trip will be more enjoyable and comfortable.

TYPICAL USAGE

Agile would work for basically any project. In particular, if you have a large team working on a complex project with no set-in-stone deadlines, Agile is the path to take.

Scrum, Kanban, Lean, and XP as Major Agile Methodologies

The list of four major Agile-based methodologies includes Scrum, Kanban, Lean, and XP. And how to choose software development methodology for your project? Let’s have a closer look at each of those.

Scrum    

BRIEF HISTORY OVERVIEW

As I mentioned earlier, the Scrum methodology was introduced in 1995 by Ken Schwaber and Jeff Sutherland. However, the term was coined in 1986 in the Harvard Business Review article “The New New Product Development Game”. 

In 2009, Schwaber and Sutherland released The Scrum Guide, which was later updated in 2016. However, to celebrate the twenty-fifth anniversary of Scrum, the authors introduced the Scrum Guide update on November 18th, 2020. And it is one of few good news pieces that the year 2020 has brought. 

A lot of people would say that Scrum is the best software development methodology — otherwise, it wouldn’t be so popular. I will not say “the best”, but this methodology is definitely a huge game-changer.

SOFTWARE DEVELOPMENT METHODOLOGY BASICS

Scrum is a software development framework that is frequently considered to be synonymous with Agile. Indeed, it is a project management model according to which developers plan their work, update their objectives, and review their progress. However, calling Agile and Scrum the same thing is a mistake.

The fundamental difference between Agile and Scrum rests in how the latter one uses certain roles (Developers, Product Owner, and Scrum Master), events (The Sprint, Sprint Planning, Daily Scrum, Sprint Review, and Sprint Retrospective), and artifacts (Product Backlog, Sprint Backlog, and Increment). 

In Scrum methodology, Developers are not necessarily software developers. Scrum can be used in other industries as well, so Developers are the team of people who execute Scrum objectives. The Product Owner is the person responsible for maintaining the Product Backlog and maximizing the product value achieved from the development process. Usually, it is a person on the project client’s side. Scrum Master is the person who makes sure that all Scrum events are held properly and all Scrum principles are implemented.

As for the artifacts, there are two backlogs. The Product Backlog is a list of tasks for future sprints, while the Sprint Backlog includes tasks assigned for the current Sprint. Scrum is an iterative managerial framework, and if the team faces an unplanned task, they need to wait until the end of the sprint to fix it. Each Backlog commits to a goal — a 2020 Scrum Guide update. That is, if all the tasks in the Product Backlog are completed, the Product Goal is achieved. If all the tasks in Sprint Backlog are finished, Sprint Goal is achieved. The third Scrum artifact is Increment — “a concrete stepping stone toward the Product Goal”. The increment is a task to be completed — a feature to be implemented or a problem to be solved. This task can be considered as completed only once it meets the Definition of Done.

The product development timeframe is called Sprint, and it starts with Sprint planning. The development team has to answer the main question: as a user of this system, which functions / services / experiences / values do I want to obtain? During Sprint planning, team members collectively decide on the Sprint Goal and Product Backlog items to pursue. Every working day, the team holds Daily Scrum meetings where they discuss achievements of the previous day and goals for the following one. At the end of each Sprint, team members conduct a Sprint Review — where they present their results to the product owner and receive feedback — and a Sprint Retrospective — during which they discuss the latest Sprint and offer improvements for the future one.

PROS AND CONS

The greatest advantage of Scrum rests in the fact that it offers clear rules of the game. Scrum is not a hard one to learn, and once the team is on board with it, everybody knows what to expect from others and what is expected from them. The Daily Standup format is as clear as day, and Sprint Retrospective questions are simple and understandable.

However, Scrum is not a methodology that works at all times. Ken Schwaber and Jeff Sutherland acknowledge that, for Scrum to be effective, the rules of the game have to be closely followed. The Scrum Guide starts with the words, 

TYPICAL USAGE

Scrum can be used for the projects that involve a team of 5 to 10 people — large enough to be productive but small enough to be easily managed. It is a suitable methodology for projects where the client is actively involved in the development process.

EXAMPLE

We have used Scrum on our Banker Advisor project. For our client, it is a passion project, so he wanted to be actively involved. The project goal was to create a valuable product rather than a commercial product, so the project requirements changed constantly.

Kanban

BRIEF HISTORY OVERVIEW

The term “Kanban” was coined in the 17th century in Japan meaning “a shop sign”. In Japanese, “kan” stands for “sign” and “ban” stands for “board”. As a project management methodology, Kanban appeared in 1956 when Taiichi Ōno —  the father of the Toyota Production System — started using paper cards for tracking demand. This system used swimlanes and WIP limits to illustrate the product cycle from manufacturing and distribution.

SOFTWARE DEVELOPMENT METHODOLOGY BASICS

Kanban is a software development framework based on Lean and Agile. It is a methodology for the teams able and willing to collaborate and to manage themselves. Kanban enables the team to identify and eliminate potential bottlenecks even in the early stages of the development process. The essential difference between Kanban and Scrum rests in the fact that, in Kanban, the work flows continuously through the system, and in Scrum, it is organized into distinct timeboxes. It makes Kanban more suitable for projects where the requirements may be suddenly changed or added.

The major Kanban concepts are Kanban Board, Kanban Cards, and Kanban Swimlanes. As for the board, it can be both physical and digital, and it is used to visually represent the development process. The cards visualize particular tasks and goals that need to be done and communicate the progress to the team. The swimlanes are the categories (visual columns) that the cards are classified into. Usually though not necessarily, these columns are titled as “To Do”, “Doing”, and “Done”.

The key idea of Kanban is to make sure that the needs and interests of the customer are met, to reduce waste to a minimum, and to prevent unnecessary features from being developed. Kanban nurtures nine essential values:

1. Transparency
2. Balance
3. Collaboration
4. Customer Focus
5. Flow
6. Leadership
7. Understanding
8. Agreement
9. Respect

Now, Kanban dictates six major principles:

1. Start with what you do now
2. Encourage acts of leadership at every level
3. Agree to pursue improvement through evolutionary change
4. Focus on your customers’ needs and expectations
5. Manage work by letting people self-organize
6. Evolve PM policies to improve business outcomes

Finally, Kanban offers six practices helping you implement these principles and values:

1. Visualize workflow with Kanban board 
2. Limit work in progress — set WIP limits for each column to avoid bottlenecks
3. Manage flow, identify and address bottlenecks and blockers
4. Write explicit, simple, well-defined, and flexible policies
5. Use feedback loops: quarterly strategy review, monthly operations review, monthly risk review, bi-weekly service delivery review, daily Kanban meeting, weekly replenishment, and delivery planning meeting.
6. Improve collaboratively and continuously

PROS AND CONS

Development teams like Kanban because of the flexibility and freedom it offers. Unlike in Scrum, in Kanban, the team does not commit to implementing a specific scope in a certain timeframe. However, such a lack of time planning and commitment requires the team and the client to have a very high level of trust.

The second possible drawback of Kanban rests in the fact that the team can make the Kanban board overly complicated. The idea of Kanban is to make the project development flow streamlined and straightforward. Yet, if the team decides to visualize each and every stage and substage of the project development process, the board will be cluttered and hard to manage.

Finally, if the Kanban team does not implement and follow work-in-progress limits, the whole idea of Kanban becomes pointless and loses its value. So, Kanban should be implemented carefully and entirely.

TYPICAL USAGE

Kanban should be used by project teams who would like to / need to:

• Release anytime
• Change priorities on the go
• Avoid estimates
• Avoid iterations
• Visualize the flow

EXAMPLE

Lean

BRIEF HISTORY OVERVIEW

The history of Lean is intertwined with the history of Kanban. In fact, Kanban emerged on the basis of the Lean philosophy, which was developed by a Japanese motor corporation Toyota in the middle of the 20th century. Toyota’s manufacturing pipeline was very long, so there was a lot of waste. Lean was developed to help the company reduce this waste.

SOFTWARE DEVELOPMENT METHODOLOGY BASICS

Optimizing efficiency and minimizing waste are the two pillars of lean software engineering. To say that differently, the Lean methodology is aimed to provide a software product of the highest quality while using the least possible amount of resources. 

The methodology recognizes three types of waste, also known as 3M’s: 

1. muda — all the activities that do not add value to the product, such as overproducing, over-processing, fixing product defects, waiting, multitasking
2. mura — non-uniform actions and irregularities. For instance, if a UX designer spends too much time creating the product design, developers may have to code in a rush and the QA process will be neglected because of a lack of time. 
3. muri — impossible tasks, emotional pressure, and overburden. For example, to optimize the efficiency of a team, it is necessary that they do not feel stressed and perfectly understand what they need to do and what tools they should use. 

Unlike other methodologies, Lean does not impose any specific processes or artifacts but dictates a set of principles that a business should abide by:

1. Define Value — understand which customer’s problem you will solve 
2. Map the Value Stream — define how exactly you will solve the customer’s problem
3. Create Flow — turn the value stream into a streamlined pipeline
4. Establish Pull — limit work in progress and enable continuous and timely delivery
5. Pursue Perfection — encourage and facilitate continuous process improvement

To ensure that these principles are met as fast as possible, the team may start with MVP development, which is a rough but important draft of the final product.

PROS AND CONS

Lean is very different from Scrum or Kanban in a way that it is descriptive rather than prescriptive. Scrum and Kanban are easier to implement and follow because there are clear rules and roles.  

TYPICAL USAGE

Lean is mostly used by teams that aim not to push something to the market fast but rather to develop a perfect idea and test it thoroughly with the market. In fact, Lean can become a perfect methodology for an MVP development team.

XP (Extreme Programming)

BRIEF HISTORY OVERVIEW

Extreme Programming as a PM methodology has been invented and shaped by a team of engineers on the basis of Chrysler Comprehensive Compensation System (C3). The first and most famous person in this team is Kent Beck — the lead software developer at C3 and the father of XP. He introduced XP in 1996. Ron Jeffries, Ward Cunningham, Don Wells, and Martin Fowler consulted Beck and championed his idea. 

SOFTWARE DEVELOPMENT METHODOLOGY BASICS

The goal of XP is to improve the quality of code and the quality of life for the project developers. Extreme Programming has the potential to apply the developers’ knowledge and skills to the fullest. XP is often referred to as “a ring of poisonous snakes” or “a house of cards” because it comprises twelve major practices and none of them should be omitted.  Similarly to Kanban and Lean, XP aims to reduce waste to a minimum and to focus on the objectives for today rather than for the future.

 The five essential XP values include:

1. Communication — in order to share knowledge with the team
2. Courage — in order to take decisive actions that will bring the team closer to the expected result
3. Respect — in order to work together successfully
4. Simplicity — in order to do only absolutely necessary things
5. Feedback — in order to identify areas for improvement

Also, there are three roles in Extreme Programming:

1. The Customer
2. The Developer
3. The Tracker (team lead or Project Manager)
4. The Coach (external consultant) 

The twelve major practices of Extreme Programming are:

1. The Planning Game — The team conducts release planning and iteration planning.
2. Small Releases — The team quite frequently releases well-tested software features to their end-users.
3. Metaphor — The team shares a common vision of how the program works.
4. Simple Design — Design is performed not before development but together with development, and design discussions involve the whole team.
5. Testing — Each feature is covered with at least one or more automated acceptance tests.
6. Refactoring — Software developers continuously work on the removal of code duplication and increasing code cohesion.
7. Pair Programming — The software is developed by two programmers using the same computer. 
8. Collective Ownership — Any pair of developers can improve any piece of code anytime.
9. Continuous Integration — The team keeps the system integrated at all times.
10. 40-hour week — The team should work no more than 40 hours per week.
11. On-site Customer (Whole Team) — The Customer is working hand-in-hand with developers for the team not to leave business needs out of attention.
12. Coding Standard — The team creates and follows a uniform code standard for all the code to look like it was written by a single, very skilled developer.

---

PROS AND CONS

Extreme Programming is the most prescriptive PM methodology among the ones mentioned in this article. It can be both advantage and disadvantage — depending on the project specifics.

TYPICAL USAGE

Extreme Programming is the most suitable for the projects with dynamically changing requirements, small, co-located development team, and the technology which allows for automated testing.

Final Remarks

Agile methodologies are not the exclusive ones that a software development team may use. Although more and more teams decide to give up on the Waterfall methodology because of its simplicity and stiffness, there are cases when this is the framework that would suit your project the most. If NASA still successfully uses Waterfall, why wouldn’t you even consider it? 

So, what is the best software development methodology? There is no “best methodology for software development”. There is a methodology that suits your project best.

If you have considered Waterfall, and you do find the suggested project flow useful, but you would want more flexibility — consider doing Scrum or Kanban. Both methodologies make use of the same project development pipeline, but they give you more freedom in terms of requirements and deadlines. At the same time, these methodologies give you tools that would help you not to turn freedom into chaos.

Once Scrum and Kanban seem to be too prescriptive for you, try doing Lean. If, by contrast, Scrum and Kanban are not prescriptive enough, go with Extreme Programming. Both Lean and XP are not as common as their counterparts, but they surely have a lot to offer.

The most important point is “do not be afraid to try”. If you have tried Kanban, and it does not work because the team ignores WIP limits despite all your efforts to enforce them, suggest moving to Scrum. Also, listen to your team members. Even though you are the project manager, developers and QAs can have a lot of great ideas as to how to improve the process. 

Final takeaway: Do not lock yourself inside a certain PM methodology. A methodology is not a castle for you and your team to hide in. It is a large and beautiful ship that helps you get closer with each passing day to your final destination.