From Scrum to Lean

While Scrum’s primary goal is organization and project management, Lean is more about optimizing processes in order to quickly produce quality products. It can be your first step toward adopting Agile principles, or it may be something that your team evolves to, when Scrum isn’t enough. I invite you to read my team’s story, and how we evolved from Scrum to a more Lean-ish development process.

A Little History

Lean is a set of principles defined by the Japanese automobile manufacturing industry in the 1980s. The Toyota quality engineer, John Krafcik, coined the term, while observing the processes and tools used to eliminate waste in mass automobile production. It wasn’t until 2003 that Mary and Tom Poppendieck introduced Lean as a software development process in their book, Lean Software Development: An Agile Toolkit.

Whereas Scrum is a set of rules and roles, Lean is a set of principles and concepts with a handful of tools. Both are considered Agile techniques, and they share the same ideology of delivering fast, while reducing defects and errors. I always emphasize Agile’s adaptability, but can’t ignore the fact that Scrum presents itself as a mandatory set of rules. In fact, Scrum’s religious fans would shout blasphemy for not following Scrum’s rules to the letter.

Lean, on the other hand, is more open; its followers present the process as a set of highly adaptable recommendations.

It encourages the team or company to make decisions, and it adapts to the decisions and every-day surprises that your team and company face.

As my team matured and exploited Scrum, we felt that some aspects of it held us back. We became a fairly disciplined and homogenous team. Some meetings were not appropriate for us anymore, and we started to realize that daily meetings were not efficient. We learned that problems should be solved faster, and we felt the need to avoid these procedures that held us back.

We moved on.

The Two Major Concepts of Lean

Lean exposes two major concepts, but the core ideas are: eliminating waste and improving the work flow.

Eliminating Waste

If something’s going to break, it will break on Friday.

Anything that stands in the way of production is waste. This includes lost time, leftover materials, and an unused work force. Defects in the final product are waste; it wastes time to fix them, wastes money to replace them, and wastes resources to find other solutions.

The concept of waste nicely translates to the world of software development. Waste can be described by late deliveries, bugs, and programmers having nothing to do (do not confuse this with “programmers should program eight hours a day without pause and youtube”).

Improving Flow

Toyota’s Lean concept concentrates on the production flow. In a production plant, the flow is a chain of procedures that transform the raw materials into their final products. These procedures can be very different from one another and take different amounts time to complete, but they can each be improved to make them more efficient. It’s a constant battle finding and improving bottlenecks in the process.

An ideal flow is one in which each step takes the same amount of time and effort to produce the same amount of products.

This does not mean that each process should cost the same amount of money, but each process should be able to complete with the same amount of ease.

Ironically, Scrum was the tool that eventually led us to realize the waste in our project. While we adhered to pure Scrum in some areas of our production, we started to identify bugs and/or delays that we could easily avoid by taking a different approach.

We knew little about Lean at that point. We read a few articles on the subject, but we did the wrong thing and ignored them, because we were so focused on our Scrum process. We were nearly convinced that Scrum was the Holy Grail, but our “Scrum machine gun” started to misfire. We needed to open our minds.

The Principles of Lean

For software development, Lean’s principles were adapted into the following seven.

1 – Eliminate Waste

The change from Scrum to Lean was actually liberating.

In software development, you can find and eliminate waste by recognizing the things that need to be improved. In a pragmatic sense, anything that is not a direct value for the customer is waste. To provide a few examples: waste is a bug, a comment in the code, an unused (or rarely used feature), teams waiting on other teams, taking a personal call… you get the idea. Anything holding you, your team, or your product back is a waste, and you should take the appropriate actions to remove it.

I remember one of our problems was the frequent need to react faster than two sprints. Developing in sprints and respecting Scrum’s rules prohibits you from changing the stories assigned to the current sprint. A team needs to adapt when a user finds a bug and needs a fix, or when the most important customer wants a feature that can easily be complted in two days. Scrum is just not flexible enough in these cases.

2 – Amplify Learning

Place a high value on education.

You have waste, and you naturally want less waste in the future. But why is there waste? It more than likely comes from a team member that doesn’t quite know how to approach a particular problem. That’s all right; nobody knows everything. Place a high value on education.

Identify the areas that need the most improvement (knowledge-wise) and start training. The more you and your team know, the easier it is to reduce waste.

For example, learning Test Driven Development (TDD) can reduce the number of bugs in your code. If you have problems with integrating different teams’ modules, you may want to learn what Continuous Integration means and implement a suitable solution.

You can also learn from user feedback; this allows you to learn how users use your product. A frequently used feature may only be accessible by navigating through five menus. That’s a sign of waste! You may initially blame such waste on programmers and designers (and you may be correct), but users tend to use your software in ways that you never intended. Learning from your users helps eliminate this kind of waste. It also helps you eliminate wrong or incomplete requirements. User feedback can drive a product on paths you would never have otherwise considered.

At some point, my team identified that certain modules could have been better written had we known more about the domain in the beginning. Of course, there is no way to turn back time, and rewriting a huge chunk of code is not practical. We decided to invest time to learn about the domain when tasked with a more complex feature. This may require a few hours or even weeks, depending on the complexity of the problem.

3 – Decide as Late as Possible

Every decision has a cost. It may not be immediate and material, but the cost is there. Deferring decisions helps you fully prepare for the problems that you need to face. Probably the most common example of delayed decision is with database design.

Decisions don’t have to be technical in nature; communicating with customers helps you make decisions that impact the way you approach your products’ features.

But users don’t always know what they want. By deferring feature decisions until the users actually need the feature, you will have more information about the problem and can provide the necessary functionality.

Choose the Agile methodology that works best for you and your team.

Fourteen years ago, the team made a decision to store the application’s configuration in a MySQL database. They made this decision at the beginning of the project, and now the current team (my team) has a difficult burden to carry. Fortunately, that product is no longer in active development, but we still have to maintain it from time to time. What should be a simple task ends up being a monumentally huge.

On the bright side, we learned from our predecessors’ mistakes. We make programming, architectural and project decisions as late as possible. In fact, we learned this hard lesson before adopting Lean. Write open and decoupled code, and write a design that is persistence – and configuration-agnostic. It is certainly more difficult to do, but ultimately saves you a lot of time in the future.

Some time ago, we added a feature to our product that compresses data on the disk. We knew it would be useful, and wanted to add it to our product as quickly as possible. We started with simple functionality, avoiding decisions regarding options and configuration until a later time. Users began providing feedback after a few months, and we took that information to make our decisions on the feature’s options and configuration. We modified the feature in less than a day, and not a single user has complained or requested more functionality. It was an easy modification to make; we wrote the code knowing that we’d make a modification in the future.

4 – Deliver as Fast as Possible

We live in a constantly changing world. The programs we write today are for computers that will be obsolete in two years. Moore’s law is still valid, and it will continue to be so.

The speed of delivery is everything in this fast-paced world.

Delivering a product in three years puts you behind the pack, so it’s very important to give value to your customers as soon as possible. History proved that an incomplete product with an acceptable amount of bugs is better than nothing. Plus you have invaluable user feedback.

Our company had a dream: deliver after each sprint. Naturally, that is impractical in most cases. Our users did not want an updated product every week or month. So, while we strive to release each version of our code, we don’t. We learned that “fast” is what the user perceives – not what we are physically able to do. In our product’s industry, fast means regular updates every few months and critical bug fixes within days. This is how our users perceives “fast”; other types of products and industries have different definitions of “fast.”

5 – Empower the Team

Programmers used to be resources encased in cubicles, silently performing their tasks for their company. This was the prominent image of a programmer in the late 1990s, but that’s certainly no longer the case.

History demonstrated that that approach and traditional waterfall project management is not suitable for software.

It was so bad at one point that only around 5% percent of all software projects were actually delivered. Million-dollar businesses and products were failing 95% of the time, leading to huge losses.

Lean identified that unmotivated programmers caused that waste. But why the lack of motivation? Well, programmers and development teams were not listened to. The company set tasks and micromanaged the employees who were seen only as resources producing source code.

Lean encourages managers to listen to programmers, and it encourages programmers to teach their managers the process of software production. It also encourages programmers to directly work with clients and users. This does not mean the developers do everything, but it does give them the power to influnce the production’s evolution. Surprisingly, having that feeling of, “You know that great feature the users love? It was my idea!” is a big motivational factor.

But don’t think this only works for large teams; our team is small. Our company has only a handful of developers, so we have always been close to our users. That relationship has allowed us to influence our product beyond what our managers might have initially envisioned.

6 – Build Integrity

Anything that stands in the way of production is waste.

Integrity is about the robustness of your product, and it’s how customers see your product as a whole. Integrity is about UI uniformity, reliability and security, and the user feeling safe using your product. Integrity is the complete image the user creates for your product. For example, UI integrity involves the look and feel between pages, screens, modules or even between the UI of your system and the company’s web site.

But integrity can also be observed and practiced at the source code level. Integrity can mean that your modules, classes and other pieces are written in a similar way. You use the same principles, patterns and techniques throughout your code base – even between different teams. Integrity means that you have to frequently refactor your code. It is continuous and endless work, and you should strive for, but never reach, it.

Maintaining integrity in our source code is a difficult task. We realized that finding duplicate code is the most difficult thing to do, and by duplication, I don’t mean a couple of lines of duplicate code in the same method or class.

It is not even about searching in different modules for the exact same code; it’s about finding those pieces of common logic, extracting them into their own classes, and using them in several places.

Finding logical duplication is very difficult and requires intimate knowledge of the source code.

I’ve been on the same project for more than a year, and I am still surprised when I find duplicate code. But that’s a good thing; we reached a point when we actually see these things and take action on them. Since we started actively removing duplicate high-level logic, our code quality increased. We’re one step closer to achieving integrity.

7 – See the Whole

Users don’t always know what they want.

When you create your application, you have to think about the third party components that you rely on in order to develop your product, as well as the other third parties your product communicates with. Your application needs to integrate with the design of a device or the operating system on a desktop PC. Isn’t it easier to use an app that integrates with your smartphone’s notification system, and the UI reflects the OS’ UI?

Yes, seeing the whole is not that easy. You have to detach yourself from the tiny details and look at your product from afar. One of the memorable moments in our product’s development was when we realized that we have to rely on what other programmers produce in other projects. We realized that the kernel of the system, programmed by others, is one of these third party components that we rely on.

At one point, that third party component changed. We could have just applied band-aids to our application, or we could’ve taken the easy route and blamed the programmers that wrote that component. Instead, we took the problem by the horns and fixed the problem in the third party kernel. Seeing the whole and working with it can be messy, but it can make the difference between a product and a great product.

Kanban – A Great Tool

There are several tools and techniques to make Lean work. I prefer Kanban, a board-based tool that is similar to Scrum’s planning board. Imagine a Kanban board as a double-funnel.

On the left are the never ending list of stories that we need to address. All the finished stories pile up on the right, and the manager or product owner determines when to publish a new release, based on this list.

In the middle is our effective Kanban process. The product should be in a stable and release-ready state when we complete a story. This doesn’t necessarily mean that a feature is done; the product may have a few partially implemented features. But the product’s stability, security and robustness should be production quality.

More Flexibility with Releases

We were doing quite well with our current sprint. It was a usual Monday, a calm day with little excitement, but we started to see some problems by Wednesday. That’s OK, it happens all the time. Overcoming these difficulties, however, required some extra time. Our product owner agreed with us to continue working on the current feature and extend the current sprint about three or four extra days.

Friday came with an surprise. You know the rule: if something’s going to break, it will break on Friday. An important and potential client required a certain feature before signing a contract with the company. We had to react (and fast!). A new release was mandatory… But wait! We are in a middle of a sprint. The product should be release-ready by the end of the sprint. What do we do? Scrum would say to do the new feature in the next sprint, but we’re already late with the current sprint! That was the moment, when we started to realize that we have to think smaller than an individual sprint. We need to be able to adapt faster, and we need to release sooner if necessary.

The Kanban Board

A Kanban board looks quite similar to a Scrum planning board, but with a few additions in order to better accommodate the Lean process.

The first column on the left is the full backlog: everything that we need to do at some point. On the far right, you have the other funnel, containing all the completed (but not released) stories.

In the middle is our process. These columns can differ depending on each team and process. It’s usually recommended to have at least one column for the next few tasks, and another column for the stories currently in development. The above image demonstrates several more columns to better present the development process.

The To-Do column lists the tasks that we need to complete. Then, we have Design, where developers work on designing the current stories. The fourth column is Development, the actual coding. Finally, the Testing column lists the tasks waiting for review by another teammate.

Limit Work in Progress

Nobody knows everything.

If you compare this Kanban board with a scrum planning board, you will immediately notice the obvious differences. First, each column has a number, representing the maximum number of stories that are allowed to reside in that column. We have four to-do’s, four in design, three in development and two in testing. The backlog and completed tasks have no such limit.

Each column’s value must be defined by the team. In the above image, I assigned arbitrary numbers to the columns; your numbers may differ significantly. Also, the numbers are not final. Adapt the numbers when you identify and remove bottlenecks.

Dynamically Allocate Resources

One of the most amazing qualities of Kanban and Lean is the importance of collaboration and effort reallocation. As you can see on the board, each column of our process contains cards with people on them. There are eight developers on the example board – quite a large team! The board always presents the current status of who is doing what at any given time.

According to our board, there are three individuals working on design, two pairs working on development and one developer testing. Stories move to the next column their the work in the current column is complete, and depending on the type of development and organization of the team, the same developer can continue working on the same story as it moves through the process.

Let’s presume that we have specialized people. So the three designers’ primary function is design, the four developers write code, and the lonely tester primarily tests the product/feature. If a designer finishes a story, the story moves to development and another story from the to-do list is pulled into design.

This is a normal process. A story was moved from design to development, and now development is at its maximum stories. But what if another designer finishes another story? That gives the developer team four stories – an unwanted situation.

Lean wants to avoid congestion. It is forbidden to move a story to the next column if it exceeds the column’s maximum. In this case, resources need to be reallocated; the designer that finished their task must chose what to do next. His first option is to pull another task from the to-do column, but he cannot do that because he needs to pass his newly finished task to the development team (which he cannot do). His only other option is to start working on a development story. He may not be the best developer, but his efforts will help maintain the process flow.

If the tester finishes the last story in his column, he could help the designer on his development task.

This is great! The team can reorganize on the fly! Do you see a waste? Do you see a bottleneck in the flow? Take immediate action!

Once a story in the development column is completed, the tester returns to testing, the designer to designing, and the developers pick up the story the designer and tester were working on. But keep in mind that you do not have to follow that exact prescription; the developer could start designing as the designer and tester finished developing their story. It’s up to you!

Our board is now back to normal.

Welcome Scrum-Ban

It is forbidden to move a story to the next column if it exceeds the column’s maximum.

I watched with nostalgia as our scrum master dismantled our board. Piece by piece, he tore down our beloved board, which then became a mountain of crumpled paper.

Another colleague entered the room, a few huge sheets of fresh white paper in his hands. Our board was about to be reborn into something different, something better suited for our needs. After the paper sheets were on the wall, we started an ad-hoc meeting to define the columns we needed to define our process. We then debated the amount of stories that should be in each column. After everything was carefully painted and arranged on the wall, we experienced that strange feeling… sadness for the old but happiness for the new.

We did something that many people call, Scrum-Ban. We kept some scrum concepts, such as the scrum master and product owner roles, and we still estimate and evaluate the stories. But we now focus on Lean and Kanban, preserving flow, and discovering and fixing waste and bottlenecks.

The change from Scrum to Lean was actually liberating. Team members became much more friendly with one another, and we understood that we should offer help as soon as there is nothing in our column. This feeling that developers matter made us think about the project as a whole; we care more for the project than we ever have before.

Final Thoughts

Lean was not always considered Agile. Even today, some Agilists refuse to recognize it as an Agile methodology. But more and more, programmers accept Lean as one of the ultimate Agile methodologies.

As one of my wise colleagues pointed out, Lean and Kanban allow you to follow this methodology on your own. So, if you are a lone developer and need some tools to make your life easier, try out some free Kanban tools.

The AgileZen website offers a free account, letting you track a single project.

I found it to be one of the best free online Kanban tools; I even use it every day for tracking and planning the progress of the articles and courses that I provide for Tuts+. Of course, you can always upgrade your AgileZen account, if you need to track more projects.

In this article, we reviewed Lean and Kanban as an evolution of Scrum. Does this mean that Lean is better than Scrum? Absolutely not! It depends on the projects and people you work with. As always, choose the Agile methodology that works best for you and your team.