Posted on December 15, 2020 | Guille
Since the publication of the Agile Manifesto almost two decades ago, agile methodologies have been slowly but steadily becoming prevalent in software development, and none more so than Scrum (according to the latest State of Agile Report, more than 75% of respondents practice Scrum or a hybrid that includes Scrum). This post analyzes some positive and negative aspects of this new reality, and sets the stage for a proposal for improvement.
The Good
Although this piece is more focused on the problems of Scrum, it is important to also credit the benefits that it brings.
Standardization and certification
The fact that Scrum is so commonplace today is precisely its best quality: it has made agile development the norm and finally displaced the waterfall model. While the Agile Manifesto sets the foundations, it doesn’t give any concrete guidance on how to achieve its principles. Scrum, on the other hand, is extremely explicit on the procedures, nomenclature, and responsibilities it prescribes. For a large company faced with the prospect of a substantial change in their way of working, rigorous specification transforms a giant leap of faith into a well-planned path. Companies love certifications like ITIL and ISO 9001 because they validate their work. Likewise, having a certified expert come in as an external consultant is almost a necessity, and no other agile development methodology can compete with Scrum in this regard.
The Product Owner
Apart from getting rid of the waterfall model, Scrum has also helped to introduce the concept of an extradepartmental project team. Large companies tend to be quite rigid in the way their workforce is organized into functional departments (HR, Operations, Sales, etc.), but for an innovative project to succeed it is essential to incorporate expertise from several departments. The Product Owner role comes close to matching the shusa role at Toyota: a person who has complete responsibility for all aspects of the product (design, production, sales, etc.), while having no formal authority over people in the functional departments (the stakeholders in Scrum). The sense of personal ownership in the project’s outcome compels the Product Owner to work with all departments to guarantee the success of the product, while keeping the product team isolated from the company’s organizational burden.
The Bad
Scrum deserves praise for what it has achieved. Software development wouldn’t be where it is today without it, and it’s definitely in a better place than before. That doesn’t mean that Scrum is perfect, especially considering that many things have changed in the past 20 years while Scrum barely has.
Sprints
Sprints are “the heartbeat of Scrum”[1], a timeboxed effort to produce the next Product Increment. The problem is that business isn’t timeboxed, so any new business demand will take approximately 2 sprint durations to be delivered (one to finish the current sprint and another to implement the new feature). For month-long sprints (the most common) this means waiting two months for results. In the fast-changing world of today, two months is an eternity. Just think of the current situation with Covid-19: in two months society might have gone from business-as-usual to complete lockdown and back again. Any plan to adapt to the circumstances would arrive when the conditions have changed and possibly require a new variation.
User Stories
Trying to get features to fit in a sprint typically involves breaking them down into “atomic” user stories, that should be independently functional (because the end result of the sprint must be a releasable product). This requirement produces myriad artificial “subfeatures” that will never be released until the epic (basically a real feature) they are grouped in is completed. Focusing on producing sprint-sized user stories adds unnecessary work to subdivide the features and obscures the business value of the original vision.
Story Points
Along with user stories come story points, a measure of the complexity of a task using the Fibonacci sequence. Although Scrum doesn’t formally propose the use of story points, it does require the development team to estimate user stories in some way. Regardless of whether it is with story points or man-hours or however else, the truth is that estimating development time is impossible: things that seemed easy can quickly become extremely complicated, and things that at first glance were very complex might actually be resolved effortlessly with an external library or a different approach. Forcing the development team to estimate work is frustrating for them, inspires false confidence for stakeholders, and provides no business value.
Scrum Ceremonies
Scrum mandates a string of meetings to be held every sprint with the entire team. In a month-long sprint, up to 20 hours are dedicated to these meetings (almost 13% of everyone’s time, considering a 40-hour work week), and it only gets worse for shorter sprints (which is why four weeks is the most common sprint duration). Not only do they incentivize longer sprints (and longer work weeks), they force the entire team into a fixed schedule (what about remote workers in a different time zone or team members with flexible work arrangements?). The worst thing is that these meeting are highly redundant:
- Sprint Planning: if everyone has already agreed on the required level of detail for user stories (the Definition of Ready) and the Product Owner has already prioritized the backlog, all that’s left is for the development team to decide how many will fit into the sprint. According to Scrum the team can refine the user stories and plan how they will implement them, but both of these tasks are better handled when starting the development, since that is when the relevant questions arise. It is hard to think of what information might be missing or how best to implement something by just looking at a definition in a meeting room.
- Daily Scrum: if communication between the team is fluent (either in person or through a tool) and there is a live display of current activities (usually via a Kanban board) there is no need to have meetings to raise issues or detail what each person is doing.
- Sprint Review: if the end result of the sprint is a working Product Increment that includes the functionalities included in the sprint, there should be no need for the development team to demo it as anyone (Product Owner and stakeholders) can do this in their own time. In fact, if each person tests the product independently the evaluation is more exhaustive and there is less pressure on the development team to defend their work.
- Sprint Retrospective: gathering feedback on the progress and quality of the project should be a continuous effort. Constructive criticism should always be welcome, and never forced.
The Scrum Master
A role whose sole purpose is to promote and support a methodology which is fully described in 3,600 words seems superfluous. If the team is new to Agile, it won’t be enough and an Agile Coach may be required. If they are experienced with Scrum (as they eventually will), there is no need for this role at all. The Scrum Master provides organizations with the security of having an “expert” on board, but ends up making the project dependent on one person. Product Owners expect the Scrum Master to act as a Project Manager, taking on part of their responsibilities, and team members rely on the Scrum Master to insulate them from the business side of the project. Both of these tendencies end up crippling the project by weakening the Product Owner and disengaging the team.
The Ugly
Apart from the specific problems inherent to Scrum, the use of Scrum promotes several misunderstandings and bad practices. Some of these are in fact consequences of the good aspects of Scrum mentioned above.
Scrum == Agile
Since Scrum is so popular, many people directly associate it to agile development and do not distinguish between the two terms. This reinforces the monopoly that Scrum holds, and doesn’t give other agile methodologies the chance of being known. Kanban is perhaps an exception to this, except for the fact that it is more recognized as a tool to be used with Scrum (the Kanban board used during Sprints) instead of a complete methodology on its own.
Multidisciplinary development teams
Although Scrum is technically a software development methodology, in its description of the Scrum Team it incorporates all the functions such as UX, design, development, QA, etc. The idea of having everyone that participates in the product together as a team is very powerful in creating a product-centered culture and having everyone working towards the same goals. The problem is that, since Scrum compresses the entire software development lifecycle into each Sprint, and each role operates mainly in one of the phases of this lifecycle, it creates a dependency chain that tends to leave some team members idle either waiting for input from others or for feedback of their output. What’s more, the further down the pipeline the less time available in the Sprint, so QA and other validation tasks tend to be neglected.
Uphold traditional paradigms
Probably one of the reasons enterprise has been eager to embrace Scrum is because it doesn’t disrupt the traditional way of working as much as other agile methodologies. By using Sprints and estimates Scrum maintains the project management efforts to track and forecast progress. The problem is that this effectively cancels some of the advantages of agile development and brings back some of the original issues that the Agile Manifesto was fighting against. For example, since Sprints provide deadlines, there is pressure on the team to deliver the software by the specified date, regardless of quality.
The Future
At Ideotec we are working on a new Agile methodology called Flow, which we will be presenting on this blog in the near future.