Testing Our Hypothesis: Results and Learnings 

Efficiency is important. But it’s also difficult to measure. How does your team measure efficiency?

Do you base it on perceived productivity, by counting the number of stories finished per sprint? Or maybe on a perception of how busy your team members are, by calculating their idle time? Without question, a team’s perceived efficiency can change based on what criteria you choose to measure.

And that begs the question: which metrics are best suited for indicating a highly efficient team? In this research blog, we explore this essential question.

Resource Efficiency – A Paradox

The most widely accepted efficiency metric is resource efficiency, a concept that comes from Manufacturing, evolved out of Frederick Taylor’s theory of Scientific Management, writes John Roets, Senior Solutions Architect at ITX. In his blog, 10 Reasons to Prioritize Flow Efficiency Over Resource Efficiency, Roets explains that resource efficiency is where we focus on minimizing people’s idle time – the theory being that idleness is one of the best leading indicators of inefficiency.

Roets challenges this notion; in fact, he contends that an overfocus on resource efficiency is harmful. In summarizing his argument, he refers to the “resource efficiency paradox” this way: the more you focus on making your team resource efficient by keeping them busy, the less efficient they become. See also: The Jevons Paradox.

One symptom of the resource efficiency paradox at work is revealed in the sprint burndown chart, below.

Chart 1: The Sprint Burndown Cliff

Here, work is assigned in a way that relates to one person working individually, rather than collaboratively as a unified team. The idea, Roets explains, is that pairing up is inefficient.

Not surprisingly, he continues, this mindset exacerbates the specialization problem, lowers the ability of team members to collaborate, and yields a web of time- and energy-devouring context switching.

Flow-Based Efficiency – A Hypothesis

Roets contends that flow-based efficiency offers a better approach – and a better measure of efficiency. His hypothesis, which comes from the Lean community – if you’re familiar with things like Kanban, for example – is that a team will be more efficient and effective if they work on fewer things in parallel and focus attention on getting items through the system at a good quick pace. In other words, “Flow.”

Many development (i.e., Scrum) teams, he says, are not as efficient and not as effective as they might be. They simply have too much work in progress – that is, user stories and epics started but not finished during a sprint. It’s also because team members tend to work too much in isolation of each other on those things.​

Flow-based development is user-centered. It’s about getting features quickly through the system and into the user’s hands. It is the ratio of value-add time to cycle time. Cycle time is the total time it takes to get something out the door.

For example, if you spend 10 hours adding value to a feature but it takes 100 hours to get it out the door, that’s a flow efficiency of 10%. The longer the wait times, the less “flow efficient” you are.

Flow-based development is not something new at ITX, Roets explains. Several teams operate effectively according to similar principles.

Testing Flow-Based Efficiency – Approach

Roets decided to test his theory: If development teams prioritize flow efficiency over resource efficiency, they will become more performant. He used his own dev team (Beta) as the subject in this experiment.

Over a recent 10-month period, Beta team adhered to a flow-based mindset and methodology. In that 38-week period, Beta worked together (pairing/mobbing) and –

• Worked on only one epic at a time,

• Worked on only one feature from that epic at a time,

• Worked on only one story in active development at a time; and

• Kept all those stories small – that is, only 2.4 points on average.

Stand up meetings – which used to provide a status update on an individual’s progress, now produced a coherent daily plan for the team. As a unit, they decided what they should work on and then worked on it together.

Results of the Experiment: Expectations

Before beginning the experiment, Beta team documented their expectations:

• Relatively low duration to get items “done” (cycle time)​

• Improved, more consistent pace (flow); a steady diet of done

• Evidence that demonstrated a higher level of focus from the team, generally; quicker learning, increased productivity, and higher-quality​ outputs

• Heightened attention brought to things that block progress​

• Potential dislike of mob dev by team members​

All experiments are successful when we learn. All the better when the experiment’s results prove the hypothesis. In this case, the quantitative results of the experiment were in line with expectations:​

• Cycle Time. 50% lower cycle time, as compared to two other dev teams (3 days vs. 6 days to get stories done.

• Pace, Cadence. In comparison to other dev teams, Beta enjoyed improved flow thanks to  smaller stories (84% of stories were 3 points or less​).

Roets highlighted these metrics specifically because current research in software delivery performance suggests that quickness and frequency of delivery predict success. Here, Roets and his Beta team are applying that same reasoning to sprints.​

Chart 2: Beta team’s tickets completed in one sprint (Flow-based)

Chart 2: Beta team’s tickets completed in one sprint (Resource-based)

Other Learnings, and a Note About Productivity and Quality

By one measure, Beta enjoyed a feature completion rate that was 30% higher than originally estimated. What’s more, we identified additional leading indicators of quality, like the (low) percentage of code reviews that resulted in re-work.​

A word of caution too: productivity and quality metrics in software development can be dubious, Roets admits.

“Since there are a lot of factors that go into success or failure of a project, we also looked for subjective qualitative evidence.”

Roets instructed his team members to complete an anonymous survey, inviting them to offer their insights as to whether the flow-based approach led to increased productivity, focus, and learning pace, among some other findings.​

Their feedback was generally positive: they felt more productive, learned at a faster pace, and were more focused. But the approach was not universally embraced: flow-based development creates some challenges to adjust to a different way of working, with pairing and mobbing in particular being difficult for some team members.

Flow-Based Development – Next Steps

Overall, the results were sufficiently positive to recommend additional experiments. One such experiment will soon be run with a larger team and a solid coach to help overcome the challenges some Beta team members experienced with pairing and mobbing.
 

---