Should Scrum teams use Prediction Markets?
As a software developer, I want to forecast my sprint, so that my promises hold.
Many software developers work in Scrum teams and forecasting comes up there.
The Scrum Guide mentions forecasting twice. Number one:
Various practices exist to forecast progress, like burn-downs, burn-ups, or cumulative flows. While proven useful, these do not replace the importance of empiricism. In complex environments, what will happen is unknown. Only what has already happened may be used for forward-looking decision making.
There is a general desire to forecast the progress of a development team (primarily by management). The popular “burn-down chart” is a plot of “work to do” over time and the expectation is that it should linearly go down to zero. By extrapolating, you get a forecast when you will reach zero.
Selecting how much can be completed within a Sprint may be challenging. However, the more the Developers know about their past performance, their upcoming capacity, and their Definition of Done, the more confident they will be in their Sprint forecasts.
A “sprint” is a duration of 2/3/4 weeks and provides some rhythm to the development team because during this time, the team is supposed to work with a stable plan (the “sprint backlog”). It only changes during “sprint planning” at the start of another iteration.
The concept of “velocity” is popular: For each backlog item (aka ticket, story, issue, bug report, etc), the team estimates the effort (in man hours or story points or whatever). The sum over all resolved item’s effort is the velocity. The assumption is that this value stabilizes after a while. Given a long prioritized list of items, the velocity then estimates that the first n items will get done in a sprint.
On a larger scale, there are approaches like Scale Agile Framework (SAFe) which features “confidence votes”. It is a form of crowd forecast if the proposed plan for the next months will work out.
Each team conducts a vote using their fingers (fist of five) or a digital tool for remote events. If the average is three fingers or above, then management should accept the commitment. If it’s less than three, the team reworks its plan. Anyone voting two fingers or fewer should be allowed to voice their concerns. These concerns might add to the risk list, require replanning, or provide information. Once each team has voted, it’s repeated for the entire ART, with everyone expressing their confidence in the collective plan (© Scaled Agile, Inc.)
While the Agile community loves simplicity (post-its on the wall), there is also an industry of tools now, so using a prediction market platform shouldn’t deter many.
How could prediction markets be involved?
Obviously, we can create markets for the team internal forecasts described above. The creation can be automated even.
Will the velocity of sprint 5 be 42 (again) or higher?
Will at least 7 backlog items be closed in sprint 5?
Will we close backlog item #53 in sprint 5?
On a higher level, management might care about different questions, like “will the burn-down chart reach zero by May 4th?” A classic deadline question. In a more formal setting you could ask, “will 305 requirements have status ‘implemented’ in June?” or “will unit test coverage exceed 90% in June?”
You can forecast relevant metrics to estimate the outcome of development efforts. For example, Manifold about its daily active users: Will Manifold "hit 15k DAU (7-day average) by Sept 30th" 2024?
If you work on your infrastructure, you can estimate the impact. For example “will we reduce our CI costs by 20% next month?”
Is that useful?
You could automatically create and resolve prediction markets for each ticket in Jira. It would probably even be fun to build this. However, in practice the estimate for a single item is usually done by handful of people within a few minutes and nobody criticizes the forecast accuracy.
Prediction markets are a crowd-sourcing approach. Unless you actually have a crowd of people involved, use something simpler. That said, a “release train” using the SAFe method can be a crowd. They might also be distributed around the world. Here prediction markets could be interesting similar to megaproject management.
Then there is the forecasting about metrics and impact. It feels to me like it should work, but I haven’t experienced it myself yet. Recently, a co-founder of Manifold,
, left and he wrote:within Manifold, we’ve failed to dogfood our markets; they do not serve as a meaningful guide our actions.
When I asked another co-founder,
, for a comment, he acknowledged that Austin has a point but also pointed to some markets which did have an impact: This one convinced them that selling Mana for USD is fine and this one to abandon trading fees. Additionally, brainstorming happens in markets. James summarized it:But overall, I would say the markets have not been that impactful for us so far. Just a little helpful.
The markets there have no overlap with the forecasting described by the Scrumguide. To return to that framing, prediction markets might helpful for the work of the product owner, as he estimates what users want. Prediction markets have no real benefit for the other members of a Scrum team though. Though you probably already knew that because of Betteridge's law of headlines.
This timeline of AI forecasts nicely summarizes a lot of prediction markets.
A Youtube video What I learned forecasting Nintendo for a year makes no reference to prediction markets but shows solid forecasting foundations.
Kalshi got Susquehanna and hopes to boost liquidity.
Probably until next week, agile readers! 😊