In the previous post, I focused on business context and how it helps absolutely everyone involved in the process of creating software. In the second part presentation for Product Camp – this time I explain how to set Success Criteria within a project and how it helps teams enhance their self-organizing skills. I will also share a few specific examples from our portfolio, giving you a more detailed insight into how we operate in these areas within Inwedo.
Listen on Spreaker or watch a video on YouTube.
The webinar took place at Product Camp 2020 – the voice and video footage are from this event. Of course, a lot of greet from us to the organizers! It was an amazing time.
The right way of finding success criteria in the software delivery process
If you haven’t already read the first part:
Let’s spend a bit of time talking about four sets of metrics that converge when you look for an answer to a question: what is the right way of finding success criteria? I will start with analyzing those metrics that are easiest to calculate but bring the least value.
Good practice: repeat a conversation with the stakeholders that you had at the beginning of the projects you’re involved in. Ask questions about how to define success.
Set 1: Naive Metrics
They are super-easy to calculate, very transparent, but sadly – usually meaningless in a broader context. We can incentivize any of them to reach them, but this way we risk they will lose their original value somewhere on the way. For example, imagine if we incentivize increasing velocity or increasing the story points burndown – then magical things happen with the story point perception.
If our goal in creating software is to increase velocity with every sprint, then we expect an increased velocity. But the question is: will the amount of work really increase? Or if we prioritize the percent of completion, then probably the old Pareto Rule will be very obvious and apparent. Why? Because we’ll run through 70% of things in quite a short time and then the remaining 30% might take days or months to complete.
The most worrying metric is lines of code written.
Do you smile while reading this, as I do? This is a metric that will not give you or the Client any meaningful information. Naive metrics are not a good alignment strategy at all.
Set 2: Vanity Metrics
They reflect some of the states of reality, but they are just focused on making someone feel good. A bit harder to calculate, Vanity Metrics are still very easy. The question is – what kind of information does the metric tell us? What is the usage of the product or the success with this product?
There’s a great difference between active users and one-time users. It’s quite often that in companies there’s a need for the application to be counted as successfully deployed. Teams define their metrics as 100% of employees log in at least once. But again, this does not reflect any truth about the reality we’re working with.
Set 3: Meaningful Metrics
Meaningful Metrics are much harder to calculate because there’s quite a lot of context involved. They are usually quite difficult to tinker with, therefore they are mostly objective. If your product or the project is pre-launch and the team has a good understanding of the context and the business value, you might try estimating business value points, together with the stakeholders or within the team.
With Meaningful Metrics, you are trying to compare different user stories, or different tasks by the significance of business value delivered.
Priority points allow to separate business value from the story point value and help to arrange tasks in the back log.
Business value point estimation for teams equipped with the context should be very easy. It brings a metric that is of course subjective but reflects the top priority of the client.
You can watch the whole episode on our Youtube.
Self-sufficient team: spending time on involving the team in the process as a good investment
Spending the time to involve and submerge the development team in the project business context (especially if they come from an external provider) is of great help on a journey to self-sufficiency within the team.
If a team is committed and focused on helping to deliver products, they actually need a way to align themselves. They will be making millions of small decisions on the way and if they are not micro-managed, they need some kind of a tool to ease the alignment.
Success criteria are the best tool to enhance teams’ self-sufficiency. If our top priority is acquisition channel through referrals, then Net Promoter Score (NPS), or any other customer satisfaction metric is super-important to be focused on. For a software product, Monthly Recurring Revenue, or any equivalent it is interesting to see how the values change over time. In products that rely on a low margin or a low margin market, Lifetime Value or Customer Acquisition Cost usually brings a lot of insight. Also, showing how this metric is calculated is very important, because it allows people to understand what’s the Lifetime Value of a customer. Or what is the typical – average behavior of a customer or a user.
Set 4: Personalized Success Criteria
The teams make hundreds and hundreds of decisions throughout every sprint and they really need a tool to align.
They can either use their intuition, or they will try to benefit from the KPIs or the goals that are set. I call them Personalized Success Criteria. They are always product-specific and they come from only context-aware teams. And that’s why the word ‘context’ has been used here so many times.
To make this story more tangible, let’s look at two real-life examples.
The first example: a mobile ticketing machine – zbiletem.pl
zBiletem.pl is a solution for selling public transportation tickets operating in 30 cities in Poland. At the beginning of the project, the idea behind was rather abstract, because the owners from Sweden came to Inwedo and said: “Please build a mobile ticketing machine.” This case was tough at the beginning because of a super-low margin world (it’s just a few cents per single ticket sold). Users don’t spend much time with the service, but if breaks – oh boy, things happen!
If you buy a ticket and it’s not there, then your rating in the stores will just go down dramatically. And that was a good thing to start – to prioritize the stakeholders.
Who is the most important stakeholder?
On one hand, we have public transportation authorities that hold in their hands the ability to sell tickets. They want to have as much data as possible.
And on the other hand, there’re passengers, who just want to have a super experience. And in this case, the passengers as stakeholders prevailed as the main focus of the team.
Success criteria in creating software and their evolution over time
zBiletem.pl had three phases of growth.
Phase 1 – introduction to the market
- 1st part – the Geographical Growth, from 0 to 30 cities, that was all about creating habits. Analyzing the percentage of returning users and focused the team to think about notifications, one-click repurchases, implementing Apple/Google Pay, creating in-app reminders, or re-engaging the users that stopped buying tickets.
- 2nd part – the Return on Investment per city. It’s about how expensive and complex it is to encourage a city transportation company to go online and in how many months the process should be expected to pay back.
- 3rd part – Customer Care, understood as responding super-quickly if anything breaks. The team was actively monitoring ticket volume on customer care requests as well as app ranking and comments in the stores, to make sure we address the most common issues first. Or automate recurring ones to save a lot of time.
Phase 2 – Going deeper into the markets
The priority in the second phase was penetrating the cities for the user base: acquiring users in cities where zBiletem was already present. That was the moment when success criteria shifted to Lifetime Value and Average Revenue per user. This is usually a phase in the project lifecycle when the team and every stakeholder are aware of the budget.
How to excel on the market and how to build a referral system?
In this phase we had to ask ourselves a few questions around metrics:
- Can we incentivize other passengers or new users to join by referral?
- What is the cost per install (it was easier to calculate this cost rather than cost per active user)?
- What is the strongest correlated metric with the conversion (the conversion was tracked along the whole user journey)?
Phase 3 is still in progress
One of the discoveries about the strongest correlation in creating software was the time needed to complete the purchase end to end. It became something that zBiletem started excelling at, being one of the quickest apps to actually close the purchase cycle. And this is still an ongoing journey.
The second example: a marketplace that supports farmers
This marketplace helps farmers in implementing regenerative farming practices on their land. What’s worse than a small unprofitable project? A big unprofitable project! 😉 My last example out there is a collaboration between our client MiljöMatematik and a Swedish Agriculture Ministry. To reach a better understanding their project (Svensk Kolinlagring), you need to know that Swedish product has two groups of interests:
- farmers (goal: to implement regenerative practices),
- climate-aware businesses (goal: providing farmers with a financial incentive for regenerative farming through purchasing the carbon dioxide that is stored within the soil as a result of the process).
We have the pleasure to work with one of the most profound uses of technology. I’m personally very proud of this project because we can work with a platform that contributes to storing tons of CO2 in the ground and regenerating farmland.
The main success criteria metrics had to deal with the following question: how many tons of CO2 are stored away? The team took on the challenge. They actually cut these success criteria into sub-criteria because the tons of CO2 sequestered depend on the number of farmers that were successfully onboarded.
Onboarding Success Rate
The number of farmers that joined the program depended on their first experiences within the platform. An Onboarding Success Rate showed us how many of those farmers got stuck in the process. One of the most important factors to improve their overall onboarding experience was to automate introducing the land geographical data – fields, areas and crops, which was achieved through uploading available government XML files.
From that point forward, the team has been engaged in watching the CO2 sequestered and their main focus is to keep developing the tool further with this high-level goal in mind.
What are the main challenges of this beyond software solutions?
To sum up I want to tell you what I’m focused on as a founder of a company that aims to create solutions, not just software.
- I feel that I already have the privilege of working with people that fall in love with project after project, product after product. But I guess the hardest part is scaling that and having more people on board that actually are ready to open their hearts in a way that I’ve already described.
- Setting the bar for the projects we start and work with. Putting pressure on Sales to actually be committed to projects that make a difference and bring long-lasting, sustainable positive change in the world.
- Encouraging everyone on the team to ask difficult questions: What does it mean to be successful? How do we know we got there? It often means challenging the customers to answer those hard questions.
- Spreading the context and aligning the minds and hearts of people who work daily with those Success Criteria. And that’s a good point to wrap up.