How I measure success in Product Operations

The connection between Product Operations and business impact is not obvious and can cause issues if it is not identified. It can be hard to determine if we are making the right decisions, or if we are achieving our goals. This lack of clarity can lead to misalignment with management and stakeholders. And ultimately, Product Ops may end up not contributing to the business as much as it could.
To prevent this, I’ve been putting a lot of effort into developing a model that works in the context of my current company. It’s not perfect and I’m continuously refining it, but it is useful to me. This approach may not work for everyone, however I hope it provides some helpful guidance.

Success starts with Strategy

To understand success and be able to measure it, I start from the strategy. Crafting a strategy for Product Operations gives direction and scope to the function within the company. It sets the context to think about which drivers can be moved by Product Ops and potentially indicate if strategic progress is made towards business objetives in a given period of time.
There are plenty of unknowns here, but if we are able to estimate the business impact of each driver and create a sensible narrative around them, we have a good starting point. To me, having a reasonable potential correlation was more important than finding a perfect one - otherwise I would’ve got stuck in analysis.

Talking about specifics, this year we want to achieve effective business growth and sustainability in our main markets. Generally, this means growing revenue, leveraging on product quality, and improving operational effectiveness. Considering the current context of the product organization, I chose MRR from new sales and expansions, and operating expenditure as my business metrics to impact through my actions on other metrics. The strategy behind it is:

• Making sure all product teams gain a deeper understanding of our markets and customers' needs, and of how our products are used.
• Optimizing our product development cycle.
• Making sure go-to-market and customer success teams have all product knowledge they need to excel at their jobs.

There are three drivers I can directly move to impact the strategy, and that benefit from influencing each other:

• Accessibility to reliable insights.
• Productivity.
• Collaborative selling.

Except for measuring productivity, I didn’t have relevant data to correlate the drivers with revenue and/or OpeEx, so I used some benchmarks from secondary research about companies in my industry and business model. The first decisions I made with it weren’t the best, but they led to better results than the ones I was making without it.

As I gather more data on the business impact of an increment in each driver, I am able to adjust the model and make it more predictable, so each decision I make is more impactful by design. Also, it allows for better conversations with my manager, who can challenge it and help me improve it, which is great not only because we are aligned but also because the value Product Ops brings is effectively conveyed.

Moving the strategic drivers: goals & execution

With a clear strategy and drivers that connect my actions to business metrics, I can now discover the problems in each area and set up goals in accordance. At the end of the day, aiming for better decision-making creates opportunities for success, without having to be right every time.

One thing that I learned as a Product Manager is that not all goals need to be outcome-based. Unless I know how to execute the solution that will move the needle and I have a minimum certainty of the impact I can expect, I won’t use outcome-based goals. Understanding this was a game-changer to me, as it helped me focus on strategic progress instead of trying to hit high-level KPIs when I didn’t even know how to. I’ll give examples.

For product teams to better understand how our products are used, they need access to behavior analytics, but they didn’t have it. I needed this resource to make progress towards the strategy, so my goal was to successfully implement behavior analytics tools and training across product and engineering teams.
I assessed progress towards the goal by setting a series of deliverables that needed to happen (e.g. tooling, event tracking plan, training plan…). And I defined success of the goal using the TARS framework (the higher the "ARS"; the more likely to lead to significant business impact). I couldn’t link this goal to revenue or OpEx as the impact was not immediate, but it would enable others which will.

To increase product knowledge across go-to-market and customer success teams (>400 people now), I carried out a series of experiments with some groups to have a greater degree of certainty about how big the problem was and whether the solutions I was considering could really move the needle. The constraint I added to myself here is to increase their knowledge without increasing their time to knowledge. The experiments reduced the risk and helped me estimate the correlation between increments in knowledge and increases in MRR from new sales and plan expansions. Now it makes sense to establish an outcome-based goal.
Knowledge is very difficult to measure, so I will assess the progress by measuring KPIs such as training completion rate per team, average knowledge per product line (from tests and self-evaluation surveys), time to knowledge, and ratio of keywords mentioned in demos with prospects and calls with existing customers. And I will use the estimated correlation with revenue resulting from the experiments to attribute impact in business. This won’t be perfect, but again I’d much rather it done than perfect, and I’ll keep adjusting it as I learn more.

The Newton's cradle

Basically, to measure the success of my specific actions, I make sure I have crystal-clear what I want to achieve and why, how I can assess progress and how I will know that I achieved the goal. Also I add to the equation what cannot change, my constraints. It is easy to focus too much on a metric that the progress on it unexpectedly harms something else. Once the strategy and the goals are framed, I build a bottom-up chain of metrics towards them, which normally starts with assumptions I continuously validate and adjust.

Imagine the Newton's cradle: when the initial sphere is lifted and let go, it hits the other spheres that are staying still, sending a force through them and pushing the last sphere up.

By connecting the dots I am not only able to measure success at many levels and quickly learn from mistakes, but I can also agree on such definition with my manager and effectively communicate the success of Product Operations across the whole organization.