Picking the Right Model Won’t Fix Your Product Roadmap

“Efficiency” is one of the most dangerous words in AI right now. Right after “story points.” Sorry, I mean “tokens.”

I was talking to my friend Larry today. Last month, his organization’s token bill was over $40,000, a fraction of its payroll. Now his job is to make that usage more efficient, including making sure people don’t use a premium frontier model for something simple.

Fair enough. I like saving money too. But once we have something to count, we can spend an awful lot of time discussing it without asking whether the work is worth doing. Can we give the roadmap the same attention as the AI invoice?

Somewhere in your organization, a feature nobody has explained the value of may be enjoying its 3rd quarter of uninterrupted funding. Fortunately, Larry is looking into the tokens.

1. Fix your value forecast

Pick something on the roadmap. Get the business owner and the team together and explain what changes when a customer can use it. More profit? An expense goes away? Less time spent doing something miserable?

Put a reasonable range on that value, say when you expect it, and write down what would prove you wrong. Let the team challenge the assumptions before you pay them to implement them.

“The executive team wants it” tells me who asked. We still have the value question.

2. Fix how you invest

Fund a small piece of working software that lets you check the forecast. Decide when you’ll review the evidence and what would make you continue, change direction, or stop.

Leadership has to make stopping possible. If a team discovers customers don’t need the feature and still has to finish it to meet the annual plan, you’ve made learning rather expensive.

If the forecast no longer holds, let the team move to something worth doing. Their salaries stay in the budget, but you stop spending their time on something customers don’t need.

3. Then put the AI bill beside the labor bill

This is why I argue that your AI cost to build software should be a function of your labor cost. Start with what the valuable work costs today. Then ask whether additional AI spending helps deliver it sooner, with less effort or fewer failures, after including review and rework.

Use a cheaper model when it does the job well. If a stronger model costs more but saves enough review time or prevents expensive mistakes, include that in the decision.

An example: $10,000 in tokens to write tests

Let’s use a hypothetical example. Customers keep reporting problems with order changes. That part of the application has no automated tests, so every release requires people to check it by hand. Changes take 4 weeks to reach customers, including 2 weeks spent waiting for and completing those checks. Defects cost $40,000 a month in refunds, service credits, and outside support charges, excluding employee salaries. Customer satisfaction is 70%.

You ask the team what would help. They propose using AI agents to write tests around order changes, starting with the failures customers already report. Engineers check that the tests describe the intended behavior and catch known defects. They fix the problems the tests expose and add the tests to the release process.

They ask for $10,000 in inference for 2 developers working with the agents for 1 week. Their combined payroll for that week is $5,000, already in the budget. The additional spending is $10,000, paid up front.

$10,000 in tokens? That’s 2 times their payroll. Clearly, it would be better to write the tests by hand. We haven’t asked how many weeks that would take, but we’ve already established that we don’t like the invoice.

I picked $10,000 to get that reaction. I doubt the inference would actually cost that much for this work. But suppose it did. Before we send the developers off to write everything by hand, let’s see what that $10,000 could buy us.

You approve the pilot and agree to measure the time from an approved change to production, monthly defect costs, and satisfaction among customers using that workflow. Leadership agrees that the team can retire the manual checks the automated tests reliably replace. Keeping both would leave people doing work you’ve just paid to automate.

Over the next 3 months, suppose comparable changes reach customers in 2 weeks instead of 4, a 50% reduction. Defect costs fall from $40,000 to $20,000 a month. Satisfaction rises from 70% to 80% using the same survey. Customers need less help changing orders, and fixes arrive sooner. The team checks whether other changes contributed to those results.

The tests now run in the existing release process, with upkeep handled within the team’s budget. The $10,000 inference bill doesn’t recur. Saving $20,000 a month on defects recovers that spending in about 2 weeks once the savings begin. Over 3 months, you avoid $60,000 in expenses and come out $50,000 ahead after the inference bill. The developers spent 1 week they could have used elsewhere; we haven’t counted their salaries as a saving.

There’s also the remaining roadmap. Suppose the next 3 improvements let customers buy an additional service, renew without calling sales, and complete purchases they currently abandon. Based on customer evidence, the business owners forecast a combined $10,000 a week in additional profit after operating costs.

If each improvement reaches customers 2 weeks earlier, each earns an extra 2 weeks of profit. Together, that adds $20,000 within the same 3 months. This is what pulling value forward means: customers can use the software sooner, so the business starts getting the benefit sooner.

While you wait, you may also be paying holding costs. Suppose 1 of those improvements replaces an old service costing $2,000 a week. Shutting it down 2 weeks earlier saves $4,000. That bill is separate from the defect expenses and profit forecast above. Until you release the replacement, you’re still paying for the old service.

Those 2 weeks of waiting would cost $20,000 in missed profit and $4,000 in bills you could have stopped paying. That $24,000 is the cost of delay. It describes the cost of waiting we just calculated, so don’t add it to the spreadsheet again.

If the forecasts hold, the business gains $74,000 over 3 months after the additional cash spending: $60,000 in avoided defect expenses, $20,000 in earlier profit, and $4,000 in retired service costs, less the $10,000 inference bill. Satisfaction improved too, but we haven’t assigned it a dollar value. Ask the team which roadmap dates actually move before using this calculation. Cutting release time by 50% won’t help a feature that’s still waiting for somebody to decide what it should do.

If a cheaper model preserves those results, great. If it saves $2,000 in tokens but adds $8,000 in review and defect costs, we’ve lost $6,000. I’m sure the token dashboard will look lovely.

But once the team has demonstrated the return, there’s probably something more valuable for Larry to do than itemize individual prompts for efficiency. Another part of the application may have the same testing problem. Another worthwhile feature may be waiting on a decision he can help get made. His time belongs in the investment calculation too.

There’s always waste in a system. Give professionals visibility into what they spend, the results they’re responsible for, and room to improve how they work. Let them reduce the waste as they find it. You can review the spending and the results without asking them to defend every prompt. We don’t need to turn a successful investment into an expense report with 40,000 line items.

You’ve made the ROI case. Check that the return holds, then help the team find the next valuable thing to improve. That might even be the roadmap.