Photo By: Nahrizul Kadri For the past 20 years, digital businesses have followed a simple growth strategy: if you have an idea, test it with real customers. It became one of the most reliable ways for companies to make decisions based on what customers actually do, rather than what teams simply believed would work. E-commerce companies test checkout pages. Media platforms test subscription offers. Marketing teams test campaigns. The basic process is familiar: come up with an idea, split customers into groups, measure the results, and keep what works. This approach has been extremely useful. But as businesses become more complex, it is becoming harder to test every decision this way.Companies now manage dynamic pricing, regional promotions, multiple advertising channels, personalized offers, and countless combinations of products and campaigns. At the same time, generative AI can create new ideas at a speed humans cannot match. This creates a new problem: the experiment tax. A/B testing is not going away. The problem is that companies cannot realistically use live customer traffic to answer every question they have. A/B Testing Has a Scaling Problem A/B tests remain one of the best ways to understand how real customers behave. The challenge is scale. Every live experiment takes engineering resources, customer traffic, time, and money. Teams may wait weeks for enough data to reach a reliable conclusion. There is also an opportunity cost. If a company tests a weak promotion or ineffective price, real customers are exposed to that decision while the experiment runs. Not every question deserves that investment.

AI Can Create More Ideas Than Companies Can Test Generative AI has made it much easier to create potential business strategies. An AI system can produce dozens of campaign ideas, hundreds of promotional offers, or thousands of targeting combinations. But companies still have limited customers, time, and resources for testing those ideas. That creates a new bottleneck.The challenge may no longer be generating ideas. It may be deciding which ideas are actually worth testing. The Case for Testing Ideas Before Testing Customers Research from Amazon Science offers an interesting example. Researchers examined 67 historical marketing A/B tests and used AI agents to simulate how customers might respond. The results showed both promise and limitations. An off-the-shelf AI model tended to overestimate the size of experimental effects. But when researchers calibrated the simulation using earlier behavioral data, prediction error dropped dramatically, by roughly 77 times. The key lesson is not that AI can replace real-world experiments. It is that simulations can become much more useful when they are grounded in actual customer behavior. This suggests a different workflow: Generate → Simulate → Prioritize → Test → Learn. Instead of testing every idea with real customers, companies could first use simulations to narrow their options. The strongest ideas could then move into live A/B tests for validation. Let Simulation Find the Shortlist Simulation cannot perfectly predict the future. Markets change, customers behave unexpectedly, and new conditions may fall outside the data used to build a model. That is why live experiments still matter. The difference is that A/B tests could become the final validation step rather than the first step for every idea. Companies could use AI to identify opportunities, simulation to estimate potential outcomes, and quantitative models to evaluate questions around pricing, demand, advertising effectiveness, and profitability. This is the approach behind companies such as Kapnova, an agentic revenue and profit optimization platform for consumer brands. Co-founded by CEO James Sun and CTO Shenbo Xu, Kapnova is building systems that combine AI agents with quantitative methods such as causal inference and Monte Carlo simulation. The idea is to use AI to monitor market signals and identify opportunities, while specialized models evaluate the potential business impact before capital or customer traffic is committed. Fewer Tests, Better Decisions This approach could deliver several benefits: Less wasted engineering time: Teams could build fewer low-value experiments. Faster decisions: Companies could spend less time waiting for weak ideas to reach statistical significance. Better margin protection: Fewer customers would be exposed to potentially harmful prices or promotions. More strategic focus: Teams could concentrate on decisions with meaningful financial impact. The competitive advantage may no longer belong to the company that runs the most experiments. It may belong to the company that knows which experiments are worth running. The A/B Test Still Has a Job The future of experimentation is unlikely to be experiment-free. Instead, companies may build decision systems in which AI generates opportunities, simulation filters the possibilities, quantitative models evaluate the economics, and targeted A/B tests validate the strongest choices. As James Sun and the team at Kapnova conclude, the opportunity is not to eliminate live experimentation. It is to make it more selective. The A/B test is not becoming obsolete. It may simply be becoming too valuable to waste on every question.