A bot that answers with sales figures: six metrics, with the arithmetic checked by AI reviewers

Why the language model doesn't do the arithmetic

This part of the bot is valuable because people trust its figures. A model with free access to the data will sooner or later produce a plausible number out of thin air. One such case in a conversation with the owner would undermine trust in every other report. So code makes the decision: keywords in two languages, a fixed set of metrics, a call into the core.

If a question isn't recognised, the metrics module stays silent and the bot answers from the knowledge base. I chose accuracy on purpose: the bot gives a figure only where it can be checked.

So that new metrics follow real demand, questions that got no figure are saved to a log. It keeps only the question text, cleaned of secrets, with no name, username or chat ID. A two-week summary shows what the bot is missing.

The bot reports three kinds of failure in three different ways

If the module is missing, it is switched off and the bot answers from the knowledge base. If the module is there and Shopify is unreachable, the bot says so plainly: "couldn't fetch live data". If the modules no longer agree on the data format, the user sees a message too. A silent failure had already killed this feature once: the module was written for a format that didn't exist, quietly fell back to the knowledge base and looked as if it worked. An error nobody sees can survive for years.

Geography: an order without an address stays in the total

The country comes from the shipping address, where the goods actually went. Orders without an address (digital goods, pickup) count in the total and aren't assigned to any country. Otherwise they would quietly end up in the US and distort the picture. The bot reads years before days, so "for 2025" gives the calendar year, and "besides the US and Canada" removes two countries from the answer. The bot now answers the owner's question word for word: orders, revenue and the number of countries for each year, with sales abroad on a separate line. In 2025 orders went to 21 countries, 19 of them outside the US and Canada.

Top products: money and units side by side

In this business the two metrics differ a lot: accessories bring about 22% of the money and about 63% of the units. A single column would lead to the wrong conclusion about the range. So each item shows both shares, and a separate block lists products that sell in large numbers without making the money top list. Products are grouped by their catalogue ID: the same product can appear under different titles, and the SKU is empty on about 87% of line items. I moved the top-products check ahead of the revenue check. That was the first of two fixes to the order of checks.

US states by sales channel

The owner asked for net sales by state, in total and for the marketplace separately, after refunds and discounts. Net sales here means line items after discounts minus refunded items, without shipping or taxes. The core counts by the same rule, so the state totals have to add up to the overall report.

The core's data had no sales channel before. I added the order source and tags to the request and cleared the old cache. The owner gets a CSV file and a bot answer built from the same numbers.

The question "by US state" contains "US", and with the wrong order of checks the bot sent it to the country breakdown. The second fix to the order: states are checked before countries.

An hour after release: four agents, four errors

Four AI agents reviewed the new code, each on its own. Here is what they found:

• The top-products module didn't subtract refunds at all, so product revenue was overstated by 8.54%.
• The collector had its own list of payment statuses, and two of the core's statuses were missing from it. That's why the report had 324 orders and the metrics 323.
• Products were grouped by a "product type" field that line items don't have, because the type lives in the catalogue. The type column had always been empty.
• The geography module counted a cancelled payment authorisation as a refund. There were no such amounts at the time, so the bug would have waited for the first cancelled authorisation.

All four errors had one cause: the collector did its own arithmetic. Now every calculation goes through the core: one rule for which orders count, refunds per line item, discounts spread across items. The geography module now uses the core's data too, with the shipping address added to it. The cache had to be cleared, or for six hours the bot would have read old data without that field. A check after the fixes on 30 days of live data: 324 orders in the report, in geography and in the top list, and a product-revenue gap of 0.00.

Five guard tests on top of the findings

A figure calculated almost correctly is impossible to catch by eye. So the match with the core is pinned down by tests. Geography and the top list count exactly the same orders as the core. A fully refunded order adds no revenue. A cancelled authorisation doesn't count as a refund. A separate test reads the source code and checks that no collector has its own filter by payment status. If such a filter appears, the automatic test run fails.

Before the state report: four more findings

The owner asked for the state report in the evening and got it the same evening. Before the figures went out, they were checked five times, each time from a different angle:

• "Draft" and "marketplace order" are two different flags. The phrase "marketplace orders are marked as drafts" invited a filter on the draft flag. I checked seven weeks of data. In one store, 81 of 106 drafts belonged to the marketplace. The other had 12 drafts and no marketplace orders at all. A draft filter would have given the marketplace 25 orders that weren't its own and invented 12 where there was no marketplace. The channel is now taken from the order tag, and drafts are shown on a separate control line.
• Gifts to bloggers were counted as sales. 264 orders over two years had a 100% discount set by hand. No money came in, but the status said "paid", so these orders were counted in the figures for every state and every country. Paid and free orders are now separate. A gift that cost nothing from the start is told apart from a sale that dropped to zero after a refund.
• Shopify stores a refund in two forms, and neither covers every case. Only one was counted, and annual revenue came out about 2% too high. Switching to the other fixed the US and Canada but broke sales abroad, where shipping and tax are part of the payment. The solution works order by order: if there is a payment record, use it; if not, use the line items. A control query confirmed it: sales abroad matched to the cent for both years, and the year total within 0.1%.
• The label "full year" covered 7.6 months. Shopify accepts an end date in the future without an error. Next to the complete previous year, growth of a third looked like a 17% drop. The current year now ends at today's date.

27 new tests pin down exactly these findings; the full run is 1,312 passing tests.

What changed day to day

The owner asks about sales in the messenger and gets live figures from both stores: countries, years, the top list by money and by units, states and channels. The state report reached him as a CSV file and as a bot answer, both built from the same numbers. Questions without a figure collect in the log, and new metrics are added according to real demand.