A bot that answers with sales numbers: six metrics and an adversarial audit of the arithmetic
The owner of a US outdoor-gear brand asked his Telegram bot about sales by country over two years and got no numbers back. The bot now answers with live figures from both Shopify stores: countries, year windows, top products by revenue and by units, US states broken down by sales channel. Every calculation runs through a single metrics core. An hour after release, four independent reviewing agents found four errors, and each one changed the meaning of an answer.
Live access to the platform was there, the numbers were missing
A US outdoor-gear brand, two Shopify stores, an owner who asks the bot questions in the messenger between other things. By then the bot already had a knowledge base over the team's correspondence and a thin live layer over the commerce platform with two metrics: revenue for a period and the split between core products and accessories.
In August the owner asked: how much did we sell in 2025 and 2026, and to which countries besides the US and Canada. The answer came from the knowledge base — text, with no figures in it. A check showed the live access itself worked: "revenue for 30 days" came back at once. Geography simply did not exist in the metric set, and the root sat deeper: the reporting module never requested the shipping address from the platform. Country did not exist in our data at all, even though the API returns it.
A second finding from the same week: asked "what are the top products by revenue", the bot replied with the store's total revenue — the word "revenue" captured the intent first. The reply looked confident and answered a different question. That kind of answer is more dangerous than silence: the owner takes it as the answer and walks off to make a decision with it.
Six metrics through one core — the language model stays out of the numbers loop
A question from the messenger is matched by keywords, lands in a fixed set of metrics, and is computed by code through a single analytics core — the same core that builds the official store report. A figure is either computed deterministically or absent: an unrecognized question goes to the knowledge base, as before.
The owner writes to the bot in Russian or English: “top products for 30 days”, “sales by state for 2026”
Keywords and an explicit order of checks: states before countries, top products before revenue, years before days
Separate modules for countries, top products, and states by channel; the window, statuses and refunds come from the core
One countable-order filter, refunds per line item, discounts by allocation, the store's time zone — one arithmetic for the bot and the report
Figures print as money and units; a question left without a live figure goes to a log with no name and no chat id
Why there is no language model in the numbers loop
The whole point of the loop is that people trust the numbers. A model with free access to the API will sooner or later produce a plausible figure out of thin air, and one such case in a conversation with the owner wipes out trust in every other report. So the decision is made by code: keywords in two languages, a fixed set of metrics, a call into the core. On an unrecognized question the layer returns nothing, and the bot answers from the knowledge base. Silence here is more honest than a guess: recall was deliberately traded for precision.
So that the metric set grows with real demand, numeric questions that got no live figure are written to a log: only the question text after secret redaction, with no name, username or chat id. A two-week summary shows what the bot is missing.
Three kinds of failure, told apart on purpose
The module is absent — the layer is off, the bot answers from the knowledge base. The module is present and the platform is unreachable — an honest "could not fetch live data". The contract with the core has drifted — a visible message as well. A silent failure had already killed this feature once: the layer was written against an imagined contract, quietly fell through to the knowledge base, and looked like it worked. A visible error gets fixed; an invisible one lives for years.
Geography: an order without an address stays in the total
Country comes from the shipping address — where the goods actually went. Orders without an address (digital goods, pickup) count toward the total and are attributed to no country: otherwise they would silently drift into the US and distort the picture. Years are parsed before days, so "for 2025" yields the calendar year, and "besides the US and Canada" subtracts two countries from the output. The owner's question is now answered word for word: orders, revenue and the number of countries for each year, with overseas on its own line. In 2025 orders shipped to 21 countries, 19 of them outside the US and Canada.
Top products: money and units print together
On this business the two metrics diverge: accessories bring about 22% of the money and about 63% of the units. A single column would push toward a false conclusion about the range, so every item prints both shares, plus a separate block for "many units, outside the money top". The aggregation key is the product id from the catalog: a product's title drifts, and the SKU is empty on roughly 87% of line items. The top-products check was moved ahead of the revenue check — the first of two fixes to the matching order.
US states by sales channel
The owner's request: net sales by state, total and marketplace, minus refunds and discounts. The base for "net" is line items after discounts minus refunded line items, with shipping and taxes excluded. The core verifies that same invariant, so the state totals are bound to match the overall report. The sales channel was absent from the core's query until then: order source and tags were added to the requested fields, and the cache version was bumped. The owner gets a CSV and a bot answer assembled from the same figures.
The question "by US state" contains "US" and, without a priority, drifted into the country breakdown — the second fix to the matching order: states are checked before geography.
The arithmetic lives in one place, collectors only aggregate
Intent matching, collectors, the metrics core and answer delivery — one language across the loop
Orders with line items, refunds, shipping address, source and channel tags; read-only access
One countable-order filter, refunds per line item, discounts by allocation, the window and the store's time zone
Russian and English, an explicit priority order: states → countries, top → revenue, years → days
Question and answer in direct messages; the same bot answers from the knowledge base
Numeric questions without a live figure, text after redaction, no identity; a 14-day summary
Four independent reviews of fresh code an hour after release; five lenses before the state report
Refunds reconciled against the platform's net payment: overseas to the cent, the year total within 0.1%
5 guard tests on the arithmetic, 27 on the state breakdown, full run — 1,312
Five Python modules — intent matching, collectors for countries, top products and states, the demand log — about 1,200 lines, plus about 600 lines of tests. Each collector fits in 150–300 lines because it borrows its arithmetic from the core.
Four agents an hour after release, five independent lenses before the state report. Reconciliation on a live 30-day window: 324 orders in all three reports, zero gap in product revenue. Full test run — 1,312 green.
Four errors in an hour and zero discrepancies after
countries, year windows, top by money and by units, states × channels
reconciled on a live 30-day window: 324 orders in all three reports
100% manual discount over two years: giveaways separated from sales
An hour after release: four agents, four errors
Four independent reviewing agents took the fresh code apart. The findings:
- The top-products collector did not subtract refunds at all — product revenue overstated by 8.54%.
- The collector had its own set of payment statuses: two of the core's statuses silently dropped out. Hence 324 orders in the report against 323 in the metrics.
- Aggregation ran on a product-type field that does not exist on line items — the type lives in the catalog. The type column had always been empty.
- Geography counted a voided payment authorization as a refund. Such amounts were zero at the time; the bug would have slept until the first voided authorization.
All four are one class: the collector did its own math. Now every calculation is a call into the core: one countable-order filter, refunds per line item, discounts by allocation. Geography is served from the same request — the shipping address was added to the core's fields and the cache key version was bumped, otherwise stale data without that field would have been read for six hours. Reconciliation after the fixes on a live 30-day window: 324 orders in the report, in geography and in the top; product-revenue gap 0.00.
Five guard tests on top of the findings
A figure computed almost the same way looks fine and slips past the eye. So the match against the single source is pinned down by tests: geography and the top count exactly the same orders as the core; a fully refunded order adds no revenue; a voided authorization is no refund; and the source code is parsed to confirm that no collector carries its own payment-status filter. A second filter breaks CI.
Before the state report: four more findings
The owner asked for the state report in the evening and had it the same evening. Before the figures went out, they passed through five independent lenses:
- Draft and marketplace are different flags. The wording "marketplace orders are marked as drafts" invited a filter on the draft flag. A seven-week measurement: in one store, 81 of 106 drafts belonged to the marketplace; in the other, 12 drafts and zero marketplace orders. A draft filter would have credited the marketplace with 25 orders that were someone else's and invented 12 where it has none. The channel is determined by the order tag; the draft flag prints as a separate control line.
- Blogger giveaways were sitting inside sales. 264 orders over two years with a 100% manual discount: no payment was taken, the status read "paid", and the countable-order filter let them through — into the counter of every state and every country. Paid and free are now separate; a gift that was zero from birth is told apart from a sale that became zero after a refund.
- A refund lives in the platform in two representations, and neither covers everything. One was counted — annual revenue was overstated by about 2%. Switching to the other fixed the US and Canada and broke overseas, where shipping and tax are part of the transaction. The rule is per order: if there is a transaction, take it; if there is none, take the line items. Reconciled against a control query: overseas matched to the cent for both years, the year total within 0.1%.
- The "full year" label sat on 7.6 months. The platform accepts a future window boundary without an error, and next to a complete previous year, growth by a third read as a 17% decline. The end of the current year is now clamped to today's date.
27 new tests pin down exactly these findings; the full run is 1,312 green.
What changed day to day
A sales question goes to the bot in the messenger and comes back as live figures from both stores: countries, years, the top by money and by units, states and channels. The state report went to the owner as a CSV and as a bot reply built from the same numbers. Questions that got no figure accumulate in the log, and the next metrics are added by real demand.
Where else the same methodology applies
Behind this case sits the standard pain of "three reports give three different revenues". It exists wherever the same figure is computed in several places: in a bot, on an analytics page, in an export for the accountant.
- → Online stores on any platform — the owner's question in a messenger is answered with a figure from the API, computed by the same code as the official report
- → Multichannel sales and marketplaces — the channel is determined by a flag verified by measurement; drafts and giveaways are kept apart from sales
- → AI assistants with access to a system of record — the model phrases and searches documents, code does the arithmetic; an unrecognized question lands in the demand log
- → Executive summaries — refunds, cancellations and discounts are subtracted in one place, and the period label matches the data inside
- → Teams with several reports — guard tests keep every collector on the single core, and a discrepancy is caught in CI
Which channel brings the orders is a question for end-to-end attribution, which lives around the click path. The question here is different: how much was sold, to whom and where, and whether that figure can be trusted.
- Keyword intent matching with an explicit priority order and tests against interception
- A single metrics core: the countable-order filter, refunds, discounts and the window in one place; collectors only aggregate
- A guard test that parses the source code and forbids a collector from carrying its own status filter
- An adversarial review of fresh code by several independent agents before figures reach the client
- A log of unanswered questions as the plan for the next metrics
If three reports in your company give three different revenues and nobody knows which one to trust — that's solvable
We start with one metrics core and one reconciliation against the official report, down to the cent. A bot with live figures arrives once the arithmetic matches, and every next metric lands on the same core.
Related cases
AI assistant over internal data: a RAG bot in Telegram
BM25 → SQLite FTS5 with a custom ranker, voice transcription, OCR, access tiers and an MCP server over the…
Autonomous bots and backtesting: reliability engineering
Ten services deciding off an external event stream. Data arrives over WebSocket the moment an event fires…
Selective traffic routing through a self-hosted VPS
Static routes by AS ranges, targeted DNS bindings, configuration automated over the router's HTTP API, config…
The 9,900 ₽ audit — with a concrete report and quote
I'll tell you what to deploy in your business first, what the payback looks like, and whether you need AI for the task at all (sometimes you don't).
Or just send your question — I reply within 2 hours