A store's own AI sales assistant beat a paid chat widget 34 : 12 in a blind test

The facts come from the catalogue and from real conversations

The base holds 562 facts: product specs, FAQ, shipping and warranty terms. The most useful source was 335 real conversations between the sales team and customers, 131 of them about stock. From them I took the answers and also the way people actually ask about size, weight, winter use and delivery times. A bot trained on marketing copy doesn't recognise questions put that way.

One gateway for three language models

The gateway sits on top of the search and streams the answer (SSE), so the customer reads it as it's written. Behind one interface sit three providers. YandexGPT answers customers on the live site. GigaChat and DeepSeek are there for comparison tests and cheap experiments. Switching models doesn't touch the chat window. The routine but essential parts all live in the gateway:

• access-token refresh and retries that respect Retry-After
• one request at a time per session, so a single user can't start five answers at once
• a spending cut-off, so abuse can't run up the model bill
• phone numbers and emails are masked before logging, so they don't reach the conversation files
• protection from bots with Cloudflare Turnstile
• voice input: a customer can simply say the question

Product cards arrive right after the answer

The text goes out first, so the customer can start reading at once. Right after it, the gateway sends a separate message with cards for the products mentioned: photo, name and link. The advice stays easy to read, and the customer can go straight from the chat to the cart. Once the conversation gets specific, the chat also saves the customer's contact and passes it to a person.

Product cards got their photos back

For a while the cards under the bot's answers showed a placeholder letter where the product photo belonged. There were two causes at once. The daily catalogue update overwrote the file with records that had no image field, and the fixed list of image links had gone out of date after the site's links changed. Now the update takes the image straight from the store's API response, the first photo in the gallery, and checks its host the same way it checks links. Each run logs a "with photo: N of M" count, and a test on the index keeps the fix in place.

The request reaches sales with the whole conversation

A manager calls the customer back using the contact request, so the conversation behind it has to travel with the request. It used to live only in a log that was regularly overwritten. By the time a real request with a phone number came in, the conversation was already gone. Now the request carries the whole session. The two are joined by session ID, and the report labels the conversation with the name the customer gave. If the text had to be shortened, the report says so: "showing 2 of 5 turns, 3 more didn't fit".

Quality is checked by a test

The first quality check used a second model to grade the answers. I dropped it. It kept missing failures on exactly the category questions the sales team had complained about, and it passed any answer that sounded coherent. Now a fixed test does the job: for each question we know which facts must appear in the answer, and the check looks for them. On top of that there are 134 pytest tests, including grounding (every claim in an answer traces back to the fact base) and abuse protection.

The check for gaps in the knowledge base stopped marking correct answers as "evasive". The chat is not allowed to quote shipping costs or instalment rates; a manager calculates them. Such answers follow the store's own policy, yet the old rules would have sent them for repair. Now only "no data" and "not specified" count as a real gap, and "a manager will calculate it" is checked against the topic of the question. The "evasive" section is gone from the daily report; "weight not specified" still counts as a gap.

The tests needed work too. Seven of them failed with a type error, so they checked nothing. The test stand-in for the model still took two arguments after the working code had gained a third, for token-spend accounting. That left the captcha, the daily budget and replay protection untested. The same review found that one test run used up 810 of the 1,000 daily calls meant for site visitors. Local checks now have a separate counter.

Hard rules in code on top of the model

The three weeks after launch went into answer accuracy and data integrity. The store sent error reports as text and as voice notes; the voice notes are transcribed automatically and join the same queue. Two of the errors could have cost a customer money. Asked about an inflatable tent, the bot first suggested a weak 2 PSI pump that can't inflate it. It also said with confidence that a clear end wall comes in the box. It had confused the wall with the windows made of the same material, listed one line above in the specs.

Both are now fixed in code. The weak pump is filtered out of tent answers, and in the general list of pumps it comes with a note on what it's for. The end wall now has a lookup table for the three models with data on file. For every other model the bot gives one fixed answer: "no data on file, a manager will confirm".

The lesson for the next project: editing the knowledge base alone didn't help here. The model kept inferring "included" from the neighbouring spec line, as it had earlier with the stove safety rules. Where a wrong answer costs the customer money, the answer is fixed in code.

The index now contains every file it should

Six knowledge-base files were written specifically to answer the store's complaints: stock and dispatch, how to give advice, what's in the box, compatibility, use and care, delivery and payment. They were missing from the list of allowed sources, and the build skipped them without a warning. However many times the index was rebuilt, the bot knew nothing from those six files. They are now on the list.

One limitation remains: the index is stored outside the repository, and rebuilding it takes a person and a cloud key. Changes to the knowledge base reach the live chat only after that rebuild.

A step-by-step rollout on a live site

The site runs on a website builder, and every edit goes live at once. So I rolled the chat out in steps. First, a backup. The chat code was added to the end of the page's HEAD, and I checked that the existing markup stayed unchanged. Then one home page, a check on the live site, and only then the remaining pages, about sixty of them. The rollback takes three steps and is written down next to the instructions.