A custom CRM for an ops team: five systems on one screen

The card queries six sources at once and shows which one failed

The customer card queries six independent sources at the same time: orders from the first store, orders from the second, shipments, call history from the cloud phone system, post-purchase survey answers, and returns together with payment disputes.

The key detail is that every source reports its own failure. Otherwise a source that failed would simply vanish from the screen, and an incomplete card would look complete. The ops person would see "no orders" when the store had just failed to respond at that moment. In those two cases you tell the customer different things.

Data from the helper server now sits next to the CRM

The CRM used to pull part of its data from a helper server, connecting over SSH every time a card was opened. A brief drop in the connection at that moment caused exactly the error described above: "no orders" on screen when the source had simply not answered.

Now a copy of the database that records where each buyer came from lives on the CRM server and is refreshed every 10 minutes. The snapshot is made by the source's own backup tool. If you simply copy a file while something is writing to it, you get a broken copy: it opens fine but shows wrong data. So a copy is accepted only if the database can be read and its size looks plausible, and the old copy is replaced in a single step.

Orders from both stores now come straight from Shopify. The access keys sit on the CRM server under the same paths as on the helper server, so the same script runs both locally and over SSH, and eight card functions did not need rewriting. The direct path turns on only when keys for both stores are present: with a key for just one store, the picture would look plausible but be incomplete. If the copy goes stale, the CRM falls back to the old, slower path and writes that to the log.

First live run: a 1.8 MB copy with 777 purchases, and over three days the source was known for 42 of 42 purchases. Figures computed from the local copies matched those from the old path byte for byte. Each part has five tests, including one that guards against injected code in database queries. All of them failed before the fix.

An internal tool needs protection too

The name of the source a buyer came from was taken from the link they arrived on and put into the page without any filtering. That meant an outsider could plant code in the link, and it would run in the admin's browser, the same browser where customer cards and outgoing mail are open. Special characters are now escaped in all six places where that name reaches the page.

The second fix concerned sessions. After a password change, 34 of 38 open sessions kept working, 26 of them admin sessions. The session fingerprint was 16 characters long, and only two of them came from the salt (a random addition), so 4,000 hashes shared just 256 possible values. The fingerprint is now 48 characters. Both fixes were checked with mutation testing: each has two tests that fail if the fix is removed.

Indexes let the CRM read only the messages it needs

The two mailboxes hold more than 100,000 messages, and the archive takes up hundreds of megabytes. The first version read all of it every time a card was opened, which is why finding a customer's emails took 5.8 seconds.

I built two indexes:

• Address → position in the archive. The CRM opens the archive, jumps straight to that position and reads only that customer's messages
• Order number → customer. Order numbers are collected from message bodies. If a buyer writes from a personal address but placed the order with a work one, the card is still found

Mail is parsed one message at a time, so memory doesn't run out

At first the parser kept all parsed messages in memory. On the earlier server with 961 MB, the system killed the process when memory ran out. I rewrote the parser so that messages are written to disk one at a time, and only a short summary of the headers stays in memory. Peak memory dropped from 985 MB to 256 MB (215 MB on the server itself). The result matched byte for byte: the same message and thread counts and the same sha256 checksum of the contents.

The knowledge base now gets mail from the server already parsed

The work computer used to download 34 GB of raw mail to get a 459 MB parsed archive. Yet the server already has such an archive. It takes 450 MB and on the whole is more complete than the local one. The switch is made separately for each mailbox, so the two mailboxes can move over one at a time.

The key part is a guard against losing history: a file is replaced only if the new copy is at least as long as the local one. It worked on the very first live run and blocked the swap for the second mailbox: 8,022 messages locally against 7,985 on the server. The messages the server was missing were kept. The whole case rests on this rule: an incomplete source must say that it is incomplete.

At the same time, after a comment from the support operator, I revised the "missed inquiries" metric. It turned out that 82% of the sample were automatic notifications from the store and the warehouse. Among messages from real people, almost half of the unanswered ones were thank-you notes. I also switched off a 103 MB index build that ran every 15 minutes although nothing used it.

Everything the team needs, in one window

• Sending and forwarding with attachments through a queue, so the interface doesn't freeze while a message goes out
• Each person has their own templates and signatures
• An AI draft of the reply based on the whole thread and the card data; you edit it before sending
• Personal accounts that remember the device, so nobody has to log in again every morning
• An action log: for every action you can see who did it, what they did and how it ended
• Warehouse stock from Lark spreadsheets, right inside the card
• A mobile layout, checked on a 390-pixel-wide screen: you can reply from your phone
• A "Hypotheses" screen: plan versus actual with a history of snapshots. Every snapshot is logged, and the trend line appears from the second point
• An "Infra & subscriptions" screen: what is paid for, until what date, and what a month costs
• A "What customers say" screen with post-purchase survey results. A separate test fails if the text of a customer's answer or their email address ever reaches the page

Six analytics screens count the same way as the evening report

The CRM now has its own analytics: the "Day", "Period", "Products", "Links", "Gaps" and "Hypotheses" screens. You can pick any dates or a ready-made calendar period, compare it with the previous one and export a CSV file that Excel opens straight away. Only admins can see these screens.

The main decision here: the screens use the same code as the evening report. Two separate calculations would give two different revenue figures, and arguing over which one is right would take more time than the screens themselves.

Measurements: the first visit used to take 6.9s. Now the screen opens instantly, because the data is loaded once, when the service restarts. The "Products" tab takes 7.5s to build from scratch and 0.00s from cache. Product totals match overall revenue to the cent, and the shares in every filter add up to exactly 100%.

• One calculation for products. The module that built the best-sellers list did its own math and overstated revenue by 8.54%. After it moved onto the shared core, a 30-day check on live data showed a difference of 0.00 across 324 orders. Rows are matched by product ID: the SKU is filled in on only 12% of rows, the ID on 97%
• A "plan versus actual" list of hypotheses. It holds 7 hypotheses with a history of snapshots. Three rules are built into the code. A verdict comes only after the period ends, because customers here take 2–3 months to decide on a purchase. A conclusion is drawn only when there are enough observations. A sale that came through a source is credited to it, but calling that revenue "additional" needs separate proof. One more check requires the note to name who approved the target
• Integration status. The page knew about 5 services out of 14. All 9 missing ones are now connected, and each has 5 possible states. If a service fails to respond, its last successful result stays on screen
• Period comparison. When two periods only partly overlapped, the comparison was wrong. A previous period with just 8 days of data produced 320% revenue growth out of nowhere. The "90 days" preset divided 69 days of revenue by 90 and understated the average day by 23%

I found both vulnerabilities described above, the code in the source name and the sessions that survived a password change, while building these screens and reviewing my own code.

The move to a dedicated server took one day

At first the CRM ran on a shared server together with the ad tracker, and any spike in load hit both. I moved it to a dedicated server behind nginx with Let’s Encrypt certificates. Mail is checked every 15 minutes, monitoring runs every hour, and backups are made every night. The move was checked against the counters: messages, threads and orders on screen matched the old installation exactly.

The watchdog scripts moved to the server

Two watchdog scripts keep an eye on stuck orders and disputed charges. They moved from a laptop to the server, where systemd timers run them. On the laptop, the order watchdog was not actually working: background jobs on macOS (launchd) had no access to the key. The script exited with an error and logged "the source returned no orders, staying quiet", and that silence was taken to mean "no stuck orders".

The very first run on the server found three orders that really were stuck: for 11, 11 and 5 days. On its first start, the dispute watchdog saw 32 disputes and, as designed, only recorded them without sending alerts about old history. This continues the chargeback story from the start of the case: a schedule now tracks the deadline for responding to a dispute.

The CRM also got a page for server and subscription costs. Each service has an end date. Once a service is switched off, it drops out of the monthly total and the reminders; it remains visible with a zero contribution to the running total. If a block is empty, the screen says why. There is also a guard against a false zero: if there are servers but the total is zero, the screen shows a dash with an explanation, because invoices are checked against this page. This data cannot be rebuilt after the fact, and before the page existed nobody could say how much had been spent last month.