Internal CRM for an ops team: one screen instead of five tabs

Six sources in parallel — and honest degradation

The customer card fans out to six independent sources at once: orders from the first store, orders from the second, shipments, call history from the cloud phone system, post-purchase survey answers, and returns with chargebacks.

The key detail is a thread-local failure counter that gets carried back into the main thread. Without it, a source that fell over would simply not render, and a partial card would silently pass for a complete one. The ops person would read "no orders" where the truth is "the store API did not answer just now". Those are different facts, and you tell the customer different things.

Off someone else's machine: local mirrors instead of an SSH hop

Part of the data used to live on a helper box, and the CRM went there over SSH on every card view. A link flicker lasting exactly one watchdog tick produced the very lie described above — "no orders" on screen while the truth was "the source did not answer".

The purchase-channel database is now mirrored locally every 10 minutes. The snapshot is taken through the source's own backup mechanism: copying the file while it is being written gives a torn snapshot that opens cleanly and lies. Only a readable database of a sensible size is accepted, and the swap is atomic. Orders from both stores now come straight over tokens placed on the CRM server under the same paths — so the same script text runs locally and over SSH, and eight card functions stayed untouched. The direct path counts as ready only when tokens for both stores are present: half the tokens would paint a plausibly truncated picture. When the mirror goes stale the old slow path takes over, and the switch is written to the log.

First production run: a 1.8 MB mirror with 777 purchases, channel completeness 42 of 42 across three days, and the numbers computed from the local mirrors matched the old path byte for byte. Five tests per part, including a guard against query injection; every one of them fails without the fix.

Security of an internal tool

The channel label for tagged sources was assembled from the address a buyer arrived on and went into the markup unfiltered. That means a stranger's code would have executed in the admin's browser — the browser that has customer cards and outgoing mail open at that very moment. Escaping now sits at the markup boundary in all six places.

Session binding got a separate fix: 34 of 38 live sessions survived a password change, 26 of them administrative. The fingerprint was 16 characters, two of which carried the salt — 256 values spread over 4,000 hashes; it is 48 now. Both fixes were verified by mutation, two red tests each.

Indexes instead of streaming the whole corpus

The two mailboxes hold 94,800 parsed messages — a corpus in the hundreds of megabytes. The first version read it end to end on every card view: hence the 5.8 seconds to pull a customer's email history.

I built two indexes:

• Address → line offsets — open the corpus, seek to the offset, read only the messages that matter
• Order number → customer — an index of order numbers extracted from message bodies: the buyer writes from a personal address while the order sits under a work one, and the card is still found

Streaming parser: otherwise OOM-kill

The mail parser originally held the parsed corpus in memory — on the earlier 961 MB box that ended in an OOM-kill. I rewrote it to stream: messages are written to disk one at a time and only a light header projection stays in memory. Peak memory dropped from 985 MB to 256 MB (215 MB on the server itself), with a byte-identical result: the same message and thread counts, and a matching sha256 of the contents.

The knowledge-base corpus now arrives from the server ready-made

The workstation used to pull 34 GB of raw mail to end up with 459 MB of parsed corpus. The server already holds a parsed corpus: it weighs 450 MB and is more complete than the local one — 94,800 parsed messages. The download switch is per-mailbox, so the two mailboxes can move over one at a time.

The important part is the guard against losing history: a file is replaced only when the incoming copy is at least as long as the local one. It earned its keep on the very first production run — it rejected the swap for the second mailbox (8,022 against 7,985) and preserved 34 messages the server was missing. It is the same principle the whole case rests on: an incomplete source has to declare itself incomplete.

The "missed leads" metric was revised at the same time, on a remark from the support operator. 82% of the sample turned out to be machine notifications from the store and the warehouse, and among human messages almost half of the unanswered ones were thank-you notes. A 103 MB index build that ran every 15 minutes with no consumer at all was retired along the way.

Everything needed to never leave the window

• Sending and forwarding with attachments through a queue — the UI never blocks on SMTP
• Per-person templates and signatures
• AI reply draft — built from the thread context and the card data, edited before sending
• Personal accounts with device memory — no re-login every morning
• An audit log of "who / what / how it ended" for every action
• A warehouse stock block pulled from Lark — inside the card, no separate spreadsheet
• Full mobile layout — verified at 390px, replies can go out from a phone
• A "Hypotheses" screen — plan against fact with a history of snapshots; every snapshot goes to the log, and the trend line starts from the second point
• An "Infra & subscriptions" screen — what is paid for, until when, and what a month costs
• A "What customers say" screen — post-purchase survey answers; a guard test goes red if survey text or an email address ever reaches the markup

Six analytics screens on the evening-digest engine

The CRM grew its own analytics: the "Day", "Period", "Products", "Links", "Gaps" and "Hypotheses" screens. Arbitrary date windows, calendar presets, comparison against the previous window, CSV export with a BOM and semicolons so Excel opens the file straight away. Access is limited to the admin role.

The key design decision: the screens compute through the same engine as the evening digest. Two independent counters would produce two truths about one revenue figure, and arguing over which one is real would cost more time than the screens themselves.

Measurements: the first visit took 6.9s and is now instant — the warm-up is paid once, at service restart. The "Products" tab: 7.5s cold, 0.00s from cache. Product totals reconcile with overall revenue to the cent, and the shares inside every filter add up to exactly 100%.

• One accounting path for product metrics — the top-sellers collector used to count on its own and overstated revenue by 8.54%; after moving it onto the shared core, a live 30-day window reconciled at 0.00 across 324 orders. The row key is the product id: SKU is filled in on 12% of rows against 97% for the id
• A "plan against fact" hypothesis register — 7 hypotheses with snapshot history. Three rules are hard-coded: a verdict lands once the window has actually passed, given a 2-3 month purchase cycle; a conclusion waits for enough observations; attribution counts as attribution, and incremental revenue needs its own proof. A guard requires the note to name whoever approved the target
• A single integration trail — the page knew the state of 5 services out of 14. All 9 missing ones are wired in, the status now carries 5 states, and a failure leaves the last success on screen
• Window comparison — it lied on partial overlap: a window measured against a previous one holding only 8 days of data showed 320% revenue growth out of thin air, while the "90 days" preset divided 69 days of revenue by 90 and understated the average day by 23%

Both holes described above — the executable source channel and the session that outlived a password change — surfaced on these very screens, while reading my own code.

Move to a dedicated server — in one day

The CRM initially lived on a shared box alongside the tracker — a neighbourhood where any load spike hits both. I moved it to a dedicated server behind nginx + Let’s Encrypt, with a 15-minute IMAP sync, an hourly monitor and nightly backups. The migration was accepted on a parity check: messages, threads and dashboard orders matched the old installation one for one on every counter.

Operations: the watchdogs moved to the server

Two watchdogs — one for stuck orders, one for disputed charges — moved off a laptop onto systemd timers on the server. On the laptop the order watchdog was dead: processes under launchd have no access to the key, so it exited with an error and logged "the source returned no orders, staying quiet", and that quiet read as "there are no stuck orders".

The first run on the server found three genuinely stuck orders, aged 11, 11 and 5 days. The dispute watchdog saw 32 disputes on its first start and, by design, only recorded the state, sparing everyone an avalanche of alerts about history. That finishes the chargeback story this case opened with: the deadline to contest is now watched by a schedule.

The CRM also gained a page for infrastructure and subscription spend. A service end-date field went in: the service leaves the monthly figure and the reminders, stays in cumulative spend and stops growing it. The reason a block is empty is printed out loud, and a guard stands against a confident zero: if servers exist while the total reads zero, the screen shows a dash with an explanation — invoices get reconciled against this page. Data in this area cannot be reconstructed after the fact, and before the page existed the question "how much did we spend last month" had no answer.