↓Skip to main content

Matchworks reply system Feb – Nov 2026 Operator who built it

An inbox that answered itself

Matchworks got its prospects answered in a median 5.6 minutes, down from 27.1. Nobody had to read routine replies before they went out any more: 6,968 of 8,820 were sent with no person first.

AI agentsPythonAutomationRaycast

This case is on the role pages for Go-to-market and outbound ops, Operations lead, Automation and AI, Marketing, Support.

27.1 → 5.6 median minutes to reply, July to October 2026
8,820 every reply sent from the inbox, hand-sent included, 22 Jul to 2 Nov; the system’s own sends were 6,968
$1.72 → $0.92 AI cost per lead after the rebuild
“Thanks for all your hard work this week and appreciate you jumping right into the new system and rocking.”
Mike, who runs Matchworks, after my first week moving the inbox onto new systems, Jul 2026

From a prospect’s reply to a graded send

  1. 01 · Inbox Prospect replies A reply to a cold email starts the clock
  2. 02 · Agent Read and classify Full thread read, funnel stage worked out
  3. 03 · Agent Draft A reply in the brand’s voice; the AI only drafts
  4. 04 · Sender Checks Run by a separate program on the draft as posted
  5. 05 · Sender Auto-send Sent with an audit row when no blocking check fails
  6. 06 · Operators Held drafts Refused drafts carry a written reason
  7. 07 · Me Review after sending Sent replies graded, and the notes fed back into the drafts
  8. 08 · Me Cost and quality Measured rebuilds, from $1.72 to $0.92 a lead

Automation or AI Done by a person

Built with: Python, Claude, Better Stack monitoring, Raycast and TypeScript. I wrote the pipeline and the operator console. I also helped move human review into Langfuse, a tool where people grade the AI’s replies; Mike, who runs Matchworks, and Liam built that with me.

The problem

Matchworks is an outbound lead-generation agency, and it runs on email replies. A prospect answers a cold email, and the clock starts. On 28 July 2026 at 10:02 pm Mike wrote “we have analtyics showing < 15m response time conversion goes way up”, then “is it possible to get it super fast?” In July the median wait was 27.1 minutes, because every draft waited for me to read it and press send. Booking data was never wired into the system, so the result below shows speed, not conversion. I joined the two once, on 27 September, to test the 15-minute rule itself.

Volume reached up to about 1,000 replies a week. One person reading every draft could not keep that under 15 minutes, so I built to the rule first and tested it later.

What I built

Planning began in February 2026, and from 29 July the system answered prospects on its own. It read the full thread, worked out how ready the prospect was, drafted a reply in the brand’s voice, and sent it on its own when its checks passed. When they didn’t, it held the draft with a written reason a person could act on in seconds.

Checks before every send. The diagram below is every check, what it caught and when it blocked a reply. Every result was saved for every draft, so “why did this send?” took one lookup to answer.

How one reply goes out on its own Every check runs on every draft. A waived check still runs, and its result is written to the audit row.
01
Reply arrives A prospect answers a cold email. The thread joins the queue.
02
Claude drafts Reads the whole thread, rates how ready the prospect is, writes in the brand’s voice.
03
Checks
guardbanned phrases, broken booking links, and the other brand’s link, sign-off or signatureon
brandthe thread belongs to one brand onlyon
patterna known reply patternon to 6 Aug, then waived and recorded; removed 20 Aug
flagthe drafter’s own doubton to 6 Aug, then waived and recorded
recipientthe draft’s note says wrong recipientalways, from 2 Oct
holdthe AI marks it to hold, such as a declinealways, from 8 Oct
Not a check: an empty draft never sends.
04
Send cap From 7 August, at most 100 sends an hour. Zero stops every send.
05
Sent and logged The reply goes out and its audit row is written.
Any blocking check failing: the draft is held with its reason in writing. An operator reads the reason first and decides in seconds. Later: a person reviews the sent reply and grades it good, acceptable or wrong. The grade is saved on the same row.
Audit row One auto-sent reply, sample data
Prospect
Sam Sample · trailnotes.example
Brand
Brand A
Sent
3 Sep, 4:12 am
Guard
pass
One brand
pass
Flag
waived: “asked twice”
Review
good
How one reply went out on its own, and its audit row. Sample data.

Held drafts explained themselves. A draft the checks refused got a written reason: the wrong brand’s sign-off, a thread that crossed brands, or template phrases that needed a person’s eye. The reviewer read the reason first, so a review took seconds.

Tested before every change. The system sent email at 4:00 am with nobody watching. By its last change on 24 October, 170 automated tests covered the send path alone, every branch that decides whether a prospect gets one email, two or none, and they were how I changed it safely.

A console for the operators. A Raycast console (Raycast is a Mac launcher, like a command bar) gave operators who aren’t engineers the reply queue, each thread’s history and bulk fixes, so they could run the system without me.

The Raycast console with the reply queue open. Names covered.
The Raycast console with the reply queue open. Names covered.

Decisions

Intent before drafting

In February 2026 I proposed an intent step before every draft. The AI would offer two or three readings of what the prospect wants, each with a confidence score, and a person would pick one. The system I shipped in July dropped the person’s pick: it worked out each thread’s intent on its own, then drafted the reply.

Send first, review after

On 28 July I was the bottleneck, because I still read every draft before it went out. Weeks of my feedback had already shaped the drafts, so I switched to sending first and grading what went out afterwards, and auto-send went on the next morning. The risk was real. A reply nobody had read could reach a prospect, and I turned down a few days of test runs against hand-sent replies so Mike would see the number that week. Three limits held it. The checks ran on every draft, every refused draft waited in a held queue for a person with its reason in writing, and I graded what went out. From 7 August a cap of 100 sends an hour also stopped a loop or a bad update from sending in bulk. In the first week, 29 July to 4 August, at least 98 replies went out on their own (the send log missed some on the first two days). I read 31 of them one at a time and marked 1 wrong, a referral reply that went only to the original sender. The other 67 were never read one at a time.

Block on our mistakes, not on the AI’s opinion

All four launch checks blocked a reply at first. By 6 August 247 drafts had waited for a person, and 173 of them were held by the AI’s flag, which is an opinion about the prospect, not a fault in our reply. So I kept the guard and the one-brand check blocking, because they check our own output, and let the pattern and flag checks run without holding a reply. Of the two holds added in October, the wrong-person hold came straight from three replies sent to the wrong person on 1 and 2 October.

Built fast, to gather data

I built it fast to gather data for the next version Mike would build: which replies could go out alone, which needed a person, and why.

The AI drafts, a separate program sends

The AI wrote drafts and had no way to send them. A separate program re-read each draft as it sat in the inbox and ran the checks on that text, not on the AI’s own copy, so what got checked was exactly what went out.

A strict field for the pattern check, and two-brand threads wait

The old pattern field was free text that nothing checked. Gating on it would have auto-sent anywhere from 35% to 77% of drafts, depending on how the AI happened to phrase it. I added a field that takes one name from a fixed list instead. Any thread that touched both brands always waited for a person, because a reply that crosses brands is the one mistake nobody can take back.

Two replies, side by side

Here are two replies from August 2026, one sent on its own on 4 August and one held for a person on 19 August, each followed from the prospect’s email to my grade. The names, brands and domains on all 6 screens are made up; the times and check results come from the 8,820-row send log and the review journal.

1. The prospect writes back

On Tuesday 4 August 2026 at 4:37 am a prospect asked how we found her and who the partners were before she would book. On Wednesday 19 August at 3:07 am another said he was the right person and open to a talk.

Two prospect replies, one per brand. Recreated, original data private.
Two prospect replies, one per brand. Recreated, original data private.

2. Intent read, then the draft

The drafting AI read each thread and named what the prospect wanted. The first was a request for partner details, so the draft answered by category and kept names for the call. The second was a plain first reply, so it got the house template.

What the AI read in each thread and the draft it posted. Recreated, original data private.
What the AI read in each thread and the draft it posted. Recreated, original data private.

3. Every check, on the draft as it sat in the inbox

On 4 August all four checks could still block, and all four passed. On 19 August only the guard and the one-brand check could block, and the one-brand check failed: the thread looked like it touched both brands.

The checks each draft ran, and which ones could block that day. Recreated, original data private.
The checks each draft ran, and which ones could block that day. Recreated, original data private.

4. Sent, or held with its reason

The first reply went out at 4:43 am, 6.6 minutes after the email arrived, with nobody reading it first. The second was held at 3:11 am, and the Raycast menu bar showed the reason, “cross-brand thread”, above the send button.

Left, the send. Right, the held draft in the Raycast menu bar with its reason. Recreated, original data private.
Left, the send. Right, the held draft in the Raycast menu bar with its reason. Recreated, original data private.

5. The audit row

The first send left a row at 4:43 am with its arrival time, send time, sender and all four check results. The held reply was sent from the menu bar at 3:53 am, 46.5 minutes after it arrived, word for word, and its row marks it as sent by a person.

The audit row each reply left. Recreated, original data private.
The audit row each reply left. Recreated, original data private.

6. Graded after sending

I graded the first reply good at 4:48 am, five minutes after it went out, with a note. I graded the second acceptable on 21 August. On 19 and 20 August I pulled all 40 drafts held as “cross-brand thread”: none carried two brands, so I split the reason in two, one for an address the system could not read and one for a real two-brand thread.

The grade on each reply, and what the held one changed. Recreated, original data private.
The grade on each reply, and what the held one changed. Recreated, original data private.

The result

  • Median wait from a prospect’s reply to ours: 27.1 minutes in July 2026, 5.6 minutes in October. The system’s own replies in October took 4.5 minutes (2,027 that month). Most of the drop came from replies no longer waiting for a person to read them before sending.
  • 8,820 replies sent from 22 July to 2 November: 6,968 went out alone, and 1,852 waited for a person.
  • How much people changed those 1,852 drafts is mostly unknown: 1,625 were sent by hand inside the inbox, and their text was never saved. Where both the system’s draft and the sent text survive, for 71 replies, 70 went out word for word and 1 was rewritten. I compared the two texts with spacing ignored.
  • I read and graded 324 sent replies one at a time, all between 3 and 25 August 2026, and marked 6 wrong. Five have a note: a referral reply that went only to the original sender, a reply to a lead already disqualified, one that offered the prospect an option it shouldn’t have, a malformed booking link, and one that pivoted too confidently when the prospect named another company. One has no note. The review journal’s other 2,689 grades were whole-list approvals, one click that cleared a list without reading it, so they are not counted here.
  • After 25 August nothing measured quality reply by reply. The only quality record after that date is three replies addressed to the wrong person, sent on 1 and 2 October. They are why, from 2 October, a draft addressed to the wrong person was always held.
  • Every auto-send carried a full audit row.
  • From 27 October 2026 Mike built the next version in Langfuse on the data this one gathered, the review handbook I wrote on 23 October and my review notes.

How it is measured: the send log records when each prospect’s reply arrived and when ours went out, and marks who sent it. The median counts every reply from every sender, by calendar month in Philippine time; the 4.5 counts only the system’s own.

What the records said

The thread read, June 2026

On 11 June 2026 I pulled every reply thread with a send since 27 May, 4,000+ of them, and read them before writing the reply playbook. About 2,900 were real exchanges once sequence emails and warmup sends were removed. One line, telling prospects we had spoken with possible buyers the day before, went out 250+ times in a week, and two groups of replies walked it back when prospects challenged it. At least ten groups of prospects said the booking form showed no times.

Testing the 15-minute rule, September 2026

On 27 September 2026 I tested Mike’s rule that every lead gets an answer inside 15 minutes. I chose what to test and read the result; Claude agents did the maths. The test matched about 3,600 first answers, from 21 July to 20 September, to booking form submissions within 7 days.

  • Answers under 15 minutes beat slower ones on submissions by 1.9 points, with a 95% margin of 0.0 to 3.6. That margin touches zero, so speed inside 15 minutes is unproven.
  • The loss sat in answers over 6 hours: 7.0 points worse on submissions than answers under 5 minutes.
  • Under 5 minutes never beat 5 to 15 minutes.
  • A follow-up on 3 October found a fair test would need 21 leads a day for 90 days to see a 3.6-point lift.

The 27 September records also showed where the slow answers came from. The sender was down 32% of the 67 days before 27 September. Since 20 September, 37.1 of its 40.8 down-hours were the Mac asleep. A lead-minute is one lead waiting one minute: with the Mac awake, the 90-second check for new replies cost about 1,200 a week, and with the real sleep and outages about 11,800. So the advice from 27 September was to keep the Mac awake or move the sender off it, and to stop chasing speed under 15 minutes in favour of the answers that waited over an hour.

What broke, and the fix

A lead here is one prospect’s reply: one run of the drafting AI (Claude Opus 4.8) that reads the thread and writes a draft. A step is one action in that run, such as reading or searching a file. The run was a loop of steps because it copied the path I used by hand: read the thread, check the business fits, draft against the rules, then post the draft. The first version cost too much per lead and kept hitting the provider’s usage limit mid-shift. I measured where the cost came from before cutting it. It took 17.8 steps because nothing was shared between leads: every run reloaded the full drafting instructions and searched a 480 KB file of past reply decisions three to six times before it could write. I put the rules one draft needs into one file that every run shared, moved the old decisions out of the drafting path, then re-measured on 23 August, 489 runs against the 603 before. Steps per lead fell by almost two thirds and AI cost per lead by almost half:

AI cost per lead

Before, 603 runs $1.72
After, 489 runs $0.92

Steps per lead

Before, 603 runs 17.8
After, 489 runs 6.4
AI cost and steps per lead before and after the rebuild, re-measured 23 August 2026

What I’d change

It was my first time building a system like this. I understood the theory, and now I have the firsthand experience. Next time I’d start on proper infrastructure like Langfuse. I’d build a test set from the hand-sent replies and run it before turning on auto-send, the test I skipped so Mike would see the number that week. And I’d grade every cost test. In October I tested a cheaper setup, where code gathered the thread first and the AI wrote the reply in one prompt; over about 1,080 runs on each side it cost a third less per lead than the loop. Its replies were never graded before Mike’s version took over, so I can’t say it was as good.

Check this

Email me the process you’d hand off first. In my working hours I reply within 30 minutes.
Email me ckwater123@gmail.com LinkedIn