Search that stopped guessing.

01 — The problem

The marketplace matches pharmaceutical and healthcare clients to specialist freelancers: medical writers, regulatory writers, market-access consultants, creative directors and dozens of narrower roles. Each brief arrives in the client's own language. The same job is a "core visual aid" to one recruiter and a "detail aid" to another; a "payor" specialist and a "market access" specialist are frequently the same person.

Keyword search could not bridge that vocabulary gap. Good candidates were missed because a single word differed, and irrelevant ones surfaced because a generic term matched. An earlier LLM-based prototype from another team was more accurate but took 30 to 60 seconds per query, failed on one query in five, and cost money on every call. Recruiters had stopped trusting the results and were searching by hand.

02 — What we did

• Embedding retrieval. Every profile is embedded with a large open-weight model and served from a vector index with filter paths for therapy area, work area, role, country and experience, so hard constraints are applied before ranking rather than after.
• Cross-encoder reranking on GPU. The top candidates are rescored by a reranker running in fp16 on a single GPU, with a lock to keep concurrent requests from starving each other.
• A domain vocabulary layer. We compiled the industry's taxonomy of roles and aliases into a query-expansion and canonicalisation module, so "CVA", "detail aid" and "core visual aid" now produce identical top-ten results. Every alias is versioned and each release is checked against the previous one.
• A fine-tuned role classifier. A RoBERTa model trained on labelled profiles tags each freelancer against a 106-role taxonomy. On the validation set it beat a general-purpose LLM on macro F1 while producing zero invented role names, which the LLM could not manage even with a closed-world prompt.
• Query understanding. Typo-tolerant filter extraction, query rewriting and hypothetical-document expansion, all run locally. No third-party LLM is called at query time.

03 — How we proved it

Accuracy claims on search are cheap. We built the evidence layer first and treated it as the release gate.

• A generated regression suite of 328 fixtures covering real briefs, alias pairs, filter edge cases and geography. It runs on the production box before and after every deploy, and a red gate blocks the release.
• Replay proofs. Every change is replayed against a fixed set of 75 real briefs. When the vocabulary layer went live, 71 of 75 rankings were byte-identical and the four that moved were the four that were meant to move, each with a written explanation.
• Independent judging. Roughly 330 query-and-profile pairs were rated by an LLM judge at build time only, on a 0 to 2 relevance scale. Against the earlier prototype, the new engine won six of eight ranking metrics, including a 22% higher NDCG@10 and 64% higher recall at ten.
• Founder-led live audits. The client's founder tested every release on live data and issued rulings on disputed results. Each ruling became a gate, so the system could not regress on a decision that had already been made.

04 — The outcome

Exact-match accuracy on the client's own benchmark of real briefs rose from 46.5% to 89%. Median query time fell from tens of seconds to about two and a half seconds, with no failed queries in the benchmark and no per-query LLM spend. The evaluation harness now runs as part of normal operation, so the client can change vocabulary, retrain a model or swap a component and know within minutes whether search got better or worse.

The engine has since been extended with output provenance, so every candidate card shows where each claim came from, and a phase-two measurement suite is in progress.

05 — What this means for you

If your product already has an AI feature that nobody fully trusts, the fastest fix is usually not a bigger model. It is an evaluation layer that turns "it seems better" into a number, and a gate that stops regressions from shipping. That is the shape of our AI Reliability Audit, and it is where most of our AI engagements start.