What eight years of production NLP taught me about LLM agents

In late 2018, a few months after Google released BERT, my team at Spotify started fine-tuning it on user text — support tickets, emails, social media — and clustering it into intents. That system became company-wide infrastructure and, as far as I know, is still in production. In 2026 I build LLM agent systems with client teams. The models are unrecognizable. The list of things that decide whether the system survives is almost unchanged.

2018: when "language model" meant 340 million parameters

The intent model was a classic of its era. Four of us took BERT — brand new, exciting, badly understood — and put it to work on a real problem: Spotify received enormous amounts of user text, and nobody could read all of it. We fine-tuned, clustered messages into intents (hacked account, cancellation, playback problems, and hundreds more), indexed the corpus so any team could search it, and watched intent spikes become an early-warning system for incidents.

Here is the part people forget about that generation of NLP: the model was the smallest line item. The work was everything around it — getting labeled data, deciding what the taxonomy of intents even was, serving predictions reliably, noticing when language drifted and the model quietly got worse, and convincing other teams to build on top of it. Fine-tuning was weeks; the system was years.

Keep that ratio in mind. It's the thesis of this post.

2022–2024: ML with revenue on the line

At SeenThis I built the data platform end to end — real-time ingestion from the edge at around 10,000 events per second — and then the ML on top of it: models tuning a streaming algorithm and forecasting the impression volumes the company priced its product on. When your model's output feeds pricing, you learn a specific kind of humility. Nobody asks how novel the architecture is. They ask whether the number was right, and what you'll do when it isn't.

That period taught me the operational half of the job: monitoring is a feature, not an afterthought; a forecast that can't explain itself gets overridden by a spreadsheet; and the failure you didn't enumerate is the one that pages you.

2024–2026: LLMs as the product

At Saltfish — the startup I co-founded and ran as CTO until our exit in 2026 — LLMs weren't a feature, they were the factory. The platform turned one capture of a software product into interactive demos, videos and documentation, with generation pipelines spanning multiple models and providers, and an MCP integration so customers could drive the product from their own AI assistants. Two engineers, production responsibility, paying customers.

Running multi-provider LLM pipelines in production for two years compresses a lot of opinions into you. The main one: model progress is real and wonderful and it will not save you. Every capability jump moved our floor up — and left every hard problem (evals, integration, failure handling, cost control, trust) exactly where it was.

What transfers from 2018 to agents, almost embarrassingly directly

1. The model is still the smallest part of the system

In 2018 the ratio of "surrounding system" to "model work" was maybe 10:1. With agents in 2026 it feels higher, not lower. Tool design, context management, permissions, evals, monitoring, the integration into the workflow where the agent lives — that's the project. Teams that evaluate agents by comparing frontier models in a chat window are measuring the 5%.

2. Evals are the new labeled dataset

In 2018 nothing happened until we had labels. In 2026 nothing should happen until you have an eval suite: real tasks from your real workflow, with graded answers. It's the same discipline wearing new clothes — and it's still the difference between engineering and vibes. When I build an agent with a client team, the eval suite is the first artifact we create and the last thing I hand over.

3. Intent classification didn't die — it became routing

The unglamorous 2018 problem — "what is this text actually about?" — is now the router at the front of every serious agent system: which tool, which workflow, which cases to escalate to a human. If your agent misroutes, everything downstream is garbage. The old NLP instincts (taxonomy design, confusion analysis, thresholds for "I don't know") apply verbatim.

4. Drift is eternal

Language drifted under our intent model; the world drifts under your agent. New products, renamed fields, a partner that changes its reporting format. The 2018 answer is still the answer: monitor outputs, sample failures into your eval set, and treat "quietly getting worse" as the default trajectory that engineering effort must fight.

5. Production is a place, not a milestone

The intent model survived eight years because it became infrastructure other teams depended on — searchable, reliable, boring. Demos don't survive; dependencies do. An agent survives the same way: embedded in a workflow, cheaper to use than to bypass, owned by a team that can fix it at 3 a.m. That's why I insist on building with the client's team rather than shipping them a black box.

What's genuinely new: the model acts

One thing did change fundamentally. In 2018, the worst case was a wrong label. In 2026, an agent's worst case is a wrong action. That asymmetry should drive the whole design: read-only tools first, narrow permissions, human sign-off where actions are irreversible, and guardrails treated as product features rather than compliance chores. The teams that get this right earn the license to automate more; the teams that skip it generate the horror stories that set everyone else's programs back a year.

Eight years in, my summary is short. The models finally got good enough to be genuinely useful for almost any company. The work that makes them useful is the same work it always was. Do the boring parts, in the right order, with the people who'll own the system — and the shiny part takes care of itself.