Every engineering team shipping AI products in 2026 has seen it: a prototype that impresses in a Friday demo, and a system that embarrasses in Monday production. The causes are almost always the same — and they are solvable if you identify them early.

1. Evaluation gaps

Most teams test their LLM product with a few hand-crafted examples. Production surfaces thousands of edge cases those examples never anticipated. Without systematic evaluation — coverage of topics, styles, intents, languages, and adversarial inputs — you are flying blind. We invest heavily in eval frameworks before a product ships.

2. Latency budgets ignored during development

A model that takes 4 seconds to respond is fine in a demo. It is unusable in a customer-facing product. Production LLM systems need streaming responses, aggressive caching of common query patterns, and fallback strategies when a primary model is overloaded. These must be designed in from the start, not bolted on later.

3. No observability

You cannot improve what you cannot measure. We instrument every production LLM system with: per-request tracing, prompt and response logging (with PII controls), latency breakdowns by component, user feedback capture, and automatic flagging of low-confidence outputs.

4. Guardrails as afterthoughts

Output safety, topic restrictions, hallucination controls, and sensitive content filters cannot be retrofitted without significant rework. They must be designed as first-class components of the system architecture — with their own testing, monitoring, and improvement loops.

5. Cost surprises

Token costs at demo scale feel trivial. At production scale they can be the dominant infrastructure line item. Effective production AI teams design for cost from the start: prompt compression, caching, routing to smaller models for simpler tasks, and monitoring spend per user or workflow.

When developers spend 40% of their time on environment setup, dependency conflicts, and deployment pipelines, the productivity ceiling is obvious long before AI enters the picture. Internal Developer Platforms do not just remove friction — they create the conditions for AI-augmented teams to operate at their ceiling.

What an IDP actually is

An Internal Developer Platform is a curated set of self-service capabilities built on top of your infrastructure. At minimum, it provides: environment provisioning, CI/CD abstraction, secrets management, observability bootstrapping, and service templates. Done well, it compresses the time from idea to running service from days to hours.

Why it matters more in the AI era

AI-augmented development means faster iteration and more experimentation. Without a platform, that speed generates chaos: inconsistent environments, untested deployments, ungoverned dependencies. A well-designed IDP provides the guardrails that let teams move fast with confidence rather than fast with fear.

Common mistakes

Building too much too soon is the most common failure. The teams that succeed start with the highest-friction points — usually environment setup and deployment — and build out from there based on actual developer feedback, not assumed needs.

Fastify has always been the performance-conscious choice in the Node.js ecosystem. Version 5 takes that foundation further with a redesigned plugin architecture, improved TypeScript inference, and significant throughput gains that make it particularly well suited for the new class of AI API workloads.

What changed in v5

The plugin encapsulation model has been tightened significantly. Plugins now declare their dependencies explicitly, making large plugin graphs easier to reason about and debug. Schema compilation has been overhauled, reducing cold-start latency by 30–40% in typical service configurations. The TypeScript types are now generated directly from the Fastify core rather than maintained separately, eliminating the type drift that affected complex deployments in v4.

Why this matters for AI APIs

LLM-powered API workloads have a different profile to traditional web services. Request bodies are large and structurally complex. Response streaming is the default, not an edge case. Throughput matters at the per-connection level because token generation is inherently slower than database queries. Fastify v5’s improved schema handling and reduced per-request overhead translate directly into better p99 latencies for these patterns.

Migration from v4

For most services, migration from v4 to v5 is straightforward. The breaking changes are well-documented and primarily affect applications using undocumented internal APIs. The Fastify team has published a comprehensive migration guide, and the performance improvements justify the effort for any service processing more than a few hundred requests per second.

The phrase “AI security” is used to describe everything from antivirus updates to model red teaming. That breadth is part of the problem. When everything counts as AI security, nothing gets the focused engineering attention it needs.

Here is the core issue: enterprise AI systems are probabilistic, context-sensitive, and increasingly autonomous. The security tools most organisations have were designed for deterministic software — systems where the same input reliably produces the same output, and where threat detection can rely on known signatures and rule patterns.

Agentic AI systems do not work that way. An AI agent with access to your CRM, your document stores, and your outbound email API can be directed to do harmful things by an adversarially crafted input that looks, to every existing monitoring tool, like completely normal traffic. That is a fundamentally different threat surface.

What “provable trust” means in practice

Saying you trust your AI system is not a security posture. Provable trust means you can produce evidence — at any point in time — that the model behaved within defined boundaries, that its outputs can be traced to their inputs and context, and that any deviation from expected behaviour was detected and handled.

That evidence must be continuous and automatic, not produced by a manual quarterly audit. Systems operating at enterprise scale make thousands of AI-driven decisions per hour. Human review cannot provide assurance at that cadence — engineering controls must.

The two moments that matter

We structure our Secure AI work around two distinct moments where trust is either established or broken.

Build-time is where most of the leverage sits. Securing the training data, validating the model pipeline, testing for adversarial inputs, and designing appropriate access controls before deployment means you are not trying to patch a fundamentally unsound system in production. This is the phase most organisations underinvest in — they treat security as a deployment gate rather than a design discipline.

Run-time is where reality diverges from assumption. Models encounter inputs they were not trained for. Agents take actions that were technically in scope but contextually wrong. Data distributions shift. New adversarial patterns emerge. Run-time monitoring needs to detect these not as point failures, but as signals in a continuous assurance system that updates your confidence in the deployed system over time.

What the regulatory environment is asking for

The EU AI Act, NIST AI RMF, and sector-specific guidance from financial regulators and healthcare authorities are converging on a common expectation: organisations must be able to explain what their AI systems do, demonstrate that controls exist, and produce audit evidence when asked. Vague governance documentation and one-off risk assessments will not satisfy these requirements as enforcement matures.

The organisations that will find this manageable are the ones that built traceability and policy enforcement into their AI systems from the start — not the ones scrambling to retrofit governance frameworks onto systems that were never designed with auditability in mind.

The agentic escalation problem

Autonomous agents introduce a specific risk pattern that deserves its own engineering response: cascading action. An agent that can reason, plan, and act across multiple systems can turn a single manipulated instruction into a chain of consequential actions — each individually within policy, but collectively outside any intended boundary.

Addressing this requires circuit-breaker controls, action scoping by task and context, and human-in-the-loop checkpoints for high-consequence operations. These must be designed into the agent architecture, not added as policy recommendations in a governance document.