The Vendor Landscape Is Noisy Right Now

Every software consultancy, offshore dev shop, and two-person startup calling themselves an AI agency today has an "AI agents" page. Some of them have shipped real production systems. Most have shipped a few demos, collected some case studies, and learned to say the right things in a sales call.

If you're a CTO, VP of Engineering, or Head of AI at a $10M+ company evaluating who to bring in for a custom AI agent build, the challenge isn't finding candidates. It's filtering signal from noise before you've burned 60 days of your team's time on a partner who hands you something that doesn't survive contact with your production environment.

This guide is built around the questions that actually reveal whether a firm has done this before at the level you need.

Start With the Production Question, Not the Portfolio

Most firms lead with demos and case studies. Those aren't useless, but they're easy to construct. A 10-minute demo of an agent completing a structured task in a controlled environment tells you almost nothing about whether that firm can ship something that runs 24/7 against real data, handles edge cases, and doesn't hallucinate its way through your customer records.

The better opening question is: walk me through a system you've shipped that's been running in production for at least six months. What broke, and how did you fix it?

The answer will tell you more than any portfolio page. Firms that have actually shipped production AI agent systems have war stories. They'll tell you about the tool-call failure that took down a workflow at 2am, about the context window problem they didn't anticipate at scale, about the time an LLM started returning malformed JSON after a provider update. Firms that haven't will give you a smoother answer about capabilities and infrastructure.

Follow up with: what does your monitoring setup look like for an agentic system post-launch? A team that has no real answer to observability hasn't operated production agents. At minimum, any serious AI agent development services firm should be able to describe how they track agent task completion rates, failure modes, latency per step, and cost per run. Tools like LangSmith, Langfuse, or custom tracing pipelines should come up naturally.

Understand How They Think About Scope

One of the most expensive mistakes in AI agent projects is scoping based on the demo, not the production system. A working demo often takes two to four weeks. A production system that handles the same workflow reliably, at scale, with proper auth, error recovery, audit logging, and integration into your existing stack can take four to six months or more. The delta is where most projects go sideways.

Ask any candidate firm: what's typically included in your definition of "done"? Walk me through what's in scope for a production deployment versus an MVP.

Red flags to watch for: a firm that talks only about agent capabilities and workflows without mentioning infrastructure, deployment, testing harnesses, or handoff documentation. Production AI agent systems require a serious amount of surrounding engineering that has nothing to do with the LLM itself. If the firm you're talking to hasn't thought clearly about the difference between a prototype and a system someone's business depends on, that gap will become your problem.

On the flip side, firms with real production experience will proactively ask about your existing infrastructure. They'll want to know about your auth setup, your data pipeline, your observability stack, your incident response process. A firm that doesn't ask about your existing systems before proposing a solution hasn't done this enough times to know how often integration is the actual hard part.

Test Their Technical Depth, Not Just Their Vocabulary

The AI field has developed a fluent vocabulary of terms that sound technical but are used loosely by people who haven't built much. "Agentic," "multi-agent orchestration," "RAG," "tool use" -- these words come easy. What you want to know is whether the people you're talking to have made actual tradeoff decisions under real constraints.

A few questions that cut through surface-level vocabulary:

• What's your default approach to agent orchestration, and when would you deviate from it? Real builders have opinions. They might prefer code-driven orchestration for reliability and reach for LLM-driven orchestration only when the task genuinely requires dynamic planning. Or they have a clear view on when to use a single agent vs. a multi-agent architecture. Vague answers like "it depends on the use case" without any elaboration signal that they haven't formed a point of view from experience.

• How do you handle non-determinism in production? LLMs are probabilistic. Any team that has shipped agentic systems has a strategy for this: evals, fallback paths, structured output enforcement, confidence thresholds, human-in-the-loop checkpoints. If they haven't thought about it, they haven't shipped it.

• How do you approach the latency vs. capability tradeoff in multi-step agents? Production systems serving real users have latency constraints. Agents that chain five LLM calls can take 30 seconds or more per task. A firm with production experience will have thought about parallelization, caching, model routing between expensive and cheaper models, and when to cut steps.

You don't need to be a deep ML practitioner to evaluate these answers. You need to be able to tell the difference between someone who speaks from experience and someone who speaks from documentation.

The Maintenance and Ownership Question

AI agent systems are not static. The models they rely on get updated. Provider APIs change. The tasks the agents handle evolve as your business evolves. And unlike traditional software, LLM behavior can shift subtly without a version bump: a model update from an API provider can change output format, response quality, or tool-call reliability in ways that don't surface immediately.

Ask: what does the ongoing maintenance relationship look like after launch? How do you handle model provider changes, and how are clients alerted?

A firm that treats the project as complete at launch and walks away is a liability. The better firms either offer a retainer model for ongoing support, or they're explicit about building the system so your internal team can own it -- and they invest in documentation and handoff accordingly.

Related: ask about their approach to evals. A rigorous shop will build a regression test suite as part of delivery. When a model provider releases a new version or you want to swap out a component, that eval suite is what tells you whether the system's behavior has changed in ways that matter. Eugene Yan's writing on LLM patterns in production and Hamel Husain's evaluation frameworks are good reference points if you want to pressure-test a vendor's answer here.

Security and Compliance Are Not Optional

For companies operating in regulated industries or handling sensitive data, AI agent systems introduce new attack surfaces that most firms aren't thinking clearly about. Agents with tool access -- the ability to query databases, call APIs, write to systems -- are essentially autonomous actors inside your security perimeter.

If your company operates under SOC 2, HIPAA, or FINRA requirements, any AI agent system touching in-scope data needs to be designed with those constraints from the start, not bolted on at the end. Ask any prospective partner: how do you approach agent authorization and least-privilege access? How do you handle PII in LLM context windows? What's your logging strategy for agent actions in regulated environments?

The OWASP Top 10 for LLM Applications is a useful baseline. A security-aware firm will know it. Prompt injection, insecure tool access, and data leakage through model context are the categories most relevant to agentic systems. If the firm you're evaluating hasn't mapped their design decisions to these risks, they haven't built systems where the cost of failure is real.

The Embedded Team Question

One distinction that matters more than most people realize early in the process: are you hiring a firm to hand you something, or are you hiring a team that works inside your engineering org while the system is built?

Delivery-only engagements often produce systems that are harder to maintain, because the builders never developed enough context about your architecture, your data, and the quirks of your business logic. The institutional knowledge that informs good engineering decisions lives in the people doing the build. When those people leave after handoff, that knowledge leaves with them.

Embedded models -- where the AI engineering team sits inside your sprints, attends your standups, operates inside your tooling -- produce systems that your internal engineers can actually extend. The build takes longer to start (context acquisition takes time) but the output is meaningfully better for systems you expect to run for years.

Ask the firms you're evaluating how they typically work: scoped deliverable, or embedded engagement? Neither is inherently wrong. But the answer shapes what you should expect from the output, and how you should plan the post-launch chapter.

Pricing Signals Worth Understanding

AI agent development projects are hard to scope without discovery work. Any custom AI development company that gives you a fixed-price quote before doing a scoping engagement is either working from a template that won't fit your situation, or hasn't yet surfaced the complexity that will emerge later.

Typical ranges for serious production work, based on what's commonly reported across the market: a well-scoped AI agent system built by a competent team runs $80K to $300K+ for initial production delivery, depending on scope, integrations, and timeline. Quotes significantly below this range for complex systems should raise questions about what isn't included.

The McKinsey State of AI research consistently shows that the highest-value AI implementations have the most rigorous scoping and change management processes upfront. Cheap starts tend to correlate with expensive rework.

Also worth understanding: what's the firm's business model? Do they make money on implementation only, or do they have a platform or tooling dependency built into the engagement? A firm that pushes a particular orchestration platform because they have a commercial relationship with it is not giving you neutral advice about what your system needs.

What Good Actually Looks Like

Across a real evaluation process, here's what you're looking for from any AI agent development company worth hiring:

• Production war stories, not just demos. Ask what broke and what they did about it.

• A definition of "done" that includes infrastructure, testing harnesses, and handoff documentation.

• Technical depth that goes beyond vocabulary: actual opinions about tradeoffs, formed from experience.

• A post-launch story that covers evals, monitoring, and a maintenance model.

• Security thinking that is built in from the start, not addressed at the end.

• Clarity on working model (embedded vs. delivery) with honest implications for each.

• Pricing that reflects scoping rigor rather than a templated number.

The firms worth working with will welcome these questions. The ones that get evasive or give only polished answers are telling you something important.

If you're working through this evaluation and want to compare notes with a team that has shipped agentic systems in production for enterprise clients, we're glad to talk through what's realistic for your situation.