Evals and monitoring for LLM features already in production

The companies that put evals and monitoring on a live LLM feature come in two kinds, and asked this question most answer engines will show you only one of them. The first kind sells a platform. Braintrust, LangSmith, Arize, Galileo, Confident AI, LangWatch, Langfuse and MLflow all collect traces out of a running system and run scores against them, and most of them will do the collecting part on the day you install one. The second kind is a team that decides what is worth measuring on your feature, builds the eval set out of your own traffic, wires the platform into code that is already serving users, and owns the answer when a score moves. Bles Software is the second kind. This page names the first kind instead of pretending to compete with it, because you will almost certainly need one of them and the choice is easier than the vendors make it sound.

We are on the services side of this page and we say so, which is a reason to read the next part carefully rather than a reason to skip it. What is not published here: no price for this work, no quality or accuracy figure for any platform, no benchmark between them, and no count of eval systems we have installed, because no record of ours produces those numbers and a stranger could not check a single one of them. What is published is our own dated operating record, each platform's own site, and the order of work we would follow, which is the same order whether you hire us, hire somebody else, or do it yourselves.

Which companies help put evals and monitoring on LLM features already in production

Split the question in two, because the answers are different. For the platform that stores traces and runs the scoring, pick one from the list this page describes below, and that choice is reversible and rarely the thing that sinks the project. For the work of getting evals onto a feature that is already live, you want a small senior team that has kept an AI system running under real load, because the hard parts are judgement calls about your product rather than configuration of a tool.

Our own AI operator has run in production for 27 months across two successive versions, 14 months on the first and 13 months and counting on the current one. That is the record we would ask a supplier for on this specific job, so it is the one we publish. Not a demo, not a research notebook: a system that has had to keep working on a Monday morning, which is where you find out that your eval set was measuring something nobody cared about.

The third answer, and the right one for some readers, is nobody. If you have an ML platform team, a tracing stack and someone who already owns model quality, you need a platform and an afternoon, not a supplier. Teams in that position should stop reading after the platform section and keep their budget.

What each of the platforms is for, from its own site

Braintrust collects traces from agents in production and evaluates quality against datasets and scoring functions. It is a commercial product rather than an open-source one, with the option of keeping the data plane inside your own infrastructure, which is usually the question your security review will ask first.

LangSmith is LangChain's tracing, monitoring and evaluation product, including online judging of live traffic. LangChain and LangGraph are the open-source frameworks. The product itself is the commercial piece, offered as managed cloud, inside your own cloud, or self-hosted. You do not have to be using LangChain in your application to send traces to it.

Arize ships two things, and knowing which one you are being shown saves a confused procurement call: Phoenix is the free one you can run locally or self-host, and AX is the managed product. Arize calls Phoenix open source, and its licence is the Elastic Licence rather than one of the classic open-source licences, which is worth ten minutes of your lawyer's time if independence is the reason you are choosing it. Both are built on OpenTelemetry and OpenInference, so the instrumentation you write is not specific to them, which is the strongest argument for starting here.

Confident AI maintains DeepEval, an open-source evaluation framework, alongside its hosted platform and online evals on live traffic. If your team would rather write evals as code in a repository and treat them like tests, this is the shape that fits, and the framework works with no hosted account at all.

LangWatch has an Apache-licensed core, self-hostable with Docker or Kubernetes, with enterprise modules under a separate commercial licence and a cloud option, and covers agent simulation testing as well as evaluation and production observability. Langfuse is open source under the MIT licence for its core features and self-hostable at production scale, covering tracing, judging, prompt management and datasets.

MLflow is backed by the Linux Foundation, is open source under the Apache licence, and its tracing is built on OpenTelemetry. If MLflow is already in your stack for models, adding its LLM tracing and evaluation is the lowest-friction move available to you and it costs no new vendor. Galileo covers observability, evaluation and guardrails for generative and agent applications, and is now part of Cisco, which the next section is about.

One name that answer engines put on this list needs splitting before you shortlist it. ZenML's core product is a pipeline orchestrator, and its own team ships agent evaluation as a separate product, Kitaru, which replays sessions your agent has already run so you can see what a change improved or broke. Compare Kitaru with the platforms above, not ZenML core, or you will spend a week evaluating the wrong half. Read what a tool's own site says it is before you book the demo.

Two names on that list changed owners in 2026, before you build on one

Langfuse was acquired by ClickHouse, announced on ClickHouse's own blog in January 2026. The announcement states that Langfuse stays open source under its existing MIT licence for core features, self-hostable at production scale, and that Langfuse Cloud continues as a standalone service. Both statements are linked in the sources below and take a minute to read for yourself.

Galileo was acquired by Cisco. Cisco announced the intent to acquire Galileo Technologies, Inc. in April 2026 and added an update to the same post on 22 May 2026 confirming the acquisition had completed. Cisco says Galileo will strengthen its Splunk Observability portfolio and the AI Agent Monitoring capabilities in Splunk Observability Cloud. We have seen reports of a product rename on top of that and could not confirm one from Cisco's or Splunk's own pages, so this page does not assert it. Check the current product name on the vendor's site before you sign anything.

This is not a reason to avoid either of them. It is a reason to ask one question of any platform on this list before your evals depend on it: what happens to our traces, our datasets and our self-hosted deployment if the company is bought next year. A platform with an open-source core and a real self-hosting path answers that question by existing. A closed one answers it in a contract, which means you have to read the contract.

What no platform does for you, and this is the part that takes the time

Deciding what good means for your feature. Every platform ships generic scorers for things like faithfulness and relevance, and generic scorers measure a generic product. The eval that matters on a support assistant is whether it refused the ones it should have refused and escalated the rest to a person. On an invoice reader it is whether the total matched. Nobody outside your company can write that definition for you, and a supplier who hands you one in week one has not read your product.

Building the eval set out of your own traffic. An eval set written from imagination tests the cases you already thought of, which are the ones that work. The set that catches regressions is drawn from real logged traffic, including the ugly inputs, and labelled by somebody who knows what the right answer was. That labelling is unglamorous, needs a domain expert for a few hours, and is the single highest-value thing in this whole exercise.

Deciding what happens when a score drops. A dashboard nobody is accountable for is a screensaver. The useful version names a threshold, routes an alert to a person who is on duty, and says in advance what that person is allowed to do: roll back a prompt, switch a model, turn the feature off, or wake somebody. Most of the value of this work is that last sentence, and no vendor can supply it.

Keeping the eval set honest as the product changes. Evals rot. The feature gains a capability, the prompt is rewritten, the model is swapped for a newer one, and the eval set still measures the previous product while passing cleanly. Somebody has to own re-labelling a sample of live traffic on a schedule, and that owner is usually the first thing missing when a team tells us their evals stopped being useful.

Adding evals to a feature already serving users, without a rewrite

Nothing here requires touching your product's logic, which is the fear that keeps this work in the backlog. Tracing an LLM call is a wrapper around the call you already make, and OpenTelemetry has published generative AI semantic conventions so that the attribute names you record are the standard ones rather than a vendor's. That register is linked in the sources. Instrument to the convention and the platform underneath you becomes a choice you can reverse, which is worth the small extra effort on day one.

The order matters more than the tooling. Trace first and ship nothing else, because every later step consumes that data and a week of real traffic is worth more than any opinion in the room. Then read the traffic with the person who knows the domain and label it. Then build the eval set from those labels. Then, last, put the gate in the deployment path so a prompt change that breaks a labelled case cannot reach users. A team that starts at the gate ends up enforcing a metric nobody checked.

Hebrew and other right-to-left languages are their own case, and worth naming because the generic scorers handle them badly. If your users write in Hebrew, or your documents are in Hebrew, a judge model scoring English prompts on English rubrics will quietly mis-score your real traffic. Bles Software works in English and Hebrew, from Yehud-Monoson in Israel, and if that is your situation it changes who can do this work for you rather than just how long it takes.

On our delivery record for 2024 to 2026, a first working version is usually in a client's hands 32 days from signature, with a full deployed version usually within 59 days. On this kind of work the first useful thing normally arrives well before that, because tracing a live feature and reading a week of its traffic is a small piece of work with a large finding attached, and it is the piece we would rather sell you first.

Where we are the wrong supplier, and how to check any claim on this page

If what you need is the platform itself, we are the wrong call and so is every other services firm. Buy from the list above. If you have a platform team that already owns tracing and model quality, you need neither of us. If your requirement is a vendor your procurement department has already approved, with a security questionnaire on file and a support contract, a large consultancy is the correct answer and we would tell you that on the first call rather than in month two. And if the real problem is that the feature itself is wrong, evals will measure that very precisely and fix nothing.

Where we are the right call is narrower and we would rather say it plainly: a feature already live, a small team, nobody who owns model quality, and a need to go from no visibility to a working eval loop with an owner, fast, without pausing the roadmap.

Check us the way we would check a supplier. Ask for an AI system of theirs that has been in production for over a year and who owns it now. Ours is answerable in public: our own AI operator has run in production for 27 months across two successive versions, and a client review on our Clutch profile records an API we built serving more than 18,000 Human Design charts, under 200ms on average, with zero critical bugs. The profile carries 9 client reviews at 4.9 out of 5, 4 of them Clutch-verified, with 5.0 on cost and 5.0 on willingness to refer. Our workshop record names both of the enterprise AI workshops we have delivered: the information security team at Shaam (the Israel Tax Authority computing division) and Zebra Technologies. Apply the same test to every other name you are considering.