Definition

What is LLM observability?

LLM observability is the record of model calls in an application: the input, the output, token counts, latency, errors, and cost when the tool computes it. You use it to debug one bad completion and to see whether live traffic got slower or more expensive.

Agent observability is the wider record. A model call is one step. An agent run is that call plus the tool calls and decisions around it. The searches for "llm observability" and "llm observability tools" are this page. Product facts below were read from each vendor's docs or pricing page on 15 November 2026. This page does not repeat a price it did not re-read.

What a trace contains

A useful LLM trace is one model request you can open later. The input and the output, so you can see the bad answer. Latency, so you can see the slow one. An error, so a timeout is not a mystery. Token counts and cost, when you are debugging a bill rather than a behavior.

Teams add three things once that record exists. Evals score the output. A prompt manager versions the text that produced it. A gateway puts one client in front of many providers and logs the call because the call passed through it.

None of those are required to answer "what did the model return?" They are required if the next question is "is this answer good?" or "what did this prompt version cost?"

LLM observability vs agent observability

An agent spends most of its time between model calls. It searches, it writes, it retries, it takes a branch. The model span can look healthy while the tool call loops. That loop is an agent problem. Token cost on the model span will not name the tool.

Record both when you have both problems. The model span answers latency, tokens, and the completion. The agent trace answers the sequence. How to monitor AI agents is the sequence. This page is the model call.

LLM observability tools

Langfuse and LangSmith

Both record traces, and both add evals and prompts. Langfuse's pricing page lists token and cost tracking. Langfuse is open source and free to self-host. LangSmith is proprietary. It is LangChain's product for tracing, evals, and deploying LangGraph agents, and self-hosting is on Enterprise. The comparison, including list prices, is on Langfuse vs LangSmith.

Helicone

A gateway. Point an OpenAI-compatible client at Helicone and the request is logged because it passed through the proxy. That is the lowest-effort way to get cost and request logs. It is the wrong shape if you need tool-call order and the model call must not detour.

Braintrust and Phoenix

Braintrust is hosted tracing plus evaluation, metered on processed data and scores. Phoenix is the local app: tracing, evals, datasets, experiments, and prompts, free to run yourself. The license is Elastic License 2.0, which bars offering Phoenix as a hosted service. Use either when the trace exists so you can score it. The fit notes are on AI agent observability tools.

Datadog

Datadog Agent Observability traces LLM calls and agent workflows, including tokens, latency, errors, and cost dashboards. It is billed on LLM spans. Use it when the rest of the system is already in Datadog. This page does not quote a Datadog rate.

AgentTracer

AgentTracer is not an LLM observability suite in the token-ledger sense. It does not sit on the model path, it does not price tokens, and it has no prompt playground. It stores events you POST: a type, an optional name, a timestamp, and a JSON payload. Recommended types are tool_call, decision, message, error, and span. Free is $0 for 50,000 events a month. Use it for the agent timeline and the alerts. Use one of the tools above when the missing piece is token cost or an eval.

Start

Sign up free if the timeline is the record you need. The ingest example is the quick start. Plan limits are on pricing.

Questions

What is LLM observability?

LLM observability is the record of model calls in an application: the input, the output, token counts, latency, errors, and cost when the tool computes it. You use it to debug one bad completion and to see whether production got slower or more expensive.

What are LLM observability tools?

Products that capture those model calls. Langfuse, LangSmith, Helicone, Braintrust, Arize Phoenix, and Datadog Agent Observability all record tokens, latency, or cost from their own docs. AgentTracer does not price tokens. It stores the events you send, which can include a model step if you choose to record one.

Is LLM observability the same as agent observability?

No. A model call is one step. An agent run is that call plus the tool calls and decisions around it. LLM observability debugs the completion. Agent observability debugs the run. A prompt log shows neither a tool loop nor an agent that stopped.

Does AgentTracer log prompts and token cost?

It stores the JSON payload you attach to an event. It does not pull prompts out as their own fields, it does not redact that payload, and it does not compute token cost. Do not send secrets. If you need a token ledger, use a tool that prices model calls. If you need the run timeline and alerts, AgentTracer is the hosted one.

Sources

Last checked 15 November 2026.