Lunary

57%

COVERAGE

Prompt/Completion Tracing

Record the complete lifecycle of every LLM request — prompts, completions, tool calls, retrieval steps — with structured parent-child span relationships.

Latency Monitoring

Track response times at each pipeline step with p50/p95/p99 breakdowns and historical trends.

Multi-model Support

Trace across multiple LLM providers and frameworks (LangChain, LlamaIndex, Vercel AI SDK) with auto-instrumentation.

Agentic Observability

Dedicated tracing for multi-step agent workflows — tool call visualization, decision tree inspection, agent-specific metrics, and multi-turn threading.

Cost Tracking

Calculate per-request and aggregate costs. Attribute spend to teams, features, users, or projects.

Token Analytics

Monitor input/output token counts, context window utilization, and token efficiency.

Alerting & SLOs

Configure alerts for latency spikes, error thresholds, cost overruns, and quality degradation.

Built-in Evals

Pre-built evaluators for hallucination, relevance, toxicity, faithfulness, coherence.

Custom Evals

Custom evaluation metrics, LLM-as-a-judge prompts, code-based scorers, domain-specific criteria.

User Feedback

Collect user feedback (thumbs up/down, ratings) linked to specific traces.

RAG-specific Metrics

Specialized metrics for retrieval-augmented generation: context relevance, groundedness, answer faithfulness, retrieval precision.

Annotation & Labeling

Annotation queues, human-in-the-loop review workflows, SME feedback collection, and golden dataset creation from production traces.

Dataset Mgmt

Create, version, and manage evaluation datasets from production traces or manual curation.

A/B Testing

Run experiments comparing prompts, models, or configs against datasets with statistical rigor.

Playground

Interactive environment to test prompts, replay failed traces, and iterate on configurations.

Prompt Management

Version control, deploy, cache, and collaboratively iterate on prompts as first-class assets. Track which prompt version produced which output.

Drift Detection

Detect changes in model behavior, output quality, or input distribution over time.

Self-hosted

Deploy on your own infrastructure for data residency, compliance, and air-gapped environments.

OpenTelemetry Native

Built on OpenTelemetry standards vs proprietary instrumentation. Prevents vendor lock-in and exports traces to any compatible backend.

Guardrails Integration

Built-in or pluggable content safety, PII detection, toxicity filtering, and output validation within the observability pipeline.