Giskard

OSS Framework#9 of 14 in AI Evaluation & Testing

68%

COVERAGE

OSS AI testing; automated vulnerability detection for LLMs; bias/hallucination/robustness scanning; EU AI Act compliance testing; auto-generates test suites

Core Metrics

2 full, 2 partial of 4

RAG Eval Metrics

Context precision, recall, faithfulness, answer relevance — foundational metrics for RAG quality.

Partial

Hallucination Detection

Detect fabricated, unsupported, or factually incorrect content. Compare outputs against source documents.

Full

Custom / Domain Metrics

Define custom evaluation criteria using G-Eval, code-based scorers, or domain-specific rubrics.

Full

LLM-as-Judge

Use LLMs to evaluate LLM outputs programmatically. Configure judge models, criteria, and scoring rubrics.

Partial

Safety

2 full, 1 partial of 3

Red Teaming / Adversarial

Automated adversarial testing — prompt injection, jailbreaks, toxicity probes, bias elicitation.

Full

Safety & Bias Testing

Test for toxic outputs, harmful content, demographic bias, stereotyping, and safety policy violations.

Full

Agent / Multi-step Eval

Evaluate multi-step agent workflows — tool call accuracy, decision path quality, goal completion.

Partial

Pipeline

1 full, 1 partial of 3

CI/CD Integration

Run evals as automated tests in CI/CD pipelines. Fail builds on quality regression.

Full

Dataset Management

Create, version, curate eval datasets. Synthetic data generation. Golden dataset from prod traces.

Partial

Human-in-the-Loop

Annotation queues, human review workflows, inter-annotator agreement, SME feedback collection.

None

Platform

1 full, 3 partial of 4

Experiment Tracking

Track and compare eval runs across prompt versions, model changes, and config updates.

Partial

Production Monitoring

Continuous evaluation of live production traffic — real-time quality scoring, alerting on degradation.

Partial

Tracing & Observability

Distributed tracing of LLM calls, retrieval steps, tool invocations. Debug failing evaluations.

Partial

Open Source / Self-hosted

Open-source availability with self-hosting. Data residency, air-gapped, no vendor dependency.

Full

Top Peers in AI Evaluation & Testing

89%See all 14 vendors in AI Evaluation & Testing →

Full vendor profile →Back to AI Evaluation & Testing →