llm-as-judge

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform

Design visual, review-gated agent loops for Claude Code before you run them.

Hands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set — model APIs, structured output, tool calling, RAG, evals-as-the-spine, agents (loop from scratch, tool design, guardrails, MCP, Skills), fine-tuning vs LoRA, prompt-injection/security, LLMOps, and customer craft. Runs on the free Groq API.

Pairwise LLM judges (A/B/tie): budget-aware multi-turn packing, position-bias correction, pseudo-label distillation. Generalized from the 4th-place (gold) solution to Kaggle LMSYS Chatbot Arena.

agentevals is a framework-agnostic evaluations solution based on OpenTelemetry traces

Academic Personal AI Infrastructure

The trust layer for your AI agent on any platform. Trace and evaluate your agent. Full Observability. Run Model portability analysis. Autotune your agent.

Lumen — learner-owned AI education platform. Tell the AI what you want to learn: it builds you a private course in ~a minute, tutors you with course-scoped RAG + citations, and lets you share, clone & remix via a moderated catalog. BYOK, custom no-LangChain multi-agent orchestrator, golden evals in CI, MCP server. Live demo + public /eval.

LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.

AI evaluators CLI for your AI apps and Agents - Dynatrace AI Observability

Causal Judge Evaluation: calibrate LLM-as-judge scores against oracle labels with valid uncertainty.

A dependency-free Python genetic-algorithm engine with pluggable fitness criteria — the reusable search core for an LLM/AI autoresearcher.

Python SDK for Modaic

HonestyMeter: An NLP-powered framework for evaluating objectivity and bias in media content, detecting manipulative techniques, and providing actionable feedback.

Practical LLM evaluation for teams that ship to production. Deterministic + LLM-as-judge evaluators, dataset support, CI/CD integration.

Evidence-first multi-agent debate skill: get a second opinion by pitting Codex × Claude Code (or GLM/DeepSeek/Qwen) to independently review, red-team & judge high-stakes code and architecture decisions.

Code and data for Koo et al's ACL 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"

Diagnosing policy grounding in LLM judges via causal erasure. Companion code to the AIES 2026 submission READ.

Toward Expert-Level Medical Text Validation with Language Models