Turn your team’s judgment
into evals that scale
Your team defines what good looks like. AI applies that standard across production. Human review keeps it aligned.
Not sure what to measure? We’ll build your first eval with you, free.
Book free evals working sessionEval
Uses customer history
Makes reasonable exception
Chooses proportionate remedy
Standard alignedAI can now apply this refund judgment across production.
Every eval starts with a decisiononly your team can make
Ask a question worth answering
Skip vague questions. Ask something specific, like: would we send this to an actual customer?
Let your experts set the standard
Domain experts answer based on how your business works. That becomes the reference.
Let AI apply the standard at scale
Use an LLM to judge in one pass, or an agent to investigate the context before deciding.
Keep AI judgment anchored to people
Send the same question to humans and AI. Track their agreement and recalibrate.
Eval
Live evalRefund qualityCalibration set · 120 runs
Did the response use customer history and apply the repeat-delivery exception?
YesNeeds review8 of 9 human votes
Sarah, Jon, Chris, Mark +4Cross functional reference panelReferenceNeeds reviewThe agent ignored two prior delivery failures and refused the refund.
Needs reviewCustomer history was available, but the repeat-delivery exception was not applied.
Build an evalin minutes
Is the new policy in accordance with the board meeting last weekShort answer textT1 of 4
Should support make an exception for this customer?
Step 01
Write the questions
Write your own in the form builder, or describe what matters and let neatlogs draft them.
Assigned evaluatorsChoose who reviews this question
SarahReviewer
MarkReviewer
JonReviewer
ChrisReviewerStep 02
Choose who answers
Route each question to a person, an AI evaluator, or both. Keep human judgment in the loop.
Refund
Step 03
Choose what they review
Evaluate complete runs or specific spans across all traces or a controlled sample.
EvalEvaluator test
ReadyTraceCustom data
TraceSpan
Recent samplerefund_agent · run_1842
Needs reviewRefund eligibility was not checkedNo
Step 04
Test before you trust
Run the AI evaluator on an actual trace. See how it performs before it reviews production.
EvalIncident
Opensupport_runrefund_agent
Eval result attached
Step 05
Investigate failures
A failed eval becomes an incident with the run attached. From there, neatlogs investigates it.
Reviewers never have toread a trace
Review in Slack
Quick calibration for Refund quality:
“The undelivered order isn’t eligible for a refund.”
Did the response use customer history and apply the repeat-delivery exception?
3 Replies
Needs review. The customer has two prior delivery failures, so the exception should apply.
Needs review. Agreed. The refund decision should use that customer history.
Needs review. customer_history never reached decide_resolution on this path.
Evals ›Refund policy review
In progress854 of 1500 answers received.
EvaluatorQuestionFilter by answer
northline_support_turnTrace · 01c4f2
No, customer history was ignored
Needs review
northline_support_turnTrace · 02c5f3
Yes, customer history was used
Needs review
northline_support_turnTrace · 03c6f4
No, customer history was ignored
Needs review
Read the resultsfour ways.
The same evaluation, organized around the question you need to answer.
StatusEval
Refund response quality review
Repeat-delivery exception review
Customer history review
A.Should support make the repeat-delivery exception?
YesReviewNoB.
C.
D.
By subject
See how one run or span performed across every question.
Choose a question
Did the response use customer history?
Was the repeat-delivery exception applied?
QuestionDid the response use customer history?
3 fully correct75%
1 needs review25%
By question
Find standards that fail repeatedly across many runs.
Choose an evaluator
AISupport Trajectory
Sarah
Jon
Mark
EVALUATORMark4 answersBy evaluator
Compare reviewers and track whether AI judgment still agrees with your team.
In progress0 of 16 answers · 50%
9 Sep 2026 · DispatchedDue · 16 Sep 2026
4 tracesIn this batch
How it was chosen
3 evaluatorsAggregate
Follow completion, verdicts, scores, agreement, and trends across the evaluation.
One system.Four different questions
CapabilityThe question it answers
01Trace
What happened in this run?
02Detection
Where did a known condition appear?
03Eval
Did the behavior meet an explicit standard?
04Investigation
Why did it happen, and what should change?
FAQs01 / 10
What is an eval?
An eval is a set of questions asked about an agent’s output, answered against a standard your business defines. Detections identify events; evals decide whether the behavior was acceptable.
FAQs
An eval is a set of questions asked about an agent’s output, answered against a standard your business defines. Detections identify events; evals decide whether the behavior was acceptable.
Still unsure?
We’ll look at your stack with you.