Platform

Audits & evals

An audit runs every prompt in your test set against every candidate model, scores each output with an LLM judge, and returns a routing recommendation - primary, fallbacks, and an optional cost route. Cran audits are faithful: they use the workflow's real prompt, params, tools, and rubric unless you override them.

What an audit does

For each (prompt × model)pair, Cran calls the candidate with the audit’s system prompt and params; records latency, tokens, and real dollar cost; scores the output with an LLM judge (1–5 plus optional dimensions); aggregates per-model averages; and picks a routing recommendation. When a workflow_idis set, it also stamps the workflow’s last score and runs drift detection.

text

prompts[]  ×  models[]  →  cells[] (each scored)
                  ↓
            perModel[] aggregates
                  ↓
   recommendation { primary, fallbacks, costRoute }

Recommended lifecycle

For registered workflows, run these in order for the most defensible result:

cran_submit_manifest - register call sites.
cran_classify_workflow - set workflow class + judge strategy.
cran_generate_test_suite - build adversarial stress tests.
cran_run_audit with useGeneratedTests: true.
cran_get_routing_policy - export deployable routing.
cran_propose_change - submit for human approval → publish.

Workflow vs ad-hoc

Workflow mode (pass workflow_id) inherits the workflow’s system_prompt, judge_strategy, params, tools, and tool_choice. Prompts come from the cached generated test suite (recommended), sample_inputs via the template, or explicit prompts[].

Ad-hoc mode passes prompts[] + models[] directly with no inheritance - good for one-off experiments before registering a call site.

Request

POST https://trycran.in/api/audits · Bearer token · scope read.

bash

curl -X POST https://trycran.in/api/audits \
  -H "Authorization: Bearer cran_…" -H "Content-Type: application/json" \
  -d '{
    "workflow_id": "wf_abc123",
    "use_generated_tests": true,
    "models": ["openai/gpt-5.4", "anthropic/claude-haiku-4-5", "google/gemini-2.5-flash-lite"]
  }'

Field	Constraints
prompts	Max 100. text ≤ 20k chars, expected ≤ 8k.
models	Required. 1–6 catalog ids (provider/short-name).
system	≤ 8k chars. Applied to every candidate run.
judgeStrategy	One of: general, classification, sentiment, extraction, schema, semantic
workflow_id	DB id. Enables inheritance + score stamp.
use_generated_tests	Requires a prior generate-tests call.

Response

POST /api/audits and cran_run_audit return camelCase. The recommendation: primary = highest avgScore (tie-break lower cost, then latency); fallbacks = next two by quality; costRoute = cheapest usable model distinct from primary.

json

{
  "audit_id": "aud_abc123",
  "perModel": [{ "modelId": "anthropic/claude-haiku-4-5", "avgScore": 4.7,
                 "p95LatencyMs": 580, "totalCostUsd": 0.00042 }],
  "recommendation": {
    "primary": "anthropic/claude-haiku-4-5",
    "fallbacks": ["openai/gpt-5.4"],
    "costRoute": "google/gemini-2.5-flash-lite",
    "reason": "Highest avg score at acceptable cost"
  }
}

Judge strategies

The judge always runs on OpenAI (default gpt-4o-mini). Each strategy injects a different rubric:

Strategy	Best for
general	Chat, tool use, open-ended tasks
classification	Label / category assignment
sentiment	Sentiment and tone analysis
extraction	Structured field extraction
schema	JSON / schema validation
semantic	Reasoning, generation, summarization

cran_classify_workflow maps a workflow class → judge strategy automatically; override with judgeStrategy.

Faithful benchmarking

A faithful audit reproduces production conditions: same system prompt, same generation params, same tools, the current model always included as a baseline, and stress tests from generate-tests rather than happy-path samples. Without a workflow_id you must pass params and tools yourself - useful for exploration, but not a basis for production routing.

MCP tools

Tool	Purpose
cran_classify_workflow	Set workflow class + judge strategy
cran_generate_test_suite	Build adversarial prompt suite (cached)
cran_run_audit	Run a faithful benchmark
cran_get_audit	Reload an audit (normalized shape)
cran_get_routing_policy	Export deployable routing code
cran_propose_change	Submit a change for human approval

Routing policy export

After an audit, export a deployable policy:

bash

GET /api/audits/:id/policy?format=json|litellm|vercel-ai-sdk|openai-sdk

For production traffic through the proxy, publish a managed architecture in the dashboard instead (see Control plane).