Platform
How grading works
Every recommendation Cran makes is backed by a score. This page explains exactly how those scores are produced, how a model switch clears the bar, and — just as important — where the method's limits are.
The automated grader
When Cran tests your AI feature, it runs each of your prompts against every candidate model and asks a separate model — the automated grader— to score each output on a 1–5 scale. The grader sees the prompt, the expected answer (when you provided one), and the candidate's output, and returns a score plus a short written rationale. That rationale is shown on every cell of the test result, so you can read why a model scored the way it did rather than trusting a bare number.
The grader is an LLM (OpenAI's gpt-4o-miniby default). It is a different job from generation: judging "did this answer the prompt?" is far easier and more consistent than producing the answer, which is why a small, cheap model grades reliably.
Six class-aware rubrics
A sentiment classifier and a long-form drafter should not be graded the same way. When Cran classifies your AI feature, it picks the matching rubric, and the grader is told exactly what "good" means for that kind of work:
| Rubric | Used for |
|---|---|
| general | Open Q&A and mixed work — correctness, completeness, formatting. |
| classification | Label/category outputs — is the predicted label right? |
| sentiment | Polarity or numeric scoring — does the sentiment/score match? |
| extraction | Pulling entities/fields from input — precision and recall of the values. |
| schema | Structured JSON output — conforms to the required shape, right values. |
| semantic | Open-ended prose — meaning and usefulness, not exact wording. |
You can override the rubric per test if the auto-pick is wrong. The prompt, params, and tools used in the test are your workflow's real ones — the score reflects your actual product, not a synthetic benchmark.
Multi-pass averaging
Integer 1–5 scores tie easily. Cran grades each output more than once (the default is two passes) with a little grader temperature, then averages — yielding finer-than-integer scores that separate models a single pass would call equal. Dimension sub-scores (correctness / completeness / formatting) are averaged the same way, and only when every pass reported them.
When the grader can't score
If the grader is unavailable for a response (an outage, a rate limit), that response is excludedfrom every average — it is never counted as a fabricated neutral 3. A test that couldn't score enough of its responses returns no recommendation rather than a misleading one, and says so. A wrong recommendation is worse than none.
The bar a switch has to clear
Cran only recommends switching models when the cheaper model is not meaningfully worse— not merely "higher average." It computes the paired, per-prompt score difference between the candidate and your current model and takes the lower bound of a 95% confidence interval on that difference. The switch is recommended only when that lower bound clears a small non-inferiority margin (epsilon). In plain terms: we have to be statistically confident quality holds, not just lucky on average. The same test gates prompt-optimization savings.
Honest limits
The grader is an LLM, and these scores are comparative signals — not absolute truth. Specifically:
- Scores are only meaningful relative to each otherwithin one test. A 4.2 is not a universal grade; it means "better than the 3.8 next to it on your prompts."
- The default grader is an OpenAI model, which can carry mild self-preference when grading other OpenAI models. Mitigations: the written rationale is always shown (you can spot a bad call), multi-pass averaging, and class-specific rubrics that pin grading to observable criteria. A cross-grader option (grading with a different provider's model) is on the roadmap for teams that want it.
- Grading is only as good as your test set. A handful of unrepresentative prompts yields a confident but unrepresentative score — use your real traffic samples or generated tests that cover the edges.
The point of showing you the rationale, the confidence bound, and these caveats is that you never have to take a Cran recommendation on faith — you can check the working.