What are decision models? Uses, examples, and tradeoffs

A decision model is a specialized AI model that evaluates inputs against predefined outcomes, returning a structured choice, classification, or score. Unlike large language models (LLMs), which handle a broad range of generative and reasoning tasks, decision models are optimized for tasks like routing requests, selecting agent actions, and evaluating AI outputs.

Decision models like TypeSafe’s Jev, OpenAI’s Decisions API, and Microsoft’s Decision-1 offer a middle ground between hard-coded application logic and general-purpose LLMs. They can interpret language and context that would be difficult to capture in code, while potentially making decisions faster and at lower cost than an LLM, without requiring developers to train a separate classifier for each task.

When should you use a decision model instead of conventional application logic or an LLM? In this post, we’ll look at how decision models work, where they offer the most value, and how to put them to use in applications, agent workflows, and CI pipelines.

What is a decision model?

A decision model is a type of AI designed to make predictions from a limited set of possible outcomes. Instead of generating a new answer, it evaluates the information it’s given and determines which outcome is most likely, or how strongly the input meets a particular criterion.

Decision models can handle questions that require interpreting language, intent, or context rather than following explicit rules. Developers specify the question and allowed outcomes, and the model returns a prediction that application code can use to determine what happens next.

For example, an application might use a decision model to route an incoming support request:

I was charged twice for my subscription.

The application supplies three possible destinations: billing, technical_support, and general. The model evaluates the message and selects billing, along with probabilities for the other options.

Decision models commonly support three types of tasks:

Decision type	Example	Output
Binary classification	Does this message contain a refund request?	Probability that the condition is true
Categorical selection	Which team should handle this request?	Selected category and probabilities
Scoring	How urgent is this issue on a scale of 1–5?	Score based on defined levels

Classification isn’t new. Developers already use specialized models to filter spam, detect intent, and categorize incoming requests. But traditional classifiers generally need training data tailored to the task they’re meant to perform.

Newer decision models let developers handle different classification and scoring tasks through an API. Instead of training and maintaining a separate classifier for each task, you can specify the question and possible answers in the request.

That means the same underlying model can potentially handle different kinds of decisions across an application, from classifying messages to scoring generated responses.

How do decision models work?

Decision models take an input, evaluate it against a question and a set of possible outcomes, and return a structured prediction. Depending on the model, the response might include a selected category, probabilities for each option, or a score.

For the support-ticket example, a simplified request might look like this:

{
  "input": "I was charged twice for my subscription.",
  "question": "Which team should handle this request?",
  "options": [
    "billing",
    "technical_support",
    "general"
  ]
}

The model evaluates the message and returns a result:

{
  "choice": "billing",
  "probabilities": {
    "billing": 0.96,
    "technical_support": 0.01,
    "general": 0.03
  }
}

The application can use billing to route the ticket, while the probability estimates help determine whether the request needs another look. A clear billing issue might go directly to the billing team. A message that could belong to several categories might be sent for review instead.

Once the model returns a prediction, the application decides what to do with it. A support ticket might go straight to the billing team, while a request that could belong to several categories gets flagged for review. The application can also check the prediction against other information before taking action.

Decision model use cases

Decision models can handle everything from routine request classification to tool selection in AI agents and evaluation of generated outputs. They’re particularly useful when a decision requires interpreting context rather than following a fixed rule.

Classification, routing, and prioritization

Applications can use decision models to interpret incoming information and decide where it should go, how urgent it is, or whether it needs review.

Common examples include:

• Routing customer requests to the appropriate team.
• Flagging messages that may require security or compliance review.
• Categorizing incoming issues by type or urgency.
• Prioritizing work based on information in a request.

A customer reporting that someone accessed their account without permission probably needs security assistance, even if the message never uses the word “security.” A decision model can recognize the intent without requiring developers to anticipate every possible way someone might describe the problem.

Probabilities can also help determine when to ask for a second opinion. For example, an application might flag a possible account compromise for review even when the model assigns another category a higher probability. Missing a security incident is usually worse than reviewing an extra message.

If a condition can be checked directly in code, check it in code. There’s no reason to ask a model whether an invoice has been paid when the application can look up its payment status. Decision models are better suited to questions that require interpreting ambiguous information.

Agent workflows and tool selection

Coding agents make decisions throughout a task: which files to inspect, when to run tests, how to investigate failures, and when to ask for developer input.

Some steps require substantial reasoning. Others involve choosing from a known set of actions.

For example, a coding agent investigating a failed integration test might have these options:

Task: Fix a failing integration test

Available actions:
- inspect_failure
- modify_code
- rerun_tests
- request_human_review

Decision: inspect_failure

A decision model could evaluate the task, the CI failure report, and the actions the agent has already taken, then recommend what to do next.

The agent’s orchestration logic determines how to carry out that recommendation. Developers can still require approval before agents take consequential actions, such as changing production configuration or deploying a fix.

You can test whether the model recommends the right next step without changing how the agent executes it. Separating action selection from execution also makes it easier to require human approval for sensitive operations.

Decision models can help with another common routing problem: choosing which AI model should handle a request. Straightforward tasks might go to a smaller model, while complex work goes to a more capable reasoning model.

If the routing decisions are accurate, an application can avoid unnecessary calls to expensive models. The savings need to outweigh the added routing time and the cost of mistakes, so it’s worth measuring the approach against simply using one model for everything.

Evaluating LLM and agent outputs

Decision models can also evaluate AI-generated outputs against predefined criteria, making them useful for automated testing and quality checks.

Developers already use LLM-as-a-judge techniques to grade generated responses against instructions, reference answers, or evaluation rubrics. Decision models offer another option when the judgment can be expressed as a category, a yes-or-no answer, or a score.

For example, an evaluator might ask:

• Does the response contain the information requested?
• Did an agent follow the specified constraints?
• Is an answer supported by the supplied documentation?
• Does the output satisfy a defined quality criterion?
• Should the result pass, fail, or require review?

A decision model evaluates the generated output and returns its judgment. An evaluation framework can collect those results across many examples to show whether an AI application is improving or getting worse.

Decision models are particularly useful when you need a repeatable judgment without a detailed written explanation. More complex evaluations may still need a general-purpose LLM, especially when the evaluator has to explain its reasoning or provide nuanced feedback.

For generated code, model-based evaluations should complement executable tests. A model can assess whether a proposed change appears to follow instructions, but running the code and its tests is still necessary to check whether it behaves correctly.

Decision models vs. LLMs

Decision models and LLMs can perform many of the same tasks, but they have different strengths. Decision models specialize in selecting and scoring defined outcomes, while general-purpose LLMs handle a broader range of generation and reasoning tasks.

Feature	Decision models	General-purpose LLMs
Primary purpose	Evaluate inputs against defined outcomes	Generate text, code, and other content
Outputs	Categories, probabilities, scores, or Boolean judgments	Free-form or structured generated outputs
Common applications	Classification, routing, prioritization, grading	Writing, reasoning, code generation, extraction, tool use
Confidence information	Often return probabilities for predefined outcomes	May expose token probabilities; generated confidence statements need independent validation
Performance	Can reduce latency and cost for bounded decisions	Depends on model size, task complexity, and generated output
Limitations	Restricted to supported decision types and outcomes	Generative inference may add overhead for simple decisions

A decision model might be the more efficient choice when all you need is to determine which team should handle a support request. But if the application also needs to explain an invoice, summarize previous interactions, and draft a response, an LLM is better suited to the work.

And if your LLM already handles classification and response generation in a single call, adding a decision model may not be worth it. Another API call introduces cost and complexity, so the benefits need to justify the change.

Using decision models and LLMs together

You can also use both kinds of models within the same workflow:

• Decision model: Selects the appropriate action or workflow.
• LLM: Performs the task, such as drafting a response or proposing a code change.
• Decision model: Evaluates a defined property of the generated output.

For an application that generates content and makes frequent routing or scoring decisions, combining models may be more efficient than relying on one LLM for everything.

There’s still a tradeoff. Each additional model creates another dependency, so compare the combined approach with a simpler implementation before committing to it.

When should you use a decision model?

A decision model is worth considering when an application needs to interpret information and choose or score predefined outcomes. It’s an especially good candidate when the decision happens frequently enough that latency, cost, or reliability become important.

Start by asking whether you need AI at all. A conditional statement, database lookup, or validation function is usually the better choice when the correct answer can be computed directly.

If the task requires interpreting language or context, compare a decision model with a traditional classifier, an LLM using structured outputs, or whatever your application already uses.

Focus on four things:

• Accuracy: How often does the model choose the right answer? Which mistakes does it make, and how serious are they?
• Latency and cost: Is the decision model faster or cheaper at the volume you expect?
• Probability calibration: Do its probabilities match real-world accuracy? For example, predictions assigned about 90% probability should be correct roughly 90% of the time if the model is well calibrated.
• Integration: Does the model work with your inputs and required outputs? Can you control model versions and monitor changes in behavior?

The best way to find out is to test the options on your own data. Start with real examples where you know the correct answer, then compare how accurately and quickly each approach handles them.

You can also turn those examples into regression tests. Changing a model version, question, or list of allowed outcomes can affect predictions even when the rest of your application hasn’t changed.

Running the evaluations in CI helps catch those changes before they reach production.

Evaluating AI output with decision models

Decision models can serve as automated evaluators, checking whether LLM and agent outputs meet predefined quality criteria.

Traditional software tests compare behavior against known expectations. Testing an AI application can be harder because several responses might be acceptable, and exact string matching won’t tell you whether an answer is relevant, accurate, or follows instructions.

To measure those qualities, developers often use nondeterministic evaluations, or evals, which assess AI-generated outputs against defined criteria rather than requiring an exact match. When an evaluation can be expressed as a choice or score, a decision model can serve as the evaluator.

Testing decision model accuracy

Before relying on a decision model in production, you need to know how accurately it makes decisions. A model that routes support tickets correctly most of the time might still miss urgent security requests or assign high confidence to incorrect predictions.

Start with a labeled dataset containing inputs and their known correct answers. Run the examples through the model, then check its performance in three areas:

• Accuracy: How often does the model choose the correct answer?
• Errors by category: Which mistakes does it make, and how serious are they? For example, how often does a security request end up in the general support queue?
• Probability calibration: Do the model’s confidence estimates match its actual accuracy? Predictions assigned roughly 90% probability should be correct about 90% of the time if the model is well calibrated.

Running these evaluations in CI lets you check for regressions whenever you change the model version, question, or allowed outcomes. You can track whether accuracy drops, new classification mistakes appear, or confidence estimates become less reliable.

Using decision models as judges in CI

An AI assistant that answers questions using company documentation should provide answers supported by its sources. But changes to the assistant’s prompts or underlying model can affect the quality of those answers.

A CI/CD pipeline automatically builds and tests code when changes are made, making it an ideal place to run AI evaluations. You can include decision-model evaluations in your pipeline to catch regressions whenever you update an application’s prompts, models, or agent workflows.

A CI workflow could:

1. Run test cases: Submit representative questions to the assistant.
2. Evaluate responses: Pass each answer and its reference documentation to a decision model.
3. Record judgments: Ask whether each response is supported by the supplied sources.
4. Calculate metrics: Measure how often the answers meet the evaluation criteria.
5. Enforce a quality gate: Fail the pipeline when performance falls below an established threshold.

You can track the results across builds, just like other tests. If a prompt change or model update causes evaluation scores to drop, you can investigate which cases failed before shipping.

The decision model grades the generated responses, while CircleCI runs the evaluation workflow and checks whether the results meet your acceptance criteria.

Example: Running decision-model evaluations in CircleCI

A CircleCI configuration for the evaluation workflow might look like this:

version: 2.1

orbs:
  evals: circleci/evals@2.0.0

jobs:
  evaluate-ai:
    docker:
      - image: cimg/python:stable
    steps:
      - checkout
      - run:
          name: Evaluate AI responses
          command: |
            pip install -r requirements.txt
            python evals.py > eval-results.json
      - evals/test:
          assertions: .circleci/eval-test.json
          metrics: eval-results.json
          results: eval-test-results.xml

workflows:
  evaluate:
    jobs:
      - evaluate-ai:
          context: model-api

The evals.py script represents your evaluation code. It runs the test cases, sends the generated responses to a decision model for grading, and writes the results to eval-results.json. The model-api context supplies the API credentials.

The CircleCI Evals orb checks those metrics against the acceptance criteria in .circleci/eval-test.json. For example, you might require at least 95% of evaluated responses to meet your correctness criteria. If the score falls below that threshold, the CI job fails.

The same pattern works for other kinds of AI evaluations, including instruction-following, answer relevance, and agent behavior. Your evaluation code handles the model judgments; CircleCI runs the checks whenever the application changes.

Before letting these evaluations determine whether a build passes or fails, test the decision model against examples you’ve already reviewed. That gives you a way to measure its accuracy and catch mistakes that could lead to the wrong result.

Keep using deterministic tests for things you can check directly. Decision-model evaluations are useful for assessing harder-to-measure qualities like relevance, instruction-following, and whether an answer is supported by source material.

Building reliable applications with decision models

Decision models fill a useful gap between hard-coded application logic and general-purpose LLMs. You can use them to classify requests, direct agent workflows, and evaluate AI-generated outputs without invoking a generative model for every decision.

Before relying on a decision model in production, test it against real examples from your application. Pay particular attention to mistakes that could affect users, and rerun those tests whenever you change the model or its instructions.

CI makes those checks part of your normal development workflow. You can test decision models themselves, use them to evaluate AI-generated outputs, and catch regressions alongside other software failures.

Ready to bring AI evaluations into your development workflow? CircleCI lets you run model evaluations alongside your existing tests, so you can check AI-generated outputs and catch regressions before they reach production.

Install the CircleCI CLI and run:

circleci onboard

The guided setup walks you through signing up, generating your pipeline configuration, and getting started in minutes. From there, you can begin testing and validating decision models alongside the rest of your application.