Skip to content

feat(Decision): use AI SDK decisions, configure decision models in ai.decisionModel - #5742

Merged
DavertMik merged 5 commits into
4.xfrom
feat/decision-ai-sdk
Oct 9, 2026
Merged

DavertMik merged 5 commits into
4.xfrom
feat/decision-ai-sdk

Conversation

@DavertMik

@DavertMik DavertMik commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Decision helper now uses AI SDK experimental_decide with decision models from the OpenRouter provider, replacing hand-written fetch calls.

Decision models are configured only one way, in codecept.conf.js:

import { createOpenRouter } from '@openrouter/ai-sdk-provider'

const openrouter = createOpenRouter({
  apiKey: process.env.OPENROUTER_API_KEY,
})

export default {
  ai: {
    decisionModel: {
      model: openrouter.evaluationModel('typesafe/jev-1.13'),
      confidence: 0.8,
    },
  },
  helpers: {
    Playwright: { url: 'http://localhost' },
    Decision: {},
  },
}

I.decideVisually uses a separate model with image input, set as visualModel: openrouter.evaluationModel('cloudflare/clef').

Changes

  • ai.decisionModel.model and visualModel take AI SDK decision models. If one is missing, the step fails with an error that shows the config above.
  • Statements are sent as boolean questions.
  • Breaking: these are removed:
    • the provider and apiKey options;
    • model ID strings;
    • default models.
  • Breaking: ai is bumped from ^6 to 7.0.132 (pinned), because experimental_decide requires v7. From 7.0.133 on, experimental_decide wraps state into SDK parts that @openrouter/ai-sdk-provider 3.1.0 forwards untranslated, which breaks screenshot input. Unpin once the OpenRouter provider supports the new state format. Users who bring their own @ai-sdk/openai or @ai-sdk/anthropic for ai.model need provider versions compatible with ai@7.
  • @openrouter/ai-sdk-provider is a dev dependency. Users install it themselves, like other AI SDK providers.
  • Behavior is otherwise unchanged:
    • No retries (maxRetries: 0), and the timeout still applies.
    • Connection failures and timeouts raise a retryable DecisionConnectionError. HTTP errors fail the step and report the status and body.
    • decideVisually still sends the screenshot.
  • Missing answers are now caught by the SDK, so the error message changed to "Decision must return exactly one answer for every question."

Docs

  • ai.md:
    • Configure Decision Models: what to import and how to configure model.
    • Visual Decision Model: how to configure visualModel.
  • assertions.md: links to the ai.md config; options table updated.
  • agents.md: links to the ai.md config.
  • helpers/Decision.md: regenerated from the helper JSDoc.

Test plan

  • npm run test:unit: 882 passed. Tests configure models with createOpenRouter and a fake fetch.
  • Live check against OpenRouter with a real key

🤖 Generated with Claude Code

DavertMik and others added 3 commits October 8, 2026 02:03
Replace hand-written fetch calls to OpenRouter/TypeSafe decision endpoints
with AI SDK experimental_decide via @openrouter/ai-sdk-provider and
@ai-sdk/typesafe-ai. Bumps ai to v7, which experimental_decide requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cisions

Drop the decision provider option and @ai-sdk/typesafe-ai: all decision
models, including TypeSafe Jev, are served by the OpenRouter Decisions API.
Document how decisions use AI SDK experimental_decide and how to ask
choice/score questions from a custom helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ai.decisionModel.model and visualModel accept an OpenRouter model ID or an
AI SDK decision model, e.g. createOpenRouter().evaluationModel(id).
Document configuring decision models in ai.md; drop custom decisions section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DavertMik DavertMik changed the title feat(Decision): use AI SDK experimental_decide feat(Decision): use AI SDK decisions, configure decision models in ai.decisionModel Oct 8, 2026
DavertMik and others added 2 commits October 8, 2026 03:35
ai.decisionModel.model and visualModel must be AI SDK decision models
created with createOpenRouter().evaluationModel(). Drop model ID strings,
default models and apiKey option. @openrouter/ai-sdk-provider becomes a
dev dependency; users install it like other AI SDK providers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ai 7.0.133 wraps decision state into SDK parts that
@openrouter/ai-sdk-provider 3.1.0 forwards untranslated, so OpenRouter
receives wrapped state and screenshots lose the image_url format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DavertMik
DavertMik merged commit a459969 into 4.x Oct 9, 2026
15 checks passed
@DavertMik
DavertMik deleted the feat/decision-ai-sdk branch October 9, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant