diff --git a/docs/agents.md b/docs/agents.md
index 9f20b5b9a..a216c0b75 100644
--- a/docs/agents.md
+++ b/docs/agents.md
@@ -186,6 +186,8 @@ I.decide([
The agent runs the statement on the live page like any other command and keeps it only if it passes. The test then checks it on every run.
+To let the agent use decisions, [configure a decision model](/ai#configure-decision-models) and enable the Decision helper.
+
Decision models like [Jev](https://openrouter.ai/typesafe/jev-1.13) are built for this. They answer in a fraction of a second, cost a fraction of a cent per request, and return a probability instead of free text. That makes them fast and cheap enough to run on every CI build, and predictable enough to keep in a test.
## Skills bundle
diff --git a/docs/ai.md b/docs/ai.md
index e156818fc..140685869 100644
--- a/docs/ai.md
+++ b/docs/ai.md
@@ -353,7 +353,47 @@ I.decideVisually('sidebar is shown')
It uses a [decision model](https://openrouter.ai/models?output_modalities=decisions) like [Jev](https://openrouter.ai/typesafe/jev-1.13) instead of a chat model. A decision model reads the page and returns the probability that a statement is true. It is fast, costs a fraction of a cent per request, and gives a probability instead of free text, so a step passes or fails on a confidence threshold you set.
-Configure it in `ai.decisionModel`. Decisions call the decisions API directly, so they don't need `ai.model` or the `--ai` flag. See [Decision Assertions](/assertions#decision-assertions) for setup and usage.
+Decision models are separate from `ai.model`: decisions don't need `ai.model` or the `--ai` flag. CodeceptJS calls them with AI SDK [`experimental_decide`](https://ai-sdk.dev/docs/ai-sdk-core/decisions).
+
+### Configure Decision Models
+
+Decision models are served by the [OpenRouter Decisions API](https://openrouter.ai/models?output_modalities=decisions). Install the [OpenRouter provider](https://ai-sdk.dev/providers/community-providers/openrouter) `@openrouter/ai-sdk-provider`, set `OPENROUTER_API_KEY`, and create the decision model with `evaluationModel()` in `codecept.conf.js`:
+
+```js
+import { createOpenRouter } from '@openrouter/ai-sdk-provider'
+
+const openrouter = createOpenRouter({
+ apiKey: process.env.OPENROUTER_API_KEY,
+})
+
+export default {
+ ai: {
+ decisionModel: {
+ model: openrouter.evaluationModel('typesafe/jev-1.13'),
+ confidence: 0.8,
+ },
+ },
+ helpers: {
+ Playwright: { url: 'http://localhost' },
+ Decision: {},
+ },
+}
+```
+
+`model` is used by `I.decide`.
+
+### Visual Decision Model
+
+`I.decideVisually` sends a screenshot, so it needs a separate decision model with image input. Set it as `visualModel`:
+
+```js
+decisionModel: {
+ model: openrouter.evaluationModel('typesafe/jev-1.13'),
+ visualModel: openrouter.evaluationModel('cloudflare/clef'),
+},
+```
+
+See [Decision Assertions](/assertions#decision-assertions) for all options and usage.
## Advanced Configuration
diff --git a/docs/assertions.md b/docs/assertions.md
index d18107b2e..5707fb09d 100644
--- a/docs/assertions.md
+++ b/docs/assertions.md
@@ -407,15 +407,9 @@ A [decision model](https://openrouter.ai/models?output_modalities=decisions), li
- **Cost-efficient.** A request costs a fraction of a cent. You can run decision assertions in every CI build.
- **Reliable.** The answer is a probability, not free text. There is nothing to parse, and you choose how confident the model must be for the step to pass.
-Set `OPENROUTER_API_KEY`, configure the decision model in the `ai` section, and enable the helper next to your browser helper:
+Configure the decision model in `ai.decisionModel` as described in [Configure Decision Models](/ai#configure-decision-models), and enable the helper next to your browser helper:
```js
-ai: {
- decisionModel: {
- model: 'typesafe/jev-1.13',
- confidence: 0.7,
- },
-},
helpers: {
Playwright: { url: 'http://localhost' },
Decision: {},
@@ -426,15 +420,13 @@ helpers: {
| Option | Default | Description |
|---|---|---|
-| `provider` | `openrouter` | `openrouter` reads `OPENROUTER_API_KEY`, `typesafe` reads `TYPESAFE_API_KEY` |
-| `apiKey` | | API key, overrides the environment variable |
-| `model` | `typesafe/jev-1.13` | model for `I.decide` |
-| `visualModel` | `cloudflare/clef` | model with image input for `I.decideVisually`, OpenRouter only |
+| `model` | | decision model for `I.decide` |
+| `visualModel` | | decision model with image input for `I.decideVisually` |
| `confidence` | `0.7` | minimal probability for a statement to pass |
| `timeout` | `15000` | request timeout in ms |
| `maxLength` | `12000` | maximal length of ARIA snapshot or HTML sent to the model |
-Decisions don't need the `--ai` flag, and `ai.model` is not required.
+Any model from the [OpenRouter decision models](https://openrouter.ai/models?output_modalities=decisions) list can be used with `openrouter.evaluationModel()`. Decisions don't need the `--ai` flag, and `ai.model` is not required.
Then assert statements about the current page:
@@ -457,7 +449,7 @@ I.decideVisually('sidebar is shown')
expected page to satisfy "success message is shown" (12%) with confidence of 70%
```
-`I.decideVisually` also sends a screenshot, so it needs a model with image input. It uses `visualModel`, which is [Clef](https://openrouter.ai/cloudflare/clef) by default. Visual decisions are experimental.
+`I.decideVisually` also sends a screenshot, so it needs a [visual decision model](/ai#visual-decision-model) with image input, like [Clef](https://openrouter.ai/cloudflare/clef), set as `visualModel`. Visual decisions are experimental.
To turn decisions off without removing the helper, set its `mode`:
diff --git a/docs/helpers/Decision.md b/docs/helpers/Decision.md
index 0e6de31a2..7d202371c 100644
--- a/docs/helpers/Decision.md
+++ b/docs/helpers/Decision.md
@@ -36,31 +36,44 @@ This helper must be enabled together with a web helper (Playwright, Puppeteer, W
## Configuration
-Decision model is configured in the `ai.decisionModel` section of the config:
+Decision model is created with the OpenRouter provider from `@openrouter/ai-sdk-provider`
+and configured in the `ai.decisionModel` section of the config:
```js
-ai: {
- decisionModel: {
- provider: 'openrouter',
- model: 'typesafe/jev-1.13',
- visualModel: 'cloudflare/clef',
- confidence: 0.7,
+import { createOpenRouter } from '@openrouter/ai-sdk-provider'
+
+const openrouter = createOpenRouter({
+ apiKey: process.env.OPENROUTER_API_KEY,
+})
+
+export default {
+ ai: {
+ decisionModel: {
+ model: openrouter.evaluationModel('typesafe/jev-1.13'),
+ confidence: 0.8,
+ },
+ },
+ helpers: {
+ Playwright: { url: 'http://localhost' },
+ Decision: {},
},
-},
-helpers: {
- Playwright: { url: 'http://localhost', browser: 'chromium' },
- Decision: {},
}
```
-* `provider` (default: `openrouter`) - decision API to call: `openrouter` (reads `OPENROUTER_API_KEY`) or `typesafe` (reads `TYPESAFE_API_KEY`).
-* `apiKey` (optional) - API key, overrides the environment variable.
-* `model` (default: `typesafe/jev-1.13`) - decision model used by `decide`. Use `jev-latest` with the `typesafe` provider.
-* `visualModel` (default: `cloudflare/clef`) - decision model with image input used by `decideVisually`. Available on OpenRouter only.
+* `model` - decision model used by `decide`.
* `confidence` (default: `0.7`) - minimal probability, between 0 and 1, for a statement to pass.
* `timeout` (default: `15000`) - request timeout in ms.
* `maxLength` (default: `12000`) - maximal length of ARIA snapshot or HTML sent to the model.
+`decideVisually` uses a separate decision model with image input, configured as `visualModel`:
+
+```js
+decisionModel: {
+ model: openrouter.evaluationModel('typesafe/jev-1.13'),
+ visualModel: openrouter.evaluationModel('cloudflare/clef'),
+},
+```
+
The helper has one option:
* `mode` (default: `assert`) - how decisions are executed:
diff --git a/lib/ai.js b/lib/ai.js
index dcff6af94..e90b725a7 100644
--- a/lib/ai.js
+++ b/lib/ai.js
@@ -3,7 +3,7 @@ const debug = debugModule('codeceptjs:ai')
import output from './output.js'
import event from './event.js'
import { removeNonInteractiveElements, minifyHtml, splitByChunks } from './html.js'
-import { generateText } from 'ai'
+import { generateText, experimental_decide as decide, APICallError } from 'ai'
import { fileURLToPath } from 'url'
import path from 'path'
import { fileExists, resolveImportModulePath } from './utils.js'
@@ -271,15 +271,7 @@ function parseCodeBlocks(response) {
return modifiedSnippets.filter(snippet => !!snippet)
}
-const DECISION_ENDPOINTS = {
- openrouter: { url: 'https://openrouter.ai/api/alpha/decisions', keyName: 'OPENROUTER_API_KEY' },
- typesafe: { url: 'https://api.typesafe.ai/v1/systemone', keyName: 'TYPESAFE_API_KEY' },
-}
-
const defaultDecisionConfig = {
- provider: 'openrouter',
- model: 'typesafe/jev-1.13',
- visualModel: 'cloudflare/clef',
confidence: 0.7,
timeout: 15000,
maxLength: 12000,
@@ -291,97 +283,68 @@ class DecisionAI {
constructor(config = {}) {
this.config = { ...defaultDecisionConfig, ...config }
- const { provider, confidence } = this.config
- this.endpoint = DECISION_ENDPOINTS[provider]
- if (!this.endpoint) throw new Error(`Unknown decision provider "${provider}" in ai.decisionModel, use one of: ${Object.keys(DECISION_ENDPOINTS).join(', ')}`)
+ const { confidence } = this.config
if (!(confidence > 0 && confidence < 1)) throw new Error(`ai.decisionModel.confidence must be between 0 and 1, got ${confidence}`)
-
- this.fetchImpl = fetch
}
- checkModel() {
- if (this.config.apiKey || process.env[this.endpoint.keyName]) return
-
- const noKeyErrorMessage = `
- No API key is set for decision model.
+ checkModel(option) {
+ if (this.config[option]) return
- [!] Set ${this.endpoint.keyName} environment variable or apiKey in ai.decisionModel config.
-
- Example (connect to OpenRouter, default):
-
- export OPENROUTER_API_KEY=sk-or-...
-
- ai: {
- decisionModel: {
- model: 'typesafe/jev-1.13',
- confidence: 0.7,
- }
- }
+ const noModelErrorMessage = `
+ No decision model is set in ai.decisionModel.${option} config.
- Get a key at https://openrouter.ai/settings/keys
+ [!] Configure decision model with OpenRouter provider:
- Example (connect to TypeSafe):
+ import { createOpenRouter } from '@openrouter/ai-sdk-provider'
- export TYPESAFE_API_KEY=...
+ const openrouter = createOpenRouter({
+ apiKey: process.env.OPENROUTER_API_KEY,
+ })
ai: {
decisionModel: {
- provider: 'typesafe',
- model: 'jev-latest',
+ model: openrouter.evaluationModel('typesafe/jev-1.13'),
+ visualModel: openrouter.evaluationModel('cloudflare/clef'),
}
}
See https://openrouter.ai/models?output_modalities=decisions for all decision models.
`.trim()
- throw new Error(noKeyErrorMessage)
+ throw new Error(noModelErrorMessage)
}
- async decide(model, state, statements) {
- this.checkModel()
+ async decide(option, state, statements) {
+ this.checkModel(option)
+ const model = this.config[option]
- const apiKey = this.config.apiKey || process.env[this.endpoint.keyName]
- const questions = Object.fromEntries(statements.map((statement, i) => [`q${i}`, { type: 'noul', instructions: statement }]))
- debug('Decision request', model, statements)
+ const questions = Object.fromEntries(statements.map((statement, i) => [`q${i}`, { type: 'boolean', instructions: statement }]))
+ debug('Decision request', model.modelId, statements)
const controller = new AbortController()
const timer = setTimeout(() => controller.abort(), this.config.timeout)
let result
try {
- let response
- try {
- response = await this.fetchImpl(this.endpoint.url, {
- method: 'POST',
- headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
- body: JSON.stringify({ model, state, questions }),
- signal: controller.signal,
- })
- } catch (err) {
- throw new DecisionConnectionError(`Decision model ${model} request failed: ${err.message}`)
- }
-
- if (!response.ok) {
- const body = await response.text().catch(() => '')
- throw new Error(`Decision model ${model} responded with ${response.status}: ${body}`)
- }
-
- result = await response.json()
+ result = await decide({
+ model,
+ state,
+ questions,
+ maxRetries: 0,
+ abortSignal: controller.signal,
+ })
} catch (err) {
- if (controller.signal.aborted) throw new DecisionConnectionError(`Decision model ${model} did not respond in ${this.config.timeout}ms`)
+ if (controller.signal.aborted) throw new DecisionConnectionError(`Decision model ${model.modelId} did not respond in ${this.config.timeout}ms`)
+ if (APICallError.isInstance(err) && !err.statusCode) throw new DecisionConnectionError(`Decision model ${model.modelId} request failed: ${err.message}`)
+ if (APICallError.isInstance(err)) throw new Error(`Decision model ${model.modelId} responded with ${err.statusCode}: ${err.responseBody || err.message}`)
throw err
} finally {
clearTimeout(timer)
}
- debug('Decision response', result?.answers, result?.usage)
+ debug('Decision response', result.answers, result.usage)
- const answers = result?.answers || {}
- return statements.map((statement, i) => {
- const probability = answers[`q${i}`]?.noul
- if (typeof probability !== 'number') throw new Error(`Decision model ${model} returned no answer for "${statement}"`)
- return probability
- })
+ return Object.keys(questions).map(id => result.answers[id].probability)
}
}
diff --git a/lib/helper/Decision.js b/lib/helper/Decision.js
index e67ac2f90..4bc2bc505 100644
--- a/lib/helper/Decision.js
+++ b/lib/helper/Decision.js
@@ -38,31 +38,44 @@ import { DecisionAI, DecisionConnectionError } from '../ai.js'
*
* ## Configuration
*
- * Decision model is configured in the `ai.decisionModel` section of the config:
+ * Decision model is created with the OpenRouter provider from `@openrouter/ai-sdk-provider`
+ * and configured in the `ai.decisionModel` section of the config:
*
* ```js
- * ai: {
- * decisionModel: {
- * provider: 'openrouter',
- * model: 'typesafe/jev-1.13',
- * visualModel: 'cloudflare/clef',
- * confidence: 0.7,
+ * import { createOpenRouter } from '@openrouter/ai-sdk-provider'
+ *
+ * const openrouter = createOpenRouter({
+ * apiKey: process.env.OPENROUTER_API_KEY,
+ * })
+ *
+ * export default {
+ * ai: {
+ * decisionModel: {
+ * model: openrouter.evaluationModel('typesafe/jev-1.13'),
+ * confidence: 0.8,
+ * },
+ * },
+ * helpers: {
+ * Playwright: { url: 'http://localhost' },
+ * Decision: {},
* },
- * },
- * helpers: {
- * Playwright: { url: 'http://localhost', browser: 'chromium' },
- * Decision: {},
* }
* ```
*
- * * `provider` (default: `openrouter`) - decision API to call: `openrouter` (reads `OPENROUTER_API_KEY`) or `typesafe` (reads `TYPESAFE_API_KEY`).
- * * `apiKey` (optional) - API key, overrides the environment variable.
- * * `model` (default: `typesafe/jev-1.13`) - decision model used by `decide`. Use `jev-latest` with the `typesafe` provider.
- * * `visualModel` (default: `cloudflare/clef`) - decision model with image input used by `decideVisually`. Available on OpenRouter only.
+ * * `model` - decision model used by `decide`.
* * `confidence` (default: `0.7`) - minimal probability, between 0 and 1, for a statement to pass.
* * `timeout` (default: `15000`) - request timeout in ms.
* * `maxLength` (default: `12000`) - maximal length of ARIA snapshot or HTML sent to the model.
*
+ * `decideVisually` uses a separate decision model with image input, configured as `visualModel`:
+ *
+ * ```js
+ * decisionModel: {
+ * model: openrouter.evaluationModel('typesafe/jev-1.13'),
+ * visualModel: openrouter.evaluationModel('cloudflare/clef'),
+ * },
+ * ```
+ *
* The helper has one option:
*
* * `mode` (default: `assert`) - how decisions are executed:
@@ -119,7 +132,7 @@ class Decision extends Helper {
if (this.isSkipping) return this._skip()
return this._run(async () => {
const state = await this._grabState()
- return this._assert(statements, this.options.model, state)
+ return this._assert(statements, 'model', state)
})
}
@@ -145,7 +158,7 @@ class Decision extends Helper {
{ type: 'text', text: JSON.stringify(state) },
{ type: 'image_url', image_url: { url: `data:image/png;base64,${screenshot}` } },
]
- return this._assert(statements, this.options.visualModel, content)
+ return this._assert(statements, 'visualModel', content)
})
}
@@ -167,11 +180,11 @@ class Decision extends Helper {
}
}
- async _assert(statements, model, state) {
+ async _assert(statements, option, state) {
const list = [statements].flat()
if (!list.length) throw new Error('No statements to decide')
- const probabilities = await this.decisionAI.decide(model, state, list)
+ const probabilities = await this.decisionAI.decide(option, state, list)
const failed = []
const results = list.map((statement, i) => {
diff --git a/package.json b/package.json
index d6771005d..7d88c05d8 100644
--- a/package.json
+++ b/package.json
@@ -101,7 +101,7 @@
"@modelcontextprotocol/sdk": "^1.26.0",
"@xmldom/xmldom": "0.9.10",
"acorn": "8.15.0",
- "ai": "^6.0.43",
+ "ai": "7.0.132",
"arrify": "3.0.0",
"axios": "1.16.1",
"chalk": "4.1.2",
@@ -149,6 +149,7 @@
"@eslint/js": "9.39.2",
"@faker-js/faker": "10.3.0",
"@inquirer/testing": "^3.0.3",
+ "@openrouter/ai-sdk-provider": "^3.1.0",
"@pollyjs/adapter-puppeteer": "6.0.6",
"@pollyjs/core": "6.0.6",
"@testomatio/reporter": "^2.3.1",
diff --git a/test/unit/ai_test.js b/test/unit/ai_test.js
index 89b457e13..b70d7bbfd 100644
--- a/test/unit/ai_test.js
+++ b/test/unit/ai_test.js
@@ -1,5 +1,6 @@
import { expect } from 'chai'
import AiAssistant, { DecisionAI } from '../../lib/ai.js'
+import { createOpenRouter } from '@openrouter/ai-sdk-provider'
import config from '../../lib/config.js'
import { createMockModel, MockResponses } from '../support/mock-ai-provider.js'
import fs from 'fs'
@@ -299,36 +300,25 @@ describe('AI module with mock provider', () => {
})
describe('DecisionAI', () => {
- it('guides user to set API key', async () => {
- const key = process.env.TYPESAFE_API_KEY
- delete process.env.TYPESAFE_API_KEY
- try {
- const decisionAI = new DecisionAI({ provider: 'typesafe' })
- expect(() => decisionAI.checkModel()).to.throw(/TYPESAFE_API_KEY[\s\S]*openrouter\.ai\/settings\/keys/)
- const err = await decisionAI.decide('jev-latest', 'state', ['page is loaded']).catch(e => e)
- expect(err.message).to.include('No API key is set for decision model')
- } finally {
- if (key) process.env.TYPESAFE_API_KEY = key
- }
- })
-
- it('accepts API key from config', () => {
- expect(() => new DecisionAI({ provider: 'typesafe', apiKey: 'secret' }).checkModel()).not.to.throw()
- })
-
- it('rejects unknown provider', () => {
- expect(() => new DecisionAI({ provider: 'unknown' })).to.throw('Unknown decision provider')
+ it('guides user to configure decision model', async () => {
+ const decisionAI = new DecisionAI()
+ expect(() => decisionAI.checkModel('model')).to.throw(/ai\.decisionModel\.model[\s\S]*@openrouter\/ai-sdk-provider/)
+ const err = await decisionAI.decide('model', 'state', ['page is loaded']).catch(e => e)
+ expect(err.message).to.include('No decision model is set')
})
it('asks all statements in one request', async () => {
const calls = []
- const decisionAI = new DecisionAI({ apiKey: 'secret' })
- decisionAI.fetchImpl = async (url, options) => {
- calls.push({ url, body: JSON.parse(options.body) })
- return { ok: true, json: async () => ({ answers: { q0: { type: 'noul', noul: 0.9 }, q1: { type: 'noul', noul: 0.2 } } }) }
- }
+ const openrouter = createOpenRouter({
+ apiKey: 'secret',
+ fetch: async (url, options) => {
+ calls.push({ url, body: JSON.parse(options.body) })
+ return Response.json({ answers: { q0: { type: 'noul', noul: 0.9 }, q1: { type: 'noul', noul: 0.2 } } })
+ },
+ })
+ const decisionAI = new DecisionAI({ model: openrouter.evaluationModel('typesafe/jev-1.13') })
- const probabilities = await decisionAI.decide('typesafe/jev-1.13', { url: 'http://localhost' }, ['form is shown', 'cart is empty'])
+ const probabilities = await decisionAI.decide('model', { url: 'http://localhost' }, ['form is shown', 'cart is empty'])
expect(probabilities).to.eql([0.9, 0.2])
expect(calls).to.have.length(1)
diff --git a/test/unit/helper/Decision_test.js b/test/unit/helper/Decision_test.js
index 01693f639..0d31d14da 100644
--- a/test/unit/helper/Decision_test.js
+++ b/test/unit/helper/Decision_test.js
@@ -5,21 +5,25 @@ import path from 'path'
import Decision from '../../../lib/helper/Decision.js'
import store from '../../../lib/store.js'
import Config from '../../../lib/config.js'
+import { createOpenRouter } from '@openrouter/ai-sdk-provider'
-function createDecision(decisionModel, config = {}) {
- Config.create({ ai: { decisionModel } })
+let fetchImpl
+const openrouter = createOpenRouter({ apiKey: 'secret', fetch: (...args) => fetchImpl(...args) })
+const decisionModels = {
+ model: openrouter.evaluationModel('typesafe/jev-1.13'),
+ visualModel: openrouter.evaluationModel('cloudflare/clef'),
+}
+
+function createDecision(decisionModel = {}, config = {}) {
+ Config.create({ ai: { decisionModel: { ...decisionModels, ...decisionModel } } })
return new Decision(config)
}
function fakeFetch(answers, calls, status = 200) {
return async (url, options) => {
- calls.push({ url, body: JSON.parse(options.body), headers: options.headers })
- return {
- ok: status === 200,
- status,
- text: async () => 'boom',
- json: async () => ({ answers }),
- }
+ calls.push({ url, body: JSON.parse(options.body), headers: new Headers(options.headers) })
+ if (status !== 200) return new Response('boom', { status })
+ return Response.json({ answers })
}
}
@@ -42,18 +46,18 @@ describe('Decision helper', () => {
beforeEach(() => {
calls = []
- decision = createDecision({ apiKey: 'secret' })
+ decision = createDecision()
decision._actingHelper = () => browser
})
it('passes when probability reaches confidence', async () => {
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls)
+ fetchImpl = fakeFetch(noul(0.9), calls)
const probability = await decision.decide('submit button is present')
expect(probability).to.equal(0.9)
expect(calls).to.have.length(1)
expect(calls[0].url).to.equal('https://openrouter.ai/api/alpha/decisions')
- expect(calls[0].headers.Authorization).to.equal('Bearer secret')
+ expect(calls[0].headers.get('authorization')).to.equal('Bearer secret')
expect(calls[0].body.model).to.equal('typesafe/jev-1.13')
expect(calls[0].body.questions).to.eql({ q0: { type: 'noul', instructions: 'submit button is present' } })
expect(calls[0].body.state.url).to.equal('http://localhost/checkout')
@@ -62,7 +66,7 @@ describe('Decision helper', () => {
})
it('fails when probability is below confidence', async () => {
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.4), calls)
+ fetchImpl = fakeFetch(noul(0.4), calls)
const err = await decision.decide('cart is empty').catch(e => e)
expect(err).to.be.instanceOf(Error)
@@ -71,16 +75,16 @@ describe('Decision helper', () => {
})
it('respects configured confidence', async () => {
- decision = createDecision({ apiKey: 'secret', confidence: 0.95 })
+ decision = createDecision({ confidence: 0.95 })
decision._actingHelper = () => browser
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls)
+ fetchImpl = fakeFetch(noul(0.9), calls)
const err = await decision.decide('cart is empty').catch(e => e)
expect(err.message).to.include('95%')
})
it('checks all statements in a single request', async () => {
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.9, 0.99, 0.8), calls)
+ fetchImpl = fakeFetch(noul(0.9, 0.99, 0.8), calls)
const probabilities = await decision.decide(['form has fields', 'submit enabled', 'cancel present'])
expect(probabilities).to.eql([0.9, 0.99, 0.8])
@@ -90,7 +94,7 @@ describe('Decision helper', () => {
})
it('lists only failed statements', async () => {
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.9, 0.1, 0.2), calls)
+ fetchImpl = fakeFetch(noul(0.9, 0.1, 0.2), calls)
const err = await decision.decide(['form has fields', 'submit enabled', 'cancel present']).catch(e => e)
expect(err.message).to.include('"submit enabled" (10%)')
@@ -101,7 +105,7 @@ describe('Decision helper', () => {
it('sends screenshot to visual model', async () => {
const outputDir = store.outputDir
store.outputDir = fs.mkdtempSync(path.join(os.tmpdir(), 'decision-'))
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.8), calls)
+ fetchImpl = fakeFetch(noul(0.8), calls)
try {
await decision.decideVisually('sidebar is shown')
expect(fs.readdirSync(store.outputDir)).to.be.empty
@@ -120,39 +124,29 @@ describe('Decision helper', () => {
it('sends html when aria snapshot is not supported', async () => {
decision._actingHelper = () => ({ ...browser, grabAriaSnapshot: undefined, grabSource: async () => 'Checkout
' })
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls)
+ fetchImpl = fakeFetch(noul(0.9), calls)
await decision.decide('heading is shown')
expect(calls[0].body.state.aria).to.be.undefined
expect(calls[0].body.state.html).to.include('Checkout
')
})
- it('uses typesafe endpoint', async () => {
- decision = createDecision({ apiKey: 'secret', provider: 'typesafe', model: 'jev-latest' })
- decision._actingHelper = () => browser
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls)
- await decision.decide('page is loaded')
-
- expect(calls[0].url).to.equal('https://api.typesafe.ai/v1/systemone')
- expect(calls[0].body.model).to.equal('jev-latest')
- })
-
it('reports http errors', async () => {
- decision.decisionAI.fetchImpl = fakeFetch({}, calls, 402)
+ fetchImpl = fakeFetch({}, calls, 402)
const err = await decision.decide('page is loaded').catch(e => e)
expect(err.message).to.include('402')
})
it('reports missing answers', async () => {
- decision.decisionAI.fetchImpl = fakeFetch({}, calls)
+ fetchImpl = fakeFetch({}, calls)
const err = await decision.decide('page is loaded').catch(e => e)
- expect(err.message).to.include('returned no answer for "page is loaded"')
+ expect(err.message).to.include('Decision must return exactly one answer for every question')
})
it('times out hanging requests', async () => {
- decision = createDecision({ apiKey: 'secret', timeout: 50 })
+ decision = createDecision({ timeout: 50 })
decision._actingHelper = () => browser
- decision.decisionAI.fetchImpl = (url, { signal }) => new Promise((_, reject) => signal.addEventListener('abort', () => reject(new Error('aborted'))))
+ fetchImpl = (url, { signal }) => new Promise((_, reject) => signal.addEventListener('abort', () => reject(new Error('aborted'))))
const err = await decision.decide('page is loaded').catch(e => e)
expect(err.message).to.include('did not respond in 50ms')
@@ -160,13 +154,12 @@ describe('Decision helper', () => {
})
it('times out stalled response body', async () => {
- decision = createDecision({ apiKey: 'secret', timeout: 50 })
+ decision = createDecision({ timeout: 50 })
decision._actingHelper = () => browser
- decision.decisionAI.fetchImpl = async (url, { signal }) => ({
- ok: true,
- status: 200,
- json: () => new Promise((_, reject) => signal.addEventListener('abort', () => reject(new Error('aborted')))),
- })
+ fetchImpl = async (url, { signal }) => {
+ const body = new ReadableStream({ start: controller => signal.addEventListener('abort', () => controller.error(new Error('aborted'))) })
+ return new Response(body, { headers: { 'content-type': 'application/json' } })
+ }
const err = await decision.decide('page is loaded').catch(e => e)
expect(err.message).to.include('did not respond in 50ms')
@@ -174,13 +167,13 @@ describe('Decision helper', () => {
})
it('marks failed decisions as not retryable', async () => {
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.1), calls)
+ fetchImpl = fakeFetch(noul(0.1), calls)
const err = await decision.decide('cart is empty').catch(e => e)
expect(err.isTerminal).to.equal(true)
})
it('marks http errors as not retryable', async () => {
- decision.decisionAI.fetchImpl = fakeFetch({}, calls, 500)
+ fetchImpl = fakeFetch({}, calls, 500)
const err = await decision.decide('page is loaded').catch(e => e)
expect(err.isTerminal).to.equal(true)
})
@@ -193,40 +186,46 @@ describe('Decision helper', () => {
})
it('keeps connection errors retryable', async () => {
- decision.decisionAI.fetchImpl = async () => {
- throw new TypeError('fetch failed')
+ fetchImpl = async () => {
+ throw new TypeError('fetch failed', { cause: new Error('connect ECONNREFUSED') })
}
const err = await decision.decide('page is loaded').catch(e => e)
- expect(err.message).to.include('request failed: fetch failed')
+ expect(err.message).to.include('request failed: Cannot connect to API: connect ECONNREFUSED')
expect(err.isTerminal).to.be.undefined
})
it('reads config from ai.decisionModel', () => {
- decision = createDecision({ apiKey: 'secret', model: 'jev-latest', confidence: 0.9 })
- expect(decision.options.model).to.equal('jev-latest')
+ decision = createDecision({ confidence: 0.9 })
+ expect(decision.options.model.modelId).to.equal('typesafe/jev-1.13')
+ expect(decision.options.visualModel.modelId).to.equal('cloudflare/clef')
expect(decision.options.confidence).to.equal(0.9)
- expect(decision.options.visualModel).to.equal('cloudflare/clef')
})
it('validates config', () => {
- expect(() => createDecision({ apiKey: 'secret', provider: 'unknown' })).to.throw('Unknown decision provider')
- expect(() => createDecision({ apiKey: 'secret', confidence: 1.5 })).to.throw('between 0 and 1')
+ expect(() => createDecision({ confidence: 1.5 })).to.throw('between 0 and 1')
})
- it('requires api key only when deciding', async () => {
- const key = process.env.TYPESAFE_API_KEY
- delete process.env.TYPESAFE_API_KEY
- try {
- decision = createDecision({ provider: 'typesafe' })
- decision._actingHelper = () => browser
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls)
+ it('requires decision model only when deciding', async () => {
+ decision = createDecision({ model: undefined })
+ decision._actingHelper = () => browser
+ fetchImpl = fakeFetch(noul(0.9), calls)
- const err = await decision.decide('page is loaded').catch(e => e)
- expect(err.message).to.include('TYPESAFE_API_KEY')
- expect(calls).to.be.empty
- } finally {
- if (key) process.env.TYPESAFE_API_KEY = key
- }
+ const err = await decision.decide('page is loaded').catch(e => e)
+ expect(err.message).to.include('No decision model is set in ai.decisionModel.model')
+ expect(err.message).to.include("openrouter.evaluationModel('typesafe/jev-1.13')")
+ expect(calls).to.be.empty
+ })
+
+ it('requires visual model for visual decisions', async () => {
+ decision = createDecision({ visualModel: undefined })
+ decision._actingHelper = () => browser
+ const outputDir = store.outputDir
+ store.outputDir = fs.mkdtempSync(path.join(os.tmpdir(), 'decision-'))
+
+ const err = await decision.decideVisually('sidebar is shown').catch(e => e)
+ fs.rmSync(store.outputDir, { recursive: true, force: true })
+ store.outputDir = outputDir
+ expect(err.message).to.include('No decision model is set in ai.decisionModel.visualModel')
})
describe('mode', () => {
@@ -236,7 +235,7 @@ describe('Decision helper', () => {
it('skips decisions without requests', async () => {
decision = createDecision({}, { mode: 'skip' })
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.1), calls)
+ fetchImpl = fakeFetch(noul(0.1), calls)
store.currentStep = { comment: '' }
expect(await decision.decide('cart is empty')).to.be.undefined
@@ -246,9 +245,9 @@ describe('Decision helper', () => {
})
it('reports results in step comment without failing', async () => {
- decision = createDecision({ apiKey: 'secret' }, { mode: 'report' })
+ decision = createDecision({}, { mode: 'report' })
decision._actingHelper = () => browser
- decision.decisionAI.fetchImpl = fakeFetch(noul(0.9, 0.1), calls)
+ fetchImpl = fakeFetch(noul(0.9, 0.1), calls)
store.currentStep = { comment: '' }
const probabilities = await decision.decide(['form has fields', 'submit enabled'])
@@ -259,9 +258,9 @@ describe('Decision helper', () => {
})
it('reports api errors in step comment without failing', async () => {
- decision = createDecision({ apiKey: 'secret' }, { mode: 'report' })
+ decision = createDecision({}, { mode: 'report' })
decision._actingHelper = () => browser
- decision.decisionAI.fetchImpl = fakeFetch({}, calls, 500)
+ fetchImpl = fakeFetch({}, calls, 500)
store.currentStep = { comment: '' }
expect(await decision.decide('page is loaded')).to.be.undefined