fix(consult): stop the model spending its step budget on invalid tool params
A staging consultation calculated the chart and then returned nothing but the ensureFinalResponseText fallback. The model had made four calls to run-jyotish-consultation, and two of them never reached a calculation: they set both domains and theme, which canonicalDomainPlan rejects at execution. The schema declared those two fields as independent optionals, the description never mentioned the constraint, and the instructions actively told the model to use theme for a single-domain retry. Each attempt therefore bought a rule the contract never stated, and because the throw happens before the step-recording try/catch, it left no trace in the receipt either. Make the constraint unrepresentable instead of enforced. The model-facing schema keeps only question and domains, so Mastra refuses the pair before the tool body runs; the description states the single-array contract, and the instruction that advertised theme is gone. canonicalDomainPlan still resolves the single-value form for callers that build a plan without that schema, and is now exported so that path has its own tests. maxSteps and the abort timeout bound the same run but were hard-coded apart. One calculation takes about 20s against a 110s budget, so time is the binding constraint and three failed calculations exhaust it whatever the step count. The budget only has to cover the longest useful shape, so it moves to 8 beside the timeout with that reasoning recorded, and the recorded step list is sized to match so an exhausted run cannot truncate its own evidence. Step exhaustion was only ever inferable by counting events, since finishReason was recorded nowhere and progressive-disclosure reads never reach the public stream. Capture it as a closed enum plus a step count, normalizing anything unrecognized, and log both as controlled fields. Neither may enter the client receipt, whose step schema is strict and would fail a successful run. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -36,6 +36,7 @@ import { streamAgentResponse } from "@/lib/stream-agent-response";
|
||||
import type { AgentExecutionReceipt, WorkflowReceipt } from "@/lib/consultation-agent-events";
|
||||
import {
|
||||
createConsultationAgentContext,
|
||||
consultationModelStepTelemetry,
|
||||
consultationStepBudgetReceipt,
|
||||
createConsultationRuntimeHooks,
|
||||
createConsultationRuntimeState,
|
||||
@@ -62,6 +63,16 @@ import { z } from "zod";
|
||||
export const runtime = "nodejs";
|
||||
export const maxDuration = 120;
|
||||
|
||||
// These two limits bound the same Agent run and must be changed together. One
|
||||
// chart calculation takes about 20s and maxDuration caps the request near 120s,
|
||||
// so wall-clock time, not steps, is the binding constraint: three failed
|
||||
// calculations exhaust the timeout no matter how many steps remain. The step
|
||||
// budget therefore only has to cover the longest useful shape—skill load, a
|
||||
// couple of progressive-disclosure reference reads, one calculation plus one
|
||||
// retry, and the answer turn—since a larger budget cannot buy more time.
|
||||
const AGENT_MAX_STEPS = 8;
|
||||
const AGENT_TIMEOUT_MS = 110_000;
|
||||
|
||||
const chatRequestMetadataSchema = z.object({
|
||||
requestId: z.string().uuid(),
|
||||
sessionId: z.string().uuid(),
|
||||
@@ -513,7 +524,10 @@ export async function POST(request: Request) {
|
||||
name: string,
|
||||
generalDailyContext: GeneralDailyPanchangaContext | null,
|
||||
) {
|
||||
const state = createConsultationRuntimeState();
|
||||
// The recorded step list has to be able to hold everything the model loop
|
||||
// can produce, otherwise a run that exhausts its steps also truncates the
|
||||
// evidence of having done so.
|
||||
const state = createConsultationRuntimeState({ plannedSteps: AGENT_MAX_STEPS });
|
||||
const hooks = createConsultationRuntimeHooks(state);
|
||||
const usages: Promise<Usage>[] = [];
|
||||
const agentStartedAt = Date.now();
|
||||
@@ -581,6 +595,7 @@ export async function POST(request: Request) {
|
||||
},
|
||||
],
|
||||
retryCount: Math.max(0, usages.length - 1),
|
||||
...consultationModelStepTelemetry(state),
|
||||
...(errorCode === undefined ? {} : { errorCode }),
|
||||
inputTokens,
|
||||
outputTokens,
|
||||
@@ -621,10 +636,10 @@ export async function POST(request: Request) {
|
||||
].filter(Boolean).join("\n"),
|
||||
},
|
||||
];
|
||||
const agentAbortSignal = AbortSignal.timeout(110_000);
|
||||
const agentAbortSignal = AbortSignal.timeout(AGENT_TIMEOUT_MS);
|
||||
const streamOptions = {
|
||||
runId: requestId,
|
||||
maxSteps: 6,
|
||||
maxSteps: AGENT_MAX_STEPS,
|
||||
abortSignal: agentAbortSignal,
|
||||
hooks,
|
||||
};
|
||||
|
||||
@@ -47,6 +47,37 @@ export const billingSettlementResults = [
|
||||
export type AgentBillingSettlementResult = (typeof billingSettlementResults)[number];
|
||||
export type AgentSettlementResult = Exclude<AgentBillingSettlementResult, "not_applicable">;
|
||||
|
||||
/**
|
||||
* Closed vocabulary for why the model stopped stepping. These are the provider
|
||||
* finish reasons plus the two Mastra adds; anything else normalizes to
|
||||
* `unknown` so an unrecognized provider string can never become a log field.
|
||||
*
|
||||
* `tool-calls` on a run that produced no answer means the step budget ran out
|
||||
* while the model still wanted to call a tool, which is otherwise only
|
||||
* inferable from the recorded step list.
|
||||
*/
|
||||
export const agentModelFinishReasons = [
|
||||
"stop",
|
||||
"length",
|
||||
"content-filter",
|
||||
"tool-calls",
|
||||
"error",
|
||||
"other",
|
||||
"tripwire",
|
||||
"retry",
|
||||
"unknown",
|
||||
] as const;
|
||||
|
||||
export type AgentModelFinishReason = (typeof agentModelFinishReasons)[number];
|
||||
|
||||
const knownModelFinishReasons = new Set<string>(agentModelFinishReasons);
|
||||
|
||||
export function toAgentModelFinishReason(value: unknown): AgentModelFinishReason {
|
||||
return typeof value === "string" && knownModelFinishReasons.has(value)
|
||||
? value as AgentModelFinishReason
|
||||
: "unknown";
|
||||
}
|
||||
|
||||
export type AgentSettlementTelemetryOutcome = Readonly<{
|
||||
billingSettlementResult: AgentSettlementResult;
|
||||
errorCode?: string;
|
||||
@@ -97,6 +128,10 @@ export const agentObservabilityEventSchema = z.object({
|
||||
contractPhases: z.array(agentObservabilityContractPhaseSchema).max(64).optional(),
|
||||
retryCount: z.number().int().min(0).max(100).optional(),
|
||||
errorCode: machineCodeSchema.optional(),
|
||||
// Why the model stopped, and how many model steps the run consumed across
|
||||
// every attempt. Both are enum-like machine values, never provider text.
|
||||
modelFinishReason: z.enum(agentModelFinishReasons).optional(),
|
||||
modelStepCount: countSchema.optional(),
|
||||
|
||||
inputTokens: tokenCountSchema.optional(),
|
||||
outputTokens: tokenCountSchema.optional(),
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
type AgentExecutionReceipt,
|
||||
type ConsultationAgentPublicEvent,
|
||||
} from "./consultation-agent-events.ts";
|
||||
import { toAgentModelFinishReason } from "./agent-observability.ts";
|
||||
import { createVisibleTextTransformer } from "./stream-text-response.ts";
|
||||
|
||||
type Chunk = { type?: string; payload?: Record<string, unknown>; data?: unknown };
|
||||
@@ -58,6 +59,25 @@ function activity(value: unknown): ConsultationAgentPublicEvent | null {
|
||||
return { type: "activity", phase: phase.data, label: data.label.slice(0, 120) };
|
||||
}
|
||||
|
||||
/**
|
||||
* The runtime only reveals how the model loop ended through the stream: one
|
||||
* `step-finish` per model step, then a terminal `finish` carrying the reason
|
||||
* the model stopped and the authoritative step list. Without this, a run that
|
||||
* exhausted its step budget is indistinguishable from one that chose to stop,
|
||||
* because progressive-disclosure reads never reach the public event stream.
|
||||
*/
|
||||
function finishTelemetry(chunk: Chunk) {
|
||||
const payload = chunk.payload as {
|
||||
stepResult?: { reason?: unknown };
|
||||
output?: { steps?: unknown };
|
||||
} | undefined;
|
||||
const steps = payload?.output?.steps;
|
||||
return {
|
||||
reason: toAgentModelFinishReason(payload?.stepResult?.reason),
|
||||
stepCount: Array.isArray(steps) ? steps.length : null,
|
||||
};
|
||||
}
|
||||
|
||||
function safeToolError(error: unknown) {
|
||||
if (error instanceof DOMException && error.name === "AbortError") return "cancelled" as const;
|
||||
if (error instanceof DOMException && error.name === "TimeoutError") return "timeout" as const;
|
||||
@@ -193,8 +213,17 @@ export function streamAgentResponse(options: StreamAgentResponseOptions) {
|
||||
if (/\S/.test(held)) emitted = true;
|
||||
held = "";
|
||||
};
|
||||
// A retry runs a second model loop under the same step budget, so the run
|
||||
// total accumulates while the finish reason describes the latest attempt.
|
||||
const stepCountBeforeAttempt = options.state.modelStepCount;
|
||||
for await (const chunk of readChunks(stream)) {
|
||||
for (const event of mapChunk(chunk, options, startedAt, jyotishSkillCallIds)) send(controller, event);
|
||||
if (chunk.type === "step-finish") options.state.modelStepCount += 1;
|
||||
if (chunk.type === "finish") {
|
||||
const finish = finishTelemetry(chunk);
|
||||
options.state.modelFinishReason = finish.reason;
|
||||
if (finish.stepCount !== null) options.state.modelStepCount = stepCountBeforeAttempt + finish.stepCount;
|
||||
}
|
||||
if (chunk.type === "text-delta" && typeof chunk.payload?.text === "string") {
|
||||
await outputText(visible.push(chunk.payload.text));
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@ import { applyBirthTimeModeToWorkflowContext, type ConsultationBirthTimeMode } f
|
||||
import type { ServerChartConsultation } from "../lib/consultation-route-service.ts";
|
||||
import { createConsultationPlan, type ConsultationPlan } from "../lib/consultation-plan.ts";
|
||||
import type { WorkflowReceipt } from "../lib/consultation-agent-events.ts";
|
||||
import type { AgentModelFinishReason } from "../lib/agent-observability.ts";
|
||||
import {
|
||||
consultationInputSchema,
|
||||
consultationWorkflowFailureCode,
|
||||
@@ -20,11 +21,13 @@ import {
|
||||
const MAX_CONSULTATION_DOMAINS = 6;
|
||||
const domainPlanValueSchema = z.string().trim().min(1).max(64);
|
||||
|
||||
// The model may only express a domain plan one way. A second, mutually
|
||||
// exclusive field was representable here but rejected at execution, so every
|
||||
// call that set both spent a model step to learn a rule the schema never
|
||||
// stated. Internal callers keep the single-value form; see canonicalDomainPlan.
|
||||
const consultationToolInputSchema = z.object({
|
||||
question: z.string().trim().min(1).max(500),
|
||||
domains: z.array(domainPlanValueSchema).min(1).max(MAX_CONSULTATION_DOMAINS).optional(),
|
||||
// Transitional compatibility for older Agent calls. New calls must use domains.
|
||||
theme: domainPlanValueSchema.optional(),
|
||||
}).strict();
|
||||
|
||||
const MAX_RECORDED_STEPS = 32;
|
||||
@@ -57,6 +60,10 @@ export type ConsultationRuntimeState = {
|
||||
steps: ConsultationRuntimeStep[];
|
||||
stepBudget: ConsultationStepBudget;
|
||||
stepsTruncated: boolean;
|
||||
// Diagnostics for the model step budget. Internal only: the public receipt is
|
||||
// strict and would reject them, so they never enter it.
|
||||
modelStepCount: number;
|
||||
modelFinishReason?: AgentModelFinishReason;
|
||||
};
|
||||
|
||||
export function createConsultationRuntimeState(options: { plannedSteps?: number; reservedValidationSteps?: number } = {}): ConsultationRuntimeState {
|
||||
@@ -72,6 +79,19 @@ export function createConsultationRuntimeState(options: { plannedSteps?: number;
|
||||
steps: [],
|
||||
stepBudget: { planned, reservedValidation, total: planned + reservedValidation },
|
||||
stepsTruncated: false,
|
||||
modelStepCount: 0,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The controlled fields that report how the model loop ended. Kept beside the
|
||||
* public allowlist so both directions of the boundary are visible: these go to
|
||||
* the observability log only, never to the client receipt.
|
||||
*/
|
||||
export function consultationModelStepTelemetry(state: ConsultationRuntimeState) {
|
||||
return {
|
||||
modelStepCount: state.modelStepCount,
|
||||
...(state.modelFinishReason === undefined ? {} : { modelFinishReason: state.modelFinishReason }),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -128,7 +148,13 @@ export function createConsultationAgentContext(context: ConsultationAgentContext
|
||||
return Object.freeze(context);
|
||||
}
|
||||
|
||||
function canonicalDomainPlan(
|
||||
/**
|
||||
* Resolves the one domain plan a call may express. The model-facing schema
|
||||
* declares only `domains` and rejects anything else before execute() runs, so
|
||||
* the single-value `theme` form and its mutual exclusion remain the contract
|
||||
* for callers that build a plan without that schema.
|
||||
*/
|
||||
export function canonicalDomainPlan(
|
||||
input: { domains?: readonly unknown[]; theme?: unknown },
|
||||
context: Pick<ConsultationAgentContext, "plan" | "theme">,
|
||||
): ConsultationDomain[] {
|
||||
@@ -202,7 +228,7 @@ export function createConsultationTools(ctx: ConsultationAgentContext) {
|
||||
let calculation: Promise<ReturnType<typeof toModelDomainPlanContext>> | null = null;
|
||||
const consultationTool = createTool({
|
||||
id: "run-jyotish-consultation",
|
||||
description: "Run one server-validated plan of up to six allowlisted personal Jyotish consultation domains. Use domains in priority order; birth data is server-bound and must never be supplied by the model.",
|
||||
description: "Run one server-validated plan of up to six allowlisted personal Jyotish consultation domains. Send only question and domains. The single ordered domains array is the only way to select domains: list them in priority order, or omit it entirely to accept the domain the server already selected for this consultation. Birth data is server-bound and must never be supplied. One calculation is executed per request and reused, so repeating the call with different parameters cannot change the result.",
|
||||
inputSchema: consultationToolInputSchema,
|
||||
execute: async (input, context) => {
|
||||
const domains = canonicalDomainPlan(input, ctx);
|
||||
|
||||
@@ -35,7 +35,7 @@ const jyotishInstructions = `You are the guide for a conversational Vedic astrol
|
||||
Write in concise Simplified Chinese as a natural conversation, not a report or fixed template. Use Markdown only when it improves scanning; tables are allowed only for genuinely comparative information.
|
||||
For Vedic astrology questions, load the jyotish-vedic-astrology skill before deciding which calculation tool or workflow to use. Follow the skill's method and truth boundaries, but use run-jyotish-consultation for actual chart calculations instead of inventing results.
|
||||
For questions that require a new chart claim, call run-jyotish-consultation before answering. Simple conversational follow-ups may use the existing context.
|
||||
When a question spans multiple supported consultation domains, submit one ordered domains plan to run-jyotish-consultation. The server canonicalizes aliases, rejects unsupported/product domains, executes each accepted domain, and returns the actual domains in the tool context and receipt. Use the legacy theme field only for a single-domain compatibility retry.
|
||||
Select consultation domains only through the single ordered domains array of run-jyotish-consultation, whether the question covers one domain or several; omit it to accept the domain the server already selected. The server canonicalizes aliases, rejects unsupported/product domains, executes each accepted domain, and returns the actual domains in the tool context and receipt. A rejected domain plan is final for this run: correct the domains once, and never re-send the same call with extra parameters.
|
||||
Activity, progress, tool status, and execution receipts are server-owned. Never imitate data-jyotish-activity, activity events, tool-started/tool-completed messages, or receipts in the answer text.
|
||||
Treat the server-provided current time as authoritative for words such as today, now, this year, and the next few months. Never infer the current date from model knowledge or the birth date.
|
||||
Treat consumer_context as the authoritative answer policy:
|
||||
|
||||
Reference in New Issue
Block a user