root/Jyotisha
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix: harden multi-model chat selection

Browse Source
This commit is contained in:
Jesse_Chen
2026-07-17 14:09:57 +08:00
parent 4968c961f7
commit 6c02f27d1c
13 changed files with 338 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/plans/2026-07-17-multi-model-chat-selection.md
+36 -32
View File
@@ -35,7 +35,7 @@
- Produces public shape: `{ id, label, description, creditCost: 1, isDefault }`.
- Consumes: `MastraModelConfig`, Zod, `NodeJS.ProcessEnv`.
- [ ] **Step 1: Write failing catalog tests**
- [x] **Step 1: Write failing catalog tests**
Add tests for a two-model catalog, secret redaction, invalid entries, unknown defaults, and legacy single-model fallback:
@@ -90,15 +90,15 @@ test("resolves two configured models while returning sanitized public metadata",
});
```
- [ ] **Step 2: Run the catalog test and verify RED**
- [x] **Step 2: Run the catalog test and verify RED**
Run: `cd frontend && node --test tests/model-catalog.test.ts`
Expected: FAIL because `resolveLanguageModelCatalog` is not exported.
- [ ] **Step 3: Implement the catalog parser and resolver**
- [x] **Step 3: Implement the catalog parser and resolver**
Use a Zod boundary for each raw catalog item and return immutable resolved entries. OpenAI entries produce a Mastra model string; OpenAI-compatible entries produce `{ providerId, modelId, url, apiKey }`. Resolve `apiKeyEnv` only on the server. Invalid items are excluded with redacted issue codes. If `LLM_MODELS_JSON` is absent, derive one entry from the shipped `LLM_*` or `OPENAI_*` variables.
Use a Zod boundary for each raw catalog item and return immutable resolved entries. Catalog entries produce an explicit Mastra configuration object containing the resolved server-side key; OpenAI-compatible entries also include their fixed URL. Resolve `apiKeyEnv` only on the server. Invalid items are excluded with redacted issue codes. If `LLM_MODELS_JSON` is absent, derive one entry from the shipped `LLM_*` or `OPENAI_*` variables.
The catalog result must have this contract:
@@ -123,17 +123,17 @@ export type LanguageModelCatalog = {
};
```
- [ ] **Step 4: Run catalog tests and type checking**
- [x] **Step 4: Run catalog tests and type checking**
Run: `cd frontend && node --test tests/model-catalog.test.ts && npx tsc --noEmit`
Expected: all catalog tests PASS and TypeScript exits `0`.
- [ ] **Step 5: Document configuration**
- [x] **Step 5: Document configuration**
Update `frontend/README.md` and `deploy/README.md` with `LLM_MODELS_JSON`, `LLM_DEFAULT_MODEL_ID`, one secret environment variable per provider, and the existing single-model fallback. Use redacted values only. Do not add `frontend/.env.example`: the repository intentionally ignores all `.env*` files.
- [ ] **Step 6: Commit the catalog task**
- [x] **Step 6: Commit the catalog task**
```bash
git add frontend/src/mastra/model.ts frontend/tests/model-catalog.test.ts frontend/README.md deploy/README.md docs/superpowers/plans/2026-07-17-multi-model-chat-selection.md
@@ -158,7 +158,7 @@ git commit -m "feat: add server model catalog"
- Produces: `getJyotishAgent(model)` and `getOnboardingAgent(model)` process-local caches.
- Consultation request consumes `modelId: string`.
- [ ] **Step 1: Write failing public payload tests**
- [x] **Step 1: Write failing public payload tests**
Create a Zod client boundary that accepts only the sanitized response and rejects routing fields:
@@ -189,17 +189,17 @@ test("parses a sanitized public model catalog", () => {
});
```
- [ ] **Step 2: Run the payload test and verify RED**
- [x] **Step 2: Run the payload test and verify RED**
Run: `cd frontend && node --test tests/public-models.test.ts`
Expected: FAIL because the parser module does not exist.
- [ ] **Step 3: Implement the public parser and authenticated route**
- [x] **Step 3: Implement the public parser and authenticated route**
`parsePublicModelCatalog(value: unknown)` must use strict Zod objects so additional secret or routing fields are rejected. `GET /api/models` must authenticate through `createServerSupabaseClient`, return `401` when logged out, `503` when the catalog has no default, and otherwise return the sanitized catalog.
- [ ] **Step 4: Refactor Mastra Agent construction**
- [x] **Step 4: Refactor Mastra Agent construction**
Move the existing shared instructions into constants and build Agents through keyed factories:
@@ -224,17 +224,17 @@ export function getJyotishAgent(model: ResolvedLanguageModel) {
Create the onboarding Agent with the default resolved model and keep its existing instructions unchanged.
- [ ] **Step 5: Select the model before credit reservation**
- [x] **Step 5: Select the model before credit reservation**
Extend `chatRequestSchema` with `modelId: z.string().trim().min(1).max(64)`. Resolve the ID after authentication, request parsing, and prompt-extraction blocking, but before `begin_consultation_credit`. Return `409` with a safe message for an unavailable model. Use `getJyotishAgent(resolvedModel)` for streaming and record `resolvedModel.id` in `credit_transactions`.
- [ ] **Step 6: Run tests, type checking, and lint**
- [x] **Step 6: Run tests, type checking, and lint**
Run: `cd frontend && npm test && npx tsc --noEmit && npm run lint`
Expected: all tests PASS; type checking and lint exit `0`.
- [ ] **Step 7: Commit the endpoint task**
- [x] **Step 7: Commit the endpoint task**
```bash
git add frontend/src/app/api/models/route.ts frontend/src/lib/public-models.ts frontend/tests/public-models.test.ts frontend/src/mastra/index.ts frontend/src/app/api/consult/route.ts frontend/src/app/api/onboarding/route.ts
@@ -256,7 +256,7 @@ git commit -m "feat: route consultations by model"
- Produces: `resolveSessionModelId(saved, catalog) -> { modelId, fellBack }`.
- Persists: `chat_sessions.model_id text`.
- [ ] **Step 1: Write failing session fallback tests**
- [x] **Step 1: Write failing session fallback tests**
```ts
test("falls back to the configured default when a saved model is removed", () => {
@@ -280,31 +280,31 @@ test("falls back to the configured default when a saved model is removed", () =>
});
```
- [ ] **Step 2: Run the fallback test and verify RED**
- [x] **Step 2: Run the fallback test and verify RED**
Run: `cd frontend && node --test tests/public-models.test.ts`
Expected: FAIL because `resolveSessionModelId` does not exist.
- [ ] **Step 3: Implement fallback and migration**
- [x] **Step 3: Implement fallback and migration**
Add nullable `model_id text` to `public.chat_sessions` and grant authenticated users column-level insert/update access. Do not store labels, provider fields, or secrets.
`resolveSessionModelId` returns the saved ID when it is in the catalog and otherwise returns the default with `fellBack: true`.
- [ ] **Step 4: Wire persistence into the page**
- [x] **Step 4: Wire persistence into the page**
Extend `ChatSession` with `modelId`. Fetch `/api/models` during bootstrap, parse it through `parsePublicModelCatalog`, normalize loaded sessions, and persist fallback replacements once. New sessions use `defaultModelId`; `persistSession` reads/writes `model_id`; consultation requests include the active session's `modelId`.
Preview mode must install a deterministic two-model catalog so browser QA can run without provider keys.
- [ ] **Step 5: Run focused tests and build**
- [x] **Step 5: Run focused tests and build**
Run: `cd frontend && npm test && npx tsc --noEmit && npm run build`
Expected: tests PASS and production build exits `0`.
- [ ] **Step 6: Commit persistence**
- [x] **Step 6: Commit persistence**
```bash
git add frontend/supabase/migrations/20260717010000_chat_session_model.sql frontend/src/app/page.tsx frontend/src/lib/public-models.ts frontend/tests/public-models.test.ts
@@ -325,17 +325,17 @@ git commit -m "feat: persist session model choice"
- Consumes: `readonly PublicLanguageModel[]`, selected ID, disabled state, selection callback.
- Produces: accessible Base UI Popover with native radio inputs.
- [ ] **Step 1: Add the model-selector primitive to `DESIGN.md`**
- [x] **Step 1: Add the model-selector primitive to `DESIGN.md`**
Document the compact trigger, upward warm-canvas popup, radio rows, 44px touch target, focus behavior, disabled request states, and existing motion/token usage before writing JSX or CSS.
- [ ] **Step 2: Add the component in preview mode and observe RED behavior**
- [x] **Step 2: Add the component in preview mode and observe RED behavior**
Render a temporary import of the not-yet-created `ModelSelector` in the composer footer and run `cd frontend && npx tsc --noEmit`.
Expected: FAIL because `frontend/src/components/model-selector.tsx` does not exist.
- [ ] **Step 3: Implement the Base UI Popover**
- [x] **Step 3: Implement the Base UI Popover**
Use `Popover.Root`, `Trigger`, `Portal`, `Positioner side="top" align="start"`, and `Popup`. Render a `role="radiogroup"` whose rows contain controlled native radio inputs. Selecting an item closes the popup and invokes the supplied callback. Base UI owns Escape, outside press, focus restoration, and collision positioning.
@@ -350,21 +350,21 @@ type ModelSelectorProps = {
};
```
- [ ] **Step 4: Persist selection from the page**
- [x] **Step 4: Persist selection from the page**
Place the trigger below `.composer` and before the status line. Optimistically update the active session, persist it immediately, retain the visible choice on sync failure, and show a retryable composer notice. Disable selection while undo, streaming, cancellation, session creation, or model loading is active.
- [ ] **Step 5: Style entirely from existing design tokens**
- [x] **Step 5: Style entirely from existing design tokens**
Add `.composer-tools`, `.model-selector-*` rules using current canvas, border, ink, radius, spacing, shadow, type, and 120/180ms motion tokens. Constrain the popup to the viewport and keep each row at least 44px. Add reduced-motion behavior through the existing media query.
- [ ] **Step 6: Run static verification**
- [x] **Step 6: Run static verification**
Run: `cd frontend && npm test && npx tsc --noEmit && npm run lint && npm run build`
Expected: all commands exit `0`.
- [ ] **Step 7: Commit the UI task**
- [x] **Step 7: Commit the UI task**
```bash
git add frontend/src/components/model-selector.tsx frontend/src/app/page.tsx frontend/src/app/globals.css frontend/DESIGN.md
@@ -382,7 +382,7 @@ git commit -m "feat: add chat model selector"
- Consumes the complete feature.
- Produces fresh test, browser, migration, security, and deployment evidence.
- [ ] **Step 1: Run the complete relevant verification set**
- [x] **Step 1: Run the complete relevant verification set**
```bash
cd frontend
@@ -406,17 +406,19 @@ Expected: all relevant checks PASS. Name any unrelated pre-existing failure with
- [ ] **Step 2: Apply and verify the Supabase migration**
Deferred for the local trial: the migration file is verified, but the linked remote database is intentionally unchanged until the user approves deployment preparation.
Run: `cd frontend && npx supabase db push --linked`
Then run: `npx supabase migration list --linked`
Expected: local and remote both list `20260717010000`.
- [ ] **Step 3: Run real browser QA**
- [x] **Step 3: Run real browser QA**
Start the production-like app with preview data, then drive it through the in-app browser or Playwright at 375px, 768px, and 1280px. Verify open/close, radio keyboard behavior, Escape, focus return, model switching, per-session persistence, disabled state during undo/streaming, no horizontal overflow, and no console errors. Inspect the `/api/models` payload to confirm no provider routing or secret fields are present.
- [ ] **Step 4: Run the requested AI-slop audit**
- [x] **Step 4: Run the requested AI-slop audit**
Run:
@@ -426,14 +428,16 @@ node ../.agents/skills/kill-ai-slop/scripts/scan.mjs frontend/src --json
Review every hit against `frontend/DESIGN.md`; fix confirmed slop and retain only deliberate, documented patterns.
- [ ] **Step 5: Run final review and debugging gates**
- [x] **Step 5: Run final review and debugging gates**
Review goal coverage, QA evidence, code quality, security, and missed context. Record at least three runtime hypotheses and the evidence that ruled each in or out. Fix every blocking finding and rerun only the checks whose inputs changed.
- [ ] **Step 6: Commit verification fixes**
- [x] **Step 6: Commit verification fixes**
If verification required changes, stage only feature-owned files and commit them with a focused `fix:` message. If no files changed, do not create an empty commit.
- [ ] **Step 7: Publish through the user-selected Git workflow**
Deferred by user request: keep the branch local and unpushed until the local service trial is accepted.
After fresh verification, preserve unrelated work, inspect branch/upstream state, and use the finishing-a-development-branch workflow. Push only after the feature commits and migration evidence are complete; if merged to `main`, monitor CI and production deployment through the existing workflows and run the production smoke checks documented in `deploy/README.md`.