Models and credentials

Before Pi can send a request it must know two things: what the model is (its API, endpoint, limits and prices) and who is paying (an API key, an OAuth token, cloud credentials). The first comes from a generated model catalog overlaid with fresher data at runtime; the second from a credential store plus provider-specific resolution rules and OAuth login flows. This chapter also covers the faux provider, the fake LLM that makes the whole system testable.

Learning goals. Explain how the catalog is produced and kept current; trace credential resolution for a provider; walk through the OAuth 2.0 authorization-code flow with PKCE and the device-code flow; explain why OAuth refresh must happen under a lock; write a deterministic test using the faux provider.

5.1 What a model record holds#

A chat Model (Chapter 1) extends BaseModel with reasoning, thinkingLevelMap, promptCache, contextWindow, maxTokens, sampling defaults and an API-specific compat block whose type is chosen by a conditional type on TApi. Prices are TypeBox schemas in providers/model-schema.ts:

const ModelCostRatesProperties = {
	input: Type.Number({ description: "Input cost in USD per million tokens." }),
	output: Type.Number({ description: "Output cost in USD per million tokens." }),
	cacheRead: Type.Number({ description: "Cache-read cost in USD per million tokens." }),
	cacheWrite: Type.Number({ description: "Cache-write cost in USD per million tokens." }),
};
export const ModelCostSchema = Type.Object({
	...ModelCostRatesProperties,
	tiers: Type.Optional(Type.Array(ModelCostTierSchema, {
		description: "Request-wide pricing tiers. The highest matching input threshold applies to the full request.",
	})),
});
export type ModelCost = Static<typeof ModelCostSchema>;

The last line is the TypeBox idiom: write the schema once, derive the static type with Static<>. The same object validates JSON at runtime (a user's models.json, a downloaded catalog) and types code at compile time, so the two can never drift apart.

The catalog also distinguishes model kinds. ModelTypeMap maps chat, image and classifier to their interfaces; isModelType() narrows a mixed list. The README notes a policy decision: "The chat catalog only includes models that support tool calling (function calling), as this is essential for agentic workflows."

5.2 Where the catalog comes from#

packages/ai/scripts/generate-models.ts builds the catalog. It pulls public metadata (models.dev, OpenRouter's model list, Vercel AI Gateway pricing, vendor model lists), then applies a long series of hand-maintained corrections with comments explaining each one ("Keep current OpenAI prices authoritative until models.dev and passthrough catalogs catch up", "Retired preview id — models.dev may still list it after GA ships", "Baseten's GLM-5.2 endpoints are text-only despite models.dev reporting image input"). It also derives compat metadata, such as which Anthropic models accept mid-conversation system messages. Its outputs are per-provider JSON data files plus small TypeScript wrappers such as providers/anthropic.models.ts, which begin "This file is auto-generated by scripts/generate-models.ts. Do not edit manually".

This is code generation used to turn a slowly-changing external dataset into typed, tree-shakable modules. The generated wrapper imports its JSON with an import attribute (import values from "./data/anthropic.json" with { type: "json" }) so bundlers include only the providers you import. Because MODELS is generated as a literal, TypeScript knows every provider and model id: getBuiltinModel("anthropic", "<id>") is checked at compile time, and the return type is narrowed to the model's API through CatalogApi<…>, a conditional type with infer.

A catalog frozen at release time goes stale in weeks, so the coding agent overlays live data. withRemoteCatalog() in packages/coding-agent/src/core/remote-catalog-provider.ts wraps a static provider and, on refresh, fetches https://pi.dev/api/models/providers/<id>. It is a careful HTTP client:

• It skips the request entirely if the stored copy was checked less than a refresh interval ago (a TTL cache).
• It sends the stored ETag as a validator — but "Only revalidate when a cached body backs the validator, so a 304 can never leave the overlay empty." A 304 Not Modified response then only moves the freshness window.
• Remote models are merged over static ones by (type, id), so the overlay can correct a price without dropping models the server does not mention.

The same mechanism serves dynamic providers whose catalogs exist only at runtime: createProvider({ fetchModels }) restores the stored list first, publishes it, then fetches a fresh list if the network is allowed and the signal has not fired.

5.3 Credential resolution#

Every provider declares an auth object with an apiKey method, an oauth method, or both. ApiKeyAuth.resolve() turns a stored credential and the ambient environment into a ModelAuth — deliberately just apiKey, headers and baseUrl. The interface comment draws the line: "If a value cannot be expressed as apiKey, headers, or baseUrl, it is provider config, not auth."

The Anthropic resolver (providers/anthropic.ts) shows a typical precedence order:

1. a stored credential's key;
2. ANTHROPIC_AUTH_TOKEN, sent as an Authorization: Bearer header;
3. ANTHROPIC_OAUTH_TOKEN, then ANTHROPIC_API_KEY;
4. workload identity federation, if all of its required variables are present ("Last in line so keys and ANTHROPIC_AUTH_TOKEN keep winning, as in the SDK");
5. otherwise undefined — the provider is not configured.

Each step calls signal.throwIfAborted(), and environment access goes through an injectable AuthContext (env(name), fileExists(path)) — dependency injection that lets tests and browsers supply their own environment. Even reading Node modules is careful: env-api-keys.ts builds specifiers as "node:" + "fs" and imports them dynamically only when running under Node or Bun, so bundlers targeting browsers do not try to include them.

The result carries a human-readable source ("stored credential", "ANTHROPIC_API_KEY", "workload identity federation") so the UI can tell the user which credential is in use — an underrated debugging aid.

5.4 The credential store and the double-refresh problem#

Stored credentials live behind a small interface:

export interface CredentialStore {
	read(providerId: string, options?: AuthOperationOptions): Promise<Credential | undefined>;
	list(options?: AuthOperationOptions): Promise<readonly CredentialInfo[]>;
	modify(
		providerId: string,
		fn: (current: Credential | undefined) => Promise<Credential | undefined>,
		options?: AuthOperationOptions,
	): Promise<Credential | undefined>;
	delete(providerId: string, options?: AuthOperationOptions): Promise<void>;
}

modify is "the only write path, so every mutation is a serialized read-modify-write" with "Mutual exclusion per provider id, cross-process too where the backing store supports it (e.g. a file lock)." In the coding agent the store is auth.json, guarded by the proper-lockfile package, written with file mode 0o600 (owner read/write only) inside a directory created with 0o700.

Why so much ceremony? OAuth access tokens expire, and many providers rotate refresh tokens: using a refresh token returns a new one and invalidates the old. Imagine two Pi processes (or two parallel requests) noticing an expired token at the same moment. Both refresh; the first succeeds and stores token B; the second presents the now-invalid token A and fails — or worse, succeeds with a different rotation and overwrites B. refreshStoredOAuthCredential() in auth/resolve.ts prevents this with the classic double-checked pattern under a lock:

post = await credentials.modify(
	providerId,
	async (current) => {
		signal.removeEventListener("abort", cancelLockWait);
		signal.throwIfAborted();
		if (current?.type !== "oauth") return undefined; // logged out meanwhile
		if (!needsRefresh(current)) return undefined; // another process/request refreshed
		try {
			return await oauth.refresh(current, AbortSignal.timeout(DEFAULT_OAUTH_REFRESH_TIMEOUT_MS));
		} catch (error) {
			throw new ModelsError("oauth", `OAuth refresh failed for ${providerId}`, { cause: error });
		}
	},
	{ signal: lockWait.signal },
);

The condition is checked once outside the lock (cheap) and again inside it (correct). The cancellation semantics are subtle and documented: the caller's signal cancels only the wait for the lock. "Once a refresh starts, the provider may already have rotated the refresh token, so the refresh and its persistence ignore signal and are bounded only by a timeout. Otherwise a cancelled caller could discard the only valid refresh token." Tokens are refreshed proactively when fewer than five minutes of validity remain (DEFAULT_OAUTH_MINIMUM_VALIDITY_MS), so a long request does not start with a token about to expire.

5.5 OAuth login flows#

Several providers let you log in with a subscription account instead of an API key. Pi implements two standard OAuth 2.0 flows in auth/oauth/.

Authorization code with PKCE (RFC 7636), used for example by auth/oauth/anthropic.ts. A command-line program cannot keep a client secret, so it proves continuity between the start and end of the flow with a one-time secret instead:

1. Generate a random code verifier and its SHA-256 hash, the code challenge (auth/oauth/pkce.ts, using the Web Crypto API so it works in Node and browsers).
2. Start a temporary HTTP server on localhost (callback-server.ts) to receive the redirect.
3. Open the provider's authorization URL with code_challenge, code_challenge_method=S256, the redirect URI and a state value. (The Anthropic flow reuses the verifier as the state.)
4. The user signs in in a browser; the provider redirects to localhost with an authorization code and the state.
5. Pi checks the state ("OAuth state mismatch" otherwise), then exchanges the code plus the original verifier for access and refresh tokens.

export async function generatePKCE(): Promise<{ verifier: string; challenge: string }> {
	const verifierBytes = new Uint8Array(32);
	crypto.getRandomValues(verifierBytes);
	const verifier = base64urlEncode(verifierBytes);
	const encoder = new TextEncoder();
	const data = encoder.encode(verifier);
	const hashBuffer = await crypto.subtle.digest("SHA-256", data);
	const challenge = base64urlEncode(new Uint8Array(hashBuffer));
	return { verifier, challenge };
}

An attacker who intercepts the authorization code (say, another program on the machine watching redirects) cannot redeem it without the verifier, and cannot derive the verifier from the challenge because SHA-256 is a one-way function. The state parameter defends against a different attack, cross-site request forgery: it ties the redirect to the session that started it. If the browser runs on another machine (SSH sessions), the flow races the callback server against a prompt where the user can paste the final redirect URL; whichever arrives first wins and the other is cancelled through the prompt's signal.

Device authorization grant (RFC 8628), used by providers such as GitHub Copilot, for devices that cannot receive a redirect. The program shows a short user code and a URL, the user enters the code on any device, and the program polls the token endpoint. device-code.ts implements the polling rules from the RFC, citing sections in comments: default interval 5 seconds if the server omits one (§3.2), and on slow_down increase the interval by 5 seconds (§3.5) — or adopt the server's new interval when given, since "trusting only a client-tracked value risks polling early forever". An error message even anticipates a field problem: repeated slow_down is "often caused by clock drift in WSL or VM environments".

Login interacts with the user only through an AuthInteraction (prompt() for text, secrets, selections or a pasted code; notify() for events such as auth_url and device_code). The same flows therefore work in the terminal UI, in RPC mode, and in a web app — the flows are written against an interface, not a UI.

5.6 Constrained sampling#

A tool may request constrainedSampling: { type: "json_schema", strict: "prefer" | "require" }, asking the provider to restrict sampling so the arguments always match the schema, or { type: "grammar", variants } with an OpenAI Lark or regex grammar. api/constrained-sampling.ts converts schemas to the strict subset providers accept and checks grammar tools ("requires exactly one required string property"). Under the hood, the provider masks out tokens that would make the output invalid at each step, so the result is valid by construction. prefer degrades gracefully (Chapter 4's strict-tools fallback); require fails with an explicit error when the provider cannot comply.

5.7 The faux provider: a deterministic LLM for tests#

Testing an agent against a real model is slow, costly and nondeterministic. providers/faux.ts implements a test double: a provider whose responses are scripted. A test registers it, queues responses with setResponses() or appendResponses(), and runs the real agent code:

// Shape of a faux script (helpers from providers/faux.ts)
faux.setResponses([
  fauxAssistantMessage([fauxToolCall("read", { path: "README.md" })]),
  fauxAssistantMessage("The README describes Pi."),
]);

A step may also be a function (context, options, state, model) => AssistantMessage, so a test can assert on what the agent sent or compute a reply from it. The faux stream is realistic in the ways that matter: it splits text, thinking and tool-call JSON into chunks of 3–5 "tokens", emits the full event protocol, optionally paces chunks with tokensPerSecond, estimates usage (including simulated prompt-cache reads), honours the abort signal, and can even simulate deferred responses that stay pending for a configurable number of polls. AGENTS.md makes it mandatory for the coding agent's suite: "use test/suite/harness.ts + the faux provider. No real provider APIs, keys, or paid tokens."

5.8 Concepts in this chapter#

• Code generation. Produce source from data so that facts (ids, prices) become types the compiler checks. Never hand-edit generated output; change the generator.
• Schema-first types. One definition yields both a runtime validator and a static type (TypeBox Static<>).
• HTTP caching. TTLs avoid requests; ETag and If-None-Match turn a full download into a 304 when nothing changed.
• Precedence chains. Resolve a setting from an ordered list of sources and record which one won.
• Read-modify-write races and double-checked locking. Check, lock, re-check, act. Without the re-check, two writers both act.
• Advisory file locks and file permissions. Cooperating processes serialize through a lock file; 0o600 keeps secrets readable only by their owner.
• OAuth 2.0, PKCE and the device grant. Delegated authorization without sharing passwords; PKCE replaces a client secret with a one-time proof; the device grant replaces a redirect with polling.
• Cryptographic hash functions. Easy to compute, infeasible to invert or collide; the basis of PKCE.
• Polling with backoff and server-directed rate limits. Respect the interval the server asks for.
• Constrained decoding. Restrict generation to a formal language (JSON Schema, grammar) by masking invalid tokens.
• Test doubles (fakes). A working, simplified implementation that replaces a dependency in tests and makes them deterministic.

Exercises#

1. Draw a sequence diagram of the PKCE flow with four actors: Pi, the localhost callback server, the browser, and the authorization server. Mark where the verifier and the challenge travel.
2. Explain precisely what goes wrong if refreshStoredOAuthCredential omitted the needsRefresh(current) re-check, using two processes and a rotating refresh token.
3. Why must a cancelled caller not cancel an in-flight refresh? Construct the failure.
4. Using the faux provider, write a test in which the model calls a tool with invalid arguments, receives the validation error, and corrects itself on the next turn.
5. The remote catalog refuses to send an ETag when it has no stored models. What bug does this prevent?
6. Add a hypothetical provider to a Models collection with createProvider(), an apiKey auth that reads MY_API_KEY, and the openai-completions api. What else would you need to make it appear in getAvailable()?