Managed AI

Give every notebook a working AI assistant without asking users for their own API key. When managed AI is on, marimohub points marimo's assistant at a provider you configure and pays for. Chat, autocomplete, and "generate with AI" work when someone opens a notebook.

It's optional and off by default. Turn it on by setting one upstream provider; provider credentials stay on the server and are never exposed to notebook code.

Configuration

Select Bedrock or an API-key-backed OpenAI-compatible upstream. When configured, managed AI is injected into every session deployment-wide.

Amazon Bedrock

Use Amazon Bedrock's OpenAI-compatible endpoint with the hub's AWS identity. The hub signs upstream requests with SigV4; no Bedrock API key or AWS credential is written into a sandbox.

MARIMOHUB_AI_BACKEND=bedrock
MARIMOHUB_AI_AWS_REGION=eu-west-1
MARIMOHUB_AI_MODEL=eu.anthropic.claude-opus-4-7
# MARIMOHUB_AI_ALLOWED_MODELS=model-a,model-b

The runtime identity needs bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream for the configured inference profiles or foundation models. On EKS, use IRSA or EKS Pod Identity; the standard AWS credential chain also supports local development credentials.

When MARIMOHUB_AI_ALLOWED_MODELS is unset, Bedrock is restricted to MARIMOHUB_AI_MODEL. Set an explicit comma-separated allowlist to expose more models.

OpenAI-compatible provider

Point the hub at any OpenAI-compatible provider. Notebooks get the AI assistant with no key of their own — the hub holds the real key server-side and proxies requests through /api/ai/v1.

MARIMOHUB_AI_BACKEND=openai-compatible
MARIMOHUB_AI_UPSTREAM_BASE_URL=https://api.openai.com/v1
MARIMOHUB_AI_UPSTREAM_API_KEY=sk-...          # real key, held server-side
MARIMOHUB_AI_MODEL=gpt-4o-mini                 # default model offered to users
# MARIMOHUB_AI_ALLOWED_MODELS=gpt-4o-mini,gpt-4o   # optional allowlist

Reuses the session secret

Per-session tokens are signed with MARIMOHUB_AUTH_SESSION_SECRET, so that must be set too (it also signs login cookies). The real upstream key is never written into a sandbox.

Works with OpenAI, OpenRouter, LiteLLM, W&B Inference, or Anthropic's OpenAI-compatible endpoint. See Managed AI for the provider table.

The full set of variables:

Variable	Required	Description
MARIMOHUB_AI_BACKEND	—	none (default), bedrock, or openai-compatible.
MARIMOHUB_AI_AWS_REGION	Bedrock	AWS region for Bedrock. Falls back to AWS_REGION or AWS_DEFAULT_REGION.
MARIMOHUB_AI_UPSTREAM_BASE_URL	API key	Upstream OpenAI-compatible base URL, e.g. https://api.openai.com/v1. The proxy POSTs to <base>/chat/completions.
MARIMOHUB_AI_UPSTREAM_API_KEY	API key	The real upstream key. Held server-side; never injected into a sandbox.
MARIMOHUB_AI_MODEL	yes	Default model id surfaced to marimo, e.g. gpt-4o-mini.
MARIMOHUB_AI_ALLOWED_MODELS	no	Comma-separated allowlist; off-list requests fall back to the default model. Unset allows any model on OpenAI-compatible upstreams; on Bedrock it restricts to MARIMOHUB_AI_MODEL.
MARIMOHUB_AI_UPSTREAM_PROJECT	no	Optional OpenAI-Project header forwarded upstream (e.g. W&B Inference entity/project attribution).
MARIMOHUB_AI_MAX_TOKENS	no	[ai] max_tokens written into the notebook config.
MARIMOHUB_AI_RULES	no	[ai] rules — custom assistant instructions.
MARIMOHUB_AI_TOKEN_TTL_SECONDS	no	AI session-token lifetime in seconds (default: 3600). Shorter lifetimes reduce the revocation window.

Managed AI also requires MARIMOHUB_AUTH_SESSION_SECRET — the per-session tokens are signed with it (the same secret that signs login cookies).

Providers

For API-key-backed providers, any OpenAI-compatible endpoint works. Set MARIMOHUB_AI_UPSTREAM_BASE_URL to the provider's base and MARIMOHUB_AI_MODEL to one of its model ids:

Provider	Base URL	Notes
OpenAI	https://api.openai.com/v1	gpt-4o-mini, gpt-4o, …
OpenRouter	https://openrouter.ai/api/v1	One key, hundreds of models across vendors.
W&B Inference	https://api.inference.wandb.ai/v1	Set MARIMOHUB_AI_UPSTREAM_PROJECT=entity/project.
Anthropic (compat.)	https://api.anthropic.com/v1	Anthropic's OpenAI-compatible endpoint.
LiteLLM / self-hosted	https://<your-litellm-host>/v1	Front many providers behind one gateway.

How it works

1. Inject. At session start, marimohub sets XDG_CONFIG_HOME and writes a marimo.toml into that sandbox-local config directory, outside the notebook's files. The config registers a custom AI provider pointed at marimohub's own proxy using a short-lived, session-scoped token as the api_key.
2. Proxy. marimohub hosts an OpenAI-compatible endpoint at /api/ai/v1. It verifies the session token, authenticates the upstream request server-side, and streams the response back.

Notebook kernels run untrusted code, so provider credentials are never written into a sandbox — only a minted, expiring, session-scoped token. This mirrors how Workload Identity Federation avoids long-lived storage keys.

Token revocation is not immediate

AI session tokens are self-contained. The proxy checks the signature and expiration of each token without reading object storage. This avoids an object-storage read for each AI request.

An issued token remains valid until it expires. Stopping the session, removing project access, or suspending the user does not invalidate the token.

The default lifetime is one hour. Set MARIMOHUB_AI_TOKEN_TTL_SECONDS to a lower value to reduce the revocation window. AI access ends when the token expires, even if the notebook session remains active.

Proxy contract

The proxy implements the subset of the OpenAI API that marimo calls server-side. All endpoints require a valid bearer session token; upstream authentication is applied server-side.

• POST /api/ai/v1/chat/completions — forwards to the upstream, streaming SSE when stream: true. The request model is normalized to a managed model.
• POST /api/ai/v1/responses — the same passthrough for the OpenAI Responses API, so a client pointed at marimo's built-in [ai.open_ai] provider also works.
• GET /api/ai/v1/models — returns the configured/allowed models.

Each request body can contain at most 10 MB. This total includes embedded file content. The proxy rejects larger requests with HTTP 413 and does not send them upstream.

This is a deliberate allowlist, not a generic OpenAI passthrough: the session token authorizes untrusted notebook code, so the proxy exposes only the endpoints clients need. Others (/v1/embeddings, /v1/images, …) are added only on demand.

What the user can override

The injected config sits at the user-config tier, so a user who explicitly sets [tool.marimo.ai] in their own pyproject.toml still overrides it — a deliberate bring-your-own-key escape hatch. With managed AI off (MARIMOHUB_AI_BACKEND=none), the assistant only works for users who supply their own key in marimo's settings.

OpenCode uses the same proxy through a temporary marimohub provider. Project opencode.json files and /connect providers can override it. The token expires at the configured TTL. Restart OpenCode to get a new token.