Architecture

XSI is a single-process HTTP service written in Go. One port serves both model calls (the data plane) and account, key, and configuration management (the control plane). State lives in PostgreSQL and Redis, and two static consoles talk to the service through its HTTP API.

Components

                 β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
   Apps/SDKs  ──▢│                              │──▢ OpenAI / Azure
   Codex      ──▢│   XSI API (Go, :3443)        │──▢ Anthropic
   Claude Code ─▢│  β”œβ”€ Data plane: /v1/...      │──▢ Gemini / Vertex
   Consoles   ──▢│  └─ Control: /x-*, /dashboard│──▢ DeepSeek / xAI / ...
                 β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜
                         β”‚               β”‚
                  PostgreSQL           Redis ──▢ Mail service (SMTP)
         Accounts, keys, ledger   Config, cache, usage,
                                  counters, messages, mail queue
ComponentImageResponsibility
APIproxyxai/xsiAuthentication, admission, routing, forwarding, metering, billing, and all management endpoints
Consolesproxyxai/xsi-pagesTwo static sites, configuration management (Admin) and account management (Manage), each in Chinese and English
Mail serviceproxyxai/mailConsumes the Redis mail queue and delivers over SMTP; the API never connects to SMTP directly
PostgreSQLpostgres:18Accounts, upstream keys, top-up ledger, notification outbox
Redisredis:alpineTenant configuration, account cache, usage statistics, rate limit counters, sticky bindings, cross-instance messaging; DB 1 holds the mail queue

Request lifecycle

Request ─▢ Size check ─▢ Auth ─▢ Account admission ─▢ Model resolution ─▢ Model admission ─▢ Level ─▢ Key selection ─▢ Forward
                                                                                                                          β”‚
        Settlement ◀─ Usage extraction (side-channel parse of response / SSE) ◀─ Retry (next key in group, may cross groups) β—€β”˜
  1. Size check: if the declared Content-Length exceeds the limit, the request is rejected with 413 before authentication.
  2. Authentication: the key is read from Authorization, x-api-key, or x-goog-api-key. An in-memory hit passes immediately; otherwise Redis and then PostgreSQL are queried. Failed authentication attempts are counted per IP and trigger a temporary ban.
  3. Account admission: IP allowlist, account status and freeze, daily/monthly quota, balance, account-level rate limits, and resource allowlist.
  4. Model resolution: a single scan of the top-level JSON fields (model, stream, prompt_cache_key, speed, service_tier) applies the mapping rules to produce the upstream model name, then rewrites the request body or URL in place.
  5. Model admission: model allowlist and per-model limits.
  6. Level and key selection: LEVEL_MAPPER determines the group, and a key is chosen from that group's key pool by sticky session or round robin.
  7. Forwarding: caller credentials and forwarding headers are stripped, and the path and auth headers are rewritten for the upstream host.
  8. Retry: when the upstream returns a retryable error, the request is retried with another key before any byte is written to the client.
  9. Metering and settlement: non-streaming responses are parsed for usage after being read in full; SSE streams are parsed chunk by chunk while being forwarded, without buffering the whole stream. Settlement runs once at the end of the request, and a client that disconnects mid-stream is still billed for actual usage.

State and caching

LayerLocationNotes
Hot account stateProcess memory β†’ Redis β†’ PostgreSQLReads fall back in this order; balance and usage accumulate in memory and are flushed to Redis and the database in batches
Tenant configurationIn-process snapshotLoaded from Redis, copy-on-write; synced across instances via pub/sub plus a version check every 3 minutes
Key poolIn process, one round-robin ring per tenant per LevelLazily loaded and rebuilt when keys change
Runtime mappingsExact-match cache per account and tenantWildcard resolutions are stored as exact entries to avoid repeated matching
PricesIn process, one effective price table and model match index per tenantpricing.json is loaded at startup; tenant overrides are merged on demand
Usage statisticsIn process + RedisOne record per account per day and per month, broken down by model
Sticky bindingsMemory for a single instance, Redis for multiple instancesRetained for 24 hours by default

PostgreSQL holds only authoritative data such as accounts, keys, and the ledger; fast-changing usage statistics are kept in Redis.

Encryption model

  • Each tenant has a randomly generated tenant key. Upstream provider keys and Vertex private keys are encrypted with the tenant key before being stored, using XChaCha20-Poly1305.
  • The tenant key itself is encrypted with the account's own API key and stored in the account record. The server keeps no plaintext material that can directly unlock a tenant key.
  • As a result, a tenant's key pool can only be decrypted and loaded when an account in that tenant makes a request; if the database alone is exfiltrated, upstream keys cannot be decrypted directly.
  • When an account key is rotated, the tenant key is re-encrypted with the new key and the old key is invalidated immediately.
  • Core dumps are disabled at process startup.

Single instance and multiple instances

SINGLE_MODE defaults to true. In single-instance mode, rate limit counters, sticky bindings, and change notifications all stay in process for the lowest latency. Set it to false to scale horizontally:

AspectSingle instance (default)Multiple instances (SINGLE_MODE=false)
Rate limit countersIn processRedis
Key and account change notificationsIn-process channelRedis pub/sub
Sticky bindingsMemoryRedis
Usage writes to RedisOn maintenance and flushOn every update
Key auto-disable/delete broadcastβ€”Redis channel

In multi-instance deployments, group switchover (SWITCH_OVER), model failover, and failure counts are still evaluated independently per instance, and auto-sleep applies only to the instance that triggered it.

Performance design

  • Connection reuse: all HTTP upstream requests share one http.Transport (WebSocket uses a separate dialer; up to 256 idle and 2048 concurrent connections per host, with HTTP/2 and TLS session caching enabled), with 64 KiB read and write buffers reused through sync.Pool.
  • Zero-copy streaming: SSE responses are copied straight from the upstream to the client while the usage tracker parses chunks on the side, without buffering the whole response (XSI requests uncompressed responses from upstreams; if an upstream still returns a compressed stream, metering buffers that response).
  • Minimal parsing: the request body is scanned once for top-level fields, and model rewrites use in-place JSON edits rather than full deserialization.
  • Asynchronous settlement: usage calculation, charging, and state persistence run in background workers sharded by account, off the request path.
  • Hot-path caching: authentication, configuration, mappings, key pools, and prices are all served from process memory; configuration and key pool loads use singleflight to coalesce concurrent fetches.
  • No timeout truncation: the HTTP server sets only a read-header timeout (15 seconds) and an idle timeout (2 minutes), so long streaming responses are never cut off by the server; the wait for upstream response headers defaults to a maximum of 1800 seconds.

Persistence and shutdown

  • SIGHUP: runs a checkpoint (flushes the usage queue, persists account state, runs Redis SAVE) without interrupting service.
  • SIGTERM/SIGINT: stops accepting new requests and waits for in-flight requests (up to 20 seconds), closes WebSocket sessions and waits for their settlement, then runs a final checkpoint (up to 30 seconds). The deployment template's stop_grace_period: 60s covers both phases.
  • If the process is killed forcibly (SIGKILL, OOM), in-flight requests are not settled.