Overview
oMLX is a native macOS inference server built on Apple's MLX framework, designed for the way coding agents actually work. Its block-based KV cache spans two tiers — a hot tier in RAM for frequently used blocks, and a cold tier that offloads to SSD in safetensors format — so when an agent circles back to a prefix it has seen before, the cache is restored from disk instead of recomputed, even after a server restart. It serves LLM, VLM, OCR, embedding, and reranker models simultaneously with continuous batching, LRU eviction, model pinning, and per-model settings, and exposes drop-in OpenAI and Anthropic-compatible APIs (including a native /v1/messages endpoint). A signed, notarized menu bar app and a web dashboard handle model downloads, chat, benchmarking, and real-time metrics.
Minimum macOS: 15.0 (Sequoia)
Architecture: Apple Silicon
Key Features
- Tiered KV cache: hot tier in RAM, cold tier offloaded to SSD and restored on prefix match
- Cache survives server restarts, with prefix sharing and copy-on-write
- Continuous batching for concurrent requests, with configurable max concurrency
- Native macOS menu bar app (signed, notarized, in-app auto-update — not Electron)
- Multi-model serving: LLM, VLM, OCR, embedding, and reranker models at once
- LRU eviction, model pinning, per-model TTL, and a process memory ceiling
- Per-model sampling settings and named profiles served without a reload
- OpenAI + Anthropic drop-in APIs, including native /v1/messages, embeddings, and rerank
- Claude Code optimization: context scaling for auto-compact timing and SSE keep-alive
- One-click integration setup for OpenClaw, OpenCode, Codex, Hermes Agent, Copilot, and Pi
- Admin dashboard for model downloads from HuggingFace, chat, benchmarking, and live metrics
- Experimental multi-Mac distributed inference over Ring or Thunderbolt RDMA (source builds)
- Install via .dmg or Homebrew tap; Apache 2.0 licensed
