kvpack
An open-source Python tool that turns repositories, docs sites, and folders into trained KV caches for self-hosted language models.
The idea
Give a language model reusable context without sending the full documents with every request. kvpack trains a compact KV cache, called a cartridge, from a repository, documentation site, or folder. The model’s weights stay unchanged.
How it works
Build a cartridge from your sources, chat with it locally, or serve it through an OpenAI-compatible API. The project includes a command-line interface and a self-hosted Studio web app for connecting sources and keeping cartridges in sync.
Engineering focus
kvpack brings context distillation into a Python tool built with PyTorch and Hugging Face Transformers. It implements the Cartridges research method and includes an evaluation harness to compare cartridges with retrieval and full-document context.
The project is early and research-grade. Small-model benchmarks show that a cartridge alone recalls fewer exact facts than retrieval, so the evaluation tools are part of understanding where the approach is useful.