Chorus LLM is a high-performance embedded LLM runtime for your game.

It succeeds at making local inference work in compute-starved environments when many resources are asking for inference. Chorus is built from the ground up to work for applications: video games, firstly.

Run a whole village of NPCs with minimal hardware and a single model. Prioritize player interactions, give NPCs stochastic motivations, or schedule and efficiently run background agents.

This is a personal passion project of mine. Expect me to iterate on it and make it better over time; in the meanwhile, report any issues you experience or feature requests on my Issues channel.

Full README and a guide for Godot.

Requirements

• Godot 4.4 or later.
• Linux x86-64 or Windows x86-64 with Vulkan (the CPU path needs AVX2), or macOS on Apple silicon with Metal. Intel Macs are not supported.
• A GGUF model with a chat template. Chorus does not ship with a model and never will!

Features

• Concurrent agents. Many requests share one loaded model. In my testing, I had a town of 40+ NPCs conversing, each with an internal monologue.
• Priority scheduling. Keep urgent work moving first.
• Continuous batching. New requests join mid-stream; your game doesn't wait on Chorus unless you want it to.
• Conversation control. Keep separate histories per instance. Edit, regenerate, import, and export them.
• Structured output. Constrain replies to JSON Schema or a GBNF grammar you can parse. Tools aren't (yet) available, but you could construct tools with these, I guess, but I'll eventually implement them so if you need tool use just poke me.
• Reliable outcomes. Every accepted request ends in exactly one success or error, and failed turns roll back their history.
• Godot-native API. A GodotChorus node, typed request resources, signals, Project Settings, and built-in editor docs. Godot was (and continues to be) my first target.

See FEATURES.md for features planned and implemented.