LitServe
Python framework for serving AI models as APIs with batching and scaling
Steal this
An AI server in one Python class, with batching and streaming in code and no config files.
What it is
A minimal Python framework for building your own inference servers, for models, agents, RAG or pipelines. You control batching, routing and streaming in code, with no config files.
What it’s for
For serving a model or agent as an API with batching and multiple GPUs, without MLOps glue.
Benefits
- Inference logic written in plain Python
- Batching, streaming and multi-GPU scaling
- Runs locally, self-hosted or on Lightning AI
Caveats
- According to its maker, it is twice as fast as FastAPI
- Apache-2.0 license; managed hosting on Lightning AI is paid
