continuous-batching

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

A high-performance API server that provides OpenAI-compatible endpoints for MLX models. Developed using Python and powered by the FastAPI framework, it provides an efficient, scalable, and user-friendly solution for running MLX-based vision and language models locally with an OpenAI-compatible interface.

RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching

An Efficient and Versatile Inference Engine for Distributed LLM Serving

面向开发者与初学者的 nano-vLLM 交互式源码教程 - 通过 13 个 HTML 互动实验 + 13 章中文教程理解大模型推理引擎

Reproducible llama.cpp CPU inference profiling and a deterministic LLM serving simulator with continuous batching, KV cache, prefix caching, and workload-driven latency analysis.

A local LLM runtime for many-agent applications, built on llama.cpp. One model, shared inference, many agents, embedded. C++/C, Vulkan, Metal, Godot.

Miniature LLM serving runtime with continuous batching, chunked prefill, KV-cache management, preemption, prefix caching, and real Llama-family execution.

A from scratch LLM inference engine build in PyTorch with custom GPT2 transformers, kv cache, paged kv cache, continuous batching and A100 benchmarks

A lightweight, educational LLM inference engine for studying continuous batching, paged KV cache, chunked prefill, and online serving.

An AI inference engine from scratch. Like nanoGPT, but for serving.

Continuous batching for TTS — like vLLM, but for voice. Serve 10+ simultaneous text-to-speech requests on a single GPU.

🌱 A tiny, readable LLM serving engine with vLLM/SGLang-style features.

Mini-SGLang port for Ascend NPUs with FIA, paged KV cache, ragged continuous batching, and request lifecycle safety.

vLLM-style continuous batching for iPhone. Native Swift on MLX, no Python. The fastest multi-agent inference stack on iOS.

A mini-vLLM: a from-scratch LLM inference server with a paged KV cache and continuous batching (GPT-2, OpenAI-compatible streaming API).

用约 680 行 Python 实现 vLLM 核心：页式 KV Cache、连续批处理与抢占。A readable vLLM core in ~680 lines.

An educational LLM inference engine built from scratch: custom decoding loop, KV cache, continuous batching, paged memory with preemption, prefix caching, speculative decoding, OpenAI-compatible API, real benchmarks.