# miniVLLM

## Docs

- [Attention](https://mintlify.wiki/Wenyueh/MinivLLM/api/attention.md): Triton-accelerated attention kernels for prefill (flash attention) and decode (paged attention), plus the unified Attention module that routes between them.
- [Layers](https://mintlify.wiki/Wenyueh/MinivLLM/api/layers.md): Neural network building blocks for miniVLLM: activations, layer normalization, tensor-parallel linear layers, embeddings, positional encodings, and token sampling.
- [LLMEngine](https://mintlify.wiki/Wenyueh/MinivLLM/api/llm-engine.md): High-level engine that orchestrates tokenization, scheduling, and multi-GPU model execution for batched text generation.
- [ModelRunner](https://mintlify.wiki/Wenyueh/MinivLLM/api/model-runner.md): Per-rank model execution backend with paged attention, CUDA graph replay, and inter-process communication for multi-GPU inference.
- [Models](https://mintlify.wiki/Wenyueh/MinivLLM/api/models.md): Causal language model implementations for Qwen3 and Llama 3, including constructor parameters, forward interfaces, checkpoint loading, and a guide for registering new models.
- [SamplingParams](https://mintlify.wiki/Wenyueh/MinivLLM/api/sampling-params.md): Dataclass for controlling token sampling behavior — temperature, output length limits, and EOS handling.
- [Scheduler](https://mintlify.wiki/Wenyueh/MinivLLM/api/scheduler.md): Token-level batch scheduler with paged KV cache management, prefix caching, and preemption support.
- [Engine Architecture](https://mintlify.wiki/Wenyueh/MinivLLM/architecture/engine.md): End-to-end walkthrough of LLMEngine: how requests flow from add_prompt through scheduling, model execution, and sampling back to the caller.
- [Neural Network Layers](https://mintlify.wiki/Wenyueh/MinivLLM/architecture/layers.md): The primitive building blocks in myvllm/layers: activation, normalization, tensor-parallel linear, embeddings, attention, positional encoding, and sampling.
- [Model Implementations](https://mintlify.wiki/Wenyueh/MinivLLM/architecture/models.md): How Qwen3 and Llama 3.2 are assembled from miniVLLM layers, how checkpoint weights are mapped to model parameters, and how to add a new model.
- [Multi-GPU Inference](https://mintlify.wiki/Wenyueh/MinivLLM/architecture/multi-gpu.md): How miniVLLM shards model weights across GPUs using tensor parallelism, and how worker processes are coordinated at runtime.
- [Decoding Benchmark](https://mintlify.wiki/Wenyueh/MinivLLM/benchmarks/decoding.md): Compare Naive PyTorch, Optimized PyTorch, and a Triton kernel for paged attention during the decode phase, including block table lookup mechanics and CUDA chunk processing.
- [Prefill Benchmark](https://mintlify.wiki/Wenyueh/MinivLLM/benchmarks/prefill.md): Compare PyTorch Standard, Naive Triton, and Flash Attention implementations during the prefill phase, including crossover analysis and kernel launch overhead.
- [Tokens Per Second](https://mintlify.wiki/Wenyueh/MinivLLM/benchmarks/tps.md): Measure end-to-end inference throughput of miniVLLM and compare it against vLLM and Hugging Face Transformers using Qwen3-0.6B.
- [Flash Attention](https://mintlify.wiki/Wenyueh/MinivLLM/concepts/flash-attention.md): How miniVLLM's Triton kernel implements tiled online-softmax attention with O(N) HBM access for the prefill phase.
- [KV Cache](https://mintlify.wiki/Wenyueh/MinivLLM/concepts/kv-cache.md): How miniVLLM stores, retrieves, and reuses key-value tensors across decode steps using paged physical blocks and xxhash-based prefix caching.
- [Paged Attention](https://mintlify.wiki/Wenyueh/MinivLLM/concepts/paged-attention.md): How miniVLLM eliminates GPU memory fragmentation by mapping KV cache to fixed-size physical blocks with prefix caching.
- [Scheduling](https://mintlify.wiki/Wenyueh/MinivLLM/concepts/scheduling.md): How miniVLLM's iteration-level scheduler prioritizes prefill, manages running sequences, and preempts when GPU memory runs out.
- [Installation](https://mintlify.wiki/Wenyueh/MinivLLM/installation.md): Step-by-step guide to installing miniVLLM and its dependencies.
- [Introduction](https://mintlify.wiki/Wenyueh/MinivLLM/introduction.md): miniVLLM is a from-scratch replication of the vLLM inference engine, featuring custom Triton kernels for paged attention and flash attention.
- [Quick start](https://mintlify.wiki/Wenyueh/MinivLLM/quickstart.md): Run your first miniVLLM inference in under five minutes.