Building a Lightweight LLM Inference Engine in Rust: Candle

技术架构AI agents in production

Why Rust for LLM Inference?

Python is the king of AI training, but has fatal flaws for inference deployment: GIL, high memory overhead, no zero-copy. Rust's advantages for inference:

Dimension Python (PyTorch) Rust (candle)
Runtime overhead Interpreter + framework, heavy extra memory No runtime; memory is mostly model weights
Cold start typically 10-30s ~2-5s (measured in an example environment, for reference)
Memory safety Manual Compile-time guaranteed
Concurrency GIL-limited Unlimited
Distribution size runtime + deps often 500MB+ single binary ~15MB (example build, for reference)

Rust isn't replacing Python for training — it's the lightweight deployment form of inference: serving trained models as a single binary with minimal dependencies. Note that throughput depends on the implementation; this article does not claim candle beats vLLM on throughput — see "Honest Performance Positioning" below.

candle Framework: HuggingFace's Rust ML

candle core features:
├── Pure Rust, no Python dependency
├── CUDA and Metal GPU acceleration
├── Zero-copy tensor operations
├── GGML/GGUF quantization support
├── Supports: Llama, Qwen, Mistral, Phi, etc.
└── Compiles to single binary

Building the Inference Engine

Cargo.toml

[dependencies]
candle = { git = "https://github.com/huggingface/candle.git", features = ["cuda", "flash-attn"] }
candle-transformers = { git = "https://github.com/huggingface/candle.git" }
tokio = { version = "1", features = ["full"] }
axum = "0.7"
serde = { version = "1", features = ["derive"] }

Core Engine

pub struct InferenceEngine {
    model: Arc<RwLock<Model>>,
    tokenizer: Tokenizer,
    device: Device,
    config: EngineConfig,
}

impl InferenceEngine {
    pub async fn generate(&self, prompt: &str, max_tokens: usize) -> anyhow::Result<String> {
        let tokens = self.tokenizer.encode(prompt, true)?;
        let mut input_tokens = tokens.get_ids().to_vec();
        let mut generated_tokens = Vec::new();

        let model = self.model.read().await;

        for _ in 0..max_tokens {
            let input = Tensor::new(&input_tokens, &self.device)?.unsqueeze(0)?;
            let logits = model.forward(&input, input_tokens.len() - 1)?;
            let next_token = self.sample_token(&logits, &generated_tokens)?;
            generated_tokens.push(next_token);
            input_tokens.push(next_token);

            if next_token == self.tokenizer.get_eos_token_id() {
                break;
            }
        }

        let output = self.tokenizer.decode(&generated_tokens, true)?;
        Ok(output)
    }
}

HTTP API

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let device = Device::cuda(0)?;
    let engine = InferenceEngine::new("models/qwen2.5-7b-instruct-q4_0.gguf", device)?;
    let state = Arc::new(engine);

    let app = Router::new()
        .route("/v1/chat/completions", post(chat_handler))
        .with_state(state);

    let listener = tokio::net::TcpListener::bind("0.0.0.0:8000").await?;
    axum::serve(listener, app).await?;

    Ok(())
}

Honest Performance Positioning

Conclusion first: the engine in this article targets teaching and lightweight scenarios. It does not represent candle's throughput ceiling, and it cannot beat vLLM.

Why no tok/s comparison table?

The engine code above serializes the forward pass through a global RwLock and runs token-by-token with no batching — it physically cannot reach thousands of tokens/s. Numbers circulating online like "candle 8,200 tok/s crushes vLLM's 2,500" have no credible source; an earlier version of this article cited them, and they have been removed.

Industry consensus and independent benchmarks point the other way:

  • High-concurrency throughput: vLLM is clearly stronger thanks to PagedAttention + continuous batching; candle-style single-request implementations typically reach only ~70% of vLLM's throughput (varies with model, hardware, and quantization format — for reference only).
  • candle's real advantage is deployment form: single binary, no Python dependency, fast cold start, low runtime memory overhead.

Qualitative comparison (deployment dimensions)

Dimension vLLM This candle engine
High-concurrency throughput Strong (PagedAttention + continuous batching) Weak (no batching, serialized per request)
Single-request latency Low Low
Deployment dependencies Python + CUDA ecosystem Single binary, no Python
Cold start typically 10-30s ~2-5s (example environment, for reference)
Runtime memory overhead Python runtime + framework Mostly model weights + KV cache
Best fit Production high-concurrency serving Teaching, edge/embedded, lightweight services

Honest memory numbers

For Qwen2.5-7B-Instruct Q4_0:

  • Weights: ~4GB after Q4_0 quantization (~0.55 bytes per parameter; 7B × 0.55 ≈ 3.9GB)
  • KV cache: grows with context length and concurrency, typically +1-2GB
  • Total: ~5-6GB

Any claim that "a 7B Q4_0 model uses only 1.1GB of memory" violates basic arithmetic — the weights alone are ~4GB. An earlier version of this article contained that figure in error; it has been corrected.

Production Deployment

FROM rust:1.78 as builder
WORKDIR /app
COPY . .
RUN cargo build --release

FROM nvidia/cuda:12.4-runtime-ubuntu22.04
COPY --from=builder /app/target/release/rust-llm-engine /usr/local/bin/
EXPOSE 8000
CMD ["rust-llm-engine"]

Summary

Rust's core advantages for LLM inference:

  1. Lightweight deployment: single ~15MB binary (example build), no Python dependency, ~2-5s cold start (example environment)
  2. Memory safety: No data races, no null pointers, no buffer overflows
  3. Low runtime overhead: memory is mostly model weights + KV cache — no interpreter or framework overhead
  4. No GIL: True multi-threaded concurrency

To be honest: throughput is not the selling point of this setup. For high-concurrency production serving, choose an engine with PagedAttention and continuous batching such as vLLM; the candle approach fits teaching, edge deployment, and lightweight services where the deployment form matters.

Rust doesn't exist to out-muscle anyone on throughput — it adds one more deployment form for inference: train in Python, serve at high concurrency with vLLM, and go lightweight/edge with Rust + candle.

#Rust#LLM推理#轻量部署#candle#推理引擎#内存安全