Edge AI Model Optimization in Practice: A Full-Pipeline Guide to Quantization, Pruning, and Deployment

AI与大数据

Summary

  • Edge AI is the largest growth market in 2026: global edge AI chip shipments are expected to exceed 5 billion, and model optimization is the key bottleneck for deployment
  • Quantization in 3 stages: FP16→INT8→INT4, INT8 quantization accuracy loss <1%, INT4 requires GPTQ/AWQ algorithms
  • Structured pruning in 2 approaches: magnitude pruning is simple and efficient, distillation pruning offers better accuracy, combining both can compress 70%+ parameters
  • Deployment in 3 frameworks: ONNX Runtime for best versatility, TensorRT for best performance, TFLite as the top choice for mobile
  • This article provides full-pipeline practical code from model optimization to edge deployment

Table of Contents

Edge AI: The Last Mile of AI Deployment

Edge AI Market Overview

Device Type 2026 Shipments Typical Compute Typical Applications
Smartphones 1.5B 30-60 TOPS Voice assistants, photo enhancement
Smart cameras 500M 2-8 TOPS Face recognition, behavior detection
Automotive (ADAS) 100M 50-200 TOPS Autonomous driving, DMS
IoT gateways 300M 1-4 TOPS Predictive maintenance, anomaly detection
Industrial controllers 50M 5-20 TOPS Quality inspection, robot control

Core Challenges of Edge AI

Challenge Cloud Edge Gap
Compute 1000+ TOPS 2-60 TOPS 50-500×
Memory 80-640GB 2-16GB 5-80×
Power 300W+ 5-30W 10-60×
Latency 50-200ms (network) 1-10ms (local) Edge advantage
Privacy Data goes to cloud Processed locally Edge advantage

Edge AI Chip Landscape

Chip Vendor Compute Power Ecosystem
Jetson Orin NX NVIDIA 100 TOPS 25W CUDA+TensorRT
Hailo-8 Hailo 26 TOPS 2.5W Proprietary compiler
Google Edge TPU Google 4 TOPS 2W TFLite
Rockchip RK3588 Rockchip 6 TOPS 10W RKNN
Huawei Ascend 310P Huawei 22 TOPS 8W MindSpore
Horizon Robotics J5 Horizon Robotics 128 TOPS 15W Tiangong Kaiwu

Model Quantization in 3 Stages

Quantization Principles and Accuracy Impact

┌──────────────────────────────────────────────────────────────┐ │ Quantization in 3 Stages │ │ │ │ FP32 → FP16 (Half Precision) │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Accuracy loss: <0.1% │ │ │ │ Size reduction: 50% │ │ │ │ Speedup: 1.5-2× │ │ │ │ Method: Direct conversion, no calibration needed │ │ │ └──────────────────────────────────────────────────────┘ │ │ ↓ │ │ FP16 → INT8 (8-bit Integer Quantization) │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Accuracy loss: 0.5-2% │ │ │ │ Size reduction: 75% (vs FP32) │ │ │ │ Speedup: 2-4× │ │ │ │ Method: PTQ (Post-Training Quantization) / QAT │ │ │ │ (Quantization-Aware Training) │ │ │ └──────────────────────────────────────────────────────┘ │ │ ↓ │ │ INT8 → INT4 (4-bit Integer Quantization) │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Accuracy loss: 2-5% (requires GPTQ/AWQ compensation)│ │ │ │ Size reduction: 87.5% (vs FP32) │ │ │ │ Speedup: 3-6× │ │ │ │ Method: GPTQ / AWQ / AQLM / HQQ │ │ │ └──────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────┘

INT8 PTQ Quantization in Practice

`python import torch from transformers import AutoModelForCausalLM, AutoTokenizer from neural_compressor import QuantizationAwareTraining, PostTrainingQuantConfig

def int8_ptq_quantize(model_path, output_path, calibration_data): model = AutoModelForCausalLM.from_pretrained( model_path, torch_dtype=torch.float16, device_map="auto", ) tokenizer = AutoTokenizer.from_pretrained(model_path)

config = PostTrainingQuantConfig(
    approach="static",
    tuning_criterion={
        "max_trials": 100,
        "objective": "accuracy",
    },
    quant_format="QDQ",
    calibration_sampling_size=512,
)

def calibration_dataloader():
    for text in calibration_data:
        inputs = tokenizer(text, return_tensors="pt", max_length=512)
        yield {"input_ids": inputs["input_ids"]}

from neural_compressor import quantize
q_model = quantize(
    model,
    config,
    calibration_dataloader=calibration_dataloader,
)

q_model.save(output_path)
return q_model

def int8_gptq_quantize(model_path, output_path, calibration_data): from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoGPTQForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float16,
    device_map="auto",
)

quantize_config = BaseQuantizeConfig(
    bits=8,
    group_size=128,
    desc_act=True,
    damp_percent=0.01,
    sym=True,
)

examples = []
for text in calibration_data:
    tokenized = tokenizer(text, return_tensors="pt", max_length=2048)
    examples.append(tokenized["input_ids"])

model.quantize(examples, quantize_config=quantize_config)
model.save_quantized(output_path)
return model

`

INT4 Quantization in Practice (GPTQ/AWQ)

`python def int4_gptq_quantize(model_path, output_path, calibration_data): from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoGPTQForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float16,
    device_map="auto",
)

quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=True,
    damp_percent=0.01,
    sym=False,
)

examples = []
for text in calibration_data[:256]:
    tokenized = tokenizer(text, return_tensors="pt", max_length=2048)
    examples.append(tokenized["input_ids"])

model.quantize(examples, quantize_config=quantize_config)
model.save_quantized(output_path)
return model

def int4_awq_quantize(model_path, output_path, calibration_data): from awq import AutoAWQForCausalLM

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",
}

model.quantize(
    tokenizer,
    quant_config=quant_config,
    calib_data=calibration_data[:256],
)

model.save_quantized(output_path)
return model

`

Quantization Results Comparison

Quantization Scheme 7B Model Size Inference Speed Accuracy Loss GPU Memory
FP32 28GB 1× Baseline 32GB
FP16 14GB 1.8× <0.1% 16GB
INT8 PTQ 7GB 3.2× 0.5-2% 8GB
INT8 GPTQ 7GB 3.5× 0.3-1% 8GB
INT4 GPTQ 3.5GB 5.0× 2-4% 4GB
INT4 AWQ 3.5GB 5.5× 1-3% 4GB

Structured Pruning in Practice

Pruning Methods Comparison

Method Granularity Accuracy Loss Hardware-Friendly Complexity
Unstructured pruning Individual weight Low Poor (sparse compute) Low
Structured pruning Channel/layer Medium Good (dense compute) Medium
Distillation pruning Channel + distillation Low Good High
Auto pruning (NAS) Search space Low Good Very high

Magnitude Pruning in Practice

`python import torch import torch.nn as nn from transformers import AutoModelForCausalLM

class MagnitudePruner: def init(self, model, sparsity_ratio=0.5): self.model = model self.sparsity_ratio = sparsity_ratio self.masks = {}

def compute_masks(self):
    for name, module in self.model.named_modules():
        if isinstance(module, nn.Linear):
            weight = module.weight.data.abs()
            threshold = torch.quantile(
                weight.flatten(), self.sparsity_ratio
            )
            self.masks[name] = (weight > threshold).float()

def apply_masks(self):
    for name, module in self.model.named_modules():
        if name in self.masks and isinstance(module, nn.Linear):
            module.weight.data *= self.masks[name]

def structured_prune(self):
    for name, module in self.model.named_modules():
        if isinstance(module, nn.Linear):
            weight = module.weight.data
            row_norms = weight.norm(dim=1)
            threshold = torch.quantile(
                row_norms, self.sparsity_ratio
            )
            keep_mask = (row_norms > threshold).float()
            module.weight.data *= keep_mask.unsqueeze(1)
            if module.bias is not None:
                module.bias.data *= keep_mask

def fine_tune(self, dataloader, epochs=3, lr=1e-5):
    optimizer = torch.optim.AdamW(self.model.parameters(), lr=lr)
    
    for epoch in range(epochs):
        for batch in dataloader:
            outputs = self.model(**batch)
            loss = outputs.loss
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
            self.apply_masks()
    
    return self.model

def prune_model(model_path, output_path, sparsity=0.5): model = AutoModelForCausalLM.from_pretrained( model_path, torch_dtype=torch.float16, device_map="auto" )

pruner = MagnitudePruner(model, sparsity_ratio=sparsity)
pruner.structured_prune()

model.save_pretrained(output_path)
return model

`

Pruning + Distillation Combination

`python class PruneDistillPipeline: def init(self, teacher_path, student_path, sparsity=0.5): self.teacher = AutoModelForCausalLM.from_pretrained( teacher_path, torch_dtype=torch.float16, device_map="auto" ) self.student = AutoModelForCausalLM.from_pretrained( student_path, torch_dtype=torch.float16, device_map="auto" ) self.pruner = MagnitudePruner(self.student, sparsity)

def train(self, dataloader, epochs=5):
    optimizer = torch.optim.AdamW(self.student.parameters(), lr=2e-5)
    kl_loss = nn.KLDivLoss(reduction="batchmean")
    
    for epoch in range(epochs):
        for batch in dataloader:
            with torch.no_grad():
                teacher_out = self.teacher(**batch)
            
            student_out = self.student(**batch)
            
            hard_loss = student_out.loss
            
            soft_loss = kl_loss(
                F.log_softmax(student_out.logits / 2.0, dim=-1),
                F.softmax(teacher_out.logits / 2.0, dim=-1),
            ) * 4.0
            
            total_loss = 0.3 * hard_loss + 0.7 * soft_loss
            
            total_loss.backward()
            optimizer.step()
            optimizer.zero_grad()
            self.pruner.apply_masks()
    
    return self.student

`

Pruning Results Comparison

Pruning Scheme Parameters Retained Accuracy Loss Inference Speedup Size Reduction
Magnitude pruning 50% 50% 3-5% 1.8× 50%
Magnitude pruning 70% 30% 8-12% 2.5× 70%
Distillation pruning 50% 50% 1-3% 1.8× 50%
Distillation pruning 70% 30% 4-6% 2.5× 70%
Pruning + Quantization 30%+INT8 5-8% 5× 85%

ONNX Runtime Deployment in Practice

PyTorch → ONNX Export

`python import torch from transformers import AutoModelForCausalLM, AutoTokenizer

def export_to_onnx(model_path, output_path, opset=17): model = AutoModelForCausalLM.from_pretrained( model_path, torch_dtype=torch.float16 ) tokenizer = AutoTokenizer.from_pretrained(model_path) model.eval()

dummy_input = tokenizer("Hello", return_tensors="pt")
input_ids = dummy_input["input_ids"]
attention_mask = dummy_input["attention_mask"]

torch.onnx.export(
    model,
    (input_ids, attention_mask),
    output_path,
    opset_version=opset,
    input_names=["input_ids", "attention_mask"],
    output_names=["logits"],
    dynamic_axes={
        "input_ids": {0: "batch", 1: "seq_len"},
        "attention_mask": {0: "batch", 1: "seq_len"},
        "logits": {0: "batch", 1: "seq_len"},
    },
    do_constant_folding=True,
)
print(f"Exported to {output_path}")

`

ONNX Runtime Inference

`python import onnxruntime as ort import numpy as np

class ONNXInferenceEngine: def init(self, model_path, provider="CUDAExecutionProvider"): sess_options = ort.SessionOptions() sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL sess_options.intra_op_num_threads = 4 sess_options.inter_op_num_threads = 4

    self.session = ort.InferenceSession(
        model_path,
        sess_options=sess_options,
        providers=[provider],
    )
    
    self.io_binding = self.session.io_binding()

def infer(self, input_ids, attention_mask):
    input_ids_np = np.array(input_ids, dtype=np.int64)
    attention_mask_np = np.array(attention_mask, dtype=np.int64)
    
    outputs = self.session.run(
        ["logits"],
        {
            "input_ids": input_ids_np,
            "attention_mask": attention_mask_np,
        },
    )
    
    return outputs[0]

def infer_with_io_binding(self, input_ids, attention_mask):
    input_ids_tensor = ort.OrtValue.ortvalue_from_numpy(
        np.array(input_ids, dtype=np.int64), "cuda", 0
    )
    attention_mask_tensor = ort.OrtValue.ortvalue_from_numpy(
        np.array(attention_mask, dtype=np.int64), "cuda", 0
    )
    
    self.io_binding.bind_ortvalue_input("input_ids", input_ids_tensor)
    self.io_binding.bind_ortvalue_input("attention_mask", attention_mask_tensor)
    self.io_binding.bind_output("logits", "cuda", 0)
    
    self.session.run_with_iobinding(self.io_binding)
    
    return self.io_binding.get_outputs()[0].numpy()

`

ONNX Runtime Performance Comparison

Configuration Latency (ms) Throughput (req/s) GPU Memory
PyTorch FP16 45 22 8GB
ORT FP16 32 31 6GB
ORT INT8 18 55 4GB
ORT INT8+IO Binding 14 71 4GB
ORT INT8+Batch=8 8/batch 125 6GB

TensorRT Optimization in Practice

ONNX → TensorRT Conversion

`python import tensorrt as trt

def build_tensorrt_engine( onnx_path, engine_path, fp16=True, int8=False, max_batch_size=8, max_workspace=4 << 30, ): logger = trt.Logger(trt.Logger.WARNING) builder = trt.Builder(logger) network = builder.create_network( 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH) ) parser = trt.OnnxParser(network, logger)

with open(onnx_path, "rb") as f:
    if not parser.parse(f.read()):
        for i in range(parser.num_errors):
            print(f"Parse error: {parser.get_error(i)}")
        return None

config = builder.create_builder_config()
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, max_workspace)

if fp16:
    config.set_flag(trt.BuilderFlag.FP16)

if int8:
    config.set_flag(trt.BuilderFlag.INT8)
    config.int8_calibrator = None

profile = builder.create_optimization_profile()
profile.set_shape(
    "input_ids",
    min=(1, 1),
    opt=(1, 128),
    max=(max_batch_size, 2048),
)
profile.set_shape(
    "attention_mask",
    min=(1, 1),
    opt=(1, 128),
    max=(max_batch_size, 2048),
)
config.add_optimization_profile(profile)

engine = builder.build_serialized_network(network, config)

with open(engine_path, "wb") as f:
    f.write(engine)

print(f"TensorRT engine saved to {engine_path}")
return engine

class TensorRTInference: def init(self, engine_path): self.logger = trt.Logger(trt.Logger.WARNING) self.runtime = trt.Runtime(self.logger)

    with open(engine_path, "rb") as f:
        self.engine = self.runtime.deserialize_cuda_engine(f.read())
    
    self.context = self.engine.create_execution_context()

def infer(self, input_ids, attention_mask):
    import pycuda.driver as cuda
    import pycuda.autoinit
    
    input_ids_np = np.array(input_ids, dtype=np.int32)
    attention_mask_np = np.array(attention_mask, dtype=np.int32)
    
    d_input_ids = cuda.mem_alloc(input_ids_np.nbytes)
    d_attention_mask = cuda.mem_alloc(attention_mask_np.nbytes)
    
    cuda.memcpy_htod(d_input_ids, input_ids_np)
    cuda.memcpy_htod(d_attention_mask, attention_mask_np)
    
    output_shape = self.engine.get_binding_shape(2)
    d_output = cuda.mem_alloc(
        trt.volume(output_shape) * np.dtype(np.float32).itemsize
    )
    
    bindings = [int(d_input_ids), int(d_attention_mask), int(d_output)]
    
    self.context.execute_v2(bindings)
    
    output = np.empty(output_shape, dtype=np.float32)
    cuda.memcpy_dtoh(output, d_output)
    
    return output

`

TensorRT Performance Comparison

Configuration Latency (ms) Throughput (req/s) Speedup
PyTorch FP16 45 22 1×
ORT INT8 18 55 2.5×
TRT FP16 15 66 3×
TRT INT8 8 125 5.6×
TRT INT8+Batch=8 3/batch 333 15×

Edge Device Inference Strategies

Tiered Inference Architecture

┌──────────────────────────────────────────────────────────────┐ │ Edge-Cloud Collaborative Inference Architecture │ │ │ │ Cloud (Large Model) │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ 70B model → Complex reasoning, long text generation,│ │ │ │ multi-turn dialogue │ │ │ │ Latency: 200-500ms | Cost: High │ │ │ └──────────────────────────────────────────────────────┘ │ │ ↕ Network │ │ Edge Gateway (Medium Model) │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ 7B quantized model → Daily reasoning, simple chat, │ │ │ │ intent recognition │ │ │ │ Latency: 20-50ms | Cost: Medium │ │ │ └──────────────────────────────────────────────────────┘ │ │ ↕ Local Bus │ │ End Device (Small Model) │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Pruned + quantized small model → Keyword detection, │ │ │ │ wake word, simple │ │ │ │ classification │ │ │ │ Latency: 1-5ms | Cost: None │ │ │ └──────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────┘

Edge Deployment Code (Jetson)

`python import jetson.inference import jetson.utils

class EdgeInferencePipeline: def init(self, model_type="classification", threshold=0.7): self.model = jetson.inference.initialize(model_type) self.threshold = threshold self.fallback_enabled = True

def infer_local(self, image):
    try:
        results = self.model.Detect(image)
        confident = [r for r in results if r.Confidence >= self.threshold]
        
        if not confident and self.fallback_enabled:
            return self.infer_cloud(image)
        
        return confident
    except Exception as e:
        if self.fallback_enabled:
            return self.infer_cloud(image)
        raise

def infer_cloud(self, image):
    import requests
    _, compressed = cv2.imencode(".jpg", image, [cv2.IMWRITE_JPEG_QUALITY, 85])
    
    response = requests.post(
        "https://api.example.com/infer",
        files={"image": compressed.tobytes()},
        timeout=5.0,
    )
    return response.json()

def benchmark(self, image, iterations=100):
    import time
    
    latencies = []
    for _ in range(iterations):
        start = time.perf_counter()
        self.infer_local(image)
        latencies.append((time.perf_counter() - start) * 1000)
    
    return {
        "mean_ms": sum(latencies) / len(latencies),
        "p50_ms": sorted(latencies)[len(latencies) // 2],
        "p99_ms": sorted(latencies)[int(len(latencies) * 0.99)],
    }

`

Edge Device Performance Benchmarks

Device Model Quantization Latency (ms) Accuracy
Jetson Orin NX BERT-base FP16 12 99.2%
Jetson Orin NX BERT-base INT8 6 98.8%
Jetson Orin NX ResNet50 FP16 8 99.5%
Jetson Orin NX ResNet50 INT8 4 99.1%
RK3588 MobileBERT INT8 25 97.5%
RK3588 MobileNetV3 INT8 8 98.2%
Raspberry Pi 5 TinyBERT INT8 120 95.8%
Raspberry Pi 5 MobileNetV2 INT8 45 97.0%

Summary and Resources

Key Takeaways

  1. Quantization: INT8 PTQ offers the best cost-effectiveness; INT4 requires GPTQ/AWQ for accuracy compensation
  2. Pruning: Structured pruning + distillation can compress 70%+ parameters with controlled accuracy loss
  3. Deployment: ONNX Runtime for versatility, TensorRT for extreme performance, TFLite for mobile
  4. Edge Strategy: Tiered inference + cloud fallback balances latency and accuracy

Edge AI Optimization Roadmap

Stage Optimization Method Expected Result
Step 1 FP16 export + ONNX Runtime 2× speedup
Step 2 INT8 quantization 4× speedup
Step 3 Structured pruning Size -50%
Step 4 TensorRT optimization 6× speedup
Step 5 Edge-cloud collaboration Flexible deployment

Need to compress AI-generated images? Try our Image Compression Tool and Video Compression to quickly optimize edge deployment assets.

Further Reading

Try these browser-local tools — no sign-up required →

#边缘AI优化#模型量化#ONNX Runtime#TensorRT部署#边缘推理优化#2026