AI Chip Interconnect in Practice: NVLink, NVSwitch, and High-Speed Interconnect Architecture

AI与大数据

Summary

  • Interconnect is the lifeline of AI chip clusters: 80%+ of large model training time is spent on communication, and interconnect bandwidth determines cluster efficiency
  • NVLink 5 is the most powerful interconnect in 2026: 1.8TB/s bidirectional bandwidth, 14× that of PCIe 6.0, with NVSwitch enabling full interconnect
  • 3 major interconnect technology paths: NVLink (tightly coupled), Ultra Ethernet (loosely coupled), CXL (cache coherent), each with optimal scenarios
  • 4 key techniques for multi-GPU communication optimization: gradient compression (reduces 90% communication), communication-computation overlap, topology-aware scheduling, RDMA passthrough
  • This article provides NVLink+NCCL configuration practice and a full-stack solution for multi-GPU training communication optimization

Table of Contents

Interconnect: The Lifeline of AI Chip Clusters

Communication Proportion in Large Model Training

Model Scale GPU Count Compute Time % Communication Time % Communication Bottleneck
7B 1 95% 5% None
70B 8 70% 30% Moderate
405B 64 45% 55% Severe
1T+ 256+ 25% 75% Extremely Severe

Interconnect Bandwidth Requirements

Operation Communication Pattern Data Volume Bandwidth Requirement
Data Parallel AllReduce All2All 2×Model Params×4 High
Tensor Parallel AllReduce AllReduce 2×Activations×4 Very High
Pipeline Parallel Communication P2P Micro-batch Activations Medium
Expert Parallel All2All All2All Token Routing Very High

Interconnect Technology Evolution

Year Technology Bandwidth Latency Application
2016 PCIe 3.0 32GB/s 500ns General
2017 NVLink 1.0 160GB/s 200ns P100
2019 NVLink 2.0 300GB/s 150ns V100
2020 NVLink 3.0 600GB/s 100ns A100
2022 NVLink 4.0 900GB/s 80ns H100
2024 NVLink 5.0 1800GB/s 50ns B200
2026 Ultra Ethernet 800Gbps 2μs Cluster
Parameter NVLink 1.0 NVLink 2.0 NVLink 3.0 NVLink 4.0 NVLink 5.0
Per-link Bandwidth 40GB/s 50GB/s 50GB/s 50GB/s 100GB/s
Link Count 4 6 12 18 18
Total Bandwidth (Bidirectional) 160GB/s 300GB/s 600GB/s 900GB/s 1800GB/s
Signal Rate 20Gbps 25Gbps 50Gbps 50Gbps 100Gbps
Channel Encoding NRZ NRZ PAM4 PAM4 PAM4
Power per Link 2W 2.5W 3W 3.5W 5W

┌──────────────────────────────────────────────────────────────┐ │ NVLink 5 Architecture Innovations │ │ │ │ 1. Channel Rate Doubled │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ PAM4 Signal: 100Gbps/channel (vs NVLink4 50Gbps) │ │ │ │ 18 channels × 100Gbps = 1.8Tbps unidirectional │ │ │ │ Bidirectional: 1.8TB/s │ │ │ └──────────────────────────────────────────────────────┘ │ │ │ │ 2. Low Latency Optimization │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Latency: 50ns (vs NVLink4 80ns) │ │ │ │ Credit-based flow control optimization │ │ │ │ Adaptive routing │ │ │ └──────────────────────────────────────────────────────┘ │ │ │ │ 3. Multi-protocol Support │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Native support: NVLink, CXL, PCIe │ │ │ │ Unified interconnect architecture │ │ │ │ CPU-GPU coherence protocol │ │ │ └──────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────┘

Dimension PCIe 6.0 NVLink 5 Speedup
Bidirectional Bandwidth 128GB/s 1800GB/s 14×
Latency 500ns 50ns 10×
Connected GPU Count 1 18+ 18×
Cache Coherence CXL Native -
Cost Low Very High -

NVSwitch and Full Interconnect Architecture

NVSwitch Evolution

Parameter NVSwitch 1.0 NVSwitch 2.0 NVSwitch 3.0 NVSwitch 4.0
Port Count 18 36 64 72
Per-port Bandwidth 50GB/s 50GB/s 50GB/s 100GB/s
Total Bandwidth 900GB/s 1.8TB/s 3.2TB/s 7.2TB/s
Supported GPU Count 8 8 8 72
Multi-level Interconnect No No Yes Yes

NVSwitch Full Interconnect Architecture

┌──────────────────────────────────────────────────────────────┐ │ NVSwitch 4.0 Full Interconnect Architecture │ │ │ │ Single Node 8×GPU Full Interconnect │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ GPU0 ←→ NVSwitch ←→ GPU1 │ │ │ │ GPU2 ←→ NVSwitch ←→ GPU3 │ │ │ │ GPU4 ←→ NVSwitch ←→ GPU5 │ │ │ │ GPU6 ←→ NVSwitch ←→ GPU7 │ │ │ │ Between any two GPUs: 1.8TB/s │ │ │ └──────────────────────────────────────────────────────┘ │ │ │ │ Multi-node 72×GPU Full Interconnect (NVLink Spine) │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Node0 NVSwitch ←→ NVLink Spine ←→ Node1 NVSwitch │ │ │ │ Node2 NVSwitch ←→ NVLink Spine ←→ Node3 NVSwitch │ │ │ │ ... │ │ │ │ Between any two GPUs: 900GB/s (cross-node) │ │ │ └──────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────┘

NVSwitch Topology Configuration

`yaml

DGX H100 NVLink Topology

gpu_topology: nvlink: bandwidth: 900GB/s connections: - [GPU0, GPU1, 4_links] - [GPU0, GPU2, 4_links] - [GPU0, GPU3, 4_links] - [GPU0, GPU4, 4_links] - [GPU0, GPU5, 4_links] - [GPU0, GPU6, 4_links] - [GPU0, GPU7, 4_links] - [GPU1, GPU2, 4_links] - [GPU1, GPU3, 4_links] - [GPU1, GPU4, 4_links] - [GPU1, GPU5, 4_links] - [GPU1, GPU6, 4_links] - [GPU1, GPU7, 4_links] nvswitch: count: 4 ports_per_switch: 18 total_cross_section: 7.2TB/s `

Comparison of 3 Major Interconnect Technology Paths

Path Comparison

Dimension NVLink 5 Ultra Ethernet CXL 3.0
Bandwidth 1.8TB/s 800Gbps (100GB/s) 64GB/s
Latency 50ns 2μs 1μs
Connection Distance <2m 100m+ 10m
Cache Coherence Native No Yes
Cost Very High Medium Medium
Ecosystem NVIDIA Lock-in Open Open
Applicable Scenario Intra-node Inter-node CPU-GPU

Selection Decision

`` Inter-GPU communication distance? ├── <2m (intra-node) → NVLink ├── 2-100m (intra-rack) → Ultra Ethernet / InfiniBand └── >100m (inter-cluster) → Ethernet / WAN

Need cache coherence? ├── Yes → CXL (CPU-GPU shared memory) └── No → NVLink (GPU-GPU high-speed communication)

NVIDIA ecosystem? ├── Yes → NVLink + NVSwitch └── No → Ultra Ethernet + RDMA ``

Interconnect Solution Combinations

Cluster Scale Intra-node Inter-node Typical Configuration
1-8 GPU NVLink - DGX H100
8-64 GPU NVLink InfiniBand DGX Cluster
64-512 GPU NVLink Ultra Ethernet Large-scale Cluster
512+ GPU NVLink RoCE v2 Ultra Large-scale

4 Key Techniques for Multi-GPU Communication Optimization

Technique 1: Gradient Compression

`python import torch import torch.distributed as dist

class GradientCompressor: def init(self, compression_ratio=0.1): self.ratio = compression_ratio

def compress(self, tensor):
    k = max(1, int(tensor.numel() * self.ratio))
    values, indices = torch.topk(tensor.abs().flatten(), k)
    signs = torch.sign(tensor.flatten()[indices])
    return signs * values, indices

def decompress(self, compressed, shape):
    signs_values, indices = compressed
    decompressed = torch.zeros(shape, device=signs_values.device, dtype=signs_values.dtype)
    decompressed.flatten().scatter_(0, indices, signs_values)
    return decompressed

class CompressedAllReduce: def init(self, compression_ratio=0.1): self.compressor = GradientCompressor(compression_ratio)

def all_reduce(self, tensor):
    compressed = self.compressor.compress(tensor)
    
    signs_values, indices = compressed
    
    dist.all_reduce(signs_values, op=dist.ReduceOp.SUM)
    dist.all_reduce(indices, op=dist.ReduceOp.SUM)
    
    signs_values /= dist.get_world_size()
    
    decompressed = self.compressor.decompress(compressed, tensor.shape)
    return decompressed

`

Compression Scheme Communication Reduction Accuracy Impact Applicable Scenario
Top-K (10%) 90% 1-3% Data Parallel
Top-K (1%) 99% 3-5% Communication-intensive
Quantization INT8 75% <1% General
Sparsification 80-95% 2-5% Gradient Sparsity

Technique 2: Communication-Computation Overlap

`python class OverlapCommunicator: def init(self, model, process_group=None): self.model = model self.process_group = process_group or dist.group.WORLD

def overlapping_allreduce(self, gradient):
    chunk_size = gradient.numel() // 2
    
    chunk1 = gradient[:chunk_size]
    chunk2 = gradient[chunk_size:]
    
    handle1 = dist.all_reduce(chunk1, async_op=True)
    
    self._compute_chunk2(chunk2)
    
    handle1.wait()
    handle2 = dist.all_reduce(chunk2, async_op=True)
    
    handle2.wait()
    
    gradient[:chunk_size] = chunk1
    gradient[chunk_size:] = chunk2

def _compute_chunk2(self, chunk):
    pass

`

Technique 3: Topology-Aware Scheduling

`python class TopologyAwareScheduler: def init(self, num_gpus=8, topology="nvlink"): self.num_gpus = num_gpus self.topology = topology self.hop_matrix = self._build_hop_matrix()

def _build_hop_matrix(self):
    if self.topology == "nvlink":
        return [[0] * self.num_gpus for _ in range(self.num_gpus)]
    elif self.topology == "nvlink_switch":
        return [[0] * self.num_gpus for _ in range(self.num_gpus)]
    elif self.topology == "pcie":
        hops = [[0] * self.num_gpus for _ in range(self.num_gpus)]
        for i in range(self.num_gpus):
            for j in range(self.num_gpus):
                if i != j:
                    same_numa = (i // 4) == (j // 4)
                    hops[i][j] = 1 if same_numa else 3
        return hops

def optimize_placement(self, model_layers, num_replicas):
    placement = {}
    for i, layer in enumerate(model_layers):
        gpu = i % self.num_gpus
        placement[layer] = gpu
    return placement

def get_optimal_ring_order(self):
    return list(range(self.num_gpus))

`

Technique 4: RDMA Passthrough

`python

NCCL RDMA Configuration

nccl_config = { "NCCL_IB_DISABLE": "0", "NCCL_IB_HCA": "mlx5_0,mlx5_1", "NCCL_NET_GDR_LEVEL": "5", "NCCL_IB_GID_INDEX": "0", "NCCL_IB_TC": "106", "NCCL_IB_TIMEOUT": "22", "NCCL_IB_RETRY_CNT": "7", "NCCL_IB_SL": "0", "NCCL_PROTOCOL": "Simple", "NCCL_ALGO": "Ring,Tree", "NCCL_MAX_NRINGS": "8", "NCCL_MIN_NRINGS": "4", "NCCL_BUFFSIZE": "8388608", "NCCL_NSOCKS_PERTHREAD": "4", "NCCL_SOCKET_NTHREADS": "4", } `

Combined Effect of 4 Techniques

Optimization Method Communication Time Reduction Training Speedup Accuracy Impact
Gradient Compression 70-90% 1.5-2× 1-5%
Communication-Computation Overlap 30-50% 1.3-1.5× None
Topology-Aware 20-40% 1.2-1.3× None
RDMA Passthrough 10-20% 1.1-1.2× None
Combined Optimization 85-95% 2.5-3.5× 1-3%

NCCL Configuration and Tuning in Practice

NCCL Environment Variable Configuration

`ash #!/bin/bash

NCCL Production-grade Configuration

export NCCL_DEBUG=INFO export NCCL_DEBUG_SUBSYS=ALL

export NCCL_IB_DISABLE=0 export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 export NCCL_NET_GDR_LEVEL=5 export NCCL_IB_GID_INDEX=0

export NCCL_SOCKET_IFNAME=eth0

export NCCL_ALGO=Ring,Tree export NCCL_PROTOCOL=Simple

export NCCL_MAX_NRINGS=8 export NCCL_MIN_NRINGS=4 export NCCL_BUFFSIZE=8388608

export NCCL_NSOCKS_PERTHREAD=4 export NCCL_SOCKET_NTHREADS=4

export NCCL_IB_TC=106 export NCCL_IB_SL=0 export NCCL_IB_TIMEOUT=22 export NCCL_IB_RETRY_CNT=7

export NCCL_P2P_DISABLE=0 export NCCL_SHM_DISABLE=0 export NCCL_P2P_LEVEL=5 `

NCCL Performance Testing

`ash

NCCL Bandwidth Test

python -m torch.distributed.launch --nproc_per_node=8
-m nccl_tests.all_reduce_perf -b 8 -e 256M -f 2 -g 8

NCCL Latency Test

python -m torch.distributed.launch --nproc_per_node=8
-m nccl_tests.all_reduce_latency -b 8 -e 4M -f 2 -g 8 `

NCCL Tuning Parameters

Parameter Default Recommended Description
NCCL_MAX_NRINGS 4 8 Parallel ring count
NCCL_BUFFSIZE 4MB 8MB Communication buffer
NCCL_ALGO Auto Ring,Tree Communication algorithm
NCCL_PROTOCOL Simple Simple Communication protocol
NCCL_NSOCKS_PERTHREAD 1 4 Sockets per thread
NCCL_SOCKET_NTHREADS 1 4 Socket thread count

Summary and Further Reading

Key Takeaways

  1. Interconnect is the lifeline of AI clusters: 75%+ of large model training time is spent on communication, and interconnect bandwidth determines efficiency
  2. NVLink 5 is the most powerful interconnect: 1.8TB/s bidirectional bandwidth, 50ns latency, 14× over PCIe 6.0
  3. 3 paths each have their scenarios: NVLink for intra-node, Ultra Ethernet for inter-node, CXL for cache coherence
  4. 4 key communication optimization techniques: Gradient compression + communication overlap + topology awareness + RDMA, combined 2.5-3.5× speedup

Interconnect Solution Recommendations

Cluster Scale Intra-node Interconnect Inter-node Interconnect Communication Optimization
1-8 GPU NVLink 4 - NCCL Tuning
8-64 GPU NVLink 4 IB NDR All 4 Techniques
64+ GPU NVLink 5 Ultra Ethernet 4 Techniques + Topology

Need to quickly generate cURL commands to test GPU cluster APIs? Try our cURL to Code Tool and JSON Formatter.

Further Reading

Try these browser-local tools — no sign-up required →

#AI芯片互联#NVLink#NVSwitch#高速互联#GPU集群通信#2026