RISC-V AI Chip Ecosystem: Open-Source Instruction Set for AI Inference

Summary

• RISC-V is the "Linux moment" for AI chips: open-source instruction set breaks ARM/x86 monopoly, with AI inference chip market share exceeding 15% in 2026
• RISC-V Vector Extension (RVV 1.0) is the core of AI inference: variable-length vector instructions processing up to 2048 bits of data per instruction
• Open-source toolchains (GCC/LLVM) now maturely support RVV 1.0, significantly reducing model migration costs
• Domestic RISC-V AI chips: StarFive JH8110, VeriSilicon NPU, and T-Head Yeying 1520 form a tripartite landscape
• This article provides a complete solution from RVV programming to RISC-V AI inference deployment

---

Table of Contents

• RISC-V: The Linux Moment for AI Chips
• RVV 1.0 Vector Extension: The Engine for AI Inference
• Open-Source Toolchains and Model Migration
• Domestic RISC-V AI Chips Comparison
• RISC-V AI Inference Deployment in Practice
• Summary and Further Reading

---

RISC-V: The Linux Moment for AI Chips

Why AI Chips Need RISC-V

Dimension	ARM	x86	RISC-V
Instruction Set	Closed (licensing fees)	Closed (Intel/AMD)	Open-source (free)
Customization Freedom	Low (requires license)	Very Low	High (freely extensible)
AI Extensions	SVE/SVE2	AVX-512/VNNI	RVV 1.0 (customizable)
Licensing Cost	$5M-50M/year	N/A	$0
Ecosystem Maturity	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐
Domestic Substitution	⚠️ Restricted	❌ Restricted	✅ Self-controllable

RISC-V AI Chip Market Size

Year	Market Size	AI Inference Share	Representative Products
2024	$3.2B	8%	StarFive JH7110
2025	$5.8B	12%	T-Head Yeying 1520
2026	$9.5B	15%	VeriSilicon VIP9400
2028 (projected)	$20B+	25%+	Multiple vendors

Reference: RISC-V International

---

RVV 1.0 Vector Extension: The Engine for AI Inference

RVV 1.0 Core Features

Feature	Description	AI Inference Value
Variable Vector Length (VLEN)	128-2048 bits configurable	Flexible adaptation to different compute requirements
Variable Element Width (SEW)	8/16/32/64 bits	Supports INT8/FP16/FP32 mixed precision
Masked Operations	Conditional execution	Flexible handling of irregular data
Vector Reduction	Sum/Max/Min	Core operation for matrix multiplication
Permutation Instructions	Vector rearrangement	Data preprocessing

RVV Vector Programming Example

#include <riscv_vector.h>

void vector_add(float *dst, const float *src1, const float *src2, size_t n) {
    size_t vl;
    size_t i = 0;
    while (i < n) {
        vl = vsetvl_e32m1(n - i);
        vfloat32m1_t v1 = vle32_v_f32m1(src1 + i, vl);
        vfloat32m1_t v2 = vle32_v_f32m1(src2 + i, vl);
        vfloat32m1_t v3 = vfadd_vv_f32m1(v1, v2, vl);
        vse32_v_f32m1(dst + i, v3, vl);
        i += vl;
    }
}

void matrix_vector_mul(float *dst, const float *matrix, const float *vec,
                        size_t rows, size_t cols) {
    for (size_t i = 0; i < rows; i++) {
        size_t vl;
        size_t j = 0;
        vfloat32m1_t sum = vfmv_v_f_f32m1(0.0f, 1);
        while (j < cols) {
            vl = vsetvl_e32m1(cols - j);
            vfloat32m1_t m_row = vle32_v_f32m1(matrix + i * cols + j, vl);
            vfloat32m1_t v_col = vle32_v_f32m1(vec + j, vl);
            sum = vfmacc_vv_f32m1(sum, m_row, v_col, vl);
            j += vl;
        }
        dst[i] = vfmv_f_s_f32m1_f32(sum);
    }
}

RVV vs ARM SVE vs x86 AVX-512

Dimension	RVV 1.0	ARM SVE2	x86 AVX-512
Vector Length	Variable (128-2048 bits)	Variable (128-2048 bits)	Fixed 512 bits
Element Width	8/16/32/64 bits	8/16/32/64 bits	8/16/32/64 bits
Masking	✅	✅	✅
Reduction	✅	✅	✅
Custom Extensions	✅ Free	❌ Requires ARM approval	❌ Requires Intel
INT8 Matrix Multiply	✅ (custom)	✅ (outer product)	✅ (VNNI)

---

Open-Source Toolchains and Model Migration

RISC-V AI Toolchain

┌──────────────────────────────────────────────────────────────┐
│              RISC-V AI Inference Toolchain                      │
│                                                                │
│  ┌──────────────────────────────────────────────────────┐    │
│  │              Model Layer                               │    │
│  │  PyTorch/ONNX/TFLite → Export ONNX/FlatBuffer        │    │
│  └────────────────────────┬─────────────────────────────┘    │
│                           │                                   │
│  ┌────────────────────────▼─────────────────────────────┐    │
│  │              Compiler Layer                            │    │
│  │  TVM/MLIR-LLVM → RISC-V RVV code generation          │    │
│  │  Apache TVM (recommended) / MLIR-Affine / LLVM direct │    │
│  └────────────────────────┬─────────────────────────────┘    │
│                           │                                   │
│  ┌────────────────────────▼─────────────────────────────┐    │
│  │              Runtime Layer                             │    │
│  │  ONNX Runtime (RISC-V backend) / TFLite Runtime       │    │
│  └────────────────────────┬─────────────────────────────┘    │
│                           │                                   │
│  ┌────────────────────────▼─────────────────────────────┐    │
│  │              Hardware Layer                            │    │
│  │  RISC-V CPU + RVV 1.0 + Custom NPU coprocessor       │    │
│  └──────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────┘

Compiling ONNX Models to RISC-V with TVM

import tvm
from tvm import relay
import onnx

model = onnx.load("qwen1.5-0.5b-int8.onnx")

mod, params = relay.frontend.from_onnx(model)

target = tvm.target.Target("llvm -mtriple=riscv64-unknown-linux-gnu -mattr=+v")

with tvm.transform.PassContext(opt_level=3):
    lib = tvm.relay.build(mod, target=target, params=params)

lib.export_library("qwen_rvv.tar")

---

Domestic RISC-V AI Chips Comparison

Chip	Vendor	Core Architecture	AI Compute	Memory	Power	Use Case
JH8110	StarFive	4×U74+1×S7	2 TOPS	8GB	5W	Edge Gateway
Yeying 1520	T-Head	8×C920+4×NPU	4 TOPS	16GB	10W	Edge Inference
VIP9400	VeriSilicon	4×RV64+VIP-NNA	8 TOPS	16GB	15W	Surveillance/Industrial
SG2042	Sophgo	64×C920	16 TOPS	128GB	200W	Server

RISC-V AI Inference Performance

Model	Chip	Quantization	Latency	Throughput	Power
Qwen2.5-0.5B	JH8110	INT8	350ms	3 tok/s	4W
Qwen2.5-0.5B	Yeying 1520	INT8	180ms	6 tok/s	8W
Qwen2.5-1.8B	Yeying 1520	INT4	250ms	5 tok/s	9W
ResNet-50	VIP9400	INT8	12ms	80fps	12W

---

RISC-V AI Inference Deployment in Practice

Cross-Compilation and Deployment

FROM riscv64/ubuntu:22.04

RUN apt-get update && apt-get install -y \
    python3 python3-pip \
    libopenblas-dev \
    && rm -rf /var/lib/apt/lists/*

RUN pip3 install --no-cache-dir \
    onnxruntime==1.18.0 \
    numpy

COPY qwen_rvv.tar /app/
COPY inference.py /app/

WORKDIR /app

CMD ["python3", "inference.py"]

RISC-V Inference Script

import numpy as np
import onnxruntime as ort

class RiscVInferencer:
    def __init__(self, model_path: str):
        sess_options = ort.SessionOptions()
        sess_options.intra_op_num_threads = 4
        sess_options.inter_op_num_threads = 1

        self.session = ort.InferenceSession(
            model_path,
            sess_options=sess_options,
            providers=["CPUExecutionProvider"],
        )

        self.input_name = self.session.get_inputs()[0].name
        self.output_names = [o.name for o in self.session.get_outputs()]

    def infer(self, input_ids: np.ndarray) -> np.ndarray:
        outputs = self.session.run(self.output_names, {self.input_name: input_ids})
        return outputs[0]

---

Summary and Further Reading

RISC-V is the "Linux moment" for AI chips — the open-source instruction set has broken the ARM/x86 monopoly, and domestic chips face a historic opportunity for self-controllability. The RVV 1.0 vector extension is the core engine for AI inference, and open-source toolchains now support TVM compilation and deployment.

Key Development Takeaways:

1. RISC-V is open-source and free, with licensing cost of $0 vs ARM $5M-50M/year
2. RVV 1.0 variable vector length ensures flexibility for AI inference
3. TVM is the preferred tool for RISC-V AI model compilation
4. Domestic chips: StarFive (edge gateway), T-Head (edge inference), VeriSilicon (surveillance/industrial)
5. RISC-V AI inference performance now meets edge scenario requirements

Related Reading:

• AI Chip Inference Deployment in Practice — GPU/NPU/edge chip comparison
• WASM Component Model Cross-Language Microservices — RISC-V + WASM edge inference
• LLM Inference Acceleration Benchmark — Inference engine selection

Authoritative References:

• RISC-V Vector Extension Specification
• Apache TVM