AI Search Engine Rebuild: Semantic Search and LLM-Augmented Retrieval

Summary

AI search is disrupting traditional search: from "keyword matching" to "semantic understanding + intelligent generation," with the AI search market exceeding $5B in 2026
Semantic search 3-layer architecture: Query understanding → Vector retrieval → LLM reranking + generation, improving retrieval precision by 35%+
Vector retrieval engine selection: Milvus for large-scale, Qdrant for small-to-medium scale, Elasticsearch 8.x for hybrid search
LLM-augmented ranking (Reranking) is the killer feature of AI search: Cross-Encoder reranking can improve NDCG by 15%-25%
This article provides a complete rebuild solution from traditional search to AI search, including Elasticsearch + Milvus dual-engine architecture

AI Search: The Next-Generation Search Paradigm
Semantic Search 3-Layer Architecture
Vector Retrieval Engine Selection and Deployment
LLM-Augmented Ranking: Reranking
AI Search Production Deployment: ES + Milvus Dual Engine
Summary and Further Reading

AI Search: The Next-Generation Search Paradigm

Traditional Search vs AI Search

Dimension	Traditional Search (BM25)	AI Search (Semantic + LLM)
Matching Method	Exact keyword matching	Semantic similarity matching
Query Understanding	Tokenization + synonyms	Intent recognition + entity extraction
Ranking Signals	TF-IDF + PageRank	Semantic relevance + user intent
Result Presentation	10 blue links	Direct answers + cited sources
Complex Queries	Poor (requires exact keywords)	Strong (understands natural language)
Real-time Performance	High (pre-built index)	Medium (requires real-time inference)

AI Search Market Landscape

Product	Type	Core Technology	Feature
Perplexity	AI-native search	RAG + LLM generation	Cited sources
Google SGE	Traditional + AI-enhanced	BERT + MUM	Strongest ecosystem
Bing Chat	Traditional + AI-enhanced	GPT-4 + Prometheus	Microsoft ecosystem
Metaso AI Search	AI-native search	Self-developed RAG	Chinese-optimized
Quark AI Search	Traditional + AI-enhanced	Self-developed model	Mobile-first

Semantic Search 3-Layer Architecture

┌──────────────────────────────────────────────────────────────┐
│              Semantic Search 3-Layer Architecture               │
│                                                                │
│  Layer 1: Query Understanding                                 │
│  ┌──────────────────────────────────────────────────────┐    │
│  │ User query → Intent recognition → Entity extraction  │    │
│  │ → Query rewriting                                    │    │
│  │ "How to config GPU in K8s" → Intent: tutorial →     │    │
│  │ Entities: K8s, GPU                                   │    │
│  │ → Rewritten: "Kubernetes GPU scheduling              │    │
│  │ configuration tutorial"                              │    │
│  └──────────────────────────────────────────────────────┘    │
│                         ↓                                     │
│  Layer 2: Multi-path Retrieval                                │
│  ┌──────────────────────────────────────────────────────┐    │
│  │ Vector retrieval (Milvus) + Keyword retrieval (ES)   │    │
│  │ + Knowledge graph (Neo4j)                            │    │
│  │ Merge Top-K results from each path → Deduplicate →   │    │
│  │ Candidate set                                        │    │
│  └──────────────────────────────────────────────────────┘    │
│                         ↓                                     │
│  Layer 3: LLM Reranking + Generation                         │
│  ┌──────────────────────────────────────────────────────┐    │
│  │ Cross-Encoder reranking → Top-N → LLM generates     │    │
│  │ answer + cited sources                               │    │
│  └──────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────┘

Query Understanding Implementation

class QueryUnderstanding:
    def __init__(self, llm_client, embedding_model):
        self.llm = llm_client
        self.embedding = embedding_model

    async def understand(self, query: str) -> dict:
        intent = await self._classify_intent(query)
        entities = await self._extract_entities(query)
        rewritten = await self._rewrite_query(query, intent, entities)
        query_embedding = self._embed(rewritten)

        return {
            "original_query": query,
            "intent": intent,
            "entities": entities,
            "rewritten_query": rewritten,
            "embedding": query_embedding,
        }

    async def _classify_intent(self, query: str) -> str:
        response = self.llm.chat.completions.create(
            model="Qwen/Qwen2.5-7B-Instruct",
            messages=[{
                "role": "user",
                "content": f"Classify the intent of the following search query, output only the category name:\nQuery: {query}\nCategories: Tutorial/Troubleshooting/Product Comparison/Concept Explanation/Latest News"
            }],
            temperature=0.0, max_tokens=10,
        )
        return response.choices[0].message.content.strip()

    async def _rewrite_query(self, query: str, intent: str, entities: list) -> str:
        response = self.llm.chat.completions.create(
            model="Qwen/Qwen2.5-7B-Instruct",
            messages=[{
                "role": "user",
                "content": f"Rewrite the following search query into a form more suitable for semantic retrieval, output only the rewritten result:\nOriginal query: {query}\nIntent: {intent}\nEntities: {entities}"
            }],
            temperature=0.0, max_tokens=100,
        )
        return response.choices[0].message.content.strip()

Vector Retrieval Engine Selection and Deployment

Engine Comparison

Dimension	Milvus 2.5	Qdrant 1.13	ES 8.x (kNN)
Vector Index	HNSW/IVF/DiskANN	HNSW	HNSW
Hybrid Search	✅ (BM25 + vector)	✅ (sparse + dense)	✅ (native BM25 + kNN)
Scale	100M+	<50M	100M+
Latency (P50)	5ms	3ms	8ms
Ops Complexity	High	Low	Medium
Ecosystem	AI/ML ecosystem	Rust ecosystem	Enterprise search ecosystem

ES 8.x kNN Search

from elasticsearch import Elasticsearch

es = Elasticsearch("http://localhost:9200")

mapping = {
    "mappings": {
        "properties": {
            "title": {"type": "text", "analyzer": "ik_max_word"},
            "content": {"type": "text", "analyzer": "ik_max_word"},
            "embedding": {
                "type": "dense_vector",
                "dims": 1536,
                "index": True,
                "similarity": "cosine",
                "index_options": {"type": "hnsw", "m": 32, "ef_construction": 256},
            },
            "source": {"type": "keyword"},
            "published_at": {"type": "date"},
        }
    }
}

es.indices.create(index="knowledge_base", body=mapping)

def hybrid_search(query: str, query_embedding: list, top_k: int = 20):
    result = es.search(
        index="knowledge_base",
        body={
            "size": top_k,
            "query": {
                "bool": {
                    "should": [
                        {
                            "script_score": {
                                "query": {"match_all": {}},
                                "script": {
                                    "source": "cosineSimilarity(params.query_vector, 'embedding') + 1.0",
                                    "params": {"query_vector": query_embedding},
                                },
                            }
                        },
                        {
                            "multi_match": {
                                "query": query,
                                "fields": ["title^3", "content"],
                                "type": "best_fields",
                            }
                        },
                    ]
                }
            },
        },
    )
    return [hit["_source"] for hit in result["hits"]["hits"]]

LLM-Augmented Ranking: Reranking

Cross-Encoder Reranking

from sentence_transformers import CrossEncoder

class SemanticReranker:
    def __init__(self, model_name: str = "BAAI/bge-reranker-v2-m3"):
        self.model = CrossEncoder(model_name, max_length=512)

    def rerank(self, query: str, documents: list[dict], top_k: int = 5) -> list[dict]:
        pairs = [(query, doc["content"][:512]) for doc in documents]
        scores = self.model.predict(pairs)

        for doc, score in zip(documents, scores):
            doc["rerank_score"] = float(score)

        documents.sort(key=lambda x: x["rerank_score"], reverse=True)
        return documents[:top_k]

LLM Answer Generation

class AISearchGenerator:
    def __init__(self, llm_client):
        self.llm = llm_client

    async def generate_answer(self, query: str, contexts: list[dict]) -> dict:
        context_text = "\n\n".join(
            f"[Source {i+1}] {c['title']}\n{c['content'][:500]}"
            for i, c in enumerate(contexts)
        )

        response = self.llm.chat.completions.create(
            model="Qwen/Qwen2.5-7B-Instruct",
            messages=[{
                "role": "system",
                "content": "Answer the user's question based on the provided search results. Every fact must be annotated with a source number. If the search results are insufficient to answer, please state so."
            }, {
                "role": "user",
                "content": f"Question: {query}\n\nSearch results:\n{context_text}"
            }],
            temperature=0.3, max_tokens=1024,
        )

        answer = response.choices[0].message.content
        return {"answer": answer, "sources": contexts[:5]}

Reranking Effectiveness

Method	NDCG@10	MRR@10	Latency
Pure BM25	0.62	0.58	5ms
Pure Vector Retrieval	0.71	0.67	8ms
BM25 + Vector Hybrid	0.78	0.74	12ms
Hybrid + Cross-Encoder	0.89	0.85	35ms

AI Search Production Deployment: ES + Milvus Dual Engine

Deployment Architecture

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-search-api
  namespace: ai-search
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ai-search-api
  template:
    spec:
      containers:
        - name: search-api
          image: myregistry/ai-search-api:v1.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "2"
              memory: 4Gi
            limits:
              cpu: "4"
              memory: 8Gi
          env:
            - name: ELASTICSEARCH_URL
              value: "http://elasticsearch:9200"
            - name: MILVUS_URI
              value: "http://milvus-svc:19530"
            - name: LLM_URL
              value: "http://vllm:8000/v1"
            - name: RERANKER_MODEL
              value: "BAAI/bge-reranker-v2-m3"

Search Performance Benchmarks

Metric	Traditional Search	AI Search
Retrieval Latency (P50)	5ms	35ms
Retrieval Precision (NDCG@10)	0.62	0.89
Answer Generation Latency	N/A	800ms
User Satisfaction	3.2/5	4.5/5
Zero-result Rate	12%	2%

Summary and Further Reading

AI search is evolving from "keyword matching" to "semantic understanding + intelligent generation." The 3-layer architecture (Query understanding → Multi-path retrieval → LLM reranking + generation) improves retrieval precision from 0.62 to 0.89. The ES + Milvus dual engine is the standard for production deployment.

Key Rebuild Takeaways:

AI search 3-layer architecture: Query understanding → Multi-path retrieval → LLM reranking + generation
Query rewriting is the first step in semantic search, converting colloquial queries into retrieval-friendly forms
ES 8.x kNN supports hybrid search, suitable for upgrading traditional search to AI search
Cross-Encoder Reranking can improve NDCG by 15%-25%
AI search latency is 35ms (retrieval) + 800ms (generation), requiring trade-offs between latency and precision

Related Reading:

Vector Database Production Tuning in Practice — AI search vector retrieval layer tuning
GraphRAG in Practice: Knowledge Graph-Augmented RAG — Knowledge graph-enhanced search
LLM Inference Acceleration Benchmark — AI search inference backend optimization

Authoritative References: