Vector Search at Scale: Why One Size Doesn't Fit All | S2 E13

How AI Is Built

Ever wondered why your vector search becomes painfully slow after scaling past a million vectors? You're not alone - even tech giants struggle with this.

Charles Xie, founder of Zilliz (company behind Milvus), shares how they solved vector database scaling challenges at 100B+ vector scale:

Key Insights:

• Multi-tier storage strategy:
  • GPU memory (1% of data, fastest)
  • RAM (10% of data)
  • Local SSD
  • Object storage (slowest but cheapest)
• Real-time search solution:
  • New data goes to buffer (searchable immediately)
  • Index builds in background when buffer fills
  • Combines buffer & main index results
• Performance optimization:
  • GPU acceleration for 10k-50k queries/second
  • Customizable trade-offs between:
    • Cost
    • Latency
    • Search relevance
• Future developments:
  • Self-learning indices
  • Hybrid search methods (dense + sparse)
  • Graph embedding support
  • Colbert integration

Perfect for teams hitting scaling walls with their current vector search implementation or planning for future growth.

Worth watching if you're building production search systems or need to optimize costs vs performance.

Charles Xie:

• LinkedIn
• Zilliz
• Milvus
• Milvus Discord

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

00:00 Introduction to Search System Challenges 00:26 Introducing Milvus: The Open Source Vector Database 00:58 Interview with Charles: Founder of Zilliz 02:20 Scalability and Performance in Vector Databases 03:35 Challenges in Distributed Systems 05:46 Data Consistency and Real-Time Search 12:12 Hierarchical Storage and GPU Acceleration 18:34 Emerging Technologies in Vector Search 23:21 Self-Learning Indexes and Future Innovations 28:44 Key Takeaways and Conclusion