#030 Vector Search at Scale, Why One Size Doesn't Fit All

How AI Is Built

#Tech #Nicolay Gerold #Technology #Llm #Machine Learning #Data Engineering

30:55

A secret field that summons lightning. A massive spiral that disappears into a salt lake. A celestial observatory carved into a volcano. Meet the wild—and sometimes explosive—world of land art, where artists craft masterpieces with dynamite and bulldozers. In our Season 2 premiere, guest Dylan Thuras, cofounder of Atlas Obscura, takes us off road and into the minds of the artists who literally reshaped parts of the Southwest. These works aren’t meant to be easy to reach—or to explain—but they just might change how you see the world. Land art you’ll visit in this episode: - Double Negative and City by Michael Heizer (Garden Valley, Nevada) - Spiral Jetty by Robert Smithson (Great Salt Lake, Utah) - Sun Tunnels by Nancy Holt (Great Basin Desert, Utah) - Lightning Field by Walter De Maria (Catron County, New Mexico) - Roden Crater by James Turrell (Painted Desert, Arizona) Via Podcast is a production of AAA Mountain West Group.…

about a year ago 36:26

MP3•Episode home

Ever wondered why your vector search becomes painfully slow after scaling past a million vectors? You're not alone - even tech giants struggle with this.

Charles Xie, founder of Zilliz (company behind Milvus), shares how they solved vector database scaling challenges at 100B+ vector scale:

Key Insights:

Multi-tier storage strategy:
- GPU memory (1% of data, fastest)
- RAM (10% of data)
- Local SSD
- Object storage (slowest but cheapest)
Real-time search solution:
- New data goes to buffer (searchable immediately)
- Index builds in background when buffer fills
- Combines buffer & main index results
Performance optimization:
- GPU acceleration for 10k-50k queries/second
- Customizable trade-offs between:
  - Cost
  - Latency
  - Search relevance
Future developments:
- Self-learning indices
- Hybrid search methods (dense + sparse)
- Graph embedding support
- Colbert integration

Perfect for teams hitting scaling walls with their current vector search implementation or planning for future growth.

Worth watching if you're building production search systems or need to optimize costs vs performance.

Charles Xie:

Nicolay Gerold:

00:00 Introduction to Search System Challenges 00:26 Introducing Milvus: The Open Source Vector Database 00:58 Interview with Charles: Founder of Zilliz 02:20 Scalability and Performance in Vector Databases 03:35 Challenges in Distributed Systems 05:46 Data Consistency and Real-Time Search 12:12 Hierarchical Storage and GPU Acceleration 18:34 Emerging Technologies in Vector Search 23:21 Self-Learning Indexes and Future Innovations 28:44 Key Takeaways and Conclusion

53 episodes

#030 Vector Search at Scale, Why One Size Doesn't Fit All

How AI Is Built

published about a year ago

MP3•Episode home

Ever wondered why your vector search becomes painfully slow after scaling past a million vectors? You're not alone - even tech giants struggle with this.

Charles Xie, founder of Zilliz (company behind Milvus), shares how they solved vector database scaling challenges at 100B+ vector scale:

Key Insights:

Multi-tier storage strategy:
- GPU memory (1% of data, fastest)
- RAM (10% of data)
- Local SSD
- Object storage (slowest but cheapest)
Real-time search solution:
- New data goes to buffer (searchable immediately)
- Index builds in background when buffer fills
- Combines buffer & main index results
Performance optimization:
- GPU acceleration for 10k-50k queries/second
- Customizable trade-offs between:
  - Cost
  - Latency
  - Search relevance
Future developments:
- Self-learning indices
- Hybrid search methods (dense + sparse)
- Graph embedding support
- Colbert integration

Perfect for teams hitting scaling walls with their current vector search implementation or planning for future growth.

Worth watching if you're building production search systems or need to optimize costs vs performance.

Charles Xie:

Nicolay Gerold:

53 episodes

#Tech #Nicolay Gerold #Technology #Llm #Machine Learning #Data Engineering

All episodes

How AI Is Built

1
#048 TAKEAWAYS Why Your AI Agents Need Permission to Act, Not Just Read 7:07

5 days ago7:07

7:07

Nicolay here, most AI conversations obsess over capabilities. This one focuses on constraints - the right ones that make AI actually useful rather than just impressive demos. Today I have the chance to talk to Dexter Horthy, who recently put out a long piece called the “12-factor agents”. It’s like the 10 commandments, but for building agents. One of it is “Contact human with tool calls”: the LLM can call humans for high-stakes decisions or “writes”. The key insight is brutally simple. AI can get to 90% accuracy on most tasks - good enough for spam-like activities but disastrous for anything that requires trust. The solution isn't to wait for models to get smarter; it's to add a human approval layer for critical actions. Imagine you are writing to a database or sending an email. Each “write” has to be approved by a human. So you post the email in a Slack channel and in most cases, your sales people will approve. In the 10%, it’s stopped in its tracks and the human can take over. You stop the slop and get good training data in the mean time. Dexter’s company is building exactly this: an approval mechanism that lets AI agents send requests to humans before executing. In the podcast, we also touch on a bunch of other things: MCP and that they are (atm) just a thin client Are we training LLMs toward mediocrity? What infrastructure do we need for human in the loop (e.g. DBOS)? and more 💡 Core Concepts Context Engineering : Crafting the information representation for LLMs - selecting optimal data structures, metadata, and formats to ensure models receive precisely what they need to perform effectively. Token Bloat Prevention : Ruthlessly eliminating irrelevant information from context windows to maintain agent focus during complex tasks, preventing the pattern of repeating failed approaches. Human-in-the-loop Approval Flows : Achieving 99% reliability through a "90% AI + 10% human oversight" framework where agents analyze data and suggest actions but request explicit permission before execution. Rubric Engineering : Systematically evaluating AI outputs through dimension-specific scoring criteria to provide precise feedback and identify exceptional results, helping escape the trap of models converging toward mediocrity. 📶 Connect with Dexter: LinkedIn X / Twitter Company 📶 Connect with Nicolay: LinkedIn X / Twitter Bluesky Website My Agency Aisbach (for ai implementations / strategy) ⏱️ Important Moments MCP Servers as Clients : [03:07] Dexter explains why what many call "MCP servers" actually function more like clients when examining the underlying code. Authentication Challenges : [04:45] The discussion shifts to how authentication should be handled in MCP implementations and whether it belongs in the protocol. Asynchronous Agent Execution : [08:18] Exploring how to handle agents that need to pause for human input without wasting tokens on continuous polling. Token Bloat Prevention : [14:41] Strategies for keeping context windows focused and efficient, moving beyond standard chat formats. Context Engineering : [29:06] The concept that everything in AI agent development ultimately comes down to effective context engineering. Fine-tuning vs. RAG for Writing Style : [20:05] Contrasting personal writing style fine-tuning versus context window examples. Generating Options vs. Deterministic Outputs : [19:44] The unexplored potential of having AI generate diverse creative options for human selection. The "Mediocrity Convergence" Question : [37:11] The philosophical concern that popular LLMs may inevitably trend toward average quality. Data Labeling Interfaces : [35:25] Discussion about the need for better, lower-friction interfaces to collect human feedback on AI outputs. Human-in-the-loop Approval Flows : [42:46] The core approach of HumanLayer, allowing agents to ask permission before taking action. 🛠️ Tools & Tech Mentioned MCP OpenControl DBOS Temporal Cursor 📚 Recommended Resources 12 Factor Agents BAML Docs Rubric Engineering 🔮 What's Next Next week, we will continue going more into getting generative AI into production talking to Vibhav from BAML. 💬 Join The Conversation Follow How AI Is Built on YouTube , Bluesky , or Spotify . If you have any suggestions for future guests, feel free to leave it in the comments or write me (Nicolay) directly on LinkedIn , X , or Bluesky . Or at nicolay.gerold@gmail.com . I will be opening a Discord soon to get you guys more involved in the episodes! Stay tuned for that. ♻️ I am trying to build the new platform for engineers to share their experience that they have earned after building and deploying stuff into production. I am trying to produce the best content possible - informative, actionable, and engaging. I'm asking for two things: hit subscribe now to show me what content you like (so I can do more of it), and if this episode helped you, pay it forward by sharing with one engineer who's facing similar challenges. That's the agreement - I deliver practical value, you help grow this resource for everyone. ♻️…

How AI Is Built

1
#048 Why Your AI Agents Need Permission to Act, Not Just Read 57:03

12 days ago57:03

57:03

How AI Is Built

1
#047 Architecting Information for Search, Humans, and Artificial Intelligence 57:22

7 weeks ago57:22

57:22

Today on How AI Is Built , Nicolay Gerold sits down with Jorge Arango, an expert in information architecture. Jorge emphasizes that aligning systems with users' mental models is more important than optimizing backend logic alone. He shares a clear framework with four practical steps: Key Points: Information architecture should bridge user mental models with system data models Information's purpose is to help people make better choices and act more skillfully Well-designed systems create learnable (not just "intuitive") interfaces Context and domain boundaries significantly impact user understanding Progressive disclosure helps accommodate users with varying expertise levels Chapters 00:00 Introduction to Backend Systems 00:36 Guest Introduction: Jorge Arango 01:12 Podcast Dynamics and Guest Experiences 01:53 Timeless Principles in Technology 02:08 Interesting Conversations and Learnings 04:04 Physical vs. Digital Organization 04:21 Smart Defaults and System Maintenance 07:20 Data Models and Conceptual Structures 08:53 Designing User-Centric Systems 10:20 Challenges in Information Systems 10:35 Understanding Information and Choices 15:49 Clarity and Context in Design 26:36 Progressive Disclosure and User Research 37:05 The Role of Large Language Models 54:59 Future Directions and New Series (MLOps) Information Architecture Fundamentals What Is Information? Information helps people make better choices to act more skillfully Example: "No dog pooping" signs help predict consequences of actions Poor information systems fail to provide relevant guidance for users' needs Mental Models vs. Data Models Systems have underlying conceptual structures that should reflect user mental models Data models make these conceptual models "normative" in the infrastructure Designers serve as translators between user needs and technical implementation Goal: Users should think "the person who designed this really gets me" Design Strategies for Complex Systems Progressive Disclosure Present simple interfaces by default with clear paths to advanced functionality Example: HyperCard - visual interface for beginners with programming layer for experts Allows both novice and expert users to use the same system effectively Context Setting and Domain Boundaries All interactions happen within a context that influences understanding Words acquire different meanings in different contexts (e.g., "save" in computing vs. banking) Clearer domain boundaries make information architecture design easier Hardest systems to design: those serving many purposes for diverse audiences Conceptual Modeling (Underrated Practice) Should precede UI sketching but often skipped by designers Defines concepts needed in the system and their relationships Creates more cohesive and coherent systems, especially for complex projects More valuable than sitemaps, which imply rigid hierarchies LLMs and Information Architecture Current and Future Applications Transforming search experiences (e.g., Perplexity providing answers vs. link lists) Improving intent parsing in traditional search Helping information architects with content analysis and navigation structure design Enabling faster, better analysis of large content repositories Implementation Advice For Engineers and Designers Designers should understand how systems are built (materials of construction) Engineers benefit from understanding user perspectives and mental models Both disciplines have much to teach each other For Complex Applications Map conceptual models before writing code Test naming with real users Implement progressive disclosure with good defaults Remember: "If the user can't find it, it doesn't exist" Notable Quotes: "People only understand things relative to things they already understand." - Richard Saul Wurman "The hardest systems to design are the ones that are meant to do a lot of things for a lot of different people." - Jorge Arango "Very few things are intuitive. There's a long running joke in the industry that the only intuitive interface for humans is the nipple. Everything else is learned." - Jorge Arango Jorge Arango LinkedIn Website X (Twitter) Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter)…

How AI Is Built

1
#046 Building a Search Database From First Principles 53:29

9 weeks ago53:29

53:29

Modern search is broken. There are too many pieces that are glued together. Vector databases for semantic search Text engines for keywords Rerankers to fix the results LLMs to understand queries Metadata filters for precision Each piece works well alone. Together, they often become a mess. When you glue these systems together, you create: Data Consistency Gaps Your vector store knows about documents your text engine doesn't. Which is right? Timing Mismatches New content appears in one system before another. Users see different results depending on which path their query takes. Complexity Explosion Every new component doubles your integration points. Three components means three connections. Five means ten. Performance Bottlenecks Each hop between systems adds latency. A 200ms search becomes 800ms after passing through four components. Brittle Chains When one system fails, your entire search breaks. More pieces mean more breaking points. I recently built a system where we had query specific post-filters but the requirement to deliver a fixed number of results to the user. A lot of times, the query had to be run multiple times to achieve the desired amount. So we had an unpredictable latency. A high load on the backend, where some queries hammered the database 10+ times. A relevance cliff, where results 1-6 look great, but the later ones were poor matches. Today on How AI Is Built, we are talking to Marek Galovic from TopK. We talk about how they built a new search database with modern components. "How would search work if we built it today?” Cloud storage is cheap. Compute is fast. Memory is plentiful. One system that handles vectors, text, and filters together - not three systems duct-taped into one. One pass handles everything: Vector search + Text search + Filters → Single sorted result Built with hand-optimized Rust kernels for both x86 and ARM, the system scales to 100M documents with 200ms P99 latency. The goal is to do search in 5 lines of code. Marek Galovic: LinkedIn Website TopK Website TopK Docs Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter) 00:00 Introduction to TopK and Snowflake Comparison 00:35 Architectural Patterns and Custom Formats 01:30 Query Execution Engine Explained 02:56 Distributed Systems and Rust 04:12 Query Execution Process 06:56 Custom File Formats for Search 11:45 Handling Distributed Queries 16:28 Consistency Models and Use Cases 26:47 Exploring Database Versioning and Snapshots 27:27 Performance Benchmarks: Rust vs. C/C++ 29:02 Scaling and Latency in Large Datasets 29:39 GPU Acceleration and Use Cases 31:04 Optimizing Search Relevance and Hybrid Search 34:39 Advanced Search Features and Custom Scoring 38:43 Future Directions and Research in AI 47:11 Takeaways for Building AI Applications…

How AI Is Built

1
#045 RAG As Two Things - Prompt Engineering and Search 1:02:44

10 weeks ago1:02:44

1:02:44

John Berryman moved from aerospace engineering to search, then to ML and LLMs. His path: Eventbrite search → GitHub code search → data science → GitHub Copilot. He was drawn to more math and ML throughout his career. RAG Explained "RAG is not a thing. RAG is two things." It breaks into: Search - finding relevant information Prompt engineering - presenting that information to the model These should be treated as separate problems to optimize. The Little Red Riding Hood Principle When prompting LLMs, stay on the path of what models have seen in training. Use formats, structures, and patterns they recognize from their training data: For code, use docstrings and proper formatting For financial data, use SEC report structures Use Markdown for better formatting Models respond better to familiar structures. Testing Prompts Testing strategies: Start with "vibe testing" - human evaluation of outputs Develop systematic tests based on observed failure patterns Use token probabilities to measure model confidence For few-shot prompts, watch for diminishing returns as examples increase Managing Token Limits When designing prompts, divide content into: Static elements (boilerplate, instructions) Dynamic elements (user inputs, context) Prioritize content by: Must-have information Nice-to-have information Optional if space allows Even with larger context windows, efficiency remains important for cost and latency. Completion vs. Chat Models Chat models are winning despite initial concerns about their constraints: Completion models allow more flexibility in document format Chat models are more reliable and aligned with common use cases Most applications now use chat models, even for completion-like tasks Applications: Workflows vs. Assistants Two main LLM application patterns: Assistants : Human-in-the-loop interactions where users guide and correct Workflows : Decomposed tasks where LLMs handle well-defined steps with safeguards Breaking Down Complex Problems Two approaches: Horizontal : Split into sequential steps with clear inputs/outputs Vertical : Divide by case type, with specialized handling for each scenario Example: For SOX compliance, break horizontally (understand control, find evidence, extract data, compile report) and vertically (different audit types). On Agents Agents exist on a spectrum from assistants to workflows, characterized by: Having some autonomy to make decisions Using tools to interact with the environment Usually requiring human oversight Best Practices For building with LLMs: Start simple: API key + Jupyter notebook Build prototypes and iterate quickly Add evaluation as you scale Keep users in the loop until models prove reliability John Berryman: LinkedIn X (Twitter) Arcturus Labs Prompt Engineering for LLMs (Book) Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter) 00:00 Introduction to RAG: Retrieval and Generation 00:19 Optimizing Retrieval Systems 01:11 Introducing John Berryman 02:31 John's Journey from Search to Prompt Engineering 04:05 Understanding RAG: Search and Prompt Engineering 05:39 The Little Red Riding Hood Principle in Prompt Engineering 14:14 Balancing Static and Dynamic Elements in Prompts 25:52 Assistants vs. Workflows: Choosing the Right Approach 30:15 Defining Agency in AI 30:35 Spectrum of Assistance and Workflows 34:35 Breaking Down Problems Horizontally and Vertically 37:57 SOX Compliance Case Study 40:56 Integrating LLMs into Existing Applications 44:37 Favorite Tools and Missing Features 46:37 Exploring Niche Technologies in AI 52:52 Key Takeaways and Future Directions…

How AI Is Built

1
#044 Graphs Aren't Just For Specialists Anymore 1:03:35

11 weeks ago1:03:35

1:03:35

Kuzu is an embedded graph database that implements Cypher as a library. It can be easily integrated into various environments—from scripts and Android apps to serverless platforms. Its design supports both ephemeral, in-memory graphs (ideal for temporary computations) and large-scale persistent graphs where traditional systems struggle with performance and scalability. Key Architectural Decisions: Columnar Storage: Kuzu stores node and relationship properties in separate, contiguous columns. This design reduces I/O by allowing queries to scan only the needed columns, unlike row-based systems (e.g., Neo4j) that read full records even when only a subset of properties is required. Efficient Join Indexing with CSR: The join index is maintained using a Compressed Sparse Row (CSR) format. By sorting and compressing relationship data, Kuzu ensures that adjacent node relationships are stored contiguously, minimizing random I/O and speeding up traversals. Vectorized Query Processing: Instead of processing one tuple at a time, Kuzu processes blocks (vectors) of tuples. This block-based (or vectorized) approach reduces function-call overhead and improves cache locality, boosting performance for analytic queries. Factorization and ASP Join: For many-to-many queries that can generate enormous intermediate results, Kuzu uses factorization to represent data compactly. Its ASP join algorithm integrates factorization, sequential scanning, and sideways information passing to avoid unnecessary full scans and materializations. Kuzu is optimized for read-heavy, analytic workloads. While batched writes are efficient, the system is less tuned for high-frequency, small transactions. Upcoming features include: A WebAssembly (Wasm) version for running in browsers. Enhanced vector and full-text search indices. Built-in graph data science algorithms for tasks like PageRank and centrality analysis. Kuzu can be a powerful backend for AI applications in several ways: Knowledge Graphs: Store and query complex relationships between entities to support natural language understanding, semantic search, and reasoning tasks. Graph Data Science: Run built-in graph algorithms (like PageRank, centrality, or community detection) that help uncover patterns and insights, which can improve recommendation systems, fraud detection, and other AI-driven analyses. Retrieval-Augmented Generation (RAG): Integrate with large language models by efficiently retrieving relevant, structured graph data. Kuzu’s vector search capabilities and fast query processing make it ideal for augmenting AI responses with contextual information. Graph Embeddings & ML Pipelines: Serve as the foundation for generating graph embeddings, which are used in downstream machine learning tasks—such as clustering, classification, or link prediction—to enhance model performance. Semih Salihoğlu: LinkedIn Kuzu GitHub Kuzu Docs Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter) 00:00 Introduction to Graph Databases 00:18 Introducing Kuzu: A Modern Graph Database 01:48 Use Cases and Applications of Kuzu 03:03 Kuzu's Research Origins and Scalability 06:18 Columnar Storage vs. Row-Oriented Storage 10:27 Query Processing Techniques in Kuzu 22:22 Compressed Sparse Row (CSR) Storage 27:25 Vectorization in Graph Databases 31:24 Optimizing Query Processors with Vectorization 33:25 Common Wisdom in Graph Databases 35:13 Introducing ASP Join in Kuzu 35:55 Factorization and Efficient Query Processing 39:49 Challenges and Solutions in Graph Databases 45:26 Write Path Optimization in Kuzu 54:10 Future Developments in Kuzu 57:51 Key Takeaways and Final Thoughts…

How AI Is Built

1
#043 Knowledge Graphs Won't Fix Bad Data 1:10:59

12 weeks ago1:10:59

1:10:59

Metadata is the foundation of any enterprise knowledge graph. By organizing both technical and business metadata, organizations create a “brain” that supports advanced applications like AI-driven data assistants. The goal is to achieve economies of scale—making data reusable, traceable, and ultimately more valuable. Juan Sequeda is a leading expert in enterprise knowledge graphs and metadata management. He has spent years solving the challenges of integrating diverse data sources into coherent, accessible knowledge graphs. As Principal Scientist at data.world, Juan provides concrete strategies for improving data quality, streamlining feature extraction, and enhancing model explainability. If you want to build AI systems on a solid data foundation—one that cuts through the noise and delivers reliable, high-performance insights—you need to listen to Juan’s proven methods and real-world examples. Terms like ontologies, taxonomies, and knowledge graphs aren’t new inventions. Ontologies and taxonomies have been studied for decades—even since ancient Greece. Google popularized “knowledge graphs” in 2012 by building on decades of semantic web research. Despite current buzz, these concepts build on established work. Traditionally, data lives in siloed applications—each with its own relational databases, ETL processes, and dashboards. When cross-application queries and consistent definitions become painful, organizations face metadata management challenges. The first step is to integrate technical metadata (table names, columns, code lineage) into a unified knowledge graph. Then, add business metadata by mapping business glossaries and definitions to that technical layer. A modern data catalog should: Integrate Multiple Sources: Automatically ingest metadata from databases, ETL tools (e.g., dbt, Fivetran), and BI tools. Bridge Technical and Business Views: Link technical definitions (e.g., table “CUST_123”) with business concepts (e.g., “Customer”). Enable Reuse and Governance: Support data discovery, impact analysis, and proper governance while facilitating reuse across teams. Practical Approaches & Use Cases: Start with a Clear Problem: Whether it’s reducing churn, improving operational efficiency, or meeting compliance needs, begin by solving a specific pain point. Iron Thread Method: Follow one query end-to-end—from identifying a business need to tracing it back to source systems—to gradually build and refine the graph. Automation vs. Manual Oversight: Technical metadata extraction is largely automated. For business definitions or text-based entity extraction (e.g., via LLMs), human oversight is key to ensuring accuracy and consistency. Technical Considerations: Entity vs. Property: If you need to attach additional details or reuse an element across contexts, model it as an entity (with a unique identifier). Otherwise, keep it as a simple property. Storage Options: The market offers various graph databases—Neo4j, Amazon Neptune, Cosmos DB, TigerGraph, Apache Jena (for RDF), etc. Future trends point toward multimodel systems that allow querying in SQL, Cypher, or SPARQL over the same underlying data. Juan Sequeda: LinkedIn data.world Semantic Web for the Working Ontologist Designing and Building Enterprise Knowledge Graphs (before you buy, send Juan a message, he is happy to send you a copy) Catalog & Cocktails (Juan’s podcast) Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter) 00:00 Introduction to Knowledge Graphs 00:45 The Role of Metadata in AI 01:06 Building Knowledge Graphs: First Steps 01:42 Interview with Juan Sequira 02:04 Understanding Buzzwords: Ontologies, Taxonomies, and More 05:05 Challenges and Solutions in Data Management 08:04 Practical Applications of Knowledge Graphs 15:38 Governance and Data Engineering 34:42 Setting the Stage for Data-Driven Problem Solving 34:58 Understanding Consumer Needs and Data Challenges 35:33 Foundations and Advanced Capabilities in Data Management 36:01 The Role of AI and Metadata in Data Maturity 37:56 The Iron Thread Approach to Problem Solving 40:12 Constructing and Utilizing Knowledge Graphs 54:38 Trends and Future Directions in Knowledge Graphs 59:17 Practical Advice for Building Knowledge Graphs…

How AI Is Built

1
#042 Temporal RAG, Embracing Time for Smarter, Reliable Knowledge Graphs 1:33:44

13 weeks ago1:33:44

1:33:44

Daniel Davis is an expert on knowledge graphs. He has a background in risk assessment and complex systems—from aerospace to cybersecurity. Now he is working on “Temporal RAG” in TrustGraph. Time is a critical—but often ignored—dimension in data. Whether it’s threat intelligence, legal contracts, or API documentation, every data point has a temporal context that affects its reliability and usefulness. To manage this, systems must track when data is created, updated, or deleted, and ideally, preserve versions over time. Three Types of Data: Observations: Definition: Measurable, verifiable recordings (e.g., “the hat reads ‘Sunday Running Club’”). Characteristics: Require supporting evidence and may be updated as new data becomes available. Assertions: Definition: Subjective interpretations (e.g., “the hat is greenish”). Characteristics: Involve human judgment and come with confidence levels; they may change over time. Facts: Definition: Immutable, verified information that remains constant. Characteristics: Rare in dynamic environments because most data evolves; serve as the “bedrock” of trust. By clearly categorizing data into these buckets, systems can monitor freshness, detect staleness, and better manage dependencies between components (like code and its documentation). Integrating Temporal Data into Knowledge Graphs: Challenge: Traditional knowledge graphs and schemas (e.g., schema.org ) rarely integrate time beyond basic metadata. Long documents may only provide a single timestamp, leaving the context of internal details untracked. Solution: Attach detailed temporal metadata (such as creation, update, and deletion timestamps) during data ingestion. Use versioning to maintain historical context. This allows systems to: Assess whether data is current or stale. Detect conflicts when updates occur. Employ Bayesian methods to adjust trust metrics as more information accumulates. Key Takeaways: Focus on Specialization: Build tools that do one thing well. For example, design a simple yet extensible knowledge graph rather than relying on overly complex ontologies. Integrate Temporal Metadata: Always timestamp data operations and version records. This is key to understanding data freshness and evolution. Adopt Robust Infrastructure: Use scalable, proven technologies to connect specialized modules via APIs. This reduces maintenance overhead compared to systems overloaded with connectors and extra features. Leverage Bayesian Updates: Start with initial trust metrics based on observed data and refine them as new evidence arrives. Mind the Big Picture: Avoid working in isolated silos. Emphasize a holistic system design that maintains in situ context and promotes collaboration across teams. Daniel Davis Cognitive Core TrustGraph YouTube LinkedIn Discord Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter) 00:00 Introduction to Temporal Dimensions in Data 00:53 Timestamping and Versioning Data 01:35 Introducing Daniel Davis and Temporal RAG 01:58 Three Buckets of Data: Observations, Assertions, and Facts 03:22 Dynamic Data and Data Freshness 05:14 Challenges in Integrating Time in Knowledge Graphs 09:41 Defining Observations, Assertions, and Facts 12:57 The Role of Time in Data Trustworthiness 46:58 Chasing White Whales in AI 47:58 The Problem with Feature Overload 48:43 Connector Maintenance Challenges 50:02 The Swiss Army Knife Analogy 51:16 API Meshes and Glue Code 54:14 The Importance of Software Infrastructure 01:00:10 The Need for Specialized Tools 01:13:25 Outro and Future Plans…

How AI Is Built

1
#041 Context Engineering, How Knowledge Graphs Help LLMs Reason 1:33:35

14 weeks ago1:33:35

1:33:35

Robert Caulk runs Emergent Methods, a research lab building news knowledge graphs. With a Ph.D. in computational mechanics, he spent 12 years creating open-source tools for machine learning and data analysis. His work on projects like Flowdapt (model serving) and FreqAI (adaptive modeling) has earned over 1,000 academic citations. His team built AskNews, which he calls "the largest news knowledge graph in production." It's a system that doesn't just collect news - it understands how events, people, and places connect. Current AI systems struggle to connect information across sources and domains. Simple vector search misses crucial relationships. But building knowledge graphs at scale brings major technical hurdles around entity extraction, relationship mapping, and query performance. Emergent Methods built a hybrid system combining vector search and knowledge graphs: Vector DB (Quadrant) handles initial broad retrieval Custom knowledge graph processes relationships Translation pipeline normalizes multi-language content Entity extraction model identifies key elements Context engineering pipeline structures data for LLMs Implementation Details: Data Pipeline: All content normalized to English for consistent embeddings Entity names preserved in original language when untranslatable Custom Gleiner News model handles entity extraction Retrained every 6 months on fresh data Human review validates entity accuracy Entity Management: Base extraction uses BERT-based Gleiner architecture Trained on diverse data across topics/regions Disambiguation system merges duplicate entities Manual override options for analysts Metadata tracking preserves relationship context Knowledge Graph: Selective graph construction from vector results On-demand relationship processing Graph queries via standard Cypher Built for specific use cases vs general coverage Integration with S3 and other data stores System Validation: Custom "Context is King" benchmark suite RAGAS metrics track retrieval accuracy Time-split validation prevents data leakage Manual review of entity extraction Production monitoring of query patterns Engineering Insights: Key Technical Decisions: English normalization enables consistent embeddings Hybrid vector + graph approach balances speed/depth Selective graph construction keeps costs down Human-in-loop validation maintains quality Dead Ends Hit: Full multi-language entity system too complex Real-time graph updates not feasible at scale Pure vector or pure graph approaches insufficient Top Quotes: "At its core, context engineering is about how we feed information to AI. We want clear, focused inputs for better outputs. Think of it like talking to a smart friend - you'd give them the key facts in a way they can use, not dump raw data on them." - Robert "Strong metadata paints a high-fidelity picture. If we're trying to understand what's happening in Ukraine, we need to know not just what was said, but who said it, when they said it, and what voice they used to say it. Each piece adds color to the picture." - Robert "Clean data beats clever models. You can throw noise at an LLM and get something that looks good, but if you want real accuracy, you need to strip away the clutter first. Every piece of noise pulls the model in a different direction." - Robert "Think about how the answer looks in the real world. If you're comparing apartments, you'd want a table. If you're tracking events, you'd want a timeline. Match your data structure to how humans naturally process that kind of information." - Nico "Building knowledge graphs isn't about collecting everything - it's about finding the relationships that matter. Most applications don't need a massive graph. They need the right connections for their specific problem." - Robert "The quality of your context sets the ceiling for what your AI can do. You can have the best model in the world, but if you feed it noisy, unclear data, you'll get noisy, unclear answers. Garbage in, garbage out still applies." - Robert "When handling multiple languages, it's better to normalize everything to one language than to try juggling many. Yes, you lose some nuance, but you gain consistency. And consistency is what makes these systems reliable." - Robert "The hard part isn't storing the data - it's making it useful. Anyone can build a database. The trick is structuring information so an AI can actually reason with it. That's where context engineering makes the difference." - Robert "Start simple, then add complexity only when you need it. Most teams jump straight to sophisticated solutions when they could get better results by just cleaning their data and thinking carefully about how they structure it." - Nico "Every token in your context window is precious. Don't waste them on HTML tags or formatting noise. Save that space for the actual signal - the facts, relationships, and context that help the AI understand what you're asking." - Nico Robert Caulk: LinkedIn Emergent Methods Asknews Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter) 00:00 Introduction to Context Engineering 00:24 Curating Input Signals 01:01 Structuring Raw Data 03:05 Refinement and Iteration 04:08 Balancing Breadth and Precision 06:10 Interview Start 08:02 Challenges in Context Engineering 20:25 Optimizing Context for LLMs 45:44 Advanced Cypher Queries and Graphs 46:43 Enrichment Pipeline Flexibility 47:16 Combining Graph and Semantic Search 49:23 Handling Multilingual Entities 52:57 Disambiguation and Deduplication Challenges 55:37 Training Models for Diverse Domains 01:04:43 Dealing with AI-Generated Content 01:17:32 Future Developments and Final Thoughts…

How AI Is Built

1
#040 Vector Database Quantization, Product, Binary, and Scalar 52:12

15 weeks ago52:12

52:12

When you store vectors, each number takes up 32 bits. With 1000 numbers per vector and millions of vectors, costs explode. A simple chatbot can cost thousands per month just to store and search through vectors. The Fix: Quantization Think of it like image compression. JPEGs look almost as good as raw photos but take up far less space. Quantization does the same for vectors. Today we are back continuing our series on search with Zain Hasan, a former ML engineer at Weaviate and now a Senior AI/ ML Engineer at Together. We talk about the different types of quantization, when to use them, how to use them, and their tradeoff. Three Ways to Quantize: Binary Quantization Turn each number into just 0 or 1 Ask: "Is this dimension positive or negative?" Works great for 1000+ dimensions Cuts memory by 97% Best for normally distributed data Product Quantization Split vector into chunks Group similar chunks Store cluster IDs instead of full numbers Good when binary quantization fails More complex but flexible Scalar Quantization Use 8 bits instead of 32 Simple middle ground Keeps more precision than binary Less savings than binary Key Quotes: "Vector databases are pretty much the commercialization and the productization of representation learning." "I think quantization, it builds on the assumption that there is still noise in the embeddings. And if I'm looking, it's pretty similar as well to the thought of Matryoshka embeddings that I can reduce the dimensionality." "Going from text to multimedia in vector databases is really simple." "Vector databases allow you to take all the advances that are happening in machine learning and now just simply turn a switch and use them for your application." Zain Hasan: LinkedIn X (Twitter) Weaviate Together Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter) vector databases, quantization, hybrid search, multi-vector support, representation learning, cost reduction, memory optimization, multimodal recommender systems, brain-computer interfaces, weather prediction models, AI applications…

How AI Is Built

1
#039 Local-First Search, How to Push Search To End-Devices 53:09

16 weeks ago53:09

53:09

Alex Garcia is a developer focused on making vector search accessible and practical. As he puts it: "I'm a SQLite guy. I use SQLite for a lot of projects... I want an easier vector search thing that I don't have to install 10,000 dependencies to use.” Core Mantra: "Simple, Local, Scalable" Why SQLite Vec? "I didn't go along thinking, 'Oh, I want to build vector search, let me find a database for it.' It was much more like: I use SQLite for a lot of projects, I want something lightweight that works in my current workflow." SQLiteVec uses row-oriented storage with some key design choices: Vectors are stored in large chunks (megabytes) as blobs Data is split across 4KB SQLite pages, which affects analytical performance Currently uses brute force linear search without ANN indexing Supports binary quantization for 32x size reduction Handles tens to hundreds of thousands of vectors efficiently Practical limits: 500ms search time for 500K vectors (768 dimensions) Best performance under 100ms for user experience Binary quantization enables scaling to ~1M vectors Metadata filtering and partitioning coming soon Key advantages: Fast writes for transactional workloads Simple single-file database Easy integration with existing SQLite applications Leverages SQLite's mature storage engine Garcia's preferred tools for local AI: Sentence Transformers models converted to GGUF format Llama.cpp for inference Small models (30MB) for basic embeddings Larger models like Arctic Embed (hundreds of MB) for recent topics SQLite L-Embed extension for text embeddings Transformers.js for browser-based implementations 1. Choose Your Storage "There's two ways of storing vectors within SQLiteVec. One way is a manual way where you just store a JSON array... [second is] using a virtual table." Traditional row storage: Simple, flexible, good for small vectors Virtual table storage: Optimized chunks, better for large datasets Performance sweet spot: Up to 500K vectors with 500ms search time 2. Optimize Performance "With binary quantization it's 1/32 of the space... and holds up at 95 percent quality" Binary quantization reduces storage 32x with 95% quality Default page size is 4KB - plan your vector storage accordingly Metadata filtering dramatically improves search speed 3. Integration Patterns "It's a single file, right? So you can like copy and paste it if you want to make a backup." Two storage approaches: manual columns or virtual tables Easy backups: single file database Cross-platform: desktop, mobile, IoT, browser (via WASM) 4. Real-World Tips "I typically choose the really small model... it's 30 megabytes. It quantizes very easily... I like it because it's very small, quick and easy." Start with smaller, efficient models (30MB range) Use binary quantization before trying complex solutions Plan for partitioning when scaling beyond 100K vectors Alex Garcia LinkedIn X (Twitter) GitHub sqlite-vec sqllite-vss Website Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter)…

How AI Is Built

1
#038 AI-Powered Search, Context Is King, But Your RAG System Ignores Two-Thirds of It 1:14:24

18 weeks ago1:14:24

1:14:24

Today, I (Nicolay Gerold) sit down with Trey Grainger, author of the book AI-Powered Search. We discuss the different techniques for search and recommendations and how to combine them. While RAG (Retrieval-Augmented Generation) has become a buzzword in AI, Trey argues that the current understanding of "RAG" is overly simplified – it's actually a bidirectional process he calls "GARRAG," where retrieval and generation continuously enhance each other. Trey uses a three context framework for search architecture: Content Context: Traditional document understanding and retrieval User Context: Behavioral signals driving personalization and recommendations Domain Context: Knowledge graphs and semantic understanding Trey shares insights on: Why collecting and properly using user behavior signals is crucial yet often overlooked How to implement "light touch" personalization without trapping users in filter bubbles The evolution from simple vector similarity to sophisticated late interaction models Why treating search as a non-linear pipeline with feedback loops leads to better results For engineers building search systems, Trey offers practical advice on choosing the right tools and techniques, from traditional search engines like Solr and Elasticsearch to modern approaches like ColBERT. Also how to layer different techniques to make search tunable and debuggable. Quotes: "I think of whether it's search or generative AI, I think of all of these systems as nonlinear pipelines." "The reason we use retrieval when we're working with generative AI is because A generative AI model these LLMs will take your query, your request, whatever you're asking for. They will then try to interpret them and without access to up to date information, without access to correct information, they will generate a response from their highly compressed understanding of the world. And so we use retrieval to augment them with information." "I think the misconception is that, oh, hey, for RAG I can just, plug in a vector database and a couple of libraries and, a day or two later everything's magically working and I'm off to solve the next problem. Because search and information retrieval is one of those problems that you never really solve. You get it, good enough and quit, or you find so much value in it, you just continue investing to constantly make it better." "To me, they're, search and recommendations are fundamentally the same problem. They're just using different contexts." "Anytime you're building a search system, whether it's traditional search, whether it's RAG for generative AI, you need to have all three of those contexts in order to effectively get the most relevant results to solve solve the problem." "There's no better way to make your users really angry with you than to stick them in a bucket and get them stuck in that bucket, which is not their actual intent." Trey Grainger: LinkedIn AI Powered Search (Community) AI Powered Search (Book) Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter) 00:00 Introduction to Search Challenges 00:50 Layered Approach to Ranking 01:00 Personalization and Signal Boosting 02:25 Broader Principles in Software Engineering 02:51 Interview with Trey Greinger 03:32 Understanding RAG and Retrieval 04:35 Nonlinear Pipelines in Search 06:01 Generative AI and Retrieval 08:10 Search Renaissance and AI 10:27 Misconceptions in AI-Powered Search 18:12 Search vs. Recommendation Systems 22:26 Three Buckets of Relevance 38:19 Traditional Learning to Rank 39:11 Semantic Relevance and User Behavior 39:53 Layered Ranking Algorithms 41:40 Personalization in Search 43:44 Technological Setup for Query Understanding 48:21 Personalization and User Behavior Vectors 52:10 Choosing the Right Search Engine 56:35 Future of AI-Powered Search 01:00:48 Building Effective Search Applications 01:06:50 Three Critical Context Frameworks 01:12:08 Modern Search Systems and Contextual Understanding 01:13:37 Conclusion and Recommendations…

How AI Is Built

1
#037 Chunking for RAG: Stop Breaking Your Documents Into Meaningless Pieces 49:13

19 weeks ago49:13

49:13

Today we are back continuing our series on search. We are talking to Brandon Smith, about his work for Chroma. He led one of the largest studies in the field on different chunking techniques. So today we will look at how we can unfuck our RAG systems from badly chosen chunking hyperparameters. The biggest lie in RAG is that semantic search is simple. The reality is that it's easy to build, it's easy to get up and running, but it's really hard to get right. And if you don't have a good setup, it's near impossible to debug. One of the reasons it's really hard is actually chunking. And there are a lot of things you can get wrong. And even OpenAI boggled it a little bit, in my opinion, using an 800 token length for the chunks. And this might work for legal, where you have a lot of boilerplate that carries little semantic meaning, but often you have the opposite. You have very information dense content and imagine fitting an entire Wikipedia page into the size of a tweet There will be a lot of information that's actually lost and that's what happens with long chunks The next is overlap openai uses a foreign token overlap or used to And what this does is actually we try to bring the important context into the chunk, but in reality, we don't really know where the context is coming from. It could be from a few pages prior, not just the 400 tokens before. It could also be from a definition that's not even in the document at all. There is a really interesting solution actually from Anthropic Contextual Retrieval, where you basically pre process all the chunks to see whether there is any missing information and you basically try to reintroduce it. Brandon Smith: LinkedIn X (Twitter) Website Chunking Article Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter) Website 00:00 The Biggest Lie in RAG: Semantic Search Simplified 00:43 Challenges in Chunking and Overlap 01:38 Introducing Brandon Smith and His Research 02:05 The Motivation and Mechanics of Chunking 04:40 Issues with Current Chunking Methods 07:04 Optimizing Chunking Strategies 23:04 Introduction to Chunk Overlap 24:23 Exploring LLM-Based Chunking 24:56 Challenges with Initial Approaches 28:17 Alternative Chunking Methods 36:13 Language-Specific Considerations 38:41 Future Directions and Best Practices…

How AI Is Built

1
#036 How AI Can Start Teaching Itself - Synthetic Data Deep Dive 48:11

21 weeks ago48:11

48:11

Most LLMs you use today already use synthetic data. It’s not a thing of the future. The large labs use a large model (e.g. gpt-4o) to generate training data for a smaller one (gpt-4o-mini). This lets you build fast, cheap models that do one thing well. This is “distillation”. But the vision for synthetic data is much bigger. Enable people to train specialized AI systems without having a lot of training data. Today we are talking to Adrien Morisot, an ML engineer at Cohere. We talk about how Cohere uses synthetic data to train their models, their learnings, and how you can use synthetic data in your training. We are slightly diverging from our search focus, but I wanted to create a deeper dive into synthetic data after our episode with Saahil. You could use it in a lot of places: generate hard negatives, generate training samples for classifiers and rerankers and much more. Scaling Synthetic Data Creation: https://arxiv.org/abs/2406.20094 Adrien Morisot: LinkedIn X (Twitter) Cohere Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter) 00:00 Introduction to Synthetic Data in LLMs 00:18 Distillation and Specialized AI Systems 00:39 Interview with Adrien Morisot 02:00 Early Challenges with Synthetic Data 02:36 Breakthroughs and Rediscovery 03:54 The Evolution of AI and Synthetic Data 07:51 Data Harvesting and Internet Scraping 09:28 Generating Diverse Synthetic Data 15:37 Manual Review and Quality Control 17:28 Automating Data Evaluation 18:54 Fine-Tuning Models with Synthetic Data 21:45 Avoiding Behavioral Cloning 23:47 Ensuring Model Accuracy with Verification 24:31 Adapting Models to Specific Domains 26:41 Challenges in Financial and Legal Domains 28:10 Improving Synthetic Data Sets 30:45 Evaluating Model Performance 32:21 Using LLMs as Judges 35:42 Practical Tips for AI Practitioners 41:26 Synthetic Data in Training Processes 43:51 Quality Control in Synthetic Data 45:41 Domain Adaptation Strategies 46:51 Future of Synthetic Data Generation 47:30 Conclusion and Next Steps…

How AI Is Built

1
#035 A Search System That Learns As You Use It (Agentic RAG) 45:30

22 weeks ago45:30