Vector Databases Explained: How They Work in RAG
A vector database is a database designed to store, index, and search numerical vectors efficiently. In a RAG application, vector databases are commonly used to store embeddings of documents and retrieve the pieces of information that are most relevant to a user's question.
Introduction
A vector database is a database designed to store, index, and search numerical vectors efficiently.
In a RAG application, vector databases are commonly used to store embeddings of documents and retrieve the pieces of information that are most relevant to a user's question.
The basic flow looks like this:
Document
↓
Chunk
↓
Embedding Model
↓
Vector
↓
Vector Database
↓
Similarity Search
↓
Relevant Information
If you've already read our articles on What Is RAG? and What Are Embeddings?, you're now at the next important piece of the puzzle.
RAG needs a way to find relevant information.
Embeddings give us a mathematical representation of that information.
A vector database gives us a practical way to store and search those representations.
Let's see how it works.
Quick Answer
A vector database stores numerical vectors, usually embeddings, and provides efficient similarity search over them.
In a RAG application, documents are typically:
Documents
↓
Chunks
↓
Embeddings
↓
Vectors
↓
Vector Database
When a user asks a question:
Question
↓
Query Embedding
↓
Vector Search
↓
Relevant Chunks
↓
LLM
↓
Answer
The vector database is responsible for finding vectors that are similar to the user's query vector.
That sounds simple.
The interesting part is how it does this efficiently when you have thousands, millions, or even billions of vectors.
What Is a Vector Database?
Let's start with the simplest definition.
A vector database is a database optimized for storing and searching vector embeddings.
A normal database might store information like:
id
name
email
age
A vector database can store information such as:
id
embedding
text
metadata
For example:
{
"id": "doc_001",
"embedding": [0.21, -0.14, 0.73, 0.08],
"text": "The JDK is required to develop Java applications.",
"metadata": {
"topic": "java",
"source": "java-guide.md"
}
}
The embedding is the important part for similarity search.
The text and metadata provide additional information that the application can use later.
Why Do We Need Vector Databases?
Imagine you have a knowledge base containing 10 million document chunks.
A user asks:
How do I install Java?
The application creates an embedding for the question.
Now it needs to find the document chunks that are most similar to that embedding.
You could theoretically compare the query vector with every stored vector.
But that can become expensive as your dataset grows.
Query Vector
↓
Compare with Vector 1
Compare with Vector 2
Compare with Vector 3
Compare with Vector 4
...
Compare with Vector 10,000,000
That's where specialized vector indexing and search techniques become important.
A vector database is built to handle this kind of workload much more efficiently.
What Is a Vector?
Before discussing databases, let's quickly revisit vectors.
An embedding is usually represented as a vector of numbers.
For example:
[0.21, -0.14, 0.73, 0.08]
A real embedding can contain hundreds or thousands of values.
You can think of this vector as a point in a high-dimensional mathematical space.
Related information tends to have vectors that are closer together according to a chosen similarity or distance measure.
For example:
Java
●
/ \
/ \
JDK JVM
● ●
Cooking
●
This is only a simplified visualization.
Real embeddings don't live on a simple 2D chart.
They exist in a much larger mathematical space.
If you want to understand how these vectors are created, see our guide on What Are Embeddings?.
What Does a Vector Database Store?
A vector database doesn't necessarily store only vectors.
A typical record can contain three important pieces:
Vector
+
Original Content
+
Metadata
For example:
{
"id": "chunk_42",
"vector": [0.21, -0.14, 0.73, 0.08],
"text": "The JDK is required to develop Java applications.",
"metadata": {
"source": "java-guide.md",
"category": "java",
"version": "2026"
}
}
The vector is used for similarity search.
The text can be returned as retrieval context.
The metadata can be used for filtering and organization.
This combination is particularly useful for RAG.
How Does a Vector Database Work?
The overall process can be divided into two stages:
Indexing
You prepare and store your data.
Querying
You search that stored data.
The indexing workflow looks like this:
Document
↓
Extract Text
↓
Chunk Document
↓
Generate Embeddings
↓
Store Vectors
↓
Build Search Index
The query workflow looks like this:
User Question
↓
Generate Query Embedding
↓
Search Vector Index
↓
Find Similar Vectors
↓
Return Relevant Chunks
These two workflows form the foundation of vector search.
Vector Search Explained
Traditional database queries often look like:
SELECT *
FROM products
WHERE category = 'laptops';
The database looks for records matching a specific condition.
Vector search works differently.
Instead of asking:
Does this value exactly match?
the system can ask:
Which stored vectors are closest to this query vector?
For example:
Query:
"How do I install Java?"
↓
Query Vector
↓
Vector Search
↓
Closest Vectors
↓
Relevant Document Chunks
This is commonly called similarity search or nearest-neighbor search.
What Is Similarity Search?
Similarity search tries to find items that are mathematically close to a query.
Suppose your database contains vectors representing:
Java installation
Java garbage collection
Python installation
Docker containers
Spring Boot configuration
The user asks:
How do I set up Java?
The system converts the question into a vector.
It then searches the database.
Conceptually, the results could look like:
Java installation
████████████████████
Python installation
██████████████
Spring Boot configuration
██████████
Java garbage collection
████████
Docker containers
███
The system can return the highest-ranked results.
Those results can then be passed to the LLM.
Distance and Similarity
Vector databases need a way to determine how close two vectors are.
Different applications can use different distance or similarity measures.
Common approaches include:
Cosine similarity
Euclidean distance
Dot product
The exact choice depends on the embedding model and retrieval setup.
Cosine Similarity
Cosine similarity compares the orientation of two vectors.
It is widely used for text embeddings.
Euclidean Distance
Euclidean distance measures the straight-line distance between points in vector space.
Dot Product
Dot product can also be used to compare vectors and is common in many machine learning systems.
The important thing isn't memorizing every formula.
The important thing is understanding that vector search needs a mathematical definition of closeness.
Why Can't a Normal Database Do This?
This question comes up often.
The answer is: some traditional databases can now support vector search, so the distinction isn't as absolute as it once was.
For example, PostgreSQL can use the pgvector extension to store and search embeddings.
So you don't always need a separate vector database product.
The more useful question is:
Does your existing database provide the vector search capabilities, performance, indexing, filtering, and operational characteristics your application needs?
For a small application, PostgreSQL with pgvector may be more than enough.
For a large-scale retrieval system, a specialized vector database may provide capabilities that make the architecture easier to operate.
We'll look at this trade-off later.
Traditional Database vs Vector Search
A traditional database is generally excellent for structured queries.
For example:
Find all users
where country = India
and age > 25
A vector search system is designed for questions such as:
Find the documents
most semantically similar
to this question.
These aren't competing ideas.
Modern applications often use both.
For example:
User Question
↓
Metadata Filter
↓
Vector Search
↓
Reranking
↓
Relevant Documents
You might first filter documents by:
product = "Java"
version = "2026"
language = "English"
Then perform semantic search within that smaller collection.
This can improve both relevance and efficiency.
Metadata in Vector Databases
Metadata is one of the most useful features in a real RAG application.
Imagine a company has documentation for five versions of its product.
A user asks:
How do I configure authentication in version 5?
If you only perform semantic search, the system might retrieve documentation from version 3.
Metadata can help.
Your records could contain:
{
"source": "authentication.md",
"product": "ExampleApp",
"version": "5",
"category": "security"
}
The application can then filter:
version = 5
category = security
before or during vector retrieval, depending on the database and search strategy.
This is a major reason production RAG systems usually need more than just embeddings.
Vector Databases in RAG
Now let's connect everything we've learned so far.
Our RAG pipeline currently looks like:
INDEXING
Documents
↓
Text Extraction
↓
Chunking
↓
Embeddings
↓
Vector Database
QUERY
User Question
↓
Query Embedding
↓
Vector Search
↓
Relevant Chunks
↓
LLM
↓
Answer
The vector database sits between the embedding system and the language model.
It doesn't generate the answer.
It doesn't understand the question like an LLM.
Its job is primarily to help the application find relevant stored information.
That distinction is worth remembering.
A Practical RAG Example
Suppose we're building a developer documentation assistant.
Our knowledge base contains:
java-installation.md
java-collections.md
java-exceptions.md
spring-security.md
docker-basics.md
The documents are split into chunks.
Each chunk is embedded.
The vectors are stored in a vector database.
Now the user asks:
What should I install before writing my first Java program?
The query becomes an embedding.
The vector database searches for similar document chunks.
It might retrieve:
"The Java Development Kit is required
to develop Java applications."
The application then sends that information to the LLM.
The LLM generates the final response.
User Question
↓
Query Embedding
↓
Vector Database
↓
Relevant Java Chunk
↓
LLM
↓
Answer
That's the practical role of a vector database in RAG.
Do You Always Need a Vector Database?
No.
This is important.
RAG is an architecture, not a requirement to use a particular database product.
You can implement retrieval using several approaches.
For example:
Keyword Search
Semantic Search
Hybrid Search
Database Queries
External Search APIs
You can also use a traditional database with vector support.
The right choice depends on your data and application.
A small internal application might work perfectly well with PostgreSQL and pgvector.
A large retrieval-heavy application might benefit from a specialized vector database.
Don't introduce another database just because a tutorial uses one.
Popular Vector Database Options
You'll encounter several names when working with RAG.
PostgreSQL + pgvector
A PostgreSQL extension that adds vector similarity search capabilities.
It's particularly interesting when your application already uses PostgreSQL and you want to keep structured data and vector data together.
Qdrant
An open-source vector search engine designed around similarity search and metadata filtering.
Milvus
An open-source vector database designed for large-scale vector search workloads.
Weaviate
An open-source vector database that provides vector search and additional capabilities for building AI applications.
Pinecone
A managed vector database service designed for applications that need vector search without managing the underlying infrastructure themselves.
The important thing isn't choosing the "best" database from a list.
It's understanding what your application needs.
Managed vs Self-Hosted Vector Databases
Another decision you'll encounter is where the database runs.
Managed
A cloud provider or vendor operates the infrastructure.
You typically focus on:
Application
↓
API
↓
Managed Vector Database
The provider handles much of the infrastructure.
This can simplify operations.
Self-Hosted
You operate the vector database yourself.
For example:
Your Application
↓
Your Infrastructure
↓
Vector Database
This gives you more control, but also more operational responsibility.
You'll need to think about:
Deployment
Scaling
Backups
Monitoring
Security
Upgrades
Availability
For a small project, either approach can work.
For production, make the decision based on operational requirements rather than hype.
What Is a Vector Index?
Searching a large collection of vectors efficiently requires an indexing strategy.
A vector index is a data structure that helps the system find nearby vectors without comparing the query against every single vector in the database.
One widely used family of approaches is Approximate Nearest Neighbor, commonly abbreviated as ANN.
The idea is straightforward.
Instead of guaranteeing an exhaustive search over every vector, the index uses an efficient structure to find very good candidates much faster.
Millions of Vectors
↓
Vector Index
↓
Search Candidate Vectors
↓
Nearest Results
This trade-off between speed, memory, and search quality becomes very important at scale.
Exact Search vs Approximate Search
There are two broad approaches.
Exact Search
The system compares the query against every relevant vector.
This can provide exact nearest neighbors.
But as the dataset becomes very large, it can become expensive.
Approximate Nearest Neighbor Search
The system uses an index to find highly relevant candidates without exhaustively checking every vector.
This is usually much faster at scale.
The trade-off is that approximate search may not always return the mathematically exact nearest neighbors.
In practice, that's often an acceptable trade-off.
You care about finding good relevant results quickly, not winning a mathematical contest.
Why Indexing Matters
Imagine a database containing:
1,000 vectors
A straightforward search might be perfectly acceptable.
Now imagine:
10 million vectors
The cost of checking everything becomes much more significant.
An index helps narrow the search.
Without Index
Query
↓
Check Everything
↓
Results
With Vector Index
Query
↓
Vector Index
↓
Relevant Candidates
↓
Results
This is one of the reasons vector databases become valuable as applications grow.
Vector Database Performance
When evaluating a vector database, don't look only at raw search speed.
A real application needs to balance several factors.
Latency
How quickly does a query return?
Recall
How often does the system retrieve the information you actually need?
Throughput
How many queries can the system handle?
Storage
How much memory and disk space do the vectors require?
Filtering
Can you efficiently filter by metadata?
Scalability
What happens when the dataset grows?
Cost
How much does the infrastructure cost to operate?
These factors can pull in different directions.
For example, increasing search accuracy can sometimes increase latency or resource usage.
That's why production systems need measurements rather than assumptions.
Vector Database Cost
Vector storage isn't free.
Every embedding takes space.
Suppose your embedding contains:
1,536 numbers
and you store millions of vectors.
The storage requirement can become significant.
You may also need additional memory for indexes and metadata.
Then there are infrastructure costs such as:
Compute
Storage
Network
Backups
Replication
Monitoring
This is another reason not to store unnecessary vectors.
Good data preparation matters.
So does choosing sensible chunk sizes.
Common Vector Database Mistakes
Mistake 1: Choosing a Database Before Understanding the Workload
Don't start by asking:
Which vector database is the best?
Start with:
What does my application need?
Those are different questions.
Mistake 2: Ignoring Metadata
A pure vector search may retrieve semantically similar information from the wrong version, product, or department.
Metadata filtering can be extremely valuable.
Mistake 3: Storing Poor-Quality Chunks
A vector database can't fix bad source data.
If your chunks are badly structured, retrieval quality will suffer.
This is why our upcoming article on How Document Chunking Works matters.
Mistake 4: Retrieving Too Many Results
More results don't automatically mean better answers.
If you retrieve 50 loosely related chunks, the LLM may receive a lot of noise.
Start with a sensible retrieval count and evaluate it.
Mistake 5: Ignoring Retrieval Evaluation
A search system returning results doesn't mean it's returning the right results.
Test it with real questions.
Measure whether relevant information appears near the top.
Mistake 6: Changing Embedding Models Without Planning
If you switch embedding models, your existing vectors may no longer be compatible with the new setup.
You may need to regenerate your embeddings and rebuild the vector index.
How to Choose a Vector Database
There isn't one universal answer.
Start by asking a few practical questions.
How large is your dataset?
A few thousand vectors and several billion vectors are very different problems.
Do you already use PostgreSQL?
If yes, pgvector may be worth evaluating before adding another database.
Do you need managed infrastructure?
If you don't want to operate another database yourself, a managed service may make sense.
Do you need advanced filtering?
Metadata filtering can be critical for enterprise RAG.
What latency do you need?
A documentation assistant with a few seconds of latency has different requirements from a high-volume production search service.
What is your budget?
Infrastructure costs matter.
How much operational control do you need?
Self-hosting provides more control but also more responsibility.
These questions will narrow the options much faster than reading endless "best vector database" lists.
A Simple Vector Search Example
Let's represent the core idea using Python-style pseudocode:
documents = [
"How to install Java",
"Java garbage collection explained",
"Docker container basics"
]
vectors = embed(documents)
vector_db.insert(
vectors=vectors,
documents=documents
)
question = "How can I set up Java?"
query_vector = embed(question)
results = vector_db.search(
query_vector=query_vector,
top_k=2
)
print(results)
The exact syntax depends on the database and embedding provider.
But the architecture remains the same:
Documents
↓
Embeddings
↓
Vector Database
↓
Query
↓
Similarity Search
↓
Relevant Documents
Once you understand this flow, learning an actual vector database becomes much easier.
Vector Databases and Hybrid Search
Vector search is powerful, but it isn't always enough.
Consider a developer searching for:
HTTP 429
Exact keyword matching can be extremely useful.
Now consider:
How do I prevent my API
from exceeding request limits?
Semantic search may be more useful.
A production system can combine both.
User Query
↓
┌──────────┴──────────┐
↓ ↓
Keyword Search Vector Search
↓ ↓
└──────────┬──────────┘
↓
Combine
↓
Rerank
↓
Best Results
This is called hybrid search.
We'll cover it later in the playlist.
Vector Databases and Reranking
Sometimes the first retrieval stage produces several possible results.
You can then use a reranking model to reorder them.
For example:
User Question
↓
Vector Search
↓
Top 20 Candidates
↓
Reranker
↓
Top 5 Results
↓
LLM
This can improve retrieval quality because the first search stage is optimized for efficiently finding candidates, while the reranker can spend more computation deciding which candidates are most relevant.
Reranking becomes particularly useful as your RAG system becomes more sophisticated.
We'll explore it later in the series.
Vector Databases in Production
A prototype might have:
Python
↓
Vector Database
↓
LLM
A production system can look more like:
Documents
↓
Data Processing
↓
Chunking
↓
Embeddings
↓
Vector Database
↓
Search Index
↑
│
User → Query → Embedding
↓
Metadata Filter
↓
Vector Search
↓
Reranking
↓
Context Builder
↓
LLM
↓
Answer
At this stage, you're no longer just choosing a database.
You're designing a retrieval system.
That's an important shift.
Security Considerations
Vector databases can contain sensitive information.
For example:
Company Documents
Internal Documentation
Customer Data
Engineering Notes
Private Policies
If your RAG system retrieves this information, access control matters.
A user shouldn't receive information simply because it happens to be semantically similar to their question.
Your system may need to enforce permissions before or during retrieval.
For example:
User
↓
Authentication
↓
Authorization
↓
Allowed Data
↓
Vector Search
↓
Relevant Results
Don't treat the vector database as a public search box.
The retrieval layer needs to respect the application's security model.
The Most Important Thing to Remember
A vector database doesn't make an AI application intelligent.
It doesn't understand your question like an LLM.
It doesn't decide whether an answer is correct.
Its primary job is to make vector storage and similarity retrieval practical.
Think of the architecture like this:
Embedding Model
↓
Creates Representation
Vector Database
↓
Finds Similar Information
LLM
↓
Understands Context
and Generates Answer
Each component has a different responsibility.
Once you separate those responsibilities, RAG architecture becomes much easier to reason about.
Key Takeaways
A vector database is a system designed to store and search vector representations efficiently.
In RAG:
Documents
↓
Chunks
↓
Embeddings
↓
Vectors
↓
Vector Database
↓
Similarity Search
↓
Relevant Context
↓
LLM
↓
Answer
Remember these points:
Vector databases store and search vectors.
Embeddings are commonly stored as vectors.
Similarity search finds vectors that are close to a query vector.
Cosine similarity, Euclidean distance, and dot product are common comparison methods.
Metadata can make retrieval much more precise.
Vector indexes improve search performance at scale.
Approximate nearest-neighbor search trades some exactness for speed and scalability.
You don't always need a dedicated vector database.
PostgreSQL with pgvector can be a practical option for many applications.
Vector search and keyword search can work together.
Reranking can improve the quality of retrieved results.
Production RAG requires attention to performance, cost, security, and evaluation.
A vector database retrieves information. It doesn't generate the final answer.
The simplest mental model is:
Embedding models create vectors.
Vector databases find vectors.
LLMs generate answers.
FAQs
What is a vector database?
A vector database is a database designed to store, index, and search numerical vectors efficiently. It is commonly used to store embeddings for semantic search and RAG applications.
Why are vector databases used in RAG?
Vector databases allow RAG applications to search large collections of embeddings and retrieve information that is semantically similar to a user's question.
Is a vector database the same as an embedding model?
No. An embedding model converts information into vectors. A vector database stores and searches those vectors.
Do I need a vector database to build RAG?
No. RAG can use different retrieval systems. Vector databases are common, but keyword search, hybrid search, traditional databases, and other retrieval approaches can also be used.
Is PostgreSQL a vector database?
PostgreSQL is a relational database. With the pgvector extension, it can store and search vector embeddings, making it a practical option for many RAG applications.
What is vector similarity search?
Vector similarity search finds stored vectors that are mathematically close or similar to a query vector.
What is a vector index?
A vector index is a data structure designed to make nearest-neighbor or similarity searches more efficient over large collections of vectors.
What is ANN search?
ANN stands for Approximate Nearest Neighbor. It refers to search techniques that efficiently find vectors close to a query without necessarily exhaustively comparing the query with every stored vector.
Can vector databases store normal text?
Many vector databases can store the original text and metadata alongside vectors. However, their main purpose is efficient vector search.
Which vector database should beginners learn?
There isn't one answer for everyone. PostgreSQL with pgvector is a useful option if you're already familiar with PostgreSQL. Dedicated systems such as Qdrant, Milvus, Weaviate, and Pinecone are also worth understanding as you explore different RAG architectures.
Conclusion
Vector databases are one of the key pieces that make semantic retrieval practical.
The idea isn't complicated.
You start with documents.
You split them into chunks.
You generate embeddings.
You store those embeddings.
Then, when a user asks a question, you generate an embedding for the question and search for similar vectors.
Documents
↓
Embeddings
↓
Vector Database
↓
Similarity Search
↓
Relevant Information
↓
LLM
↓
Answer
But don't stop at the database itself.
A good RAG system depends on the entire retrieval pipeline.
Your chunking strategy affects what gets embedded.
Your embedding model affects how information is represented.
Your vector index affects search performance.
Your metadata affects filtering.
Your retrieval strategy affects what reaches the LLM.
And ultimately, all of those choices affect the quality of the answer.
That's why understanding vector databases is important, but it's only one part of building a reliable RAG system.
We've now covered three foundational pieces:
RAG → Embeddings → Vector Databases
Next, we need to answer another important question:
What happens when keyword search and semantic search approach the same problem differently?
That's where semantic search vs keyword search comes in.
Continue Learning
RAG works best when you learn the pieces in the right order.
Previous Articles
1. What Is RAG? Retrieval-Augmented Generation Explained
Learn the complete RAG architecture and understand why AI applications use retrieval before generation.
2. What Are Embeddings? A Developer-Friendly Guide to Embeddings in AI
Learn how text becomes vectors and how embeddings allow applications to compare information based on semantic similarity.
Next Articles
4. Semantic Search vs Keyword Search
Understand how traditional keyword search differs from semantic search and why modern RAG systems often combine both.
5. How Document Chunking Works
Learn how large documents are divided into smaller pieces before embeddings are generated.
6. Chunking Strategies for RAG
Explore different chunking approaches and learn how chunk size and overlap affect retrieval quality.
7. Build Your First RAG Application with Python
Put the concepts together and build your first working RAG application using Python.
8. RAG with OpenAI API
Learn how to connect a RAG pipeline with an LLM API and use retrieved information to generate responses.
Follow TheTechLane
Keep learning AI, development, and modern developer technologies with TheTechLane.
Learn • Build • Share
Follow @thetechlanehq for more developer guides, tutorials, and practical technology content.