Guides

Vector Stores

Vector stores contain vector embeddings for efficient retrieval and querying of your data.

Suggest Edits

Introduction

What is a vector store? A vector store is a database specifically for storing vector embeddings, which are numerical representations of data, including text, image, and audio. These vectors are stored such that ingestion of new vectors and retrieval of similar vectors is efficient. This mechanism is key for being able to quickly query across a large amount of data. You can use our EGP APIs to make your own vector stores for your data. Either upload vector embeddings directly, or create a knowledge base and upload files (or make data connectors) for a streamlined process.

How are vector stores related to knowledge bases? Knowledge bases serve as a wrapper around vector stores: every knowledge base has one vector store under the hood. Any data that is imported or ingested into a knowledge base (e.g. through a data connector) will be embedded and stored in its vector store. If you are using a knowledge base, it is unlikely you should ever need to make any calls directly to its underlying vector store.

How do vector embeddings work? In a vector storage system, each text chunk and query is represented as a high-dimensional vector. A common method to derive these vectors is by using OpenAI's embedding endpoint or other older techniques like word embeddings (e.g., Word2Vec, GloVe) or sentence embeddings (e.g., BERT, Universal Sentence Encoder). Each dimension in the vector corresponds to a specific term or feature, and its value signifies the importance of that term in the chunk or query.

How does vector similarity search work? We use cosine similarity search to retrieve text chunks with embeddings that are most similar to a given query (that has been converted into a text chunk with embedding vector). The text chunks are ranked based on their similarity scores, and the top-ranked chunks are returned as search results.

[Eventually include diagram here on vector store embeddings + retrieval]

Get Started with the Vector Store API

The starter code below creates an EGP vector_store using the Create Vector Store endpoint. Fill in your API key and choose a name for your vector store, and try out the starter code below!

import requests

# Replace this with your Spellbook API key
# See instructions for getting your API key here: scale-egp.readme.io/docs/getting-started
API_KEY = '[Your API key]'

# Choose a name for your vector store
VECTOR_STORE_NAME = "my_vector_store"

print(f"Creating a vector store named {VECTOR_STORE_NAME}...")
url = "https://api.spellbook.scale.com/egp/v1/vector_stores"
payload = {
  "vector_store_name": VECTOR_STORE_NAME
}
headers = {
    "accept": "application/json",
	  "content-type": "application/json",
    "x-api-key": API_KEY
}
response = requests.post(url, json=payload, headers=headers)
print(f"Response: {response.text}")

You should see a response like the following, which will return your vector store ID:
{"vector_store_id":"clk123456789"}

Keep this vector store ID—you will need it later when getting information about your vector store, uploading text chunks with embeddings, and more.

What's Next?

Now that you have a vector store, you can perform the following operations on it:

Updated 8 months ago