Vector Embeddings Explained: How Text Becomes Numbers a Computer Can Compare

Vector Embeddings Explained: How Text Becomes Numbers a Computer Can Compare

Computers cannot compare meaning. They compare numbers. An embedding is the bridge: a function that turns a piece of text into a fixed-length list of numbers, arranged so that texts meaning similar things end up with similar numbers.

"how do I cancel my subscription"  ->  [0.021, -0.114, 0.087, ... ]   (1024 numbers)
"where do I stop my plan"          ->  [0.019, -0.109, 0.091, ... ]   (very close)
"what is the boiling point of tin" ->  [-0.204, 0.377, -0.052, ... ]  (far away)

The first two sentences share no significant words. A keyword search for "cancel" would miss the second one entirely. Their embeddings sit almost on top of each other, because the model that produced them was trained on enough text to learn that stopping a plan and cancelling a subscription are the same request.

That is the whole idea. Everything below is detail.

What the numbers actually are

An embedding model is a neural network with the last layer removed. You feed it text, it processes it the way it would before predicting something, and instead of a prediction you take the internal state and keep it.

That internal state is a vector: an ordered list of floating point numbers. Its length is fixed by the model, not by the input. A three-word question and a three-hundred-word paragraph both come out as the same size vector.

Common sizes:

More dimensions means more room to encode distinctions, at the cost of storage and comparison speed. A million 1536-dimension vectors stored as 32-bit floats is about 6GB. The same million at 384 dimensions is 1.5GB. That difference decides architectures.

Individual numbers mean nothing on their own. There is no "dimension 400 is how formal the text is". The meaning lives in the whole vector's position relative to other vectors, not in any single coordinate.

Measuring closeness

Once text is a vector, "similar meaning" becomes "close together", and closeness is arithmetic.

The usual measure is cosine similarity: the cosine of the angle between two vectors. It ignores length and looks only at direction, which is what you want, because a long document and a short question about it should count as similar if they point the same way.

cosine(A, B) = (A · B) / (|A| × |B|)

where A · B  = a₁b₁ + a₂b₂ + ... + aₙbₙ    (dot product)
      |A|    = √(a₁² + a₂² + ... + aₙ²)     (magnitude)

The result runs from -1 to 1:

 1.0   identical direction, same meaning
 0.8   strongly related
 0.5   loosely related
 0.0   unrelated, perpendicular
-1.0   opposite direction

In practice, with modern models, almost nothing scores below zero. Real text is not the opposite of other real text; it is just unrelated. Useful thresholds usually sit between 0.3 and 0.7, and where exactly depends on the model, so measure rather than copy a number from a blog post.

Many models return normalised vectors, where every vector has a magnitude of exactly 1. When that holds, the denominator is 1 and cosine similarity reduces to a plain dot product, which is considerably faster. Most vector databases assume this and normalise for you.

Worked example

Three dimensions instead of a thousand, so the arithmetic is visible:

A = [1, 0, 1]      "reset password"
B = [1, 0.2, 0.9]  "forgot my login"
C = [0, 1, 0]      "shipping times"

A · B = (1)(1) + (0)(0.2) + (1)(0.9) = 1.9
|A|   = √(1 + 0 + 1)         = 1.414
|B|   = √(1 + 0.04 + 0.81)   = 1.360
cosine(A, B) = 1.9 / (1.414 × 1.360) = 0.988   very similar

A · C = 0.  cosine(A, C) = 0                    unrelated

Real embeddings do exactly this with a thousand more terms.

What they are used for

Semantic search. The original application. Embed the query, embed the documents, return the closest. Finds "how do I stop being charged" when the page says "cancelling your plan".

Deduplication. Near-identical text produces near-identical vectors. Cheaper and more robust than comparing strings, because it catches rewording. If you only need exact duplicates, a hash is faster and exact; embeddings are for "these two say the same thing differently".

Clustering. Group support tickets, feedback, or search queries by meaning and see what people actually ask about, without deciding the categories in advance.

Classification. Embed labelled examples once, then classify new text by nearest neighbour. Often good enough without training a model.

Retrieval for language models. The one that changed everything, covered in its own post on how AI answers from your own documents.

Chunking, which decides more than the model does

You cannot embed a whole manual into one vector and expect it to work. Everything averages out and the vector ends up meaning "this is a manual". Documents get split first, and how you split them matters more than which model you pick.

• Too small (a sentence) and each chunk loses the context that made it meaningful. "It costs $29 a month" is useless without knowing what "it" is.
• Too large (a whole chapter) and one relevant paragraph is diluted by ten irrelevant ones.
• Somewhere between 200 and 800 words is the usual sweet spot, split on real boundaries such as headings or paragraphs rather than a fixed character count.

Overlap helps. Repeating the last sentence or two of the previous chunk at the start of the next stops an answer being cut in half by an arbitrary boundary.

Splitting on structure beats splitting on length. If your source is Markdown, split on headings: the author already told you where the ideas begin. You can check how a document is structured before you process it with a Markdown previewer, and convert HTML to Markdown first if you are working from web pages.

Where embeddings fail

Exact identifiers. Embeddings encode meaning, and an order number has none. "Where is order 4417" and "where is order 9982" are nearly identical vectors. Anything involving codes, SKUs, error numbers or version strings needs keyword matching, not semantic matching. Most serious systems run both and merge the results, which is called hybrid search.

Negation. "The refund is available" and "the refund is not available" are closer than they should be. The word "not" is one small signal in a long sentence. Do not rely on embeddings to distinguish a claim from its opposite.

Domain drift. A model trained on general web text knows that "python" is usually a language. It may not know that in your product "Python" is a subscription tier. Jargon specific to your company is exactly where general models are weakest.

Long text. Every model has a token limit and silently truncates beyond it. Chunk deliberately rather than discovering the cutoff by accident.

Cross-language. Some models place the same sentence in two languages near each other; many do not. Test before assuming.

Cost and practical notes

Embedding is cheap compared with generation. Typical API pricing is a small fraction of a cent per thousand tokens, and a whole documentation site usually costs less than a coffee to index once.

Two things save real money:

Cache by content hash. If the text has not changed, the vector has not changed. Hash each chunk, skip anything you have already embedded, and re-indexing a site where three pages changed costs three pages, not the whole site.

Embed the right thing. For search, embedding a chunk plus its heading often beats embedding the chunk alone, because the heading carries context the paragraph assumes.

One constraint that catches people: vectors from different models are not comparable. Switching embedding models means re-embedding everything. The numbers from one model have no meaning in another model's space, and mixing them silently produces nonsense rather than an error.

Trying it yourself

The fastest way to build intuition is to embed a handful of sentences and print the similarity matrix. Take five sentences, three about the same topic and two about something else, and check that the numbers agree with your judgement. They usually will, and the cases where they do not will teach you more than the cases where they do.

If you want to work with vector data as JSON while experimenting, a JSON formatter makes large arrays readable, and JSON to CSV helps when you want to eyeball similarity scores in a spreadsheet.

Embeddings are not intelligence. They are a very good measurement of "these two texts are about the same thing", and almost everything interesting built on top of language models is that measurement applied carefully.