Text similarity with TF-IDF

Description

Qué haceTurns each text into a list of numbers that gives more weight to words that are frequent in that text but rare across the rest of the documents.

Para qué sirveCompare or search documents by content similarity (search engines, deduplication, article recommendations) without needing a trained language model.

Represents each text as a vector of per-word weights and compares vectors by cosine similarity. A lexical, deterministic pipeline, with no models or embeddings.

Tokenization, TF, IDF, and similarity

Tokenization

Normalizes (Unicode NFC, lowercase) and extracts sequences of letters and numbers.

TF

count / tokens in the document

What proportion of the document that word is.

IDF

ln((1+N) / (1+df)) + 1

How rare the word is across the corpus. Never reaches 0 or infinity.

Cosine similarity

TF-IDF · TF-IDF / (‖a‖ ‖b‖)

Angle between vectors: 1 if identical, 0 if they share no terms.

Intentional:No stemming, lemmatization, or stopwords: every token counts exactly as written (see “Lexical match vs. meaning”).

What each table shows

Query vector

Query terms present in the vocabulary, with count, TF, IDF, and weight.

Ranking and shared terms

Documents sorted by descending similarity; the shared terms explain what that number measures.

Lexical vs. meaning

Only compares exact words

Synonyms (“doctor” → “physician”) don't share a term, even when they describe the same thing — see the dedicated example.

No embeddings

An embeddings model would bring those synonyms closer together; this demo doesn't implement one.

In exchange: transparency

Every number traces back to a word count, cheap to compute, and ideal for keyword search.

Playground

Query

29 / 200 characters. The query is saved in the URL so the link is shareable: don't type sensitive information here.

recipewithpotatoesandeggs

Query vector (TF, IDF, and weight per term)

Query terms with their count, TF, IDF, and TF-IDF weight, sorted from highest to lowest weight.
Term	Count	TF	IDF	TF-IDF weight
eggs	1	0.200	1.693	0.339
potatoes	1	0.200	1.693	0.339
recipe	1	0.200	1.693	0.339
with	1	0.200	1.693	0.339
and	1	0.200	1.288	0.258

Ranking by cosine similarity

Documents sorted by descending cosine similarity to the query, with the shared terms that explain each score.
#	Document	Similarity	Shared terms
1	Doc. 2: Recipe for gazpacho with tomato, cucumber, pepper, garlic, and day-old bread.	0.354	recipewithand
2	Doc. 1: How to make a Spanish potato omelette: you need potatoes, eggs, onion, and olive…	0.316	eggspotatoesand
3	Doc. 3: The internal combustion engine converts thermal energy into mechanical energy th…	0.000	none

Corpus summary

Documents: 3
Vocabulary size: 37
Max similarity: 0.354

Vocabulary (37 terms)

aandbreadcombustionconvertscucumberdayeggsenergyengineforgarlicgazpachohowinternalintomakemechanicalneedoiloldoliveomeletteonionpepperpistonspotatopotatoesrecipespanishthethermalthroughtotomatowithyou

Text, tokens, and weights per document

Document 1

How to make a Spanish potato omelette: you need potatoes, eggs, onion, and olive oil.

howtomakeaspanishpotatoomeletteyouneedpotatoeseggsonionandoliveoil

Document 1 terms with their count, TF, IDF, and TF-IDF weight.
Term	Count	TF	IDF	TF-IDF weight
a	1	0.067	1.693	0.113
eggs	1	0.067	1.693	0.113
how	1	0.067	1.693	0.113
make	1	0.067	1.693	0.113
need	1	0.067	1.693	0.113
oil	1	0.067	1.693	0.113
olive	1	0.067	1.693	0.113
omelette	1	0.067	1.693	0.113

Document 2

Recipe for gazpacho with tomato, cucumber, pepper, garlic, and day-old bread.

recipeforgazpachowithtomatocucumberpeppergarlicanddayoldbread

Document 2 terms with their count, TF, IDF, and TF-IDF weight.
Term	Count	TF	IDF	TF-IDF weight
bread	1	0.083	1.693	0.141
cucumber	1	0.083	1.693	0.141
day	1	0.083	1.693	0.141
for	1	0.083	1.693	0.141
garlic	1	0.083	1.693	0.141
gazpacho	1	0.083	1.693	0.141
old	1	0.083	1.693	0.141
pepper	1	0.083	1.693	0.141

Document 3

The internal combustion engine converts thermal energy into mechanical energy through pistons.

theinternalcombustionengineconvertsthermalenergyintomechanicalenergythroughpistons

Document 3 terms with their count, TF, IDF, and TF-IDF weight.
Term	Count	TF	IDF	TF-IDF weight
energy	2	0.167	1.693	0.282
combustion	1	0.083	1.693	0.141
converts	1	0.083	1.693	0.141
engine	1	0.083	1.693	0.141
internal	1	0.083	1.693	0.141
into	1	0.083	1.693	0.141
mechanical	1	0.083	1.693	0.141
pistons	1	0.083	1.693	0.141

English examples

Each example replaces the current query and documents with a complete collection and its starting query.

Edit documents

3 / 6 documents.

Document 1
How to make a Spanish potato omelette: you need potatoes, eggs, onion, and olive oil.
85 / 240 characters
Document 2
Recipe for gazpacho with tomato, cucumber, pepper, garlic, and day-old bread.
77 / 240 characters
Document 3
The internal combustion engine converts thermal energy into mechanical energy through pistons.
94 / 240 characters

Shortcut: Ctrl/Cmd+Z undoes the last confirmed edit (if focus isn't in a text field).