Fundamentals: metrics, overfitting, and data splitting

Description

Qué haceIntroduces the tools used to tell whether a model works: data splitting, metrics, and controlling complexity.

Para qué sirveDistinguishing a model that generalizes from one that only looks good because it was evaluated with the wrong data or memorized the training set.

The core idea

A model isn’t evaluated by asking how much it remembers, but how well it responds to data it never saw. To check that, you need to split the data, pick an appropriate metric, and watch that complexity doesn’t turn learning into memorization.

Train, validation, and test

Training (train)

The model uses these examples to fit its parameters.

Validation

Lets you pick hyperparameters and complexity without looking at the final result.

Test

Used exactly once, at the end, to estimate how the model will perform on new data.

Data leakage

Happens when information from validation or test influences training. The result looks better than it actually is.

In this demo:Module 3 picks the polynomial degree by looking only at the validation error, never the test error.

How to read a classification

Accuracy

(TP+TN) / total

Overall proportion of correct predictions. Can be misleading if one class is much more frequent.

Precision

TP / (TP+FP)

Of everything the model flagged as positive, what proportion actually was.

Recall

TP / (TP+FN)

Of all the real positives, what proportion the model managed to find.

F1

2 × (precision × recall) / (precision + recall)

Summarizes precision and recall into one value; it will be low if either one is low.

Undefined, not zero:With a zero denominator (e.g. the model never predicted the positive class), the metric is shown as “undefined,” never as 0% or NaN.

Complexity and generalization

Underfitting · too simple

The model can't represent the pattern. It makes errors both in training and in validation.

Good balance

Error is low and similar in training and validation: what was learned transfers to new data.

Overfitting · too complex

The model memorizes details and noise. It performs very well while training but gets worse on validation data.

One split doesn't tell the whole story:The result can change depending on which samples land in each group. Cross-validation repeats this process over several splits to get a more stable estimate.

Playground

01

Split

Data splitting

Splits the dataset into three groups with different responsibilities. Test is reserved for the final evaluation and doesn't take part in any earlier decision.

Split result

90 synthetic samples · seed 7

Training: 63
Validation: 14
Test: 13

Table of synthetic dataset samples with the split (training, validation, or test) assigned to each one.
#	x	y	Split
0	-3.91	1.67	Training
1	1.59	2.04	Test
2	-0.27	-1.65	Training
3	1.84	3.09	Training
4	2.11	2.53	Training
5	-1.06	-3.18	Test
6	-1.63	-2.60	Training
7	3.02	1.52	Training
8	-2.76	-2.21	Training
9	2.09	3.49	Validation
10	0.43	0.83	Validation
11	3.73	0.37	Training
12	2.62	2.25	Test
13	-1.14	-1.58	Training
14	-3.30	-0.69	Validation
15	-0.59	-0.74	Training
16	1.63	2.35	Training
17	-1.15	-1.57	Validation
18	3.30	0.71	Validation
19	1.06	3.11	Test
20	1.47	2.36	Test
21	2.61	2.06	Validation
22	-3.06	-1.17	Training
23	3.13	0.75	Training
24	0.75	1.94	Training
25	0.39	1.36	Training
26	0.13	0.19	Training
27	0.40	1.45	Validation
28	0.46	0.82	Training
29	1.14	1.91	Training
30	-1.50	-1.58	Training
31	-3.81	0.31	Training
32	1.04	0.96	Training
33	0.76	2.28	Training
34	-1.78	-2.62	Test
35	2.44	2.14	Validation
36	-2.89	-1.15	Training
37	0.54	0.41	Training
38	-2.31	-1.80	Training
39	1.89	3.05	Test
40	0.87	2.44	Training
41	-0.09	-1.11	Training
42	2.37	2.10	Training
43	0.44	0.38	Training
44	-2.56	-2.59	Test
45	-0.18	0.66	Test
46	-2.61	-0.29	Training
47	3.01	0.97	Training
48	-2.60	-2.39	Training
49	1.86	1.98	Training
50	2.33	2.18	Training
51	0.24	0.27	Training
52	0.50	0.32	Training
53	2.21	2.17	Training
54	2.61	1.63	Training
55	-0.56	-0.63	Training
56	-1.99	-3.27	Training
57	2.64	2.32	Training
58	0.41	0.63	Training
59	0.30	1.46	Test
60	0.58	1.47	Test
61	-0.03	-0.34	Training
62	3.45	-0.19	Training
63	-2.70	-1.82	Training
64	0.88	1.75	Training
65	-1.64	-3.14	Training
66	1.86	2.46	Training
67	-1.92	-2.74	Training
68	3.41	0.42	Training
69	3.70	0.90	Validation
70	-1.74	-3.10	Training
71	1.71	2.27	Training
72	1.80	2.07	Training
73	-1.82	-2.90	Validation
74	-2.88	-1.57	Training
75	0.48	1.69	Training
76	3.62	0.89	Validation
77	-1.70	-3.12	Training
78	3.28	0.08	Validation
79	3.26	-0.52	Training
80	3.41	0.37	Test
81	-1.33	-2.26	Training
82	2.74	2.37	Training
83	2.02	3.51	Validation
84	3.29	-0.43	Training
85	-1.90	-2.57	Test
86	-3.54	0.71	Validation
87	-3.34	-1.30	Training
88	2.31	1.98	Training
89	-0.02	-1.31	Training

02

Measure

Classification metrics

Change the matrix and see which question each metric answers. The imbalanced preset shows why high accuracy doesn't always mean the model is useful.

Matrix preset

Classes of similar size; precision, recall, and F1 stay close to accuracy.

Invert positive class

Rows are the model's predictions and columns are the actual values. Diagonal cells are hits; the other two are errors.
Predicted ↓ / Actual →	Positive	Negative
Predicted positive	TPhit	FPmiss
Predicted negative	FNmiss	TNhit

Shortcut: Ctrl/Cmd+Z undoes the last confirmed edit (if focus isn't in a text field).

Metrics computed from the matrix

Accuracy: 89.0%
Precision: 88.2%
Recall: 90.0%
F1: 89.1%
Specificity: 88.0%

Change the threshold

ROC and precision-recall curves

What is the threshold?

It's the minimum score to predict “positive.” Lowering it surfaces more positives: recall goes up, but false alarms can increase too.

What does each curve show?

ROC compares recall against false alarms. Precision-recall shows how much recall you get without losing precision and tends to be more informative with imbalanced classes.

ROC curve over 100 synthetic samples derived from the current matrix. AUC = 0.905.

Precision-recall curve over the same samples. Area under the curve (average precision) = 0.870.

Area under the curve

AUC-ROC: 0.905
AUC-PR (average precision): 0.870

Note: the curves use 100 reproducible synthetic scores derived from the preset (45 TP, 6 FP, 5 FN, and 44 TN). At a 0.5 threshold they reproduce the matrix shown above.

03

Diagnose

Underfitting and overfitting

Increase the polynomial degree and compare training with validation. The goal isn't minimizing training error at any cost, but keeping a good result on unseen data.

Complexity (polynomial degree)3

Degree 1 draws a straight line. As the degree increases, the curve can fit more complex patterns — and noise too.

18

63 training samples, 14 validation samples. Degree-3 polynomial curve fitted on the training set.

Training
Validation
Fit (degree 3)

Training vs. validation error

Training MSE: 0.3768
Validation MSE: 0.7222
Training R²: 0.882
Validation R²: 0.750

Diagnosis

Reasonable balance

Training and validation error are similar: the model generalizes reasonably to data it didn't see while fitting.

The test set (split off in the partitioning module) doesn't take part here: it's reserved for a single final evaluation after choosing the degree, not for deciding which degree to use.