Research notes
The handful of equations that keep coming up, with a one-line reading of each.
The handful of equations that keep coming up, with a one-line reading of each. Notation: Σ is a sum, E[·] an expected value, log the natural logarithm.
softmax(z)_i = exp(z_i) / Σ_j exp(z_j)
Turns a vector of raw scores z into positive numbers that sum to 1 — a probability distribution over classes. The exponential exaggerates differences, so the largest score dominates.
References: Bishop, Pattern Recognition and Machine Learning (Springer, 2006), §4.3.4; Goodfellow et al., Deep Learning (2016), §6.2.2.
L = − Σ_c y_c · log(ŷ_c) (multi-class)
L = −[ y·log(p) + (1−y)·log(1−p) ] (binary)
Penalises a prediction by how much probability it put on the wrong answer. If the model is confident and right the loss is near 0; confident and wrong sends it toward infinity. This is the loss behind almost every classifier here.
References: Goodfellow et al., Deep Learning (2016), §3.13 and §5.5; Bishop, PRML (2006), §4.3.4.
θ ← θ − η · ∇θ L(θ)
Repeatedly step the parameters θ a small amount (η, the learning rate) in the direction that most decreases the loss. Backpropagation is just how ∇θ L is computed for a neural network.
References: Cauchy, “Méthode générale pour la résolution des systèmes d’équations simultanées” (1847); Ruder, “An overview of gradient descent optimization algorithms,” 2016 — arXiv:1609.04747.
min_G max_D Ex∼p_data[ log D(x) ] + Ez∼p_z[ log(1 − D(G(z))) ]
The discriminator D wants to output 1 on real data and 0 on generated data; the generator G wants to make D(G(z)) close to 1. At the ideal solution the generator’s distribution matches the data and D can only guess.
References: Goodfellow et al., “Generative adversarial nets,” NeurIPS 2014 — arXiv:1406.2661.
D_KL(P ‖ Q) = Σ_x P(x) · log( P(x) / Q(x) )
JSD(P ‖ Q) = ½ D_KL(P ‖ M) + ½ D_KL(Q ‖ M), M = ½(P + Q)
Both measure how far apart two probability distributions are. KL is asymmetric and unbounded; JS is symmetric and bounded, and the original GAN objective is (up to a constant) minimising the JS divergence between the real and generated data. These are also the natural way to score how faithful synthetic data is.
References: Kullback & Leibler, “On information and sufficiency,” Annals of Mathematical Statistics 22(1), 1951; Lin, “Divergence measures based on the Shannon entropy,” IEEE Trans. Information Theory 37(1), 1991.
AUC = P( score(x⁺) > score(x⁻) )
Sweep the decision threshold and plot true-positive rate against false-positive rate — that is the ROC curve. The area under it (AUC) has a clean reading: the probability that the model gives a randomly chosen positive case a higher score than a randomly chosen negative one. 0.5 is chance, 1.0 is perfect. Macro AUC averages this over classes equally; weighted AUC weights by class size.
References: Hanley & McNeil, “The meaning and use of the area under a ROC curve,” Radiology 143(1), 1982; Fawcett, “An introduction to ROC analysis,” Pattern Recognition Letters 27(8), 2006.
recall@k = | relevant ∩ top-k | / | relevant |
precision@k = | relevant ∩ top-k | / k
For a ranked shortlist of length k: recall@k is the share of the things that should have been found that appear in the top k; precision@k is the share of the top k that are actually relevant. Raising k trades precision for recall.
References: Manning, Raghavan & Schütze, Introduction to Information Retrieval (Cambridge, 2008), ch. 8 — nlp.stanford.edu/IR-book.
All formulas below are from the survey by Jadon & Patil (2024), covered in Related work, where I map each metric family onto my own TeleNotArzt evaluation.
cos(θ) = ( Σ A_i·B_i ) / ( √ΣA_i² · √ΣB_i² )
Angle between two vectors A, B (e.g. user or item profiles), ignoring their magnitude. Ranges −1 to 1; 1 means same direction.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.
d(A,B) = √( Σ (A_i − B_i)² )
Straight-line distance between two points in feature space; 0 means identical. Used to compare numeric-attribute profiles rather than orientation alone.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.
J(A,B) = | A ∩ B | / | A ∪ B |
Overlap between two sets (e.g. two users’ liked items) relative to their union. Ranges 0 to 1; suited to binary or implicit-preference data.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.
r = Σ(X_i−X̄)(Y_i−Ȳ) / ( √Σ(X_i−X̄)² · √Σ(Y_i−Ȳ)² )
Linear correlation between two rating vectors, e.g. two users’ ratings on the same items. Ranges −1 to 1; the classic weight for user-based collaborative filtering.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.
Novelty(R_u) = (1/|R_u|) Σi∈R_u ( 1 − popularity_score(i) )
Average “unpopularity” of a user’s recommendation list R_u. Higher values mean the system is surfacing less obvious, less mainstream items.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.
Catalog Coverage = ( |unique items recommended| / |items in catalog| ) × 100%
Share of the entire item catalog that ever gets recommended, across all users — the catalogue-wide relative of the coverage ratio I use in the TeleNotArzt paper.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.
RMSE = √( (1/N) Σ (y_i − ŷ_i)² )
MAE = (1/n) Σ | y_i − ŷ_i |
MAPE = (100%/n) Σ | y_i − ŷ_i | / y_i
Three ways to score rating-prediction error. RMSE squares errors first, so large misses cost more; MAE weighs every error equally; MAPE expresses the error as a percentage of the true value, which makes it comparable across differently-scaled datasets.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.
R² = 1 − SS_res / SS_tot
SS_res = Σ(y_i−ŷ_i)², SS_tot = Σ(y_i−ȳ)²
Share of the variance in the outcome that the model explains. 1 is a perfect fit; 0 means the model is no better than always predicting the average.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.
Precision@k = (# relevant in top-k) / k
Recall@k = | Relevant ∩ Top-k | / | Relevant |
F1@k = 2 × (Precision@k × Recall@k) / (Precision@k + Recall@k)
The survey’s own notation for the two metrics I used in the TeleNotArzt paper, plus F1@k, their harmonic mean — a single number when you need one ranking headline instead of the precision/recall trade-off.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.
MRR = (1/|Q|) Σi=1|Q| 1/rank_i
Average, over all queries/cases Q, of one over the position of the first relevant item. Rewards getting the right answer near the top rather than counting how many correct items appear.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.
DCG_p = Σi=1p (2rel_i − 1) / log₂(i + 1)
nDCG_p = DCG_p / iDCG_p
DCG rewards relevant items more the higher they rank, using graded relevance (rel_i) rather than a binary relevant/not-relevant call. nDCG normalises by the best possible ordering (iDCG) so scores are comparable across lists of different lengths.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.
MAP = (1/U) Σu=1U (1/|Relevant_u|) Σk Precision@k × rel_u(k)
Averages precision at every point a relevant item is retrieved for a user, then averages that across all U users — a single ranking-quality score for the whole system rather than one case at a time.
References: Jadon, A., & Patil, A. (2024). A comprehensive survey of evaluation techniques for recommendation systems. arXiv:2312.16015.