Ranking metrics: compare precision at k and NDCG

Precision at k counts how many of the first k recommendations are relevant. Normalized discounted cumulative gain, or NDCG, also rewards placing relevant items earlier. The two metrics describe different aspects of a ranked list.

Compute both from one list

Suppose items A and C are relevant, the system ranks [B, A, D, C], and the visible cutoff is three. The top-three hit vector is [0,1,0].

Precision@3 is one hit divided by three slots, or 0.3333. With binary relevance, DCG@3 discounts the hit at rank two by 1/log2(3), giving 0.63093. The ideal top-three vector is [1,1,0], whose DCG is 1.63093. NDCG@3 is therefore 0.38685.

Item C at rank four does not affect either metric at cutoff three. A different interface depth would produce a different evaluation question.

from unsupervised_cases import ranking_metric_case

r = ranking_metric_case()
assert r['hits'] == [0, 1, 0]
assert abs(r['precision_at_k'] - 1/3) < 1e-12
assert abs(r['ndcg_at_k'] - .38685280723454163) < 1e-12
print(r)

Run the exact arithmetic in the unsupervised lab. The example uses binary relevance; graded relevance changes the gain term.

Define relevance and averaging

Offline relevance often means a held-out interaction. Missing interaction is not necessarily dislike, and historical exposure affects what can become a positive. State whether purchased, clicked, completed or rated events count and over what horizon.

For users with no relevant test item, decide whether to exclude them, assign zero or report a separate cold-start metric. Macro averaging gives users equal weight; micro aggregation favours users with more events. Document the choice.

NDCG depends on the discount and ideal ranking. Precision@k ignores order within the first k. Recall@k measures recovered relevant items and can be more informative when users have different relevant-set sizes. Add catalogue coverage and business guardrails so relevance gains do not hide concentration or harm.

Do not tune k after seeing test results. Choose it from the interface and validation data, then evaluate the frozen ranking policy on a temporal test.

Exercise: move A to rank one, add graded relevance of three for C and compare precision, recall and NDCG at two cutoffs. Implement the formulas directly and cross-check a library without changing definitions midway.

NeuraPath's Data Science course teaches ranking metrics from exact lists and denominators. The formula is meaningful only after relevance and display depth are fixed.

Continue learning

This article is part of the Clustering, reduction and recommendations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate recommendations with temporal holdouts.
• Continue with Popularity bias in a recommendation system.