← home

Beyond accuracy - evaluating recommender systems by coverage and serendipity

2022/01/18

1. Coverage

  • The coverage of a recommender is a measure of the domain of items over which the system can make recommendations
  • 2 concepts
    1. The percentage of the items for which the system is able to generate the recommendation (referred as "prediction coverage")
    2. The percentage of the available items which effectively are ever recommended to a user [4,11] (referred as "catalog coverage")

2. Serendipity

  • Serendipity is a measure which recommended items are both attractive and surprising to users
  • 2 aspects
    1. A serendipitous item should be not yet discovered and not be expected by the user
    2. The item should also be interesting, relevant, and useful to the user
2.1 Metrics
2.1.1 Unexpected

We assume that a primitive prediction model shows high ratability and produces low unexpectedness.

Unexpected = RS/PM

RS = The recommendations generated by a recommender system
PM = A set of recommendations generated by a primitive prediction model
2.1.2 Useful
Serendipity = sum(u(RS_i)) where i is 1 through N / N

u(RS_i) is a usefulness for the recommendation  if the recommendation in an unexpected list, and usefulness function can be judge by the user.

3. Tradeoff

  • Catalog coverage usually decreases as a function of accuracy
  • Not every increase in coverage leads to serendipity
  • Increase in serendipity will lead to higher catalog coverage
  • Increase in accuracy leads to less serendipity
  • Increasing serendipity might negatively impact the accuracy
  • Effective Catalog SizeEffective Catalog Size
    A metric that describes how spread viewing is across the items in our catalog. If most viewing comes from a single item, it will be close to 1. If all items generate the same amount of viewing, it ...
  • Recommender System - CoverageRecommender System - Coverage
    Supports


    [[Beyond accuracy - evaluating recommender systems by coverage and serendipity]]

    Coverage

    Prediction Coverage
    Catalogue Coverage

    ...
  • Recommender System - SerendipityRecommender System - Serendipity
    Supports


    [[Beyond accuracy - evaluating recommender systems by coverage and serendipity]]

    Serendipity - Unexpected and usefulness (usefulness judged by user)


    [[Being accura...

Reference

M. Ge, C. Delgado-Battenfeld, and D. Jannach, “Beyond accuracy: evaluating recommender systems by coverage and serendipity,” in Proceedings of the fourth ACM conference on Recommender systems - RecSys ’10, Barcelona, Spain, 2010, p. 257. doi: 10.1145/1864708.1864761.

#recommender-system #recommender-system/quality #recommender-system/coverage #recommender-system/serendipity