← home

Measuring the business value of recommender systems

2022/01/17

What is being measured

  • Click Through RateClick Through Rate
    Click-Through-Rate (CTR) is the number of clicks that the item receives divided by the number of times the item is shown.

    clicks / impressions = CTR


    If we have 5 clicks and 100 impressions, our ...
    • Easy to measure and established
    • Not the ultimate goal
  • Adoption and Conversion RatesAdoption and Conversion Rates
    Adoption and Conversion Rates often require a domain- and application-specific definition.

    Supports

    [[The Youtube video recommendation system 2010]]

    Long CTR - only counting clicks t...
    • Easy to measure
    • Requires a domain-specific definition
    • Requires interpretation
  • Sales and RevenueSales and Revenue
    [12]

    Supports

    COMMUNITY-BASED RECOMMENDER SYSTEMS: ANALYZING BUSINESS MODELS FROM A SYSTEM OPERATORS PERSPECTIVE

    Problem
    Baseline
    Result
    Reference


    Price a...
    • Most informative measure
    • Cannot always be determined directly
  • Effects on Sales Distributions
    • Direct measurement
    • Requires understanding of the effects of the shifts in sales distribution
  • User Behavior and Engagement
    • Remain an approximation
    • Correspondence between user engagement and customer retention is assumed
  • Others
    • Effective Catalog SizeEffective Catalog Size
      A metric that describes how spread viewing is across the items in our catalog. If most viewing comes from a single item, it will be close to 1. If all items generate the same amount of viewing, it ...

Baseline algorithm

  • No recommender
  • Popularity-based
    • Age discounted click count
    • Most viewed videos within a day
    • Popular within a given timeframe
    • Top Favorited
    • Top Rated
  • Random-based
    • Random from users search
    • Random from the whole catalog
  • Simple model
    • Linear model
  • Current production baseline model
    • Co-viewed or Co-visitation
    • Collaborative filtering
    • Context-tree

The challenge of predicting business success from offline experiments

Recommender System Quality FactorsRecommender System Quality Factors

[[Recommender System - Accuracy or Relevance]]

The recommendation suggested to a user should be relevant because that a user has a high propensity to purchase the recommended items an...

Metrics, Engagement, and Personalization at SpotifyMetrics, Engagement, and Personalization at Spotify

Optimization metrics quantify how users engage within a session and act as proxy of satisfaction

save
long click
post
follow
click to stream
impression t...

Reference

X. Amatriain and J. Basilico, “Recommender Systems in Industry: A Netflix Case Study,” in Recommender Systems Handbook, F. Ricci, L. Rokach, and B. Shapira, Eds. Boston, MA: Springer US, 2015, pp. 385–419. doi: 10.1007/978-1-4899-7637-6_11.

#business-value #recommender-system