Estimation Methods for Ranking and Recommendation: Addressing Bias, Variance and Efficiency
by Norman Knyazev
Evaluation and optimization of ranking and recommendation systems involves estimating various quantities such as relevance, gradients or probabilities. Importantly, different estimation procedures may vary in terms of their statistical and computational properties, which can in turn significantly affect the downstream performance for a given task. In this thesis, we study ranking and recommendation estimation methods with a focus on some of their key properties: bias, variance, consistency and efficiency. We first explore the bandwagon effect – a cognitive bias whereby users’ ratings may be affected by those of other users. Through our theoretical analysis we investigate whether the bandwagon effect is a problem of statistical bias, or is better described in terms of other statistical properties, and propose methods to reduce its impact on relevance estimation. In the second research chapter, we focus on estimation of document placement probabilities under the Plackett-Luce (PL) ranking model for off-policy evaluation (OPE). To this end, we propose a novel, accurate and sample- and computationally efficient estimation method by leveraging the connection of the PLranking model to Gumbel and Poisson binomial distributions, allowing for more data-efficient and lower-variance OPE. In the third research chapter, we introduce the variable presentation length setting, whereby document order and presentation lengths have to be decided jointly. We explore the theoretical implications and the unique challenges of this setting, and propose a highly sample- and computationally efficient extension of the PL ranking model tailored to this task. Our final research chapter looks at the use of the variance over the predicted ratings to represent model confidence. We employ the beta distribution to jointly model user preference and prediction confidence, avoiding significant computational complexity or reduced accuracy on the original rating prediction problem, whilst simultaneously allowing us to successfully leverage the learned confidence values for downstream use. In its entirety, this thesis demonstrates that statistical and computational properties of estimators play a significant role in learning from user interactions. It highlights their importance across a wide range of settings and provides actionable guidance for developing estimation methods for ranking and recommendation that address bias, variance, consistency and efficiency.

