Understanding and improving automated collaborative filtering systems

Abstract

Automated collaborative filtering (ACF) is a software technology that provides personalized recommendation and filtering independent of the type of content. In an ACF system, users indicate their preferences by rating their level of interest in items that the system presents. The ACF system uses the ratings information to match together users with similar interests. Finally, the ACF system can predict a user's rating for an unseen item by examining his neighbors' ratings for that item. This dissertation presents a set of results with the goal of improving the effectiveness and understanding of ACF systems. The results cover four challenges: understanding and standardizing evaluation of ACF systems, improving the accuracy of ACF systems, designing and utilizing effective explanations for ACF predictions, and improving ACF to support ephemeral recommendations. To address these challenges, a combination of offline analysis and user testing is used. All of the evaluation metrics that have been proposed for ACF are examined theoretically and compared empirically. The empirical results show that all proposed ACF evaluation metrics perform similarly, which argues for the adoption of a standardized evaluation metric—for which I propose mean absolute error. With respect to improving algorithm accuracy, I present a detailed empirical examination of the neighborhood-based prediction algorithm, which has been the most successful algorithm, both in research and in commercial applications. ACF systems predict based on data of variable quantity and quality, but current ACF systems are black boxes, so users have no indication of when to trust an ACF prediction. Explanations expose some of the process and data behind the ACF prediction, allowing users to judge if a prediction is appropriate for their current context of risk. I present results showing what forms of explanation users find the most compelling, as well as indications that explanations can increase the acceptance of ACF systems. Finally, I present results from tests of a new algorithm for supporting focused ephemeral user information needs. Ephemeral information needs are those needs that are immediate, focused, and often temporary. The proposed algorithm provides support for ephemeral information needs using no additional data beyond the standard ACF ratings.

Keywords

Related Publications