Thursday, 11 July 2013

Are your recommendations any good?

If you've worked much with recommender systems you'll know that offline evaluation of recommendation algorithms can bring on a serious headache in no time.  It was nice of Netflix to offer researchers $1M to improve their recommender system, but I think many of them would have taken part in the challenge even without a serious prize simply because the competition format means that someone else has to worry about preparing training and test data, and choosing a success measure, not to mention the pure mechanics of running evaluations over and over again.

With the benefit of hindsight of course it's clear now that rating prediction accuracy isn't a great way of evaluating a recommender system: it isn't applicable at all to most domains, where people can't or won't give ratings; it's not a very good predictor of the  relevance of the top few recommendations that are shown to users, and which they actually care about; even if you can measure relevance it's far from the only thing that influences uptake; and, last but not least, small changes to your user interface or improvements to your input data are both likely to have a much bigger impact than anything you might do to fix up your algorithm.  Consequently most practitioners would much prefer to do online A/B testing, which will tell you reliably whether or not any changes you've made are going to have a positive impact on your bottom line.

But unless you work for a huge company, online testing is a limited resource, especially compared to the vast parameter space you might need to explore when setting up a new recommender system.  And if you're an academic you may not be able to reach users easily, and you have to have some formal results to publish in any event.  So the need to do offline evaluation isn't going away any time soon.

Here are some of the pain points that I've bumped into recently:

  • public datasets aren't enough for (easily) reproducible research: take a look at the results reported here, here and here, all based on the same dataset, but with different experimental setups making for a lot of extra work to compare with them.
  • toolkits need to scale: MyMediaLite, for example, is a great tutorial resource for recommender systems, with clear, well thought out implementations of a host of algorithms, and evaluation code too, but it doesn't handle even moderately large datasets.
  • baseline methods need to be reproducible too: I just came across this impressive paper by Jagadeesh Gorla which introduces a genuinely novel approach to making recommendations and also reports great results, but I can't come close to replicating his baseline results; should I believe the results for his new method?
  • evaluation needs resources: generating full similarity matrices for many different parameter settings, or recommendations for all users in a big dataset, can be way more compute intensive than running a recommender service for real.
  • Hadoop doesn't help much: map-reduce isn't usually a good fit for machine learning algorithms, and the large resources needed by most recommender systems mean that you can't even resort to map-only driver jobs to launch code in parallel.
  • distributed processing without Hadoop is hard: I got excited about StarCluster and IPython.parallel, as in principle they make it dead easy to set up a Python cluster on AWS and run code on multiple worker processes, but in I've found that processes easily get into an unrecoverable state meaning frustration and wasted time tearing down clusters and setting up new ones.  In practice I've made faster progress by running parallel processes on the single biggest machine I can find locally.

What's your experience?

Are there some great tools and frameworks that I should know about?  These issues aren't unique to recommender systems.  Have they already been solved in your field? If not, what kind of tools would you like to see made available?

I've just been invited to give a keynote at the workshop on Reproducibility and Replication in Recommender Systems Evaluation scheduled for this year's ACM RecSys conference in October, and I'd love to be able to talk about solutions and not just problems.

Let me know what you think!


  1. hi mark, iv been trying to make a reccomendation algorith using R do you have any experience on the same?

  2. Simple and direct has always worked best for me. I can attest to the fact that most of my rapid prototypes or proof of concept/up lift recsys happen on one beefy machine. Find something with quad core or more and 24gb+ ram and just run everything on that. Clusters, parallel runs, map reduce implementations are all good by example but honestly I just use that stuff to get my final training set in a form that can be run on that one machine. You can handle 100+ million line training sets on one machine in a reasonable amount of time.

    Tool wise I see a lot of people trying to fit their model in R but I just can't get there on a regular basis. Unless you are going DB => Python => pandas => munge => DB => R you won't have much luck. Just stay in python for everything and either get your data to an in-memory size on the first map reduce pull or store it in an intermediate DB.

    I haven't needed anything other than pandas + scikit-learn for anything. If you really want to have it interactive you can prototype on a 0.1% sample in R but mileage may vary.

    1. Hi Chris, pre-processing data on Hadoop and then training models on a single machine has worked well for me too, and I've developed some successful systems that way. However if you have a big parameter space to explore, and models that take some time to learn, then training multiple models on a single machine will leave you with a looong wait for results. OK the code required to train and evaluate multiple models in parallel is just plumbing - but in my experience it's not trivial to do robustly, and I haven't found much help out there in the form of existing frameworks. It's particularly a problem if you're trying to do publishable research, and you can't rely on hunches to cut down the number of models you need to train, or you need to run somebody else's slow algorithm to get comparative results...

  3. I just discovered this blog post. In StarCluster, the ipcluster plugin as companion plugin to kill and restart all the engines:

    But I agree it would be great if one could do that directly from the IPython.parallel API. Min is aware of the problem and has planned to redesign how the engines are spawned to make it easy to kill / restart stalled engines but I think right now they are focusing dev efforts on the notebook UI.

  4. Great blog Mark - as a Recruiter I'm learning a lot from here! Keep it up :)

  5. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor led live training in Data Science, kindly contact us
    MaxMunus Offer World Class Virtual Instructor led training on Data Science. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Nitesh Kumar
    Skype id: nitesh_maxmunus
    Ph:(+91) 8553912023

  6. The great service in this blog and the nice technology is visible in this blog. I am really very happy for the nice approach is visible in this blog and thank you very much for using the nice technology in this blog
    Data Science Online Training

    Hadoop Online Training

  7. This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.

    Data science training in Marathahalli|
    Data science training in Bangalore|
    Hadoop Training in Marathahalli|
    Hadoop Training in Bangalore|