Thursday, 11 July 2013

Are your recommendations any good?

If you've worked much with recommender systems you'll know that offline evaluation of recommendation algorithms can bring on a serious headache in no time.  It was nice of Netflix to offer researchers $1M to improve their recommender system, but I think many of them would have taken part in the challenge even without a serious prize simply because the competition format means that someone else has to worry about preparing training and test data, and choosing a success measure, not to mention the pure mechanics of running evaluations over and over again.

With the benefit of hindsight of course it's clear now that rating prediction accuracy isn't a great way of evaluating a recommender system: it isn't applicable at all to most domains, where people can't or won't give ratings; it's not a very good predictor of the  relevance of the top few recommendations that are shown to users, and which they actually care about; even if you can measure relevance it's far from the only thing that influences uptake; and, last but not least, small changes to your user interface or improvements to your input data are both likely to have a much bigger impact than anything you might do to fix up your algorithm.  Consequently most practitioners would much prefer to do online A/B testing, which will tell you reliably whether or not any changes you've made are going to have a positive impact on your bottom line.

But unless you work for a huge company, online testing is a limited resource, especially compared to the vast parameter space you might need to explore when setting up a new recommender system.  And if you're an academic you may not be able to reach users easily, and you have to have some formal results to publish in any event.  So the need to do offline evaluation isn't going away any time soon.

Here are some of the pain points that I've bumped into recently:

  • public datasets aren't enough for (easily) reproducible research: take a look at the results reported here, here and here, all based on the same dataset, but with different experimental setups making for a lot of extra work to compare with them.
  • toolkits need to scale: MyMediaLite, for example, is a great tutorial resource for recommender systems, with clear, well thought out implementations of a host of algorithms, and evaluation code too, but it doesn't handle even moderately large datasets.
  • baseline methods need to be reproducible too: I just came across this impressive paper by Jagadeesh Gorla which introduces a genuinely novel approach to making recommendations and also reports great results, but I can't come close to replicating his baseline results; should I believe the results for his new method?
  • evaluation needs resources: generating full similarity matrices for many different parameter settings, or recommendations for all users in a big dataset, can be way more compute intensive than running a recommender service for real.
  • Hadoop doesn't help much: map-reduce isn't usually a good fit for machine learning algorithms, and the large resources needed by most recommender systems mean that you can't even resort to map-only driver jobs to launch code in parallel.
  • distributed processing without Hadoop is hard: I got excited about StarCluster and IPython.parallel, as in principle they make it dead easy to set up a Python cluster on AWS and run code on multiple worker processes, but in I've found that processes easily get into an unrecoverable state meaning frustration and wasted time tearing down clusters and setting up new ones.  In practice I've made faster progress by running parallel processes on the single biggest machine I can find locally.

What's your experience?

Are there some great tools and frameworks that I should know about?  These issues aren't unique to recommender systems.  Have they already been solved in your field? If not, what kind of tools would you like to see made available?

I've just been invited to give a keynote at the workshop on Reproducibility and Replication in Recommender Systems Evaluation scheduled for this year's ACM RecSys conference in October, and I'd love to be able to talk about solutions and not just problems.

Let me know what you think!


  1. hi mark, iv been trying to make a reccomendation algorith using R do you have any experience on the same?

  2. Simple and direct has always worked best for me. I can attest to the fact that most of my rapid prototypes or proof of concept/up lift recsys happen on one beefy machine. Find something with quad core or more and 24gb+ ram and just run everything on that. Clusters, parallel runs, map reduce implementations are all good by example but honestly I just use that stuff to get my final training set in a form that can be run on that one machine. You can handle 100+ million line training sets on one machine in a reasonable amount of time.

    Tool wise I see a lot of people trying to fit their model in R but I just can't get there on a regular basis. Unless you are going DB => Python => pandas => munge => DB => R you won't have much luck. Just stay in python for everything and either get your data to an in-memory size on the first map reduce pull or store it in an intermediate DB.

    I haven't needed anything other than pandas + scikit-learn for anything. If you really want to have it interactive you can prototype on a 0.1% sample in R but mileage may vary.

    1. Hi Chris, pre-processing data on Hadoop and then training models on a single machine has worked well for me too, and I've developed some successful systems that way. However if you have a big parameter space to explore, and models that take some time to learn, then training multiple models on a single machine will leave you with a looong wait for results. OK the code required to train and evaluate multiple models in parallel is just plumbing - but in my experience it's not trivial to do robustly, and I haven't found much help out there in the form of existing frameworks. It's particularly a problem if you're trying to do publishable research, and you can't rely on hunches to cut down the number of models you need to train, or you need to run somebody else's slow algorithm to get comparative results...

  3. I just discovered this blog post. In StarCluster, the ipcluster plugin as companion plugin to kill and restart all the engines:

    But I agree it would be great if one could do that directly from the IPython.parallel API. Min is aware of the problem and has planned to redesign how the engines are spawned to make it easy to kill / restart stalled engines but I think right now they are focusing dev efforts on the notebook UI.

  4. Great blog Mark - as a Recruiter I'm learning a lot from here! Keep it up :)

  5. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor led live training in Data Science, kindly contact us
    MaxMunus Offer World Class Virtual Instructor led training on Data Science. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Nitesh Kumar
    Skype id: nitesh_maxmunus
    Ph:(+91) 8553912023

  6. The great service in this blog and the nice technology is visible in this blog. I am really very happy for the nice approach is visible in this blog and thank you very much for using the nice technology in this blog
    Data Science Online Training

    Hadoop Online Training

  7. This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.

    Data science training in Marathahalli|
    Data science training in Bangalore|
    Hadoop Training in Marathahalli|
    Hadoop Training in Bangalore|

  8. This comment has been removed by the author.

  9. Thanks for sharing great info with us.

    I wanted to write a little comment to support you and wish you a good continuation All the best for all your blogging efforts.Your good knowledge and kindness in playing with all the pieces were very useful.
    data science course in pune

  10. Thanks for sharing is so amazing and helpful to us.
    Buy Hydrocodone online

  11. It was really a wonderful article and I was really impressed by reading this blog. We are giving all software Courses such as Data science, big data, hadoop, R programming, python and many other course. R Programming training in bangalore is one of the reputed training institute in bangalore. They give professional and real time training for all students.


  12. Best Article buy Pain Pills online Excellent post. I appreciate this site. Stick with it! Because the admin of this web page is working, no doubt very quickly it will be well-known, due to its quality contents.This website was how do you say it? Relevant!! Finally, I’ve found something that helped me.
    Best Article buy Roxicodone online Excellent post
    buy Xanax online
    buy Oxycodone online

    Best Article buy Pain Medications online Excellent post. I appreciate this site. Stick with it! Because the admin of this web page is working, no doubt very quickly it will be well-known, due to its quality contents.This website was how do you say it? Relevant!! Finally, I’ve found something that helped me.

    buy Research Chemicals online

    buy Roxicodone online

    buy Cbd Isolate online

  13. Marijuana—also called weed, herb, pot, grass, bud, ganja, Mary Jane, and a vast number of other slang terms—is a greenish-gray mixture of the dried flowers of Cannabis sativa.

    The main active chemical in marijuana is THC (delta-9-tetrahydrocannabinol), the psychoactive ingredient. The highest concentrations of THC are found in the dried flowers, or buds. When marijuana smoke is inhaled, THC rapidly passes from the lungs into the bloodstream and is carried to the brain and other organs throughout the body. THC from the marijuana acts on specific receptors in the brain, called cannabinoid receptors, starting off a chain of cellular reactions that finally lead to the euphoria, or "high" that users experience. Feeling of a relaxed state, euphoria, and an enhanced sensory perception may occur. With higher THC levels in those who are not used to the effects, some people may feel anxious, paranoid, or have a panic attack.
    Cannabis plant used for medical or recreational purposes. The main psychoactive part of cannabis is tetrahydrocannabinol, one of the 483 known compounds in the plant, including at least 65 other cannabinoids. 
    buy real weed online
    how to buy weed online
    buy legal weed online
    buy recreational weed online
    buy weed edibles online
    can i buy weed online
    buy medical weed online
    buy weed online canada
    buying weed online reviews
    buy weed online legit
    buy weed online without medical card
    buy weed seeds online canada
    order marijuana online
    order marijuana seeds online
    how to order marijuana online
    order marijuana online without a medical card
    can you order medical marijuana online
    order marijuana online

  14. Thanks for sharing your innovative ideas to our vision. I have read your blog and I gathered some new information through your blog. Your blog is really very informative and unique. Keep posting like this. Awaiting for your further update.If you are looking for any Data science related information, please visit our website Data science training institute in bangalore


  15. Great post.I really liked you article,your writing style is simply awesome with useful information. Best python certification course in Bangalore is offering Python training with 100% Placement assistance.Get certification under certified experts

  16. Really awesome blog!!! Nice article on data science . Thanks for sharing your innovative ideas to our vision. your writing style is simply awesome with useful information. Very informative, Excellent work!

    Data Science Course

  17. This is a very amazing and informative blog....
    Thanks for sharing with us,
    We are again come on your website,
    Thanks and good day,
    Please visit our site,

  18. Attend The Business Analytics Courses From ExcelR. Practical Business Analytics Courses Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analytics Courses.
    Business Analytics Courses