Thursday 11 July 2013

Are your recommendations any good?


If you've worked much with recommender systems you'll know that offline evaluation of recommendation algorithms can bring on a serious headache in no time.  It was nice of Netflix to offer researchers $1M to improve their recommender system, but I think many of them would have taken part in the challenge even without a serious prize simply because the competition format means that someone else has to worry about preparing training and test data, and choosing a success measure, not to mention the pure mechanics of running evaluations over and over again.


With the benefit of hindsight of course it's clear now that rating prediction accuracy isn't a great way of evaluating a recommender system: it isn't applicable at all to most domains, where people can't or won't give ratings; it's not a very good predictor of the  relevance of the top few recommendations that are shown to users, and which they actually care about; even if you can measure relevance it's far from the only thing that influences uptake; and, last but not least, small changes to your user interface or improvements to your input data are both likely to have a much bigger impact than anything you might do to fix up your algorithm.  Consequently most practitioners would much prefer to do online A/B testing, which will tell you reliably whether or not any changes you've made are going to have a positive impact on your bottom line.


But unless you work for a huge company, online testing is a limited resource, especially compared to the vast parameter space you might need to explore when setting up a new recommender system.  And if you're an academic you may not be able to reach users easily, and you have to have some formal results to publish in any event.  So the need to do offline evaluation isn't going away any time soon.

Here are some of the pain points that I've bumped into recently:

  • public datasets aren't enough for (easily) reproducible research: take a look at the results reported here, here and here, all based on the same dataset, but with different experimental setups making for a lot of extra work to compare with them.
  • toolkits need to scale: MyMediaLite, for example, is a great tutorial resource for recommender systems, with clear, well thought out implementations of a host of algorithms, and evaluation code too, but it doesn't handle even moderately large datasets.
  • baseline methods need to be reproducible too: I just came across this impressive paper by Jagadeesh Gorla which introduces a genuinely novel approach to making recommendations and also reports great results, but I can't come close to replicating his baseline results; should I believe the results for his new method?
  • evaluation needs resources: generating full similarity matrices for many different parameter settings, or recommendations for all users in a big dataset, can be way more compute intensive than running a recommender service for real.
  • Hadoop doesn't help much: map-reduce isn't usually a good fit for machine learning algorithms, and the large resources needed by most recommender systems mean that you can't even resort to map-only driver jobs to launch code in parallel.
  • distributed processing without Hadoop is hard: I got excited about StarCluster and IPython.parallel, as in principle they make it dead easy to set up a Python cluster on AWS and run code on multiple worker processes, but in I've found that processes easily get into an unrecoverable state meaning frustration and wasted time tearing down clusters and setting up new ones.  In practice I've made faster progress by running parallel processes on the single biggest machine I can find locally.

What's your experience?

Are there some great tools and frameworks that I should know about?  These issues aren't unique to recommender systems.  Have they already been solved in your field? If not, what kind of tools would you like to see made available?

I've just been invited to give a keynote at the workshop on Reproducibility and Replication in Recommender Systems Evaluation scheduled for this year's ACM RecSys conference in October, and I'd love to be able to talk about solutions and not just problems.

Let me know what you think!

26 comments:

  1. hi mark, iv been trying to make a reccomendation algorith using R do you have any experience on the same?

    ReplyDelete
  2. Simple and direct has always worked best for me. I can attest to the fact that most of my rapid prototypes or proof of concept/up lift recsys happen on one beefy machine. Find something with quad core or more and 24gb+ ram and just run everything on that. Clusters, parallel runs, map reduce implementations are all good by example but honestly I just use that stuff to get my final training set in a form that can be run on that one machine. You can handle 100+ million line training sets on one machine in a reasonable amount of time.

    Tool wise I see a lot of people trying to fit their model in R but I just can't get there on a regular basis. Unless you are going DB => Python => pandas => munge => DB => R you won't have much luck. Just stay in python for everything and either get your data to an in-memory size on the first map reduce pull or store it in an intermediate DB.

    I haven't needed anything other than pandas + scikit-learn for anything. If you really want to have it interactive you can prototype on a 0.1% sample in R but mileage may vary.

    ReplyDelete
    Replies
    1. Hi Chris, pre-processing data on Hadoop and then training models on a single machine has worked well for me too, and I've developed some successful systems that way. However if you have a big parameter space to explore, and models that take some time to learn, then training multiple models on a single machine will leave you with a looong wait for results. OK the code required to train and evaluate multiple models in parallel is just plumbing - but in my experience it's not trivial to do robustly, and I haven't found much help out there in the form of existing frameworks. It's particularly a problem if you're trying to do publishable research, and you can't rely on hunches to cut down the number of models you need to train, or you need to run somebody else's slow algorithm to get comparative results...

      Delete
  3. I just discovered this blog post. In StarCluster, the ipcluster plugin as companion plugin to kill and restart all the engines:

    http://star.mit.edu/cluster/docs/0.94/plugins/ipython.html#restarting-all-the-engines-at-once

    But I agree it would be great if one could do that directly from the IPython.parallel API. Min is aware of the problem and has planned to redesign how the engines are spawned to make it easy to kill / restart stalled engines but I think right now they are focusing dev efforts on the notebook UI.

    ReplyDelete
  4. Great blog Mark - as a Recruiter I'm learning a lot from here! Keep it up :)

    ReplyDelete
  5. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor led live training in Data Science, kindly contact us http://www.maxmunus.com/contact
    MaxMunus Offer World Class Virtual Instructor led training on Data Science. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Nitesh Kumar
    MaxMunus
    E-mail: nitesh@maxmunus.com
    Skype id: nitesh_maxmunus
    Ph:(+91) 8553912023
    http://www.maxmunus.com/


    ReplyDelete
  6. The great service in this blog and the nice technology is visible in this blog. I am really very happy for the nice approach is visible in this blog and thank you very much for using the nice technology in this blog
    Data Science Online Training

    Hadoop Online Training

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. Thanks for sharing great info with us.

    I wanted to write a little comment to support you and wish you a good continuation All the best for all your blogging efforts.Your good knowledge and kindness in playing with all the pieces were very useful.
    data science course in pune

    ReplyDelete
  9. It was really a wonderful article and I was really impressed by reading this blog. We are giving all software Courses such as Data science, big data, hadoop, R programming, python and many other course. R Programming training in bangalore is one of the reputed training institute in bangalore. They give professional and real time training for all students.

    ReplyDelete
  10. Marijuana—also called weed, herb, pot, grass, bud, ganja, Mary Jane, and a vast number of other slang terms—is a greenish-gray mixture of the dried flowers of Cannabis sativa.

    The main active chemical in marijuana is THC (delta-9-tetrahydrocannabinol), the psychoactive ingredient. The highest concentrations of THC are found in the dried flowers, or buds. When marijuana smoke is inhaled, THC rapidly passes from the lungs into the bloodstream and is carried to the brain and other organs throughout the body. THC from the marijuana acts on specific receptors in the brain, called cannabinoid receptors, starting off a chain of cellular reactions that finally lead to the euphoria, or "high" that users experience. Feeling of a relaxed state, euphoria, and an enhanced sensory perception may occur. With higher THC levels in those who are not used to the effects, some people may feel anxious, paranoid, or have a panic attack.
    Cannabis plant used for medical or recreational purposes. The main psychoactive part of cannabis is tetrahydrocannabinol, one of the 483 known compounds in the plant, including at least 65 other cannabinoids. 
    buy real weed online
    how to buy weed online
    buy legal weed online
    buy recreational weed online
    buy weed edibles online
    can i buy weed online
    buy medical weed online
    buy weed online canada
    buying weed online reviews
    buy weed online legit
    buy weed online without medical card
    buy weed seeds online canada
    order marijuana online
    order marijuana seeds online
    how to order marijuana online
    order marijuana online without a medical card
    can you order medical marijuana online
    order marijuana online

    ReplyDelete
  11. Thanks for sharing your innovative ideas to our vision. I have read your blog and I gathered some new information through your blog. Your blog is really very informative and unique. Keep posting like this. Awaiting for your further update.If you are looking for any Data science related information, please visit our website Data science training institute in bangalore

    ReplyDelete


  12. Great post.I really liked you article,your writing style is simply awesome with useful information. Best python certification course in Bangalore is offering Python training with 100% Placement assistance.Get certification under certified experts

    ReplyDelete
  13. Really awesome blog!!! Nice article on data science . Thanks for sharing your innovative ideas to our vision. your writing style is simply awesome with useful information. Very informative, Excellent work!

    Data Science Course

    ReplyDelete
  14. This is a very amazing and informative blog....
    Thanks for sharing with us,
    We are again come on your website,
    Thanks and good day,
    Please visit our site,
    buylogo

    ReplyDelete
  15. Your points are cream of the crop and original in my opinion. I agree with most of your points. Being a travel blogger and lover, it was appealing and supportive for me.
    Data Science training in Mumbai
    Data Science course in Mumbai
    SAP training in Mumbai

    ReplyDelete
  16. Your points are cream of the crop and original in my opinion. I agree with most of your points. Being a travel blogger and lover, it was appealing and supportive for me.


    Data Science Training in Chennai

    Data Science Training in Velachery

    Data Science Training in Tambaram

    Data Science Training in Porur

    Data Science Training in Omr
    Data Science Training in Annanagar

    ReplyDelete
  17. Your points are cream of the crop and original in my opinion. I agree with most of your points. Being a travel blogger and lover, it was appealing and supportive for me.

    Data Science Training in Chennai

    Data Science Training in Velachery

    Data Science Training in Tambaram

    Data Science Training in Porur

    Data Science Training in Omr
    Data Science Training in Annanagar

    ReplyDelete
  18. This comment has been removed by the author.

    ReplyDelete
  19. A very awesome blog post. We are really grateful for your blog post. You will find a lot of approaches after visiting your post.
    SEO Cheltenham
    SEO Agency Gloucester
    SEO Gloucester
    Web Design Gloucester

    ReplyDelete
  20. this is an great content that helps a lot thanks for sharing it with us,
    we are always here design the best websites & logos with a discount also with an guarantee are you interested?
    Logo Designers

    ReplyDelete
  21. Learn an in-depth, real-time understanding of the Data Science domain by enrolling for the AI Patasala advanced Data Science Course in Hyderabad.
    Data Science Training Hyderabad

    ReplyDelete
  22. An amazing blog post. We sincerely appreciate your blog post. Upon viewing your post, numerous approaches will become apparent.
    Data science training Institute In Hyderabad

    ReplyDelete