FAQ

 

For full rules and regulations (“the fine print”), click here.

 

What is the eligibility criteria?

You have to register to enter.

Individuals may enter individually or as part of a team. Regardless of the number of team members, each team must designate a team leader. Individuals may be members of multiple Teams; however, Teams with an identical set of members are not permitted.

The contest is also open to teams affiliated with academic institutions; at least one member of a Team must be actively enrolled in or employed by such educational institution at the time of entry and have necessary consents in place to participate in the competition. Multiple Teams per educational institution are allowed. Multiple individuals from multiple educational institutions are allowed.
Back to top

How much are the contest prizes? What is the Institution Prize?

The Maximum Prize is $1,000,000. The winning team will receive a cash prize of $100,000 USD multiplied by the percentage that the entrant’s algorithm performed over RichRelevance’s algorithm—up to a maximum amount of $1,000,000. For example, if the winner produced 3.34% more revenue per session than RichRelevance’s algorithm, the winner will receive $330,000.

The Institution Prize can be up to $250,000. Should the winning team be affiliated with an educational institution, 25% of the winning prize (funded separately from the Team Winner’s Prize) can be allocated to the educational institution(s).
Back to top

How do I register to participate?

To register, visit http://overstockreclabprize.com/teams/register and complete the registration form in its entirety. You will be required to provide a Team name and the names and contact information for all individual Team members. You must identify a Team Leader. In addition, if you or your Team are claiming affiliation with one or more educational institutions, you must specify the name of such educational institution(s). Sponsor will communicate solely with the Team leader.
Back to top

How do I submit my code?

Please submit your code (in the proper file type) to reclabprize@richrelevance.com. Please submit source code only (no binary).
Back to top

How are submissions evaluated?

Entries will be judged in three stages:

  • Stage 1 – Peer Review: On or about April 16, 2012, a peer review committee appointed by Sponsor will select the top ten Semi-Finalists based on their expert opinions of the expected effectiveness at generating lift and novelty of design of each of the Entries. They will also attempt to ensure their selections reflect a diversity of approaches from among the Entries. Each Semi-Finalist Team (including each Team member and, if applicable, in Sponsor’s discretion, the Semi-Finalist’s affiliated educational institution) will be required to sign an affidavit of eligibility and a license agreement (described in Section 8 of the Rules & Regulations). If a Semi-Finalist Team does not submit all of the required documents within the time period specified by Sponsor, that Team will not be eligible to proceed to the Semi-Finalist Stage.
  • Stage 2 – Semi-Final Stage: Before the start of the Semi-Final Stage, models from the submitted entries will be built using thirty days of context history from the Overstock.com website—activity of the full set of shoppers who visited during that period. The code will then be run for up to one hour in a live evaluation on a randomly chosen 2% of sessions, subject to a 50ms time limit for recommendations to be generated. When results are returned, they will be displayed to shoppers who may then choose to interact with them. If the submission fails to generate recommendations within the 50ms threshold for more than 20% of the sessions it is given, then it will not move forward through the rest of the semi-final stage. Semi-Finalists will then be given a randomly chosen 2% of sessions on the Overstock.com website for a period of three weeks. The top three Semi-Finalist Teams (as measured by revenue per session) that each produce more than 1% more revenue per session than the existing algorithms will proceed to the Final Stage. If no Semi-Finalist produces more than 1% more revenue per session than Sponsor’s existing algorithms, then there will be no Final Stage and no prize will be awarded.
  • Stage 3 – Final Stage: Each of the Finalists will be given a randomly chosen 10% of sessions on the Overstock.com website for an additional three weeks. The best performing Finalist, as judged by increase in revenue per session over the existing algorithms, will be deemed the winner. Again, if no Finalist produces more than 1% more revenue per session than the existing algorithms, then no prize will be awarded.

Back to top

Do I have to be part of a team to compete?

Individuals may enter individually or as part of a team. Regardless of the number of team members, each team must designate a team leader. Individuals may be members of multiple Teams; however, Teams with an identical set of members are not permitted.
Back to top

Is there anything specific the winning team’s affiliated educational institution must do in order to secure the $250,000 separate distribution?

In order for an educational institution to be eligible to receive the Institution Prize, at least one member of a Team must be actively enrolled in or employed by an educational institution(s) at the time of entry. They also need to have obtained any and all necessary consents to participate in the competition. In order for the prize to be allocated the team needs to designate the educational institution(s) in their registration. Multiple institutions can be listed as long as the Team Leader designates which institution(s) will receive the Institution Prize and in what proportion, if the Team wins. To be eligible for the Institution Prize, educational institutions must be accredited by an accrediting agency recognized by the U.S. Secretary of Education or, for foreign educational institutions only, by an accrediting agency approved by the national government of the country where such institution is located.
Back to top

Why are some countries excluded from participation?

Those countries are on the U.S. Treasury Office of Foreign Assets Control’s list of embargoed countries. If this list changes, we’ll post a change to the rules and let you know. As of April 26, 2011 the list includes residents of Cuba, Iran, Myanmar, North Korea, Sudan, and Syria.
Back to top

Why do submissions have specific deadlines rather than a continuous evaluation?

Submissions that are chosen as semi-finalists or finalists will be run against live shopper traffic. Because shopping patterns vary seasonally, we need to evaluate all semi-finalists and finalists against one another and RichRelevance algorithms at the same time. Having fixed entry dates allows us to do this.
Back to top

Where do I find all the tools and documentation I need to start work on my entry?

All RecLab libraries, tools, and documentation are available at http://code.richrelevance.com/reclab-core. The current version of this software is 0.6.1. We anticipate regular updates will be made to enhance the functionality of RecLab and/or fix bugs, but the version we expect entries to be written against will be locked down no less than 90 days before entries are due. Additionally, we provide a set of tutorial code as described in the RecLab Core documentation. This tutorial code demonstrates how to build a simple recommender. We may release additional tutorial code that illustrates additional features of the system. We also make synthetic data sets available for testing and debugging of algorithms. We may release up to three data sets per quarter, but will not release any during the 90 days prior to entries being due. All of this source code and data is publicly available to all, whether they choose to enter the contest or not.
Back to top

What file types are accepted for code submissions?

Entrants must submit complete source code for both their model building code and their runtime algorithms. The source code should be in the form of either a gzipped tar file with an extension of .tgz, or a zip file with an extension of .zip. The tar or zip file should contain a complete project buildable with Maven 3.X (http://maven.apache.org/) and Java 1.6. The specific version of Maven will be finalized 90 days prior to entries being due, and will be the latest stable public release from the Apache Software Foundation. The name of the file should be based on the name of the institution and team leader and an optional version number. For example, University-of-Atlantis-David-Poseidon-2.5.tgz would be a reasonable name. When uncompressed and unarchived, the file should create a single directory whose name matches the original file without the final extension, for example University-of-Atlantis-David-Poseidon-2.5. Within that directory, there should be a pom.xml file as required by Maven, along with all necessary source code. We encourage entrants to use standard locations for their source code, e.g. src/main/java for source, src/test/java for unit tests, and src/main/resources for any resource files. The pom should specifiy that Java 1.6 is to be used, following the example of the tutorial pom.xml file.
Back to top

Can I use third-party code in my submission?

Yes, but only if it is open source, licensed via the Apache 2.0 license, and available via a publicly accessible Maven repository specified in your pom.xml file. If an entry relies on any external code that does not meet these criteria then the entry will not be eligible to compete. If the zip file contains any binary files, e.g. .class files, .jar files, or any other files containing byte code or machine code it will be rejected. All code must be provided as source only.
Back to top

What is the definition of “lift” as it pertains to this Contest?

For the purpose if this contest, lift is defined as revenue-per-session lift. Revenue per session means the total revenue generated by all sessions that were exposed to a given algorithm divided by the total number of such sessions. This includes both purchases of recommended products and purchases of non-recommended products. Lift is the difference of revenue per session using a submitted algorithm divided by revenue per session of the RichRelevance baseline algorithm, minus one. That is:

    RPS Lift = (Revenue Per Session For Submission) / (Revenue Per Session For {rr}) – 1.

The revenue per session is measured over the same time period using all sessions allocated to the given algorithm.
Back to top

How can I know if I will meet the 50ms time limit?

The RichRelevance runtime environment uses machines with high-performance CPUs, and more importantly, solid-state disks. For most models, the bottleneck in performance is disk access. If your models deliver recommendations in under 50ms on machines with spinning disks, you are unlikely to have problems on our hardware. If, however, your models do significant computation at runtime then you will want to be very careful that this can be accomplished in a total of only 50ms of compute time even if that computation is spread across multiple threads. In other words, do not count on having more than one core to run your code.
Back to top

Who can use the Forum?

Anyone! While we have made the forum publicly accessible we ask participants to keep their posts positive, constructive and on topic. The purpose of this forum is here to facilitate discussion and encourage innovation within the sphere of machine learning and collaborative filtering. Plus, stick to those topics. To read the forum rules in their entirety, click here.
Back to top

What’s with the perpetual and non-exclusive license grant in order to receive the prize?

In order to verify and test the code we need to be able access and use it. Secondly, if the code is effective we would like to incorporate it into the system. That said, we want to make sure that you, the creator of the code, can continue to use it in whatever way you want and even license it out to others. So, in summary, we get to use the code but you can let someone else use it too.
Back to top

Who are the judges?

The judges are a peer review committee of industry and academic thought leaders nominated by the public and then appointed by Overstock and RichRelevance. If you would like to nominate someone to be a member of the committee, please do so by June 15, 2011.
Back to top