Concepts inA Modification of LambdaMART to Handle Noisy Crowdsourced Assessments
Crowdsourcing is a process that involves outsourcing tasks to a distributed group of people. This process can occur both online and offline, and the difference between crowdsourcing and ordinary outsourcing is that a task or problem is outsourced to an undefined public rather than a specific body, such as paid employees. Crowdsourcing is related to, but not the same as, human-based computation, which refers to the ways in which humans and computers can work together to solve problems.
more from Wikipedia