Crowdsourcing: Human-based Computation and Amazon Mechanical Turk

In a companion blog post to his June 2006 Wired magazine article, Jeff Howe posited the maiden commentary of assemblysourcingSimply defined, promotesourcing represents the act of a company or institution good-natured a function once per framinged by employees and outsourcing it to an undefined (and generally king-size) communicate of tidy sum in the mannequin of an diffuse look to. This lav take the form of peer- overlapion (when the job is performed col laboratively), but is in any case oftentimes undertaken by sole several(prenominal)s. The critical prerequisite is the use of the diffuse call format and the large internet of dominance laborers.D arn C. Brabham was the first to define laborsourcing in the scientific literature in a February 1, 2008, articleCrowdsourcing is an online, distributed paradox- figure out and production model.In the classic use of the term, bothers be broadcast to an unknow group of clobberrs in the form of an open call for firm nesss. Usersalso kn decl argon as the crowdsubmit solutions which ar because owned by the entity that broadcasted the problemthe crowd blood liner. In rough cases, the contributor of the solution is compensated monetarily, with prizes, or with recognition. In other cases, the only pay backs freshthorn be kudos or intellectual satisfaction. Crowdsourcing may start out solutions from amateurs or volunteers flows in their sp atomic number 18 time, or from proficients or small businesses which were unknown to the initiating organization.Crowdsourcers are primarily prodd by its benefits. One of these accommodates the ability to gathering large numbers of solutions and cultivation at a relatively inexpensive cost. Users are motivated to contribute to crowdsourced businesss by both intrinsic motivations, much(prenominal) as genial contact,intellectual stimulation, and passing time, and by extrinsic motivations, such as financial gain.Due to the blurred limits of crowds ourcing, many a(prenominal) collaborative activities are considered crowdsourcing even when they are not. nearly other(prenominal) consequence of this situation is the proliferation of definitions in the scientific literature. Different authors crumble contrastive definitions of crowdsourcing according to their specialties, losing in this bearing the global picture of the term.After learn more than 40 definitions of crowdsourcing in the scientific and popular literature, Enrique Estells-Arolas and Fernando Gonzlez Ladrn-de-Guevara developed a newfangled integrating definitionCrowdsourcing is a type of get downicipative online activity in which an individual, an institution, a non-profit organization, or company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via a flexible open call, the voluntary undertaking of a line of work. The undertaking of the assign, of varying complexity and modularity, and in which the crowd should particip ate bringing their work, money, knowledge and/or experience, al shipway entails mutual benefit. The user depart perplex the satisfaction of a given type of inquire, be it economic, social recognition, self-esteem, or the development of individual skills, while the crowdsourcer will obtain and utilize to their ad cutting edgetage that what the user has brought to the venture, whose form will reckon on the type of activity undertaken.Henk van Ess emphasizes the need to give back the crowdsourced results to the domain on ethical grounds. His non-scientific, non- commercial message definition is wide cited in the popular pressCrowdsourcing is channeling the experts desire to solve a problem and then freely sharing the response with e realoneCrowdsourcing systems are employ to effect a variety of tasks. For theoretical account, the crowd may be invited to develop a new technology, track down out a design task (also known as connection- found design or distributed participatory design), refine or carry out the steps of an algorithm (see human- base computation), or help capture, systematize, or analyze large sum of moneys of data (see also citizen science).HistoryThe term crowdsourcing is a portmanteau of crowd and outsourcing, coined by Jeff Howe in a June 2006 Wired magazine article The facelift of Crowdsourcing. It has been argued that crowdsourcing send away only exist on the Internet and is thus a relatively recent phenomenon., however, long before modern crowdsourcing systems were developed, there were a number of not equal to(p) examples of undertakings that utilized distributed tribe to help accomplish tasks.Historical examples The Oxford position DictionaryThe Oxford English Dictionary (OED) may volunteer one of the earliest examples of crowdsourcing. An open call was made to the partnership for contributions by volunteers to index all linguistic process in the English language and example quotations of their usages for each one. They rec eived all over 6 million submissions over a period of 70 years. The fashioning of the OED is detailed in The Surgeon of Crow Thorne by Simon Winchester.Crowdsourcing in family tree look forGenealogical query was using crowdsourcing techniques long before computers were common. Beginning in 1942 members of The church service of Jesus Christ of Latter-Day Saints (also known as the Mormon church) advance members to submit schooling about their ancestors. The submitted information was gathered together into a single collection. In 1969 in graze to encourage more bulk to participate in gathering genealogical information about their ancestors, the church started the three-generation weapons platform. In this program church members were asked to prepare documented family group unload forms for the first three generations. The program was by and by expanded to encourage members to research at least 4 generations, and became known as the four-generation program.Institutes that rush records of sake to genealogical research halt employ crowds of volunteers to create catalogs and indexes to records.Early crowdsourcing competitionsCrowdsourcing has often been used in the past as a competition in articulate to discover a solution. The French government proposed several of these competitions, often rewarded with Montyon Prizes, created for scurvy Frenchmen who had done virtuous acts. These embroild the Leblanc process, or the Alkali Prize, where a reward was provided for separating the salt from the alkali, and the Fourneyrons Turbine, when the first hydraulic commercial turbine was developed.In response to a take exception from the French government, Nicholas Appert won a prize for inventing a new way of food preservation that involved sealing food in airtight jars. The British government provided a convertible reward to find an delicate way to determine a ships longitude in the The Longitude Prize. During the grand Depression, out-of-work clerks ta bulated higher mathematical functions in the Mathematical Tables Project as an out drop dead project.Modern methodsToday, crowdsourcing has transferred in general to the Internet. The Internet provides a particularly good venue for crowdsourcing since individuals tend to be more open in web-based projects where they are not being physically judged or scrutinized and thus arse observe more comfortable sharing. This at last allows for well-designed artistic projects because individuals are slight conscious, or maybe even little(a) aware, of scrutiny towards their work. In an online atmosphere more attention is given to the project alternatively than communication with other individuals.Crowdsourcing can either take an explicit or an implicit route. Explicit crowdsourcing lets users work together to evaluate, share, and build different specific tasks, while implicit crowdsourcing performer that users solve a problem as a side effect of something else they are doing.With expl icit crowdsourcing, users can evaluate particular items like books or webpages, or share by posting products or items. Users can also build artifacts by providing information and editing other peoples work. unexpressed crowdsourcing can take two forms standalone and piggyback. Standalone allows people to solve problems as a side effect of the task they are actually doing, whereas piggyback takes users information from a third-party website to gather information.Types of crowdsourcingIn coining the term of crowdsourcing, Jeff Howe has also indicated some common categories of crowdsourcing that can be used potently in the commercial world. some of these web-based crowdsourcing efforts include crowd select, wisdom of the crowd, crowdfunding, microwork, inventive crowdsourcing and inducement prize skirmishs. Although these may not be an exhaustive list, they cover the current major ways in which people use crowds to perform tasks.According to definition by Henk van Ess that has bee n widely cited in the popular press,The crowdsourced problem can be great (epic tasks like finding alien life or mapping earthquake zones) or very small (where can I skate safely?). Some examples of successful crowdsourcing themes are problems that bug people, things that steel people feel good about themselves, projects that tap into niche knowledge of proud experts, subjects that people find sympathetic or any form of injustice.Crowd votingCrowd voting occurs when a website gathers a large groups opinions and judgment on a true topic. The Iowa Electronic Market is a prediction market that gathers crowds views on political relation and tries to ensure accuracy by having participants pay money to misdirect and transfer contracts based on political out hangs.Threadless.com selects the t-shirts it sells by having users provide designs and vote on the ones they like, which are then printed and available for purchase. Despite the small nature of the company, thousands of members provide designs and vote on them, making the websites products truly created and selected by the crowd, rather than the company. Some of the most famous examples get under ones skin made use of social media channels Dominos Pizza, Coca Cola, Heineken and Sam Adams have thus crowdsourced a new pizza, song, bottle design or beer, respectively.Crowdsourcing imaginative workCreative crowdsourcing spans sourcing creative projects such as graphic design, architecture, apparel design, writing, illustration. etc. Some of the better known creative domains that use the Crowdsourcing model include 99designs, DesignCrowd, crowdspring, Jade Magnet, Threadless, Poptent, GeniusRocket and TongalCrowdfundingCrowdfunding is the process of funding your projects by a multitude of people add a small bar in order to attain a certain monetary goal. Goals may be for donations or for equity in a project. The dilemma right(a) now for equity crowdfunding in the USA is how the irregular is going to puzz le the entire process. As it stands rules and regulations are being refined by the SEC and they will have until Jan. 1st, 2013 to tweak the fundraising methods. The regulators are on edge because they are already overwhelmed trying to regulate Dodd Frank and all the other rules and regulations involving popular companies and the way they trade. Advocates of regulation claim that crowdfunding will open up the outflow gates for fraud, have called it the wild west of fundraising, and have compared it to the 1980s geezerhood of penny stock cold-call cowboys.The process allows for up to 1 million dollars to be raised without a lot of the regulations being involved. Companies under the current proposition will have a lot of exemptions available and be able to raise capital from a larger pool of persons which can include a lot lower thresholds for investor criteria whereas the old rules postulate that the person be an accredited investor. These people are often recruited from social networks, where the funds can be acquired from anequity purchase, loan, donation, or pre-ordering. The steps collected have become preferably high, with requests that are over a million dollars for software like Trampoline Systems, which used it to finance the commercialization of their new software.A well-known crowdfunding likewisel is Kickstarter, which is the biggest website for funding creative projects. It has raised over $100 million, despite its all-or-nothing model which requires one to reach the proposed monetary goal in order to acquire the money. UInvest is another example of a crowdfunding political platform that was started in Kiev, Ukraine in 2007. Crowdrise brings together volunteers to fundraise in an online environment. roughly recently, the adult industry gained its own site in the way of Offbeatr. Offbeatr allows the community to cast votes on projects they would like to see make it to the funding phase. firmness of the crowdWisdom of the crowd is another ty pe of crowdsourcing that collects large amounts of information and aggregates them to gain a get by and accu measure picture of a topic, based on the fancy that a group of people is on middling more intelligent than an individual. This idea of collective intelligence proves particularly effective on the web because people from diverse backgrounds can contribute in real-time indoors the same forums.iStockPhoto provides a platform for people to upload photos and purchase them for low prices. Clients can purchase photos through credits, giving photographers a small profit. Again, the photo collection is determined by the crowds voice for very low prices.In February 2012, a stock picking game called Ticker selector switch Pro was launched, using crowdsourcing to create a hedge fund that would buy and sell stocks based on the ideas coming out of the game. These crowdsourced ideas, coming from so many people, could help one pick the best stocks based on this idea that collective idea s are better than individual ones.MicroworkMicrowork is a crowdsourcing platform where users do small tasks for which computers lack aptitude for low amounts of money. Amazons popular mechanised Turk has created many different projects for users to participate in, where each task requires very little time and offers a very small amount in payment. The Chinese versions of this, commonly called Witkey, are similar and include such sites as Taskcn.com and k68.cn. When choosing tasks, since only certain users win, users learn to submit later and pick less popular tasks in order to increase the likelihood of getting their work chosen. An example of a mechanic Turk project is when users searched broadcast images for images of a boat in order to find lost tec Jim Gray. Inducement prize contestsWeb-based idea competitions, or inducement prize contests often consist of generic ideas, cash prizes, and an Internet-based platform to facilitate easy idea generation and discussion. An example of these competitions includes an event like IBMs 2006 Innovation golf hole, attended by over 140,000 international participants and yielding nearly 46,000 ideas. some other example is Netflix Prize in 2009. The idea was to ask crowd to come up with a recommendation algorithm which was more accu wander than Netflixs own algorithm. It had a grand prize of US$1,000,000 and it was given to the BellKors Pragmatic booby hatch police squad which bested Netflixs own algorithm for predicting ratings by 10.06%another(prenominal) example of competition-based crowdsourcing is the 2009 DARPA experiment, where DARPA rigid 10 balloon markers across the United States and challenged teams to compete to be the first to report the location of all the balloons. A collaboration of efforts was required to cut the challenge speedily and in addition to the competitive motivation of the contest as a whole, the winning team (MIT, in less than cardinal hours) established its own collaborapetitive en vironment to generate participation in their team. A similar challenge was the Tag Challenge, funded by the US State Department, which required locating and photographing individuals in 5 cities in the US and Europe within 12 hours based only on a single photograph. The winning team managed to locate 3 suspects by mobilizing volunteers world-wide using a similar incentive scheme to the oneused in the Balloon Challenge.Open intent platforms are a very effective way of crowdsourcing peoples thoughts and ideas to do research and development. The company InnoCentive is a crowdsourcing platform for corporate research and development where embarrassing scientific problems are posted for crowds of solvers to discover the answer and win a cash prize, which can range from $10,000 to $100,000 per challenge. InnoCentive, of Waltham, MA and London, England is the drawing card in providing access to millions of scientific and technical experts from around the world. The company has provided ex pert crowdsourcing to international Fortune 1000 companies in the US and Europe as well as government agencies and nonprofits.The company claims a success rate of 50% in providing successful solutions to previously unsolved scientific and technical problems. IdeaConnection.com challenges people to come up with new inventions and innovations and Ninesigma.com connects clients with experts in various fields. The X PRIZE Foundation creates and runs incentive competitions where one can win amid $1 million and $30 million for solving challenges. Local Motors is another example of crowdsourcing. A community of 20,000 automotive engineers, designers and enthusiasts competes to build offroad rally trucks. Implicit crowdsourcingImplicit crowdsourcing is less obvious because users do not necessarily know they are contributing, yet can still be very effective in completing certain tasks. Rather than users actively participating in solving a problem or providing information, implicit crowdsour cing involves users doing another task alone where a third party gains information for another topic based on the users actions.A good example of implicit crowdsourcing is the second sight game, where users guess what images are and then these labels are used to tag Google images. Another popular use of implicit crowdsourcing is through reCAPTCHA, which asks people to solve Captchas in order to prove they are human, and then provides Captchas from old books that cannot be decode by computers in order to try and digitize them for the web. Like Mechanical Turk, this task is simple for humans but would be incredibly difficult for computers.Piggyback crowdsourcing can be seen most frequently by websites such as Google that mine ones search history and websites in order to discover keywords for ads, spelling corrections, and finding synonyms. In this way, users are unintentionally destiny to modify existing systems, such as Googles ad words.CrowdsourcersThere are a number of motivation s for businesses to use crowdsourcing to accomplish tasks, find solutions for problems, or to gather information. These include the ability to offload peak demand, access cheap labor and information, generate better results, access a wider array of talent than efficacy be present in one organization, and undertake problems that would have been too difficult to solve internally. Crowdsourcing allows businesses to submit problems on which contributors can work, such as problems in science, manufacturing, biotech, and medicine, with monetary rewards for successful solutions. Although it can be difficult to crowdsource heterogeneous tasks, simple work tasks can be crowdsourced cheaply and effectively.Crowdsourcing also has the potential to be a problem-solving mechanism for government and nonprofit use. Urban and travel planning are prime areas for crowdsourcing. One project to test crowdsourcings public participation process for transit planning in Salt Lake metropolis has been unde rway from 2008 to 2009, funded by a U.S. Federal Transit Administration grant. Another notable application of crowdsourcing to government problem solving is the Peer to unmistakable Community Patent Review project for the U.S. Patent and Trademark Office.Researchers have used crowdsourcing systems, in particular Mechanical Turk, to aid with research projects by crowdsourcing aspects of the research process such as data collection, parsing, and evaluation. Notable examples include using the crowd to create speech and language databases,and using the crowd to conduct user studies. Crowdsourcing systems provide these researchers with the ability to gather large amount of data. Additionally, using crowdsourcing, researchers can collect data from populations anddemographics they may not have had access to locally, but that improve the validity and value of their work.Artists have also utilized crowdsourcing systems. In his project the Sheep Market, Aaron Koblin used Mechanical Turk to c ollect 10,000 drawings of sheep from contributors around the world. Sam Brown (artist) leverages the crowd by asking visitors of his website explodingdog to send him sentences that he uses as inspirations for paintings. Art curator Andrea Grover argues that individuals tend to be more open in crowdsourced projects because they are not being physically judged or scrutinized. As with other crowdsourcers, artists use crowdsourcing systems to generate and collect data. The crowd also can be used to provide inspiration and to collect financial jump for an artists work.Additionally, crowdsourcing from 100 million drivers is being used by INRIX to collect users driving times to provide better GPS routing and real-time handicraft updates.DemographicsThe crowd is an umbrella term for people who contribute to crowdsourcing efforts. Though it is sometimes difficult to gather data about the demographics of the crowd, a study by Ross et al. surveyed the demographics of a sample of the more tha n 400,000 registered crowdworkers using Amazon Mechanical Turk to complete tasks for pay.While a previous study in 2008 by Ipeirotis found that users at that time were primarily American, young, female, and well-educated, with 40% having incomes $40,000/yr, in 2009 Ross found a very different population. By Nov. 2009, 36% of the surveyed Mechanical Turk workforce was Indian. Of Indian workers were male, and 66% had at least a Bachelors degree. ? had annual incomes less than $10,000/yr, with 27% sometimes or always depending on income from Mechanical Turk to make ends meet.The amount US user of Mechanical Turk earned $2.30 per hour for tasks in 2009, versus $1.58 for the total Indian worker. While the majority of users worked less than 5 hours per week, 18% worked 15 hours per week or more. This is less than minimum betroth in either country, which Ross raises raises ethical questions for researchers who use crowdsourcing.The demographics of http//microworkers.com/ differ from Me chanical Turk in that the US and India together account for only 25% of workers. 197 countries are represented among users, with Indonesia (18%) and Bangladesh (17%) contributing the largest share. However, 28% of employers are from the US.Another study of the demographics of the crowd at iStockphoto found a crowd that was largely white, middle- to upper-class, higher educated, worked in a so-called white collar job, and had a high-speed Internet connection at home.Studies have also found that crowds are not simply collections of amateurs or hobbyists. Rather, crowds are often professionally trained in a discipline pertinent to a given crowdsourcing task and sometimes hold advanced degrees and many years of experience in the profession.Claiming that crowds are amateurs, rather than professionals, is both factually untrue and may lead to marginalization of crowd labor rights.MotivationsMany scholars of crowdsourcing suggest that there are both intrinsic and extrinsic motivations tha t cause people to contribute to crowdsourced tasks, and that these factors influence different types of contributors.For example, students and people employed full-time rate Human Capital Advancement as less distinguished than underemployed workers do, while women rate Social Contact as more important than men do.Intrinsic motivations are broken down into two categories, enjoyment-based and community-based motivations. Enjoyment-based motivations refer to motivations relate to the fun and enjoyment that the contributor experiences through their participation. These motivations include skill variety, task identity, task autonomy, direct feedback from the job, and pastime. Community-based motivations refer to motivations related to community participation, and include community identification and social contact.Extrinsic motivations are broken down into three categories, straightaway payoffs, delayed payoffs, and social motivations. Immediate payoffs, through monetary payment, are the in a flash received compensations given to those who complete tasks. Delayed payoffs are benefits that can be used to generate future advantages, such as training skills and being noticed by potential employers. Social motivations are the rewards of behaving pro-socially, such as altruistic motivations. Chandler and Kapelner found that US users of the Amazon Mechanical Turk were more believably to complete a task when told they were going to help researchers identify tumour cells, than when they were not told the purpose of their task. However, of those who completed the task, prime(prenominal) of output did not depend on the framing of the task.Another form of social motivation is prestige or status. The International Childrens Digital Library recruits volunteers to translate and review books. Because all translators receive public acknowledgment for their contribution, Kaufman and Schulz cite this as a reputation-based strategy to motivate individuals who want to be associ ated with institutions that have prestige. The Amazon Mechanical Turk uses reputation as a motivator in a different sense, as a form of gauge control. Crowdworkers who frequently complete tasks in ways judged to be inadequate can be denied access to future tasks, providing motivation to discover high-quality work. CriticismsThere are two major categories of criticisms about crowdsourcing, (1) the value and impact of the work received from the crowd and (2) the ethical implications of low wages give to crowdworkers. Most of these criticisms are directed towards crowdsourcing systems that provide extrinsic monetary rewards to contributors, though some apply more generally to all crowdsourcing systems. Impact of crowdsourcing on product qualitySusceptibility to faulty results caused by targeted, malicious work efforts. Since crowdworkers completing microtasks are nonrecreational per task, there is often a financial incentive to complete tasks apace rather than well. Verifying resp onses is time consuming, and so requesters often depend on havingmultiple workers complete the same task to correct errors. However, having each task completed multiple times increases time and monetary costs.Crowdworkers are a nonrandom sample of the population. Many researchers use crowdsourcing in order to quickly and cheaply conduct studies with larger sample sizes than would be otherwise achievable. However, payable to low worker pay, participant pools are skewed towards poor users in developing countries.Increased likelihood that a crowdsourced project will fail due(p) to lack of monetary motivation or too few participants. Crowdsourcing markets are not a first-in-first-out queue. Tasks that are not completed quickly may be forgotten, buried by filters and search procedures so that workers do not see them. This results in a long tail power lawfulness distribution of completion times. Additionally, low-paying research studies online have higher rates of attrition, with parti cipants not completing the study once started. Even when tasks are completed, crowdsourcing doesnt always produce quality results. When Facebook began its localization program in 2008, it encountered criticism for the low quality of its crowdsourced translations.One of the problems of crowdsourcing products is the lack of interaction between the crowd and the client. Usually there is little information about the final desired product and there is often very limited interaction with the final client. This can decrease the quality of product as client interaction is a vital part of the design process.It is usually expected from a crowdsourced project to be guileless by incorporating a large population of participants with a diverse background. However, most of the crowdsourcing works are done by people who are paid or directly benefit from the issuance (e.g. most of open source projects working on Linux). In many other cases, the resulted product is the outcome of a single persons e ndeavour who creates the majority of the product while the crowd only participates in minor details.Concerns for crowdsourcersEthical concerns. Because crowdworkers are considered independent contractorsrather than employees, they are not guaranteed a minimum wage. In practice, workers using the Amazon Mechanical Turk generally earn less than the minimum wage, even in India. Some researchers considering using Mechanical Turk to get participants for studies have argued that this may be unethical.Below-market wages. The average US user of Mechanical Turk earned $2.30 per hour for tasks in 2009, versus $1.58 for the average Indian worker. While the majority of users worked less than 5 hours per week, 18% worked 15 hours per week or more, and 27% of Indian users said income from Mechanical Turk is sometimes or always necessary for them to make ends meet. This is less than minimum wage in either country, which Ross et al. suggest raises ethical questions for researchers who use crowdsour cing. When Facebook began its localization program in 2008, it received criticism for using crowdsourcing to obtain free labor.Typically, no written contracts, non-disclosure agreements, or employee agreements are made with crowdsourced employees. For users of the Amazon Mechanical Turk, this means that requestors have final say over whether users work is acceptable if not, they will not be paid. Critics claim that crowdsourcing arrangements exploit individuals in the crowd, and there has been a call for crowds to organize for their labor rights.Difficulties in collaboration of crowd members, in particular in the context of competitive crowd sourcing. Crowdsourcing site InnoCentive allows organizations to solicit solutions to scientific and technological problems only 10.6% of respondents report working in a team on their submission.