Reward or Penalty: Aligning Incentives ofStakeholders in Crowd sourcing
ABSTRACT
Crowd sourcing is a promising platform, whereby massive tasks are broadcasted to a crowd of semi-skilled workers by the requester for reliable solutions. In this paper, we consider four key evaluation indices of a crowd sourcing community (i.e. quality, cost, latency, and platform improvement), and demonstrate that these indices involve the interests of the three stakeholders, namely requester, worker and crowd sourcing platform. Since the incentives a these three stakeholders always conflict with each other, to elevate the long-term development of the crowd sourcing community, we take the perspective of the whole crowd sourcing community, and design a crowd sourcing mechanism to align incentives of stakeholders together. Specifically, we give workers reward or penalty according to their reporting solutions instead of only nonnegative payment. Furthermore, we find a series of proper reward-penalty function pairs and compute workers personal order values, which can provide different amounts of reward and penalty according to both the workers reporting beliefs and their individual history performances, and keep the incentive of workers at the same time. The proposed mechanism can help latency control, promote quality and platform evolution of crowd sourcing community, and improve the mentioned four key evaluation indices. Theoretical analysis and experimental results are provided to validate and evaluate the proposed mechanism respectively.
Index Terms—Crowd sourcing, incentive, reward, penalty, belief.
EXISTING SYSTEM:
The common application of crowd sourcing is in the contexts of knowledge gathering (e.g. mobile crowd sensing or decision making tasks (labeling of training dataset in machine learning). In these contexts, the number of tasks to complete is too large for insufficient number of experts, and the evaluation process cannot be automatically performed very well by a computer. As a result, a feasible alternative is to resort to a crowd of individuals (i.e. workers)recruited on an online crowd sourcing platform to undertake these tasks. The person who publishes tasks and obtains the solutions through the crowd sourcing platform is called the requester. Based on the state of art of crowd sourcing industry and academia, we summarize four key evaluation indices of current crowd sourcing community, namely quality, cost, latency and platform improvement:
1) Quality. In typical crowd- sourcing settings, like MTurk and Crowd Flower, a worker is simply paid in proportion to the amount of tasks she has completed. As a result, a worker inclines to undertake tasks that she is not good at, or spends less effort and time on each task, thereby degrading the quality of her reporting. However, the requester desires workers to report high-quality solutions, as a task’s final truthful solution is elicited from the collected solutions;
2) Cost. The cost control focuses on how to motivate the workers to do their best with minimal cost. It is reasonable to assume that both the requester and workers are self-interested and rational. Hence each worker attempts to maximize her own payment, while the requester aims to achieve high-quality final solutions of tasks with minimal cost;
3) Latency. Latency control is important as the practical total completion time for the whole tasks of a requester may exceed the time constraint set by the requester.
PROPOSED SYSTEM:
The proposed mechanism is mainly based on the following hypothesis: all workers believe that in most cases they observe the real solution of each task, which is only perturbed by unbiased noise. An example to support these assumptions when some workers in the same classroom are asked to count the number of students, and they make decisions on their own and are not allowed communicating with each other. A radically different assumption from the mentioned hypothesis is that a worker can obtain the intention or preference of other workers. For instance, workers are asked to report their attitudes towards a well-known social issue(e.g. public voting), the statistical information of people’s attitudes can be obtained from the media or other ways, and a worker will report what others will report to get a payment, regardless of the real solution. In order to verify our hypothesis, we conduct a questionnaire survey a 500 workers on Crowd Flower. In this survey, we ask the workers just to report the percentage of cases where they need to take into account the attitudes of other workers on Crowd Flower when performing the crowd sourcing tasks, i.e. the probability that our hypothesis holds in real crowd sourcing community. Based on the results of the questionnaire survey, we show the probability density estimation curve. The probability density estimation curve is highly asymmetry, and it shows that our hypothesis holds in most cases in a real crowd sourcing community. Specifically, nearly 95%of workers think the probability that our hypothesis holds is large than 0:6, while more than 80% of workers think the probability is large than 0:8. Note that the hypothesis is not necessarily true for all the settings, however, it holds in most cases in a real crowd sourcing community. We continue to conduct a questionnaire survey a 400 workers on another crowd sourcing community named MTurk, which is a famous crowd sourcing community like Crowd Flower. The result shows that nearly 92% of workers think the probability that our hypothesis holds is large than 0:6, while more than 84% of workers think the probability is large than0:8
CONCLUSION:
For the future work, we will study the following potential directions: 1) We will build up a small crowd sourcing platform based on the proposed mechanism to test and promote the proposed mechanism; 2) We will further adapt the proposed mechanism to make it work properly within limited total budget; 3) We will extend the proposed mechanism to directly deal with multiple type tasks; and 4)We will also study security and privacy aspects of crowd sourcing to facilitate wide-deployment of crowd sourcing
REFERENCES:
[1] B. Guo, C. Chen, D. Zhang, Z. Yu, and A. Chin, “Mobile crowd sensing and computing: when participatory sensing meets participatory social media,” IEEE Communications Magazine, vol. 54,no. 2, pp. 131–137, 2016.
[2] R. Jurca and B. Faltings, “Error rate analysis of labeling by crowd sourcing,” in Proceedings of the 30th International Conference on Machine Learning Workshop (ICML), pp. 1–19, MIT Press, 2013.
[3] Y. Gao, Y. Chen, and K. J. R. Liu, “On cost-effective incentive mechanisms in micro task crowd sourcing,” IEEE Transactions on Computational Intelligence and AI in Games, vol. 7, pp. 3–15, 3 2015.
[4] J. Ren, Y. Zhang, K. Zhang, and X. Shen, “Exploiting mobile crowd sourcing for pervasive cloud services: challenges and solutions,”IEEE Communications Magazine, vol. 53, no. 3, pp. 98–105,2015.
[5] B. Aydin, Y. Yilmaz, Y. Li, Q. Li, J. Gao, and M. Demirbas, “Crowd sourcing for multiple-choice question answering,” in Proceedings of the 26th Innovative Applications of Artificial Intelligence (IAAI),pp. 2946–2953, AAAI, 2014.
[6] N. B. Shah and D. Zhou, “Double or nothing: Multiplicative incentive mechanisms for crowd sourcing,” in Proceedings of the 28thAdvances in Neural Information Processing Systems (NIPS), pp. 1–9,MIT Press, 2015.
[7] G. Li, J. Wang, Y. Zheng, and M. Franklin, “Crowd sourced data management: A survey,” IEEE Transactions on Knowledge and Data Engineering, vol. PP, no. 99, pp. 1–23, 2016.[8] P. Chandra, Y. Narahari, and D. Mandal, “Novel mechanisms for online crowd sourcing with unreliable, strategic agents,” in Proceedings of the 29th AAAI Conference on Artificial Intelligence,pp. 1256–1262, AAAI, 2015.
[9] N. Chen, X. Deng, B. Tang, and H. Zhang, “Incentives for strategic behavior in fisher market games,” in Proceedings of the 30th AAAIConference on Artificial Intelligence, pp. 453–459, AAAI, 2016.