This is a companion guide for my presentation on using MTurk for research in academia. Details about procedures and everything in the presentation are available at tinyurl.com/MTurk2019
The above URL is for the following:
Ghosh, Sudip and Sperling, Rayne and Hooper, Simon, Using Amazon MTurk for Research in Academia: A Beginner’s Guide for using Qualtrics, detecting VPN, limiting Countries using Geolocation & other Tips (September 17, 2019). Available at SSRN: https://ssrn.com/abstract=
When we started researching on how to use MTurk, I came across too much skepticism about the platform compared to earlier years and I was not sure what to expect. We were conducting an online survey using 9 data charts and almost 70 questions, which is quite a complex for a survey. Overall, I think it is an excellent option for conducting different research studies. MTurk respondents were very diligent in their responses.
I am quite surprised with the amount of criticism about the participant pool in MTurk as experiences in the real world are far more complex and less convenient.
Below are some of the common questions I had as well as ones I continue getting from others. The information is based on research conducted with Dr Rayne Sperling and Dr Simon Hooper.
What is MTurk?
MTurk is an online community of workers started by Amazon. Even today, computers are not good at many tasks that humans excel in. So, the idea was to use people to complement what computers do or provide data for machine learning, artificial intelligence and other research. That might explain why a task or a study on MTurk is a called HIT (Human Intelligence Task).
Can I do any type of study?
Technically, yes. You may recruit participants for surveys, trials, interviews, usability studies, dairy studies or even ask them to write papers. However, all of them may not be feasible. For example, if your task requires that participants use their own equipment, install software or apps, MTurk workers may not do it for security reasons.
Can I do surveys?
Yes. Surveys are quite easy. Note that your institution may limit how you do survey. For instance, you may have to limit yourself to using Qualtrics or approved data storage resources such as campus servers or Box. Check with your institution and IRB (Institutional Review Board). In this article, we will use Qualtrics to set up a survey.
How much should we pay?
As a rule, pay minimum wages for sure which is about $10 / hour. For very small items, you can consider 20 cents/ minute. Note that minimum wage is higher in states like California.
Most MTurkers use tools that estimate hourly wages and will report the same on forums, especially if you are paying below minimum wage. Monitor these forums for feedback and adjust your study. For instance, I was expecting a survey to take between 15-25 minutes but it was taking more in some instances. MTurkers will refer your study to others if you consider their feedback and make changes.
What are Amazon’s MTurk charges?
Typically, Amazon charges about 40% commission. So, if you are paying $5 to a worker, you will pay an additional $2 to Amazon in most cases. There may be other charges so check charges before publishing.
Can you lessen Amazon charges?
Yes, if you run batches with 9 or few participants using basic qualifications or requirements, then Amazon charges half in commission. So, if you run a batch with 9 workers, or less, and you pay them $5, you pay Amazon $1 in charges. This does not work if you are using Premium Qualifications or other criteria.
What qualifications or worker requirements should you set?
If you are using a VPN/ proxy detector and limiting studies to the US, you will get good results with 90+% HIT Approval Rate and Number of HITs Approved as 500 or higher.
Many recommend HIT approval rates of above 98%. This is not needed. More workers will be able to access your study if your criteria is lower. I do not have experience below these levels but, given our experience, I think it might be good at lower levels as well. Besides, if you are running a batch of 9 or less at a time, using one of them as a pilot study will be very useful to determine criteria.
What is the response rate?
Estimates say your survey is visible to about 7000 MTurk workers if it is public (“How many people can your lab reach on MTurk? | Experimental Turk,” n.d.).
In my experience, for a $5 survey (15-25 minutes), response rate was 1 every hour approximately. I think there are many more requests now and your audience reach may be a lot less.
Will you always get responses?
No. When we asked for MTurks from the Education field, we got 0 responses. So, don’t plan on relying on MTurk only.
Is it representative of the population?
Based on demographic data, it seems fairly representative of the US population.
What if you can’t pay enough because you don’t have budget?
Be clear about it. People are nice and honest. Be nice and honest and people do return the favor. If you can pay well to only a few, make the criteria clear.
What about international participants?
For international studies, you may adjust higher or lower based on cost of living. It is not recommended that you pay less than $5/hour in any location.
Note that most countries require that user data is maintained in the same country as the user/ respondent and there are legal issues here that go beyond IRB requirements. If you are paying only a proportion of users, be aware that it may be illegal in some countries to do this randomly as it may be considered gambling (payment must follow some logical rules).
How do you start?
Pilot study is very important. Specify if this is your pilot study so that MTurks can give you feedback accordingly. Most MTurks will give feedback and they like Requesters that listen.
Ensure you are available to respond to any queries and feedback within hours or, preferably, minutes. Forums such as TurkerView.com are very active with both good and bad publicity. MTurkers like requesters who listen to them and pay fairly and it will boost your study participation.
Are there alternatives?
Yes, google for “MTurk alternatives.” I do not have experiences with these other communities. Do look for participant recruiting companies and agencies locally as well.
