MS. DIANE VOGELFrom WAMU 88.5 at American University in Washington, welcome to "The Kojo Nnamdi Show," connecting your community with the world. I'm Diane Vogel sitting in for Kojo. Welcome to the new world of online labor on demand. Suppose you need something translated or something fact-checked, you've got an issue for your website, you've got an issue for a paper you're working on, you want to find somebody in your neighborhood who speaks the language, maybe who can translate it for you, or maybe you want -- you've got a directory of 10,000 URL listings, and you want to make sure that each one of those URLs is correct before you send your document to print, you could find someone in your neighborhood to do it, to plug in each one of those numbers or each one of those URLs and make sure it was right, or you could tap into the global pool of Web workers who could do the job for just pennies on the dollar.

MS. DIANE VOGELYou may never have heard of sites like Amazon's Mechanical Turk or a site called CrowdFlower.com, but they are new and they are growing and they are an important part of the labor workforce. They're harnessing the forces of globalization, cheap wages and quick broadband. You're posting small jobs on these digital marketplaces, asking people to translate this or rank this or fact-check that, and you tell them what you're willing to pay and usually someone maybe halfway around the world is willing to do the job for you. It's a new model for finding work and workers already available on the Web, but some people think that these marketplaces may be morphing into digital sweatshops. It's a complex technology. It's something that our complex technological world is only starting to grapple with, and I think if I start with how Amazon came up with this idea, you might better understand the conversation we're having today.

MS. DIANE VOGELMy guest today is Ben Bederson. Ben is the founder of -- co-founder and chief scientist at a company called Zumobi, a mobile app and advertising firm, and he is also a professor of computer science at the University of Maryland-College Park, a former director of the Human Computer Interaction Laboratory there. Welcome, Ben. Good to see you.

MR. BEN BEDERSONThanks. It's a pleasure to be here, Diane.

VOGELOh, it's always good to see you, Ben, and welcome back to Tech Tuesday. You're an early supporter and cheerleader for Tech Tuesday. So thank you. I understand that for a lot of people this is gonna be the first time that they hear words like crowdsourcing or human computation or CrowdFlower and they're gonna say what do these words mean? I sort of felt that way about six weeks ago, before we started talking about this, and what I realize was understanding how Amazon.com -- we all know Amazon.com -- I realize that once I understood how Amazon.com worked and tried to figure out a problem they were having, it helped me understand this whole new world of online workers. So maybe, Ben, you can walk us through what was going on at Amazon.com five years ago.

BEDERSONWell, they, like a lot of companies, had an information problem. They had various kinds of music assets, CDs and recordings, and they needed to validate some information. They needed to collect other information. That was sometimes available online, but it wasn't in their databases in the way that it should have been. And they could have done the normal thing, like hire a local employee, I mean, a regular employee...

VOGELMm-hmm.

BEDERSON...to go do their work, but Amazon is actually a technology company. Sure, they sell books, but they have an incredible technological resource -- resources. And so they said, you know, we can use our services to try and recruit people outside the company to help us do this information. All the people inside the company would be doing -- would be looking online anyway.

VOGELMm-hmm.

BEDERSONRight? They don't need to be an employee to do that. It's relatively low skilled labor.

VOGELAnd in a lot of ways what they were asking people to do was, hey, log in to our site and figure out if we have a lot of pages that sort of duplicate each other, you know, three pages that all talk about Frank Sinatra's Summer Winds album or whatever. I don't know that Frank Sinatra actually had a Summer Winds album, but they -- so they were asking people to do a task that sort of needed -- it wasn't something you could have a computer do because you needed to have enough brain power to think about, oh, here's something that is identical or almost identical to something else. And Amazon just wanted you to kinda alert them to this, to these duplications in their system, right?

BEDERSONExactly right. And so -- in fact, you hit the nail on the head which is this is like an isolated small problem where there is no automated solution available. Of course, if you could use artificial intelligence or data mining or your automated solution to do it, of course, you would do that. The problem comes in where computers are not sophisticated enough to do this kind of analysis that only humans can do, and the reality is -- thank goodness...

VOGELMm-hmm.

BEDERSON...there are still remained problems that humans can do that computers cannot, and that seems to be likely to be that way for a while.

VOGELExcellent.

VOGELJob security for some of us. The other example that I thought about as I was going through this, which is very relevant I'm sure to a lot of our audience, to a lot of listeners, is you think about how many times you get duplicate catalogs from somewhere or duplicate mailings because they identify you by your initials in one place or your, you know, your first name and last name in another place or they write north this street instead of abbreviating N. Those kinds of tasks as well could be farmed out.

BEDERSONRight. Exactly. In fact, in -- you know, that particular task is interesting because that is an active area of research 'cause it's such big business. There's even a class of algorithms called deduping...

VOGELWow.

BEDERSON...or deep duplication that address that, but they work reasonably well but not perfectly. And if you want to save your mailing money and you want a perfect answer, then you need to bring in a human.

VOGELOkay. So this is basically an idea to get some human interaction, some eyeballs on something that is a challenge, and most of us might consider some grunt work. But my understanding is that this is also something that if you're doing online work, you know, I might want to do it. I might want to sit at home. I have -- maybe I'm babysitting for, you know, maybe I have seven children or maybe I have an invalid or an ill person at home, would this be a good opportunity for our workers here in America to look at this and say, hey, here's an online job that I could do from the privacy of my own home. From what I read, it seems like that's not happening. It seems like most of this work is happening elsewhere. Why is that?

BEDERSONIt all comes down to money.

VOGELMm-hmm.

BEDERSONYou know, the reason that there's an interest in this and the reason for the vocation of the workers is because of how much it pays, and you can imagine it doesn't pay that much.

VOGELMm-hmm.

BEDERSONIn fact, I think probably the best analogy is to day laborers, right?

VOGELAha.

BEDERSONWhere, you know, all have some awareness that if you want some kind of, you know, relatively low skilled mechanical labor done, right, manual labor, you can go to certain locations and find people that are waiting to get hired to do work by, you know, by the day. Right? And they get paid some amount but not as much as a skilled employee, right? And it comes without the promise either, right?

VOGELMm-hmm.

BEDERSONYou don't quite know who you're getting, and sometimes, it works out, and sometimes, maybe it doesn't. And so I think when you have online -- sometimes, they call them Web workers, right? Online day laborers...

VOGELMm-hmm.

BEDERSON...you get sort of the same issue which is you get people with less skill but they're available. They're there when you want them, and they will many, many times work out great, but they need some oversight, and you can't be completely sure what you're gonna get.

VOGELGreat. Well, you have started me down a path. I'm looking for everyone to join this conversation. If you want to talk about online workers, Web workers, we're gonna learn more about this thing called Mechanical Turk, which I don't blame you if you've never heard of. I had never heard of it either. We're talking about the murky world of online labor and the possible kind of revolution it may bring to tech jobs with Ben Bederson, the co-founder of Zumobi, an online mobile mapping -- blah -- an online app and advertising company and also the professor of computer science at the University of Maryland-College Park. We would like you to join the conversation. You can do that now by calling 1-800-433-8850. 1-800-433-8850. Or e-mailing us at kojo@wamu.org. That's K-O-J-O-@-W-A-M-U-dot-org.

VOGELSo, Ben, walk us a little bit through how Mechanical Turk, for example, works. Now, I understand that in anticipation of your appearance here today, you actually did an experiment that we could talk about for "The Kojo Show" audience, and I understand that that experiment ran the gamut from experiments or challenges that might be easy to do, that you don't need any technological background for it to challenges that might be a little bit more complex. Give us an example. Tell me about the experiments you took on in the last few days.

BEDERSONSure. So I thought that I would try and learn something about "The Kojo Show," so we -- I asked -- I wanted to improve -- help "The Kojo Show" and said, hey, let's try and see if we can improve the website. It's already pretty good, but we can make it better. So I wanted to get people to look at the Kojo website and tell me what they thought was wrong with it.

VOGELMm-hmm.

BEDERSONHow could it better? So I posted a hit.

VOGELMm-hmm.

BEDERSONIt was called in this language a human intelligence task.

VOGELOkay. Human intelligence task.

BEDERSONRight. At Amazon's website, which is MTurk.com, M like Mary, T-U-R-K-dot-com, and listeners, if you're online, go take a look now. You can log in with your Amazon account, and you'll see what it looks like. And I designed a very simple task to ask people to give me that information, and so the way you make a task is you use their online little Web authoring form to describe what it is you want people to do. You could include links. You can include images and text boxes where people can put in their answers.

VOGELMm-hmm.

BEDERSONIt helps if you know a little HTML.

VOGELMm-hmm.

BEDERSONAnd there are more sophisticated...

VOGELWeb language.

BEDERSONWeb language, right. And for programmers, there are APIs and more sophisticated ways of scaling up, but there's a -- this Web access to create this hit. You put some money into your account. You say how much you want to pay per judgment. You say how many judgments you want, and you say submit.

VOGELMm-hmm.

BEDERSONAnd it gets published in this online marketplace.

VOGELMm-hmm.

BEDERSONSo then -- and that's it. It took me, you know, I kinda have figured out the system, but it took me maybe 10 minutes.

VOGELOkay.

BEDERSONAnd I did six of these. That was one.

VOGELOkay. Now, six hits, six human interaction...

BEDERSONIntelligence tasks.

VOGELIntelligence tasks. And when you say you did six, I know one of them might have been about our website, but I think another one of them was asking how -- what is the best way to say I love you without saying the words I love you. What...

BEDERSONThat's right.

VOGELA nonverbal way to say I love you.

BEDERSONThis was a great suggestion -- idea. This was actually a suggestion from my graduate student Alex Quinn, and the idea was, again, think about what could every human do, right? So you don't use special skills, but a lot of people can think about this.

VOGELMm-hmm.

BEDERSONSo we just wanted to get a range of thinking about it, and that's one of the approach -- one of the nice things about this is you're getting a wide, you know, audience of workers. You're gonna get a very diverse community of answers.

VOGELMm-hmm.

BEDERSONAnd this was the question that we got, I think, the best response. We got something like 20 different responses, and many of them quite thoughtful. And I should say...

VOGELI was very impressed, I have to say.

BEDERSONRight. Only a few of them said, you know, say hug your person that you want to, you know, express your love to. But we got like paragraph descriptions of how you can cook dinner and give them a massage on their feet and...

VOGELAnd how women should polish men's shoes because men don't think how -- about how important a good shoe polish is, but when they open their closet door and see all the polished shoes, they'll know you love them.

BEDERSONYou know, you actually did get a bit of a cultural understanding of our -- and some of the answers when you read these responses.

VOGELYes.

VOGELI thought that was terrific. And now, the question was, though, the same person could have done all of these 20, right? You don't know that, do you?

BEDERSONWell, this is the nature of -- the problem of working online. The answer is you're not supposed to.

VOGELMm-hmm.

BEDERSONSo each -- when you ask for, say, 20 judgments on one hit, the Mechanical Turk website sets it up so that each person represented by an account can only give one judgment. The problem is, of course, you can have one person make multiple accounts because the accounts aren't authenticated in any way. And so -- many of the people are honest, and so there's one -- they may have one account, and they'll give you one answer.

VOGELMm-hmm.

BEDERSONBut there was another task that we asked, where we clearly had some fraudulent response.

VOGELTell me a little bit more, just a sentence or two more about that and...

BEDERSONRight. So for this fraudulent one, we asked how...

VOGELIt wasn't intentionally fraudulent.

BEDERSONRight.

VOGELIt was...

BEDERSONWell, I think it was. So the -- we asked for suggested topics that "The Kojo Show" might cover in the future.

VOGELMm-hmm.

BEDERSONAnd I opened this question up to anybody in the world. Some of the other ones I -- was more restricted. But this open one, I had several good answers, and then I had 10 answers, all within a few minutes from the same computer, which I could tell, that gave me some information. And they were all one word, kind of trivial, not serious answers. And so this was clearly someone that was just trying to game in the system, get their 5 cents. But 5 cents 10 times is 50 cents. And this was a person in India where 50 cents is, you know, not...

VOGELIs -- goes a lot further than 50 cents here.

BEDERSONExactly.

VOGELYeah. Well, it's a fascinating new world, and there's a lot more to explore. We're talking with Ben Bederson, professor of computer science at University of Maryland College Park, former director of the Human-Computer Interaction Laboratory there and co-founder and chief scientist at Zumobi, a mobile app and advertising company. We're talking to him about this coming revolution in tech jobs. Or is it a coming a coming revolution in tech jobs? Hard to say, but we're looking at the murky world of online labor. We're interested in hearing from you. Please join the conversation at 1-800-433-8850, 1-800-433-8850. Have you ever used Mechanical Turk? Or have you conducted online work with someone where you didn't know the person who did it, and maybe the person who did the work didn't know you either? Did you use an online connection, maybe Elance or something else to do online work? Tell us your experiences. Share your view at 1-800-433-8850. E-mail us at kojo@wamu.org. Or go online to our website at kojoshow.org. We'll be right back after this break.

VOGELWelcome back. I'm Diane Vogel, managing producer of "The Kojo Show," sitting in for Kojo Nnamdi today. We're talking about the murky world of online labor and, perhaps, a future revolution in tech jobs with Ben Bederson, professor of computer science at the University of Maryland College Park and co-founder and chief scientist at Zumobi, a mobile app and advertising company. And we're taking your calls at 1-800-433-8850, 1-800-433-8850. I'd like to go right to the phones for a moment and start, perhaps, with Ben in Silver Spring, a namesake of yours. Ben, you're on the air.

BENHi. Thank you for taking my call, Diane. And I'm glad to be on the show. And I had a question for Dr. Bederson. It was actually a two-part question. One, if you thought that this, you know, new world of online labor had the potential for a real revolution and how work is done in, you know, in the entire world. And the second question, do you feel that there's a need to govern online activity, you know, as far as reviews and, you know, taking advantage and things of that aspect?

VOGELThat's a really good set of questions, Ben. I'll let you take the first one first, Ben. And the second one, perhaps, we'll do a little bit now, but we may save the big ethical discussions for later in the conversation.

BEDERSONSo, Ben, yeah, I agree. Those are excellent questions. So about is this really a revolution? So the answer is today, it is absolutely not a revolution. Today, it is the -- you know, a burgeoning idea where there are definitely people in the know that are using it seriously. But it's definitely a small minority of people in the know -- people -- there are definitely people especially in poorer countries that are making a living on it. There are a lot of people that have done surveys that are looking at how many people are doing, how many hours they're working. And, you know, like anything, there's, kind of, a curve. Many people don't spend that much time doing it. You know, I've earned about $2 total myself just trying it out. And that's with a fair amount of work.

BEDERSONThe average hourly wage estimated to be about $2 per hour which means that it's not really something that people in the U.S. are gonna like typically wanna do. But in India, $2 an hour for many people might be a competitive wage and there are people that are working on it. There are typically about 100,000 hits or tasks available at any given time on Mechanical Turk. But is it revolution? Today, no, but my answer is, I really see this moving forward and growing because there are a lot of tasks for which humans can do it -- do things with, you know, general skills. They don't need highly specific skills.

BEDERSONAs the systems work out these issues of accuracy and reputation, and we'll go to this governance issue later, you know, which are not well worked out today, admittedly, it will become a more reliable labor source. And the bottom line is there is a huge quantity for, you know, I'll call it white collar labor of various kinds. The stuff we've been talking to are very small, in the small, simple kinds of labor. But there's also researchers looking at doing harder, more interesting kinds of labor. I don't know if you wanna spend a minute talking about one other kind of...

VOGELSure. Go ahead, Ben.

BEDERSONSo, there is a guy, a professor at MIT. He's name is Rob Miller who's led a really creative effort around the system he called Soylent. If you remember the movie Soylent Green...

VOGELSoylent Green is people...

BEDERSONIt's all in the people. So suppose you were writing a paper, right, whether it's an academic paper or anything and it's, you know, nine pages but you need to cut it down to eight pages. How do you do it? You've got an hour. Well, you can do it. You can have your students help you do it, or you can post it out onto Mechanical Turk in very, very clever ways to not just have one person do it but to break it up into tiny little tasks, have lots of people do tiny little tasks, you know, shorten individual sentences or paragraphs, have multiple layers so other people review the first layer, and then have a third layer do, kind of, overall analysis. And he shows that he's got surprisingly good -- still expensive but surprisingly, potentially interesting good solutions there.

VOGELThat's an amazing complicated way to cut a page down from nine pages to eight pages. But I can see the efficacy or the reason for it. I understand -- oh, thank you, Ben, for that call. That was a great set of questions. I understand that Amazon.com or the Mechanical Turk way of doing this. We're talking about it today and using it as one of the examples. I know, I've seen it called a flawed platform. But I've also read that it's a really -- because it's a flawed platform in a lot of ways, it's useful for us to be having these discussions around it now because people in the future are gonna be creating more of these platforms maybe from the scratch. And so seeing what works, what doesn't, the sophisticated things that humans can do versus computers or vice versa.

VOGELMaybe, you can talk us through a little bit more. I'm betting that most of the Tech Tuesday audience has heard of the term crowdsourcing. But they're not really sure what crowdsourcing is. And to make matters worse, somewhere later in this conversation we're gonna throw out a place called CrowdFlower, which is, in theory, very similar to crowdsourcing or uses crowdsourcing but has nothing to do with the genetic -- generic term crowdsourcing. So maybe if you would walk us through a little bit of what we need to understand about human intelligence task and the way crowdsourcing works.

BEDERSONSo crowdsourcing is a term which is, you know, somewhat widely used although, admittedly, I threw it out at a party the other day and the lawyers that I was telling it to looked at me as if I was speaking in other language. So it's clearly not in everybody's parlance, yes.

VOGELWho was your audience there?

BEDERSONWell, that was a lawyer that I was talking about at a party.

VOGELOh, okay.

BEDERSONAnd so, you know, I figured, you know, I'm such a nerd, I actually think that crowdsourcing is a normal word. (laugh) So crowdsourcing is derivative of outsourcing. And outsourcing, I think, people do understand. Outsourcing, right, is the idea of taking one set of laborers -- of human laborers and replacing it with another set of human laborers, typically, laborers in another country that are charging less money. Crowdsourcing is also the idea of replacing human workers with another set of human workers. But those human workers are typically, the public. And, of course, Wikipedia is the, perhaps, the biggest, most well-known of crowdsourcing.

VOGELSure.

BEDERSONBut there's also a lot of now question and answer sites...

VOGELSure. But for those who don't realize it, there are people who don't realize that Wikipedia is not written by one person or by a team of paid people. Wikipedia is this massive crowdsourcing project that almost anyone can be allowed to become a Wiki editor and can -- uses the wisdom of the crowds, the wisdom of all the people to hopefully get something right. But when you -- just for those of you college students who are listening, you can't cite Wikipedia and believe that everything that you read there. Just FYI.

BEDERSONRight. It works pretty well, and there's lot of analysis. But like all of these things, it's not perfect. So that's crowdsourcing. So human computation, which is the term that researchers are using now to describe tools like Mechanical Turk, is a little bit different. Instead of replacing one set of humans with another broader set of humans, it replaces computing systems that can't give you the quality that you want with humans that have the potential for offering higher quality.

VOGELOkay.

BEDERSONAnd so it's really -- that's why it's called human computation, right? The focus is on solving computational problems, things that you might expect that a computer could do today or maybe at least, you know, in some future system -- future artificial intelligence systems might be able to do.

VOGELOkay. Now, I see that there is a number of ways in which we could motivate people to do this job and to do little jobs. And Wikipedia is one example where nobody gets -- these people who contribute mostly for Wikipedia don't get paid, right? It's out of the goodness of my heart that I signed up to be a Wikipedia editor or to write pieces. I understand that that is a system that sort of works on altruism. But very few other businesses are gonna be able to make a model based on altruism.

BEDERSONRight.

VOGELSo it's my understanding that there are other ways besides paying people that some companies are approaching the public and trying to crowdsource them in a way to solve problems but aren't paying them.

BEDERSONWell, there are actually -- you know, unlike anything, there's a full range of ways that systems use to motivate people. And pure enjoyment is, you know, actually relatively uncommon. There are actually a whole set of games where people play just for the fun of it.

VOGELMm-hmm.

BEDERSONRight? They were -- the games themselves actually harness the activities that people are doing to try and solve problems. So that's actually...

VOGELSo my hours spent online, playing Tetris, is helping somebody...

BEDERSONWell...

VOGEL...figure out spatial relations?

BEDERSONWell, Tetris, no. But Tetris is a simply just kind of -- we have moving these little shapes to fit them into little boxes and patterns, right?

VOGELMm-hmm.

BEDERSONWell, there is a really clever game called Foldit.

VOGELMm-hmm.

BEDERSONI think that the website is fold.it...

VOGELMm-hmm.

BEDERSON...where humans are playing a game to actually do this kind of spatial matching where they're really solving protein folding problems to help biologists and chemists. So that is...

VOGELSo "Kojo Show" listeners who go and click on our Web page and click at foldit.com or...

BEDERSONI think it's just fold.it.

VOGELOh, fold.it. Okay. Fold.it. They would get the thrill of playing or the fun of playing and online games similar to a Tetris, similar to a spatial relations game.

BEDERSONRight.

VOGELAnd their -- but their work, their results are being harnessed into something that scientists are using.

BEDERSONThat's right. And there's another whole set of games of this kind that are called -- professor in Carnegie Mellon University named Luis von Ahn has made a site called gwap.com, G-W-A-P -- games with a purpose -- dot com. And there's a whole bunch of games where there are millions of people that play these things. They're surprisingly fun. One of the funny ones, is they show you a picture, and you say whether you like it or not. And after 10 of them, it tells you whether it thinks you're a male or a female.

VOGELInteresting.

BEDERSONSo it's a game. In doing that, it's correlating preferences with images and whether males or females like them, because afterwards, you say whether you really are male or female. And doing so, it categorize images which can then help image search algorithms in the future. So that's enjoyment. Wikipedia actually goes beyond enjoyment, because there is a reputation element as well. It's not pure altruism. And it's not just enjoyment. People are very proud of the fact that they get listed as, you know, having contributed to, you know, thousand articles or having gone up to this sort of hierarchy to become, you know, a recognized editor.

VOGELMm-hmm.

BEDERSONSo various kinds of reputation are also very important.

VOGELInteresting. Let's go back to the phones. I think Jay in Frederick, Md. has a similar question. Jay, you're on the air.

JAYHi. I had actually used an online source, where it goes -- actually, my washing machine -- the pump went out on it. And I'm a pretty handy guy, so I was thinking maybe I could fix it myself, but I really had no idea how to go about it. And I went online to see if there's any resources and found a site, and I'm embarrassed to say I don't remember the name of it, where you could ask your question and then they would have somebody that was an expert in that field answer the question. And if you like the answer or if you felt it was worthwhile, then you pay the fee. And I think it cost me, like, $12.50. But I got some appliance repairman that was down in South Carolina or something that gave me very detailed instructions along with illustrations and everything else and where to find everything. And it allowed me to take what was going to be a $300 repair bill and do it myself for essentially 70 or 80 bucks. And I thought that was a real interesting online commerce that I never even knew existed.

VOGELMost definitely. Jay, do -- I'm sorry -- thank you, Jay, for your call. Ben, do you see an analogy there?

BEDERSONYeah. So this is an example of what are sometimes called Q&A sites, question and answering sites. And this is actually a growing both business and public Web service. So there are freely available ones. Programmers use a site called stockoverflow.com all the time. There's actually a great -- that's an example of this general set of tools from stockexchange.com, where they are building Q&A sites for lots of different domains. And they are based on the reputation of the questioners and the answerers. And because this reputation pays and anybody can vote up a question or an answer and then there is a commenting part, you get surprisingly high-quality questions. So that's a free one. Then there are these commercial sites like ChaCha as an example, which might be the one that Jay was referring too. And there's another really interesting one called Aardvark. I think with two A's. So...

VOGELUh-huh. I think Aardvark always has two A's, hasn't it? (laugh)

BEDERSONRight. Yeah. Well -- and I think they were made and bought by somebody, maybe by Google. So their -- this QA thing is a big part of this world. In this -- in cases, more like it's a form crowdsourcing, like, directly getting humans to help other humans.

VOGELThanks, Jay, for that question. You're listening to "The Kojo Nnamdi Show." I'm "Kojo Show" managing producer Diane Vogel, sitting in for Kojo. We're talking with Ben Bederson, professor of computer science at the University of Maryland, College Park, and co-founder and chief scientist at Zumobi, a mobile app and advertising company about this murky world of online labor. I wanted to share with you e-mail or two that we've got. This speaks to what Jay was just talking about and I think that -- well, I'll give Anna her own words.

VOGELShe says, "You see this growing? What do you mean growing? What about the ones that are already here, in the form eHow or others like it?" I think that's what Jay was talking about. "There is legitimate crowdsourcing," she says, "but in too many cases, this is creating masses of mediocre content that are clogging up our website search results. There's a high risk of plagiarism. The pay appears to be..." And she uses a word that start with S and ends with, I think, an asterisk. "How does minimum wage play into this in the U.S. and elsewhere? Is anybody watching?"

BEDERSONSo people are watching. This is a real issue. You know, that kind of human computation things I'm talking about are a little bit different, but, you know, not completely. In the end, the wages for these are very, very low, all around, the ones, you know, as I said, Am Mechanical Turk is on the order of $2 an hour. I'm sure eHow doesn't pay a lot unless you really become expert and highly focused. And there are a number of challenges. So minimum wage law is something that is applicable to almost 200 countries around the world. So it is something our society does support. However, minimum wage law, even in this country, does not apply to all labor. It only applies to certain -- broad category, but not all labor, only certain kinds of full-time employment agreements, right? Your -- every industry has piecework...

VOGELMm-hmm.

BEDERSON...right, where you're allowed to hire someone by the item of work, and in that case -- in which case are not employees. And so, I think the justification for this kind of work is that it is an example of piecework. It is not a kind of employment. And so, I suspect our society is not going to rule against piecework. That's, you know, pretty embedded even in places with strong minimum wage laws. At the same time, you know -- so one answer is the market will decide, right? If the price is so low, then it'll move to other countries where they can, you know, maybe the levels are different and it might be able to support it. And if the quality is so low, then it'll only keep on going as long as there's value. So eHow, which I agree -- I'm pretty critical of too, in many ways...

VOGELMm-hmm.

BEDERSON...generates a lot of mediocre content that has basically makes money by Web advertising.

VOGELMm-hmm.

BEDERSONSo this is a different kind of thing. They are trying to just get as many hits as possible. In this case, they're trying to get people to pay. So I don't think the answer here is the same as the eHow ones here. Here the answer, I think, has to be more reputation of workers...

VOGELSure.

BEDERSON...less anonymity, which is a real issue, Right now both the workers and the requestors on Amazon are anonymous. You basically can sign in with an e-mail address, and e-mail address is a pretty easy to make up. And so, I think they -- I can imagine that Amazon created that way because it provides safety...

VOGELMm-hmm.

BEDERSON...where people aren't exposing their personal information to a worker that they don't know...

VOGELSure.

BEDERSON...but it then makes it so that both sides can treat each other really easily.

VOGELWell, we'll talk a little bit more about the ethics of this, both the ethics from an employer standpoint and from a collective societal standpoint. I mean, is it ethical to outsource these kinds of simple jobs to people in low-wage countries? Do you see an opportunity, for you as an employee, to either buy or sell your labor on these sites? Call us at 1-800-433-8850 to continue this conversation on the Murky World of Online Labor. Or go online and put a comment at our website kojoshow.org. You can always send us a tweet @kojoshow. I'm Diane Vogel, sitting in for Kojo Nnamdi this day, this Tech Tuesday. We'll be right back after this short break.

VOGELWelcome back. You're listening to "The Kojo Nnamdi Show." I'm Kojo Show managing producer, Diane Vogel, sitting in for Kojo. We're talking today on Tech Tuesday about the murky world of online labor and a potential revolution in tech jobs that's coming from this online world of crowdsourcing and so on. Ben -- our guest today is Ben Bederson. He is a professor of computer science at the University of Maryland College Park and former head of the Human Computer Interaction Laboratory, now a co-founder and chief scientist at Zumobi, a mobile app and advertising company.

VOGELBen, I'm wondering about the companies that build these market places and the way in which they create the relationship with individuals. When we left, you were talking about the fact that it's anonymous and that the employees don't know who they're doing this work for and the employers don't necessarily know who their employees are. It seems to me that the more troublesome part of that is that the employee doesn't know who their employer is. And the reason I say that is because seems to me this opens a can of worms for people who might wanna do nefarious things. Can you give me a few examples of the way in which these online -- give me one or two examples of the way in which these can be -- websites or marketplaces can be used legitimately, and then contrast it for me with the way in which a nefarious person might wanna use this Web -- this online labor pool to pay five cents a person to do something completely wrong.

BEDERSONSure. So, there's plenty of examples on both sides, and I'll say by both requestors and by workers...

VOGELOkay.

BEDERSON...in fact, even as a worker, you have to be careful when you go look for work, you'll often -- people will look, try to sort the hits by price and do the expensive -- the ones that pay a lot...

VOGELMm-hmm.

BEDERSON...and those often are not so good.

VOGELAh.

BEDERSONFor example, they'll say, please create a Twitter account and follow me...

VOGELAh.

BEDERSON...and I'll pay you a dollar to do that. And then they don't pay.

VOGELAh.

BEDERSONOr they'll say, write an Amazon -- five-star Amazon review of my product...

VOGELAh.

BEDERSON...and then not pay...

VOGELMm-hmm.

BEDERSON...because the requestor has all the control in this game. They get to choose whether or not to pay based on the quality. So Amazon assumes that the requestor is honest. And maybe they are and maybe they aren't. I mean, I don't have any data on how many are, how many aren't, but there's clearly plenty of fraudulent.

VOGELSure.

BEDERSONAnd for workers -- let me give you an example of a specific task. Maybe I want to understand my, I don't know, followers...

VOGELMm-hmm.

BEDERSON...so I have pictures of all of them from their, you know, their little pictures on Twitter, and so I collect them all and I ask people to tell me whether they are male or female...

VOGELMm-hmm.

BEDERSON...or whether they're young or old. And maybe somebody does that work well and I get good work done.

VOGELMm-hmm.

BEDERSONOkay? Here's a kind of a weird thing. Instead, suppose you are, say a nefarious government and are trying to track people that are illegally protesting in the streets...

VOGELSure.

BEDERSON...so you take their picture and you get volunteers. So you get people on Mechanical Turk...

VOGELMm-hmm.

BEDERSON...to try and identify them or match them up with I.D. cards.

VOGELMm-hmm.

BEDERSONSo people are doing the same kind of labor, you know, identifying pictures of people.

VOGELSure.

BEDERSONIn one case, they're helping you, in one case, they're helping the government do something they would have never considered doing.

VOGELSure. So the government is -- in other words, the government of Iran or the government of Egypt if they wanted to, opens up their, say database of DMV photos, of driver's license photos and then posts to anyone who wants to search it in -- or who needs to search it and they post this task that says here is a picture of a crowd, please identify as many people in this crowd as you can.

BEDERSONRight. And so, this goes -- this whole issue of this, you know, decontextualized Web, where you're doing work without context, all you see is a picture and saying to do something that seems completely harmless, right? But if someone had come and say, hey, we wanna arrest these people that are illegally protesting, help me identify them, you'd say no way. So there are plenty of examples of requestors that can ask you to do stuff that's either clearly right not upfront or not so obviously not right.

VOGELNow, is it really in the technologist's job description to have ethical considerations in their mind when they build this kind of working place? I mean, clearly, Amazon knew that it was an honest broker, in its own mind, when it started MTurk, right?

BEDERSONMm-hmm.

VOGELBecause they were asking -- they didn't have to worry, is the owner or the person requesting this labor gonna pay? They knew themselves, and they knew they would pay if they got quality work. How much should we, as a society, ask technologists to worry that, five years down the road, 10 years down the road, this thing you created may be used unethically?

BEDERSONWell, this is, admittedly, a controversial topic, and I've talked to many technologists and, you know, academics about this very issue. And I must say I have a very strong opinion. I believe that inventors -- which, you know, I classify myself as one, you know, these people that are creating these new systems -- do have an ethical obligation. They don't necessarily have to solve the problems, but they have to be aware of them. They have to realize that there are humans, in fact, that are earning money, that are paying -- buying food for their children based on the job, based on the work that they're doing. And part of the reason for this is, first of all, you have immediate impact on humans, in which case you have an immediate ethical obligation. But you're also setting standards that kind of build up over each other. You know, the people that first decided how far apart the rail should be on railroad tracks...

VOGELMm-hmm.

BEDERSON...right, made a decision which may not have -- seem to have a big impact that we get stuck with for hundreds of years, right? You make some kinds of early technical decisions that become de facto standards and then become official standards. And they are what they are. And so if our society decides that online labor is anonymous and it works -- it's problematic but it works okay -- we might just continue along down that path.

VOGELMm-hmm.

BEDERSONSo, instead, the people that are building these systems can be aware of the complaints and think about what is the best way to solve them. And it might be that anonymous is the best way and there's some kind of mechanism to complain and to resolve those complaints, some kind of integrated arbitration. Or maybe it shouldn't be anonymous in the first place and reputation should be built in, which, by the way, is what Amazon's competitor, CrowdFlower.com, is doing. They charge a little bit more. Sometimes they even use Mechanical Turk as a back-end, but they add a reputation system that is not anonymous.

VOGELMmm. Well, we'll come back to CrowdFlower in a moment. I wanted to give John in Fairfax a chance to get in on the conversation. John, you are in the conversation. Go ahead.

JOHNHello. I'm very pleased to be on the show today. I have been listening since the beginning of the show. And actually, your speaker has addressed a couple of the issues, but, in particular, the one I originally called for was work at home. I have a couple of other comments as well. There -- with the onset of this or the growth of this industry, I guess it could even be called, even though it's a virgin -- just a starting technology, you have work-at-home frauds that have been around for a long time and will probably invest themselves in this new technology, as creative as they usually are. And it actually provides an opportunity to sort of start from the beginning, somehow, to incorporate safety measures, as you started to talk about, in the basic technology.

VOGELJohn, just a one-sentence clarification, if you would, when you say work-at-home fraud, I mean, I imagine -- I guess when I think of work-at-home in history, I think of people stuffing envelopes or doing whatever. But do you have -- is that what you have in mind or something else?

BEDERSONYes. In fact, I've been working at a skill source center, which is a place that helps people train for and get jobs. And quite a few of them have been taken in by -- I mean, I don't know that it's always the case. But in their search for jobs, there are quite a few advertisements for work-at-home, you know, earn $1,000 a day kind of stuff.

VOGELGotcha.

JOHNAnd that can be quite disheartening, of course. But, of course, it also costs the nation in diverted efforts and diverted...

VOGELSure, opportunity costs, loss and so on. Thanks, John.

BEDERSONLet me respond to that part, if I can. So there are -- as I mentioned, there are all these issues. So, not surprisingly, the Web is open, and there are, in fact, worker-led organizations that are responding to this. So there is a website, TurkerNation -- I think it's turkernation.com -- which is essentially a forum where Web workers go and discuss who are bad laborers and what is -- bad...

VOGELBad employers.

BEDERSON...requesters and what are good jobs taken, which are not. And...

VOGELAnd for those who haven't figured it out yet, by the way, turker is T-U-R-K-E-R or Mechanical Turk is MTurk, M-T-U-R-K-E-R, just to clarify.

BEDERSONRight. Thanks. And there's also even a plug-in, a free plug-in for Firefox, the Web browser, called Turkopticon, which recognizes when you're on the Amazon's MTurk site. And every time you see a requester, it automatically integrates into that view. It adds on a rating of that requester by other workers, by other turkers. And so it's a real kind of uprising of the community to say, hey, that, you know, online envelope stuffing company didn't really -- never paid the bills. So it works -- open Web works both ways.

VOGELInteresting. I got an e-mail here from Nicholas, who said, "Using my iPhone, I discovered an application called Field Agent that crowdsources field research for corporations. A common task is visit a Wal-Malt or a Target or other big box store and verify by taking a photo or answering a question about how the products are displayed, about product displays for specific products. It seems like the shampoo brand is checking up on Wal-Mart to make sure their products are laid out as agreed. Each job seems to pay 2.50 to $8, depending on requirements. I know that this is not an online job exactly, but it seems to be what you're talking about." That's Nicholas in Rockville.

BEDERSONMakes complete sense. I think that the -- you know, the world is a big place. And as different communities discover different ways of harnessing people -- whether it's mobile, whether it is charity work, whether it is pure corporate work, whether it's espionage -- it's all gonna happen and it's gonna be people who are doing it for good and for evil. iPhones are certainly a big part of it. In fact, when we're talking about wages, another potential user are younger people...

VOGELSure.

BEDERSON...who might not be able to work. So my daughter, who I need to wish a happy birthday today, she turned 12 today and got her first iPhone, for better or worse.

VOGELHappy birthday, Dana.

BEDERSONYes. Happy birthday, Dana. I'm afraid I -- who knows? Maybe I'll find her working for $2 an hour in her bedroom. Right? You never know.

VOGELAgain, it is a brand new world. We just have a few moments left. I thought maybe you could explain, because those of us who look at it, isn't this kind of a rush to the bottom, in some ways? I mean, if we're gonna pay people pennies to do these little jobs, aren't we almost inviting a marketplace of bad products and bad users? And how -- in asking that question, you can also explain the way in which a company like CrowdFlower tries to get around that.

BEDERSONRight. So one of the basic challenges, as you described, is that you don't -- not only is there bad work, but you don't -- the requester doesn't even know the quality of the work. The whole point is you're asking a human to do something that you don't know the answer to. So it's very hard to judge. So you can only judge typically on sort of an average of the overall quality. And so any individual worker has an incentive to sort of decrease their quality, and so it kind of spirals down. In fact, economic -- economists have a word for this called a marketplace of lemons.

VOGELMm-hmm.

BEDERSONAnd one solution is by this company CrowdFlower, at CrowdFlower.com, which not only has reputation for workers, but also has what -- they have requesters put in what they call gold standard data. So for any given job, where you might have many instances of work, you say what the correct answer is. And then it uses statistical techniques to figure out -- to try and estimate how good the work is and, hopefully, avoid this marketplace for lemon. So it is a problem. And you can be sure, where there's a problem and there's money to be made, there is gonna be solutions.

VOGELWell, Ben, we have just scratched the surface of this fascinating and murky world of online labor. Ben Bederson is the -- is a professor of computer science at the University of Maryland, College Park, co-founder and chief scientist of Zumobi, a mobile app and advertising company. He's also the former director of the Human-Computer Interaction Laboratory. And for those of you who are fans of the Computer Guys, you may recognize Ben's name since Allison Druin, our computer gal, often mentions her ever so dedicated husband, Ben Bederson. Anyway, thank you all for listening. This has been a fascinating conversation on the world, the murky world of online labor. I'm Diane Vogel, managing producer of "The Kojo Show," sitting in for Kojo. We'll be right -- we thank you very much for listening. Have a good day.