MR. KOJO NNAMDIFrom WAMU 88.5 at American University in Washington, welcome to "The Kojo Nnamdi Show," connecting your neighborhood with the world. It's "Tech Tuesday." The Secret Service is looking for software that can detect sarcasm on social media. Yeah, good luck with that. In fact, understanding the slang and shorthand of social media is one of the more difficult programming challenges out there. But marketers, the State Department, the FBI and many others hope there's a way to crack the code. The explosion of social media has jumpstarted the field known and sentiment analysis.

MR. KOJO NNAMDISifting through the avalanche of tweets, Facebook posts, blogs and online reviews out there to find out what we're thinking. Whether it's gauging the reaction to the latest Nike sneaker, anticipating an uprising abroad or catching a criminal here at home. Joining us to talk about mining ones and zeroes for feelings is Philip Resnik. He has joint appointments in the Department of Linguistics and the University of Maryland Institute For Advanced Computer Studies. Phil Resnik, good to see you again.

MR. PHILIP RESNIKGreat to see you.

NNAMDIAlso joining us in studio is Kristin Muhlner, CEO of New Brand Analytics. Kristin Muhlner, thank you for joining us.

MS. KRISTIN MUHLNERThank you for having me.

NNAMDIAnd Kalev Leetaru is a Fellow In Residence at the Institute For the Study of Diplomacy in the Edmund A. Walsh School of Foreign Service at Georgetown University. Kalev Leetaru, thank you for joining us.

MR. KALEV LEETARUIt's an honor to be here.

NNAMDIYou too can join the conversation. Give us a call. 800-433-8850. How much slang, how much sarcasm, how much shorthand do you use on Twitter? 800-433-8850. You can send email to kojo@wamu.org. Shoot us a tweet @kojoshow using the hashtag techtuesday. Or go to our website, kojoshow.org. Ask a question or make a comment there. Philip, can you explain for us what is sentiment analysis and why is it so important right now?

RESNIKAbsolutely. So, in its simplest form, sentiment analysis is about identifying language that indicates that the person using the language is feeling positively or negatively about what it is that they're talking or writing about. So, the standard way of thinking about this is a tweet or is a message or is a piece of a message positive, negative or neutral? But in recent years, it's actually gotten significantly broader. It's come to, as a cover term, include analysis of emotions. So, for example, joy or anger or disgust.

RESNIKAnd a wider set of phenomena. And the reason that it's important is that a lot of what happens in language, a lot of the signal in language for things we care about is not just a question of who did what to whom. It's a question of a person's -- what's called a private state. That provides clues as to what people are thinking and feeling, and perhaps more important, it provides clues as to things that are affecting their decision making.

NNAMDIAnd so, the rise of social media has made sentiment analysis even more important. Kristin, social media holds a lot of potential for companies to learn what people are saying about them. Can you talk a little about that?

MUHLNERAbsolutely. I think, to Philip's point, about understanding deeply about what consumers are saying, what they feel, how they're making decisions. What we find is that organizations are now trying to use all of this unstructured content that exists on the web in order to really get a deep nuanced understanding of what their customers feel about their experience when they walk into a store, when they buy a product, when they use that product. Not only are they interested in understanding that so that they can position their own products effectively, but to help them understand what's being said about their competitors, as well.

NNAMDIKalev, this field was once focused on simply weighing whether reactions, as Philip said, are positive, negative or neutral. But as social media becomes ever more important, what do we now want to know?

LEETARUWell, and this is one of the things, is sentiment mining today, people will seem to think of it as kind of a new thing. It's this new, hot thing. And they forget that really the first computerized tone dictionary dates to 1962, the era of punch card computers. And I think in today's world, you think about it, about 96 percent of the papers today on sentiment mining come from computer science. So, it's really CS people coming up with new algorithms, not the applications people. As tone begins to make its way into the applications area, as, you know, marketers come up with new, increasing ways of using it.

LEETARUAs national security comes up with new ways, we're really moving beyond just positive, negative. Again, to all of these different dimensions. But tying tone, for the first time, to reality. Historically, we've sort of measured tone or written papers, said hey, cool, we can measure tone. We can do these cool graphs. What you're starting to see now, over the last few years, my (word?) 2.0 paper in 2011, was the first real in-depth study to show, look, we can actually use tone for strong forcastive capabilities for national security purposes.

LEETARUAnd now you're starting to see more and more work coming out showing, look, tone -- we could actually use tone as a proxy, not into public opinion, but into the deeper dimensions of how news media and how today's social media, and other forms, are proxying the public in a way that gives us yield into what may happen. 'Cause again, people don't take actions based on facts. They take actions based on emotional interpretation of that fact. And so by refocusing...

NNAMDIIs that what tone is? Emotional interpretation of fact?

LEETARUIn the context where you're measuring it of news and social, you are measuring dimensions of that. Obviously, you're not able to step inside of someone like you can, for example, with a brain scan. But you are getting signals. The part that you want to measure, you can't. So, you're getting a signal out of that that gives you some traction into that space.

NNAMDIThis is also important in the field of diplomacy. Talk about that.

LEETARUYeah. So, you know, historically -- today, it is still considered novel for an ambassador to send a tweet, which boggles my mind. Increasingly, some countries, the Canadian foreign ministry in particular has been very adept at understanding and making use of Twitter and other social media to be able to sort of communicate needs. Not just sort of put a tweet out there and say, hey, do this. But really understanding, how are people reacting to things? Tailoring your message very, very specifically to this, which really requires understanding how are people viewing something?

LEETARUIf you put a message out there, today, you really, A, you can't just put message out there. You have to really understand how people are reacting to that. But B, there's so much information out there today that the volume has increased so much, you really need tools like sentiment mining or other automated tools to give you enough reach to be able to see that.

NNAMDIIt's a "Tech Tuesday" conversation on sentiment analysis. We're inviting your calls at 800-433-8850. Have you ever written comments online, positive or negative, about a company or a product? 800-433-8850. Why is it, Kristin, so challenging to program computers to understand what we really mean on social media?

MUHLNERWell, in fact, there are a lot of different aspects of -- and a lot of different techniques that are being used to help really derive structure from all of this unstructured content, both in terms of breaking down that information into, sort of, bite sized chunks, subjects and the associated properties with the subjects. Understanding the tone or the emotion behind them, the sentiment. So, there are a lot of different aspects of this that make it very difficult for computers.

MUHLNEROne of the things that many companies do, and certainly that we try to employ, is to allow humans to actually annotate, or test, if you will, the outcome from the computers, to ensure that once the algorithms have tried to break down the content, that they've actually done it correctly. What humans then help them, of course, to do is identify changes in tone, sarcasm, negativity, double negatives. Things that the computer itself might not be accustomed to. What humans are also great at doing, of course, is identifying new vocabulary and new trends.

MUHLNERTwitter, for example, is a language completely in and of itself. Completely different from what we might see on a review site like tripadvisor or yelp. So, the algorithms that are required to actually process that information are much like those that are required to process a completely different language other than English.

NNAMDIPhilip, why is Twitter, in particular, such a challenge?

RESNIKWell, Twitter, in particular, is a challenge, largely because of its 140 character format, so a lot of conventions have emerged where people take advantage of short hands in order to fit what they have to say in a -- you know, in short form. Another aspect of it that makes it difficult, and it relates to what Kristin was just saying, is -- has to do with context, right? One of the most difficult things about language in general is that understanding language is not just about understanding a language. We take advantage of our shared knowledge. We take advantage of our shared social context.

RESNIKTwitter is conversation. A tripadvisor review is relatively self-contained. But, you have to go beyond the individual tweet, the individual statement in order to get the full context. Because otherwise, exactly the same statement could be interpreted sarcastically or not.

NNAMDIAnd this is not my mother's shorthand that we're talking about here. This is the shorthand...

RESNIKNo. This is, and what's interesting is that it varies across languages. In China, the shorthand they use on their Twitter equivalent uses conventions that we would never even imagine for English. But no, the spelling conventions, the shortening of phrases, the use of punctuation, the use of emoticons. All of this makes it more similar to conversational speech than it is to written language, but with a twist.

NNAMDIIt's a "Tech Tuesday" conversation on sentiment analysis with Philip Resnik. He has joint appointments in the Department of Linguistics in the University of Maryland Institute For Advanced Computer Studies. He joins us in studio, along with Kristin Muhlner. She is the CEO of New Brand Analytics. And Kalev Leetaru. He is the -- a Fellow at the Institute For the Study of Diplomacy and the Edmund A. Walsh School of Foreign Service at Georgetown University. If you'd like to join this "Tech Tuesday" conversation, give us a call. 800-433-8850.

NNAMDICall it bad spelling, non-existent grammar, letter expansions when people draw out a word with multiple vowels for emphasis. These are things we comprehend easily, but how difficult is it for a computer to comprehend these things?

LEETARUYeah. Obviously, these make it tremendously more complicated. So, social media, in particular, a lot of work in sentiment mining today, a lot of the more sophisticated work, has tried to make use of all this existing, what we call natural language processing tools. Basically, tools that allow machines to read text and understand it, but most of those tools were designed for New York Times text. Beautiful, grammatically correct, pristine text. I've seen a number of sentiment mining commercial tools that if a comma is out of place, a single missing comma, the entire algorithm completely fails.

LEETARUObviously, on Twitter, people aren't using beautiful grammar and commas, but more critically, the domain that you use on Twitter -- the domain that you're speaking to on Twitter has dramatic changes. So, for example, if you -- you might say, you know, a common one in entertainment is to say it's blowing up the charts. Or, you know, it is too hot for words versus it's too hot outside. And so, if you're talking about the weather, the exact same phrase can have dramatically different context.

LEETARUAnd so, trying to understand -- and these are things that are missing from most tonal dictionaries today, really don't understand the difference or something as simple as go to hell versus hell yeah. I think something that's been very interesting is that up to this point, today, even today, a large number of the commercial products out there are still using individual words for tone, under the assumption that the word, you know, fantastic, probably means more than the word horrific. But you could say it's fantastically stupid, or fantastically, you know, awful.

LEETARUAnd so, what you're starting to see a movement toward more sophisticated techniques of understanding this, but these are still -- I think sentiment mining is still so new that today, really, all you have to be able to do is put a pretty graph up and say, hey look, I can show tone of tweets. There hasn't been enough, sort of, testing of these algorithms. I think this is the next horizon. The field's maturing enough to where we no longer say, isn't it great that computers can do something useful? To well, how accurate is this and how do we focus on the accuracy part, not just the cool algorithms.

NNAMDIWell, like the rest of us, computers apparently have an even more difficult time understanding teenagers. Part of the challenge is that slang is often not meant to be generally understood. Can you talk about how teenagers, for example, use social media?

LEETARUWell, this is actually kind of interesting. So, we did a project a couple of years ago. We took nine PhD students, political science students, and had them read New York Times editorials from political campaigns -- presidential campaigns going back to 1945. And I forget the candidates. Some were in the 50s. One of candidates was a horrific public speaker. So, all the New York Times editorials said he, once again, had this amazing speech, incredible eloquence. And these humans, all of them uniformly coded them as extremely positive.

LEETARUAnd I think this is -- this is very telling to us, because, you know, we talk about, well, computers don't get sarcasm. Well, neither do humans when you take them out of the context in which they know that. And what was interesting is all nine of these students had wildly different assessments of tone. We just said, you know, either negative, neutral or positive. Nothing complex. And these were people selected with no background in sentiment mining.

LEETARUAnd we said, by the end of the semester, go off and come up with a way so the nine of you actually agree, for the most part, on these editorials. And they had no knowledge of how computers do this, and at the end, they literally came up with a dictionary. They literally came up with a dictionary that said, if you see this word, do a plus one for positive, a minus one for negative. And that was the only way that these nine students could actually agree on these editorials.

LEETARUAnd I think, this is very fascinating and telling of -- you think about, well, what are the limits of computers? What are the limits of humans? And then you add in the things that teenagers -- they try to use words that we don't understand. Or, you know, you go back to a Laurel and Hardy comedy -- I love Laurel and Hardy, and you have all these words like, QT, and all these acronyms and things that meant, basically, you know, on the low. And, you know, a lot of these things, you read the text or you listen to them and half the time, when they're talking about, you know, emotional context, you have no clue what they're talking about.

NNAMDIThat's the point.

LEETARUCause the language has changed.

NNAMDIThey don't want anybody else, but their peers, to understand what they're talking about. You were going to say?

MUHLNERI think this is interesting, though, because I think, clearly, we have humans who actually do annotations to help create training sets, exactly to your point, to support some of the other techniques that we use and the algorithms that we use to create higher levels of performance in the solution. And yet, even amongst humans, we only find a 90 percent agreement rate. And so, one of the things that we'll often do is apply incremental techniques on top of some of these annotations to help understand context. Right?

MUHLNERFor example, if I say, this is a wicked place, and I'm in Philadelphia, it's very likely that I mean something different than I say, this is a wicked place and I am sitting in Wichita. Right? So, even the geography or the location of the speaker, other meta-data that can be gathered about the individual can help really provide some increase in performance in the system.

RESNIKYeah, and that highlights one of the things that needs to be paid attention to as these analyses get deeper, which is that simple dictionary based methods are not going to succeed. Nor are simple word based methods. Partly because of context and partly because of contextual factors like geography, but also language change. And so, the real movement in natural language processing since the 1990s, and particularly in sentiment analysis now, is not about trying to build knowledge resources, like dictionaries. Perhaps you do to boot strap. It's all about machine learning. It's all about annotating data or having data where you have information about the sentiment.

RESNIKAn Amazon product review, for example, has five stars, so you know that it's positive. And using machine learning to detect the signal in the language in ways that a human analyst simply wouldn't catch.

NNAMDIOn to the telephones. Here is Janet in Pikesville, Maryland. Janet, you're on the air. Go ahead, please.

JANETHi. I always thought that thought was more important than sentiment, unless you're trying to train computers to recognize sentiment so that computers can run things and run social life based on sentiment, which reminds one of either 1984 or Brave New World, where they had like mob rule, according to sentiment. And, I mean, you know, all the, fortunately, today, still, all the important social institutions are run, like the government, and the democratic process and the Supreme Court, et cetera, are run on thought rather than sentiment.

NNAMDIBut I think one of the things we're discussing here is we're trying to figure out the sentiment behind the thoughts. And, I guess, Philip, you can speak...

JANETYes, but why? I mean, you're valuing sentiment over thought.

RESNIKWell, I think Janet's got an interesting point, and there are really, at least, two answers. One is that there is an enormous literature emerging that we don't run things based on thought. If you look at the work of Conoman, (sp?) if you look at Dan Ariely, there's a whole cottage industry showing that a lot of our fundamental decision making is based on emotional context and emotional perception. So, one important reason to be doing this is to better understand the way that language affects our perceptions, so that we can shine a light on it rather than letting the people who manipulate language simply control the way that we think.

MUHLNERAnd certainly from a purely commercial point of view.

NNAMDIThat's what I was thinking about.

MUHLNERIf you look at businesses -- yeah. If you're looking at businesses who are really trying to understand why customers buy, why do they walk into my restaurant, why do they walk into my hotel? Often, a brand's understanding is derived from information that is imperfect. And so if they actually go to the web, if they go to these social sources, and they look at this content, what they'll find is what is both said and what is unsaid is incredibly important in helping inform what that decision making process is.

NNAMDIGotta take a short break. When we come -- you want to say something, Kalev?

LEETARUOh, yeah, I mean, real quickly. I think, oftentimes, emotion -- you think about a controversial issue like guns rights. You know, there's the factual pieces behind that, but the way in which, you know, depending on what side you are of that has a lot to do with how you emotionally perceive the facts that are in front of you. And I think a lot of this -- when you think of the national security perspective, you know, you think about -- there are many countries that have identical, you know, situations on the ground.

LEETARUIt's how people perceive that. Do they perceive this as, well, this is just the way it is and it will never change. I have no way of changing this. Versus, do they say, you know, we don't have to take this anymore. We can fight. So that emotional -- the way in which people emotionally perceive their life role in the world around them has a huge, huge impact. I mean, the way I always say it is, people don't take action based on facts. They take action based on emotional interpretation of that.

NNAMDIThe fact that you're pointing that gun at me is what's making me feel like giving you my money. We're going to take a short break right now. When we come back, we'll continue our conversation on sentiment analysis. It's "Tech Tuesday." You can call us at 800-433-8850 or send email to kojo@wamu.org. Do you think companies listen to your online feedback? Give us a call. 800-433-8850. I'm Kojo Nnamdi.

NNAMDIWelcome back to our "Tech Tuesday" conversation on sentiment analysis with Kalev Leetaru. He is a Fellow at the Institute For the Study of Diplomacy in the Edmund A. Walsh School of Foreign Service at Georgetown University. Philip Resnik has joint appointments in the Department of Linguistics and the University of Maryland Institute For Advanced Computer Studies. And Kristin Muhlner is the CEO of New Brand Analytics. You can send us a tweet @kojoshow. We got one from Raymond who says, wait, you mean sentiment analysis has moved beyond the responses of hung over undergrads in a Psychology study? Kalev, talk about trying -- the origins of trying to understand sentiment here.

LEETARUWell, I mean, and Phil can probably speak more to some of the earlier, earlier days of it, but, you know, in its current state today, certainly, I mean, I have worked with a good number of the Fortune 50s on applying sentiment analysis both to social media, but also traditional -- other types of data. Sentiment today is really heavily used for marketing. It's growing in the use now, in the national security arena, because we're increasingly finding that it's giving us -- it's giving us a feel into how populations are sort of perceiving things in the moment.

LEETARUBut it allows us to measure bursts of that. Or, for example, Doug Naquin, Director of the Open Source Center, here in the United States, which is the branch of the US government that monitors media around the world, gave a public interview, I think, two years ago, where he acknowledged that when Osama Bin Laden was killed, one of the studies that they did was to look at how the world reacted to that. In social media, mainstream media, so just purely public information streams. But looking at public tweets, public Facebook, public information, and then news media coverage.

LEETARUAnd looking at how is the world -- what were the pockets of things that reacted either way? A fascinating finding was Europe. A lot of European press being very, very negative about it, as the incursion to sovereignty. That, you know, yes, you got him. That was good. But you really shouldn't have invaded another nation to do that. That's very fascinating. That's not unexpected. But the ability to demonstrate that at scale and not say, well, I read a couple article -- a couple newspapers and this is what it seems like people are saying.

LEETARUSentiment analysis gives us the ability to finally move and put numbers behind it and say, instead of one person's gut feeling from reading five newspaper articles, to being able to say, this is broad across the Twittersphere, and that, I think, is a fundamental push forward where data's helping us.

NNAMDIIs it more challenging, in fact, because computer engineers are doing the programming where in the past, linguists and psychologists were involved?

LEETARUI think -- we need to have a connection between those. One of the things we learned is that, from testing, I've been working on a couple of projects with the Sci-Fi Channel. So, the Twitter popularity index that we did for "Opposite Worlds" and now "Face Off" coming up this season. One of the things we learned from looking at a huge number of the products that are out there today -- increasingly, people are using machine learning, to build these dictionaries. But the whole point of using machine learning is to have the machine build this enormous dictionary of huge numbers of terms.

LEETARUAnd, but right now, there's less -- there's not a lot of having humans really edit that down. So, we see, for example, the word Obama. Companies that built their products prior to Obama's -- when Obama was first elected -- very -- the word Obama has a very positive connotation. Companies that are coming out today -- they're building their algorithms today using Twitter data, Obama's a horrifically negative connotation. Economists, doctors are really horrible, because they usually give us bad news.

LEETARUAnd so these -- and so, the word so has a hugely negative connotation just by itself in many of these products. And that's because when we talk about things, we say, you are so dumb. Or you're very intelligent. So, the word so usually is an amplifier for negative words, oftentimes. These are fascinating findings about how we use language. But you don't want these built into your technology, because if it picks up -- if it says, for example, that the word Obama is horribly negative, and the word McCain is very positive, well, if you're using -- if you're a political polling company and you're using it to do polling, your results are automatically, completely skewed.

LEETARUAnd so, that, I think, is something that we need to have more of a blending of the two worlds.

NNAMDIWell, that blending is represented in person in Philip Resnik. You stand at the intersection of linguistics and computer science. What say you?

RESNIKWell, so, a couple of comments. One is that fortunately machine learning has evolved beyond simply acquiring information in dictionaries. We had Richard Socher and Geoff Hinton on for example, here on Kojo, not all that long ago, talking about more advanced techniques that capture interactions between words, phrases and terms so that you don't simply rely on learning lists of words. And that helps get around some of the problems that Kalev was talking about. Richard's work, in particular, really did a very good job of looking at syntactic context so that simply the occurrence of the word so or the occurrence of the word Obama wasn't going to mislead you.

RESNIKSo, the more sophisticated work is actually taking that into account. The other thing to note though, and this goes back to the email that you just read. Is there are long term, long standing issues that we have inherited here. The issue of sample bias being probably one of the most important. We didn't find out how Europe felt about, you know, Obama's -- Osama Bin Laden -- boy, that's a common, that's a common...

NNAMDIA common mistake.

RESNIKA common slip, isn't it?

RESNIKAbout, about that event. We found out how people who actually expressed themselves publicly on social media feel about that. In the same way that psychology experiments, you know, a room full of college sophomores may be telling you something about how the brain works, or they may be telling you something about how college sophomores' brains work. These are issues that need more attention and as Kalev pointed out earlier, need to be addressed more rigorously than they have been. In a survey on TV, if there's a political poll, they're going to give it with a confidence interval. They're gonna say plus or minus four points.

RESNIKIn most commercial applications of sentiment analysis, they're simply going to give you a number. And that's something that needs to change, because we need to actually do things more rigorously.

NNAMDIGlad you brought that up, because Kristin, many of us have encountered this with email. We receive an email that we read as angry or curt that's not meant to be. It's really tough to read tone online, as Kalev was saying earlier. So, where are we now, in terms of tools that can successfully glean emotion or tone from social media?

MUHLNERI think we've made incredible progress, and I think you only have to look at the kinds of companies that are using this data today, and how it's informing their strategy to realize. So, we work with some of the largest brands in the country. McDonald's, Hyatt, Dick's Sporting Goods. Helping them really understand why their customers walk into a store, what they buy, why, what their experience is with it. And because we're able to take this content now, break it down, categorize it into meaningful aspects of their experience. What do they think about the food, about the service, about the pricing and value? About the facilities? Are the bathrooms clean or not?

MUHLNERWill they come back? You know, they express, these individuals express all this information in this unstructured, unsolicited data, and it gives really deep insight into what's going on. And we have now have the tools and techniques to do that. Not only do we have it, but we've got a statistically relevant sample of data that we can use to really derive true judgments and decisions on. One of the most common misperceptions today is that it's only 18-year-olds who are talking in social media. In fact, when we look broadly across all of the organizations that we monitor, and it's about 150,000 businesses worldwide.

MUHLNERWhat we see is roughly two thirds of all comments are written by people ages 35 and above. So, for many industries, that's exactly the target audience that they're looking to try to achieve. The other bias that is commonly perceived is that people only go online if they're going to say something negative. That is wildly incorrect. In fact, most of the commentary online, in social media, on review sites, is on average positive. So, it's a great place to really understand both, what people are saying, but, you know, why they really love a particular experience, a particular product, a particular brand.

LEETARUI think, though, we, that it is interesting. You look at like news websites. More and more news websites are shutting down their comments sections, user comments sections, because it was just pure negativity and trolling. On this thing we did a couple of years ago showed that global news coverage has become steadily more negative, linearly more negative over the last 30 years. Something, I do think that, also we have to always be careful about is are we talking about tone mining here in the United States where Twitter population is more demographically open.

LEETARUOr are we talking about a country, say, Bangladesh, where you're looking at a fraction of a percentage of just one political party and just the elites within that party. And so, but I think, I think that is a very critical issue. But also, that's really more of the data revolution verses sentiment mining. Because sentiment mining is technology. Usually, it's often conflated with social media because that's where most companies are applying it today. But again, coming from a national security space, oftentimes, social media is less useful to us in many countries. It's news media.

LEETARUIt's, you know, blogs or it's, for example, broadcast. Or, you know, a local community radio show, say, in rural Zimbabwe. Which isn't, you know, a traditional news media program by our standards, but gives us a lot of information and a lot of things that you can mine. So, I think it's always important to separate out tone -- the concept of sentiment mining from social media, because they are very, very distinct. And again, there are many ways. We've worked with many Fortune companies where they're doing polling on the ground.

LEETARUThey are actually sending people out into areas. And what you come back is today that's assessed by a psychologist, but oftentimes, multiple psychologists may come up -- again, this is where sentiment mining came from -- wildly different assessments of that. And so it gives you sort of a -- it gives you an additional sort of devil's advocate into that conversation.

NNAMDIWe got a tweet from Charity who writes, I write online reviews whenever service, food or an experience is exceptionally bad or good. Mediocre places don't get a response. Philip, how do Amazon star ratings, for example, help computers, or help teach computers how to glean meaning from, say, an online review?

RESNIKWell, again, paying attention to the fact that, as Charity points out, different people are motivated to actually participate in this online world or not. So, recognizing that there is an issue with the data that you're getting in -- the basics of how those sorts of ratings are used is really straight forward. The concept, which is often called supervised learning, involves taking data that is tagged with a label or some variable that you're interested in, and subjecting that to analysis in order to learn signal within the data, including interactions of words, not just presence or absence of particular words or phrases.

RESNIKThat correlate or are predictive of a particular rating. And so, essentially, what these things are trying to do is model and reproduce what humans are doing in the aggregate. What signal leads to which direction, in terms of the ratings.

NNAMDIKristin, what approaches does your company use to understand what's being said about a particular restaurant, a particular hotel?

MUHLNERAbsolutely. So, we actually apply all the techniques that have been discussed here. And generally, I think most of the most successful commercial products actually do that. Looking at annotative models, looking at, you know, a variety of different methods to actually construct the most real picture of the data possible. So if, for example, someone says, I went to the Park Hyatt this weekend in Georgetown. I had an absolutely amazing experience, but it took me 20 minutes to check into my hotel. And once I got into my room, unfortunately, there wasn't any toothpaste in the bathroom.

MUHLNERWe just learned several things about that guest's experience. Overall, they thought it was really fantastic. But the check-in process was slow. And the room amenities were not as expected. Each of those individual components can be categorized and then can be scored for sentiment. And what that allows us to do then is aggregate all that information up and create a numerical score. Which becomes, essentially, a normative way for that business to really understand how they're doing. And then how they compare to the Westin across the street.

MUHLNEROr the Ritz Carlton down the street. And this becomes really important really helping derive, you know, a true objective assessment of this data to be used and actually make a decision.

NNAMDII'm listening to that experience that you created at this particular hotel and wondering where it falls along the lines of the 80/20 rule. Can you explain to our listeners what the 80/20 rule is?

MUHLNERSure. Absolutely. One of the things that we often find, particularly if you look at traditional methods of gathering consumer sentiment, is that companies have over-surveyed their populations. And they've asked all kinds of questions, trying to be as knowledgeable as possible about every different aspect of that consumer experience.

MUHLNERThe problem that they've done then is they've shined -- as a result, they've been shining a light on aspects of the experience that may not really matter. We did a major study for one of our big hotel customers, looking at their guest experience with their restaurant outlets in their hotels. And one of the things that we found was there was a dramatic difference in the survey feedback about service in the restaurants relative to the service feedback that was found online.

MUHLNERWhen we looked more deeply at the survey, what we discovered was they were asking specifically about service timeliness in the restaurants, in the hotel. When we looked online at the social data and the review data, what we discovered -- what people were really talking about was service friendliness and attentiveness. Did she say my name, did she come back quickly when I needed some help, right? So it didn't matter that they had a bunch of people running around really quickly being really grumpy.

MUHLNERWhat the guest really wanted was someone who approached them with care. And so this is the point that we try to drive with our customers, which is listen to the data and let it tell you what's important versus prospectively defining what you think is important on top of that.

NNAMDIEighty percent of the comments will be about 20 percent of the experience.

MUHLNERExactly.

NNAMDIThat's what it all boils down...

MUHLNERExactly. That's the net nut, yeah.

NNAMDIHere is Miriam in Arlington, Va. Miriam, you're on the air. Go ahead, please.

MIRIAMHi. Thank you so much. This topic is totally interesting, and I think with our data deluge in information waves everywhere, what you're doing is really important. I'm curious about the -- how automatic speech recognition, if at all, is being used and interrelated with sentiment analysis in text mining. And in particular, we've all been very aggravated with automatic phone systems. We read the newsfeeds, and they're often wrong. The two could inform one another a lot. And I'm just curious where the state of the art is. I'm using you instead of looking it up on Google.

NNAMDII don't know. Kalev, the feel of natural language processing in the field of computer science, what is it? Maybe that can help answer Miriam's question.

LEETARUWell, I mean, this question, I mean, one of the reasons that sentiment mining today has not incorporated many of these capabilities is that traditionally most of the application today is really on text, on things that has already been converted out. So, for example, you may have someone using a phone -- an automated phone system, and that's then reduced down via, you know, a speech recognition system to a line of text. Increasingly, though, many call centers are looking for vocal stress patterns. They are looking at those elements. (unintelligible) other elements there.

LEETARUSo, increasingly now, obviously not every call center, but increasingly call centers are starting to use tools that will flag that someone's becoming more angry and then immediately shunt that to a supervisor. That's a very different area of -- I mean, certainly that's a very -- and Phil will probably will speak much more to this. That's a very different area where, you know, at least my work primarily focuses on is areas where we don't have that recovery, so people who are communicating online, for example, in news media or in ways where we can't recover the vocal element of it.

LEETARUThat actually, again, you know, you think about body language. Body language conveys so, so much. You know, the vocal stress patterns, you know, obviously (unintelligible) I mean, these convey just enormous amounts. So sentiment mining today really is sort of at the world where we're dealing with things like an online review where you really don't know -- and that, again, that comes a lot to sarcasm.

LEETARUA lot of sarcasm is understanding the shared background there but then also watching the person, if they're laughing afterwards, if they're doing other pieces to it. Even laughter, or someone would say ha ha ha, sometimes you'll see that on social media. Someone will say ha ha ha at the end of their tweet.

NNAMDIMm hmm.

LEETARUBut this also is one of the things that we found. Oftentimes smiley -- I mean, online, it's very interesting. Emoticons, you know, a lot of more primitive tools, and believe it or not, a lot of commercial tools, still look at emoticons from the context of, if there's a smiley face, what does this mean? It's really interesting because we found very, very interesting interplays for sarcasm between, you know, the texts there and how people are using emoticons. And, again, I think one of the interesting elements that this question kind of peripherally relates to is that notion of the so what.

LEETARUSo we can measure emotion. What do we do with this? And, like, Fortune -- you know, working with one of the Fortune Fives a couple of years ago, this was a big question that was raised from their end, was, look, we're using them, number of commercial sentiment mining tools today. There was all these beautiful graphs. Our marketing department can show us, oh, totals going up, down, up, down, but today it still uses a virtual comment card. It's, you know, hey, the bathroom's dirty.

LEETARUThat's great, but, you know, you can hear from more -- you know, not everyone fills out the comment card and drops it in the box. And certainly the local franchise, if you're a restaurant, might, you know, not relay that up to corporate headquarters. But why do we go beyond that? And I think a lot of that is looking at, you know, beyond your own borders, you know, thinking about what are the broader issues that come into play? So things like a phone system, you can imagine being able to not just simply flag, hey, someone was upset on the call, but proactively shunt that to someone if they do it.

LEETARUOr, you know, a particular company that we're been working with in the past, one of the interesting things they were looking at was not just their own products but the field as a whole that they operated in, what was the future of that, and really looking at what are things that are coming and going? Like, one consumer products company we've been working with recently, what colors are -- so forgetting about (word?), what are the positive and negative experiences that people have in everyday life? What are colors that are part of that? So blue or red.

LEETARUAnd by aggregating across everything from shoes to food to everything, you start actually finding -- you start -- and, again, this is a whole area of psychology deals with emotional attachment to color. But we find is that actually changes -- you know, we -- actually changes month by month, season by season, year by year. We actually pick up the fact that red now is going out of favor in Asia. It's becoming more popular in Europe today. Very shortly now, we're starting to see the glimmers in the U.S. We can actually start measuring those at intervals using computers that we've never been able to see before.

NNAMDIGot to take a short break. When we come back, we'll continue this conversation. If you have called, stay on the line. We'll try to get to your calls. If you'd like to call, the number's 800-433-8850. Do you think a computer can be programmed to understand the slang and shorthand of social media? 800-433-8850. I'm Kojo Nnamdi.

NNAMDIIt's Tech Tuesday, and we're talking about sentiment analysis with Kristin Muhlner, CEO of New Brand Analytics. Kalev Leetaru is a fellow in the -- at the Institute for the Study of Diplomacy in the Edmund A. Walsh School of Foreign Service at Georgetown University. And Philip Resnik has joint appointments in the Department of Linguistics in the University of Maryland Institute for Advanced Computer Studies.

NNAMDIWe got an email from Tom in Silver Spring, who writes, "I totally agree that my experience in hotels and restaurants is all about service. But too often that aspect falls short. I'm glad to know they're listening to my comments. And I hope my next hotel stay reflects that." Kristin, companies are always asking for feedback online from customers. We write comments about everything from airline legroom to what we think of a new shampoo. Now, this surprised me. What do most companies do with all of these general comments?

MUHLNERWell, increasingly, they're using it both to help do better personalization in marketing but also to really improve operations. One of the things that's really interesting, if you look at our customer base today, roughly half of our customers are the chief operating officer, that individual who's responsible for implementing and running the business. And the other half is the chief marketing officer, that individual who's responsible for maintaining the brand.

MUHLNERAnd I think what that's showing is that there's a real unification that's happening in these disciplines across a number of organizations that's being tied by the fact that their customers are going out, talking about them online, and in fact controlling brand perception at the same time that they're providing deep insight into the customer experience.

MUHLNERSo, you know, we have customers who actually use this data both to make strategic investment decisions, strategy decisions -- what products should they offer, what should their menus look like to what color should the rooms be -- to making decisions about how they should staff their stores. How many sales associates do they need to have between the hours in 7:00 and 9:00 to ensure that they don't see negative drops in sentiment based on a lack of available personnel? It's incredible, what we're seeing in terms of these.

NNAMDIBut do most companies have the personnel to read -- to actually read these comments?

MUHLNERWell, increasingly, we're seeing analytics teams who have got responsibility for actually taking advantage of this data. And certainly one of the benefits of using this type of technology is it's helping to really aggregate and summarize and create structure from what could be millions of comments about a brand and really trying to synthesize that down into something that is consumable by a small team or an individual that can then be distributed across the organization.

NNAMDIWell, Philip Resnik, here's the question that you have been expecting. It comes from an email from Jack in Silver Spring. Jack writes, "I know I'm sounding like a Luddite here, but this is one of the most disturbing discussions I've heard about the future of the use and misuse of big data since the digital explosion started a decade or so ago.

NNAMDI"I think the fact that the guests have mentioned that the NSA and marketing are the biggest users of this field of technology is very telling as to the potential for even more intrusion and us humans being just commoditized. Please explain why I should not be worried about the social implications of the mining and selling of this kind of information. And what are the metrics that demonstrate that this information is actually accurate?" Phil.

RESNIKWell, that is a decidedly non-sarcastic question.

NNAMDIThank you.

RESNIKVery earnest, and my short answer is that you should be worried, that this is something that we do need to be paying attention to. I know that, you know, when my son reached Internet-accessibility age, I pointed out to him that things that he thought were private are not. And so the -- what's really happening here is there are really two questions here. One is about the privacy implications. And the other has to do with, even if we weren't worried about the privacy, to what extent can we believe in the accuracy?

RESNIKYou know, tackling the first of those first, briefly, people need to recognize that when they are participating in social media, they are making a choice to speak in the public square. And there is a qualitative change that comes with quantitative change now that we're able to analyze this extremely broadly. And that is worrisome. And I think people do need to be paying more attention to it. At the same time, we also need to be paying attention to the decisions that are made on the basis of this data.

RESNIKIt's not -- as both Kalev and Kristin have pointed out, it's not just enough to generate some numbers. You want to be driven by the data. You want to be actually looking not just at a particular number but at what's going on in the data, not that something is being perceived negatively according to your metric, but the themes, the topics, the clusters of attention that are being, you know, addressed there.

RESNIKSo the short answer to this is we really are in a -- and I hesitate to use the phrase -- a brave new world here where people need to recognize that what they're saying out there is reaching a lot of ears. And that's one of the reasons why my Facebook feed is friends and why my Twitter feed is professionals only because I've spent enough time in this field to have decided that I don't need the entire world seeing everything I say.

NNAMDIKristin...

MUHLNEROne of the...

NNAMDIGo ahead, please.

MUHLNEROne of the things that we do to try to help improve the voracity, if you will, of the data is to correlate sentiment with other measures, objective measures, if you will, of performance, like, point-of-sale information in a retail establishment, revenue, or revenue per available room in a hotel, for example. And what we have found is that there's a high degree of correlation at the location level between prevailing sentiment online and the actual revenue performance of a business.

MUHLNERAnd, in fact, in some cases, there's a predictive quality to the sentiment that can actually tell us that, you know, hey, a couple of weeks out or a month out, this business may be doing very well, or it may be doing very badly. That can be incredibly valuable for a business to help them understand how to make operational improvements in order to take advantage of, you know, new opportunities to either price better to be more competitive or to change the customer experience.

MUHLNERAt the end of the day, what customers -- what our customers are really looking to do with this data is to create a better, more authentic customer experience, and in turn hopefully create more wildly happy customers.

LEETARUAnd I think in terms of the privacy issue, I mean, that really doesn't have anything to do with sentiment mining because, again, if you're applying sentiment mining to, say, comment cards that were submitted, you know, in-store, I think, you know, where it comes out is we start talking about social media. But really, you know, my philosophy on that is that's really a discussion of, you know, whether sentiment mining exists or not, humans will be studying and analyzing all that.

LEETARUSo that's really a discussion of, you know, where should social media be used, who should be studying that? That's a side issue from sentiment analysis because, again, sentiment analysis is only really getting to how you're posturing, how you're describing things. It's not your actual thought. You know, I always say, you know, who knows you better, Facebook or Google?

LEETARUIf you wake up in the morning with a rash, you know, you're probably not going to post your Facebook at, you know, I mean, I've got a rash on my shoulder. You know, you're going to Google that. You're going to search that. And -- or you're going to call up your doctor. You're going to call up someone. So sentiment today, even when you're using social, what you're putting out there is still a -- it's a posture to the world. So you're getting at how someone wants you to feel.

LEETARUIt's sort of like news media. News media is a good proxy for, in a repressive regime, how the government wants, you know, what they want people to see. In terms of, you know, privacy, I see it -- I see that as a separate issue because, again, this isn't getting to some deep dark part of your mind that you're not dreaming of. It's really, again, still stewing at that surface level. But I think also, one quick comment, too, going back to the earlier call about -- the earlier question about vocal stress...

NNAMDIMm hmm.

LEETARU...there's a whole other area of sentiment mining that really hasn't been touched much, which is imagery. And even human assessment, even within, you know, the field of psychology, you don't have robust protocols yet that are good enough to be actionable where you could have a bunch of humans look at an image and say, what's the emotional connotation that this will generate in this particular person?

LEETARUAnd then I think is -- you know, another Holy Grail is, right now, sentiment mining is still just text. You know, vocal, it certainly exists. And there isn't a lot of work on vocal. But that's still kind of considered a fringe area, not -- true sentiment imagery really doesn't exist. I think, over time, we're going to start thinking of it more holistically, and that, I think, is that blending of all those, including the brain scanning and all that stuff.

NNAMDIWe're almost out of time. But I want to get in Jim in Chevy Chase's -- Chevy Chase, Md.'s observations. Jim, you're on the air. Go ahead, please.

JIMHi. I'd like to just mention that, back in 1978, when we were -- at Texas Instruments, we used to try to find out what happened to all the reader service cards, all the user service cards. And we found out that one woman was collecting those every day and then throwing them away at night. She didn't know anything about it. She didn't take any statistics, but she was just putting them in a box. And every night the box was taken away by the janitor.

NNAMDIYou think that's the equivalent of what's happening with online comments today on products?

JIMProbably.

NNAMDIWell, in this case, Kristin Muhlner, you get the last word.

MUHLNERWell, I think certainly what we're seeing is pretty much every company today has acknowledged the fact that their customers are talking about them online whether they want it to happen or not and that they need to take an active role in managing their reputation and managing the way that consumers are experiencing the brand and then directly engaging with those consumers. We feel that this technology and this medium has created incredible net positive for consumers in terms of voice and a way to engage.

NNAMDISo they're paying attention.

MUHLNERThey're paying attention.

NNAMDIKristin Muhlner's the CEO of New Brand Analytics. Kristin, thank you for joining us.

MUHLNERThank you.

NNAMDIKalev Leetaru is a fellow at the Institute for the Study of Diplomacy at the Edmund A. Walsh School of Foreign Service at Georgetown University. Kalev, thank you for joining us.

LEETARUThank you.

NNAMDIAnd Philip Resnik has joint appointments in the Department of Linguistics in the University of Maryland Institute for Advanced Computer Studies. Good to see you again, Philip.

RESNIKGreat to see you. Thank you.

NNAMDIAnd thank you all for listening. I'm Kojo Nnamdi.