Medical Avatar iPad app

Medical Avatar iPad app

Can seeing an image of a healthier you change your real-life health? Experiments in which people see 3-D images of themselves before and after long-term exposure to sun or cigarette smoke can have a powerful effect on behavior. Cutting-edge research like this demonstrates the surprising ways art and technology can transform medicine. We speak with a pioneer in this emerging field.


  • Virgil Wong Co-founder, Medical Avatar LLC; Artist and Researcher, Columbia University; Vice President of Interactive Media, Element 115; Adjunct Assistant Professor, Media Studies, The New School

The Medical Mirror

Using Microsoft Kinect, custom software and 3-D anatomical modeling, the Medical Mirror creates a moving reflection of your body from the inside out. Data from your hospital or doctor’s office is then mapped to this digital version of yourself, which is also accessible via your smart phone, tablet, or desktop computer. Your medical avatar is designed to help you better understand and manage your health in the past, present, and future. This visualization of your medical records is also an interactive form of individual and collective portraiture that you can use for art, empathy, story-telling, or creative experimentation.

TED Talk: Virgil Wong And The Medical Avatar


  • 13:06:43

    MR. KOJO NNAMDIFrom WAMU 88.5 at American University in Washington, welcome to "The Kojo Nnamdi Show." Connecting your neighborhood with the world. Later in the broadcast, think Eastern European food and you think potato dumplings and cabbage soup but this cuisine is changing with the times. We'll talk with a local restaurateur and "The Cookbook" author.

  • 13:07:10

    MR. KOJO NNAMDIBut first can seeing an image of a healthier you change your real life health? What if you could see an image of your own hand or face aged by sun and by smoking? Would it change your habits? What about a 3D anatomical rendering that incorporates information from your medical records?

  • 13:07:30

    MR. KOJO NNAMDIPioneers in this emerging field think art and technology can transform the way we think about our health. Joining me in studio is Virgil Wong. He is the cofounder of the company Medical Avatar and vice president of Interactive Media at Element 115 Technology and Design companies for the medial field.

  • 13:07:50

    MR. KOJO NNAMDIHe's also an artist and researcher at Columbia University and a professor of media studies at The New School in New York. Virgil Wong, thank you for joining us.

  • 13:07:59

    MR. VIRGIL WONGHi Kojo, thank you very much for having me today.

  • 13:08:01

    NNAMDIYou are trained as an artist, how'd you get interested in medicine?

  • 13:08:06

    WONGWell, I went to the Rhode Island School of Design for my Bachelor's of Fine Arts and I spent my final year abroad in Rome. And while all the other students were drawing the Pantheon and doing all the things that one should be doing as an art student in a wonderful city like that, I was very interested in anatomy, primarily as a form of informing, my figurative paintings and drawings.

  • 13:08:29

    WONGI was doing Renaissance painting techniques and the like and you can't help but be in that environment and studying art history, artists like Michelangelo and Kurachi to really try to understand the body from the inside out in order to paint in a very naturalistic manner, to bring life to your paintings.

  • 13:08:44

    NNAMDIYou actually drew from dissected bodies in the way classical artists did for centuries. How does that work?

  • 13:08:52

    WONGI did, in my particular circumstance I literally showed up the Anatomy Institute at the University of Rome Medical School with a portfolio of drawings and paintings, mostly figurative based and I said I was very interested in studying anatomy and left that for the director.

  • 13:09:07

    WONGThe next day at my studio he showed up, Antonio Barberini, and he says (speaks foreign language) "Who is this Virgilio person who left all this art in my room?" And literally absconded with me and the next thing I knew I was in a freezer with dead Italians and he started showing me the various dissection techniques and I spent the entire year drawing and trying to understand the human form from the inside out.

  • 13:09:29

    NNAMDI800-433-8850 is the number to call if you'd like to join this conversation with Virgil Wong. 800-433-8850, how did that experience change you and what you wanted to with your art training?

  • 13:09:43

    WONGSure, my initial intent was really to approach it from a more technical perspective. To be able to look at the structures and forms of the body in order to be able to render more effectively through paint or other media. But the experience itself, when you're really with this human, form of human being, and being exposed to this vessel that used to contain a person it's really something that makes you question your own mortality.

  • 13:10:08

    WONGIt's something that makes you look at the world, I believe, in a very different fashion and it's something that really fascinated me. and to me that was really a unique opportunity, as an artist, to really try to visualize that experience and I spent the time coming back to New York really using those drawings, crystallizing them into various pieces that I felt was some attempt to really bring that physicality back to that process, to really try and understand how these objects can really reflect some semblance of the life that preceded it.

  • 13:10:36

    WONGOne of the pieces was a collage of all the drawings that I did over the course of the year at the University of Rome Medical School and that's on exhibit right now at the Strathmore Art Center in an exhibition called "Pulse" and it's a multimedia exhibition that includes many different artists who are really looking at how art can inspire or can be inspired by medicine. As well as how medicine itself is something that can be really infused by this artistic perspective.

  • 13:11:01

    NNAMDIVirgil Wong will be presenting a lecture as part of the "Art and the Brain" series at the Strathmore tomorrow evening at 7:00 pm. You can find more information about that and a link on our website at Virgil Wong is the cofounder of the company Medical Avatar and vice president of Interactive Media at Element 115 technology and design companies for the medical field. You say you were just, in fact, indicating that through art you can image what might be possible in medicine. What do you actually mean by that?

  • 13:11:33

    WONGAbsolutely. I spent 15 years as the director of web and multimedia at New York Presbyterian Hospital and Wall Cornell Medical Center and what I was very surprised by is that so much of the information, the mechanisms, in which clinicians try to communicate with patients is very textual.

  • 13:11:47

    WONGIt's something that I felt was very distant from things that patients could more readily connect to. It's very much this kind of white tower jargon that can be very much an obstacle to understanding your own health. So my idea was to really visualize information in a format that is familiar to patients, which is their own bodies in some way.

  • 13:12:07

    WONGEven though patients may not be completely aware of their anatomical structures, there is this point of relationship that you can really be able to relate information to your health, to your general awareness of your day to day existence in way that this visualization can really empower you, it can really help you perceive that and have a perspective that would otherwise be very difficult to understand.

  • 13:12:29

    WONGI spent many years working with these electronic medical systems where the data is just so dense and so difficult to decipher and so poorly presented that I felt there's a huge opportunity to really transform that and really embody this information through these three dimensional anatomical representations of the patient themselves. And that's what I've been doing through this company called Medical Avatar, where we create mobile apps that patients can use to do just that.

  • 13:12:54

    NNAMDISo when we're looking into the eyes of our physician pretending that we understand everything that our physician is saying, meanwhile our minds are busily trying to translate and interpret exactly what that means and by the time we walk out of the doctor's office we have forgotten more than half of it. This is a way in which we can better understand, recall exactly what we're being told. Let me ask our listeners to see if they want to participate in this conversation.

  • 13:13:22

    NNAMDI800-433-8850, would seeing an image of your face aged by smoking or the sun change your habits? Do you use any medical apps for weight loss, to measure your heart rate or anything else? Call us, 800-433-8850. The reason I bring this up is because one big issue you identify is that we patients to remember very little of what is said in a doctor's office. How does your process remedy that, so to speak?

  • 13:13:47

    WONGIn the study that I'm doing right now at Columbia University I'm having patients who are all smokers for various amounts of time and when one of the conditions is the patient would place her hand inside this white box that looks like some sort of medical device, you're not quite sure what it is, and there's an iPad on the surface.

  • 13:14:05

    WONGThe iPad takes a photograph of your hand and in the course of introducing this health information to show you the effects of smoking on your body, you're actually seeing changes on your hand through this iPad. So when you're hearing about how smoking causes constriction in the blood vessels, you're seeing that happen on an image of your own hand.

  • 13:14:23

    WONGWhen you see your skin changing and wrinkles and other sort of changes to your skin tissue you can actually see the wrinkling on your hand itself. Bone density loss, that's also visualized through these x-rays superimposed upon on the image of your hand that's connected to your body.

  • 13:14:39

    WONGSo I thought this was an appropriate approach. Some of my earlier studies were really focusing on the face and that was based on other studies where you saw that patients responded to skin cancer health information more readily when it was images of wrinkling of their face rather than looking at melanomas. I thought, hey let me try something similar.

  • 13:14:56

    WONGBut when you're a smoker, you're not necessarily looking into a mirror as you're lighting up the cigarette. You're looking at your hands and some of the participants who have engaged in these studies have said, "Well now, actually, when I grab a cigarette and I start lighting I can actually -- I see my bones and I see my blood vessels and I can see them constricting."

  • 13:15:14

    WONGAnd I'm kind of I'm not ready to quit yet." Because most of my smokers are pretty die hard smokers. "But I'm starting to think about it a lot more and I'm actually remembering a lot more of what you said." At first I had one participant who said, "Well, I thought smoking actually relaxes you so it's actually good for your heart. And I know my doctor said otherwise, but I didn't kind of believe him. But because of this study I'm remembering some of the details, okay, I can see the blood constricting."

  • 13:15:40

    WONGAnd you've internalized information, it's this really interesting field of psychology called embody cognition where you're using your own body to better learn information and you're showing this self reference effect as well. Your memory's improved because you are directly related to it and that's something I find very interesting.

  • 13:15:56

    NNAMDIOnto the phones. Here's Lisa, in Loudon County, Va. Lisa, you're on the air, go ahead please.

  • 13:16:02

    LISAYes, this is fascinating conversation. I always think about Leonardo's drawings of the body and the studies that he did with the body and how that made me so much more aware of sort of big picture in how different systems relate to different systems within our body.

  • 13:16:18

    LISAAnd I wish doctors would think more whole brain like that when they are describing what a fissure is and how we should be dealing with it or what the ramifications of something are. If they could use that way of showing people visually what they're talking about I think you wouldn't have this shut-off like you were talking about when they give you all this information verbally, you lose half of it or three-quarters of it before you leave the office and you don't have anything concrete to relate back to and this guest is definitely on that same track.

  • 13:16:52

    LISAMore medicine should understand that that's how you get better information to your patients and they end up with a better result when you're trying to treat or trying to change their behavior in treatment.

  • 13:17:03

    NNAMDILisa, I think you have hit Virgil Wong's nail on the head, so to speak. You're doing a study on avatars with a person's face. Can you talk about that?

  • 13:17:13

    WONGAbsolutely. This is very similar to the other study that I described where some of these visualizations the aging process is accelerated and part of what this study is really interesting is how this kind of time traveling in your own body, having that broader perspective can really affect your understanding of your health.

  • 13:17:29

    WONGWe're all organisms that live in this kind of time continuum, that is linear, it's minute by minute, day by day, week by week, but if you can jump ahead and then get this perspective that's only possible from this type of long term, call this predicative analysis, where you can really see these potential health symptoms and other issues that may be of concern to you in the future and what if you could actually take steps today and have interventions?

  • 13:17:52

    WONGAnd preventative medicine is such a tricky topic it's really hard to talk to people about their health unless they're actively sick and that's a huge, huge problem. You know, of course, we don't want to think about being sick if we're not but what if we can just think about it a little bit? In some way that would actually prevent sickness?

  • 13:18:09

    WONGAnd that's something that I feel is a really crucial opportunity for us especially using new technologies to really provide that perspective and to really allow people to make their own decisions. I'm not talking about this kind of nanny state environment where you're going to die of cancer if you don't do this or this. But you can make your own decisions in a much more powerful way.

  • 13:18:25

    WONGMedia has so much influence on health, this whole kind of sugar, soda debate in New York City that's been going on has, I feel, been very much about the wrong issues. It's not about taking away your control but allowing you to be influenced by things that would actually inform you as opposed to saying well, I just want to drink big sodas because the beverage association is saying that I should be consuming more of this product or that product.

  • 13:18:48

    WONGIt really is about that self-determination. The other aspect to this study is also visualizing your past information. It's a huge problem today where you have this enormous deficit, enormous cost incurred by healthcare because physicians do not speak to one another. I say that in a very broad sense but it's kind of true.

  • 13:19:07

    WONGYou could be going to a cardiologist and your symptoms or condition could be related to some sort of brain disorder and if those specialists do not communicate it's going to be very difficult to provide that treatment. So currently patients have the responsibility to tell their medical stories in some sort of way to bridge that gap, to work within this healthcare system so that they can actually be able to get correct diagnosis, to be able to see the impact of different interventions.

  • 13:19:32

    NNAMDISeeing the impact, what results have you seen in your research?

  • 13:19:35

    WONGIt's been very exciting when you can actually be able to link, when you're able to link together these disparate conditions. Part of what our app is doing, this is a new iteration that will be out very shortly, is that you can see the connections between various symptoms, conditions, treatments over both a timeline and as you move across this sort of timeline and it varies between years, months, weeks and days you will see changes in your avatar and you can then pinpoint that avatar and look at those symptoms on those current days.

  • 13:20:04

    WONGSo you start to see patterns. You start to see trends that oh, when I eat this certain food I'm happy and I'm satiated afterwards. But then like a day later or two days later, a week consistently I have this other problem. And then you can identify it and go, okay well maybe I'll stop eating that, even though it's really delicious, or maybe I'll just have it very remotely knowing that these are the potential consequences.

  • 13:20:25

    NNAMDIYou mentioned the size of soft drinks in New York. It's my understanding you're looking at models like this to help those trying to overcome obesity.

  • 13:20:33

    WONGAbsolutely. That's another aspect to the visualization where you can look at this past data. And through this process -- it's a statistical process called regression analysis, where that past data, you can look at interventions that have occurred and what sort of changes have occurred in terms of your -- the resulting in weight loss or your lean body mass. And you can do additional interventions and have some sense of what you can expect.

  • 13:20:58

    WONGThere's some predictability that you can really look at and understand because weight loss doesn't happen very quickly and we're a culture that's very accommodating or very much expecting immediacy. You don't see results immediately but the medical avatar allows you to actually do that. You can see those changes and go, okay well, you know, now I can look ahead over the course of these number of months. And these are the types of changes that I could possibly expect.

  • 13:21:22

    NNAMDIHere's Lewis on the eastern shore in Maryland. Lewis, you're on the air. Go ahead, please.

  • 13:21:27

    LEWISYes. I just wanted to comment that I'm a retired family physician. And as a freshman in medical school the first month in anatomy all we did was draw the skeletons in different views. And all over the wall were pictures of drawings, anatomy and skeletons and all. And even when we started dissecting everything had to be drawn and our prof was convinced that anatomy drawn was anatomy learned. And forever when I was with a patient, if they had an ulcer or a broken bone or whatever, I just sat down and drew a picture and they loved it. So what he's doing is great.

  • 13:22:03

    NNAMDIThank you very much for your call, Lewis. The use of apps in medicine, Virgil Wong, is growing by leaps and bounds for doctors and patients. And patient-centered apps are an area that you're working on as well. Can you talk a little bit about that?

  • 13:22:16

    WONGAbsolutely. We're in the process right now of launching a new mobile app on behalf of Oakwood Health Care. They're a four-hospital system out serving the Detroit metro area. And their medical avatar -- it will be an app that patients can download directly for free, and they'll be able to record their health information onto their own avatar. So they can create this medical reflection of themselves on their mobile app and start to keep track of some of these issues that we've been talking about.

  • 13:22:40

    WONGIt's a way for them to be able to embody their own medical histories and be empowered by that information. So when they do engage with the physicians and services of an institution that it's not so much, oh I'm at the mercy of this organization, but here I am and this is what I need to get from this. This also helps the clinicians better communicate with those individuals and better help them in the way that they are trained. It was so great to hear that caller talk about his experience as a physician doing drawings for his patients. One of the...

  • 13:23:12

    NNAMDIIt's a no-brainer.

  • 13:23:14

    WONGIt seems to -- exactly. I'm so happy to...

  • 13:23:16

    NNAMDINow that I've heard it, yes.

  • 13:23:17 hear that -- I mean, there's so many different aspects to what he talked about. And body cognition also includes this amazing literature about drawing cognition where by drawing something -- and this is an experience of my own -- life as an artist that if I can draw something. I'm looking at this amazing studio that you have and there's some equipment that I recognize and some that I don't. I'm thinking, oh, if I can draw that, you know, from the inside out I could probably build it. And then that really excites me.

  • 13:23:42

    WONGAnd so what I really like about what that physician was saying -- and thank you again for calling in -- the physician draws but I've had the great experience to work with patients who also came in with their drawings. So typically it's a shoebox full of like old prescriptions and random notes that they may have collected over the years. I know my mom had one of those for myself and my brother.

  • 13:24:03

    WONGBut this amazing story that I'm going to be sharing tomorrow night at Mansion at Strathmore is a story of this designer who developed a really horrible autoimmune disorder called myothenicarobis (sp?). And her medical history was so complicated and every time she went to a different specialist she had to somehow communicate this information. And she started creating these timelines, she started doing these drawings of her body where she would actually label the symptoms when she felt like a burning sensation in her appendages. She'd actually draw this sort of burning fire on her fingers. Or when she was having gastrointestinal distress like, oh my tummy hurts here.

  • 13:24:39

    WONGSo we're working in developing that into the symptom tracker that's kind of note to self so that you can actually be able to do the same things that this artist who has this enormous design talent, this amazing sensibility about her health and her awareness, and use those same techniques to be able to communicate to your doctor just as effectively.

  • 13:24:55

    NNAMDIAnd if you want to hear more about that, as we said earlier, Virgil Wong is presenting a lecture as part of the Art and the Brain Series at the Strathmore tomorrow evening at 7:00 pm, you can go to our website You'll find more information and a link there. Virgil Wong, thank you so much for joining us.

  • 13:25:14

    WONGThank you, Kojo.

  • 13:25:15

    NNAMDIVirgil Wong is the cofounder of the company Medical Avatar and vice-president of Interactive Media at Element 115, technology and design companies for the medical field. He's also an artist and researcher at Columbia University and a professor of media studies at the New School in New York. We're going to be taking a short break. When we come back you should know it's Food Wednesday. Think Eastern European food. I'm Kojo Nnamdi.

Related Links

Topics + Tags

Most Recent Shows