Munmun De Choudhury (Georgia Tech): Conversational AI and Mental Health

This file was generated by Descript Kevin: Hi, I'm Kevin Werbach, Professor of Legal Studies and Business Ethics at the Wharton School of the University of Pennsylvania. For decades, I studied emerging technologies, from broadband to blockchain. Today, AI is promising to transform our world, but AI needs accountability, mechanisms to ensure it's developed and deployed in responsible, safe, and trustworthy ways. On this podcast, I speak with the experts leading the charge for accountable AI.

Many millions of people now turn to conversational AI for emotional support, even when tools weren't built for that purpose. Concerns about AI's mental health impacts, up to and including suicides, are increasingly significant on the public policy agenda. My guest is Munmun De Choudhury, professor at Georgia Tech, who's been studying the interaction of digital technology and mental health as long as anyone and is one of the best cited researchers in the field. We talk about what conversational AI is actually doing to users, how different stakeholders need to address the risks and opportunities of AI for mental health, what researchers do and don't know, and what responsible development of conversational AI for well-being looks like when technology and use cases are both changing so quickly.

Munmun, thank you so much for being with me on the road to accountable AI. Munmun: My pleasure. Thank you for having me today. Kevin: Tell me first a little bit about how you as a computer scientist got into the, uh, issue of mental health, uh, in people's interaction with computer systems.

Munmun: That's a great question, and I would say, uh, my story goes back to the times when I was still a, a PhD student, um- Mm-hmm at Arizona State, and, um, I was studying how social media emerged as a new platform for communication, and this is sort of the late two thousands. And, you know, Twitter was a big thing, uh, in that era. And I started to look at how sort of the digital traces that people leave behind on these platforms can tell so much about people. But a lot of the focus at that time was sort of understanding communities and populations, not as much about understanding the individual.

So by the time I f- came to finish my PhD and I, uh, joined Microsoft Research for a, a brief while, I started to ask that question. Can we actually help individuals with their well-being, with how they are feeling, what they're experiencing in their everyday lives by looking at what they're doing on these platforms? Uh, and that brought me close to mental health because turns out that a lot of these platforms are used by p- people when they are feeling in a cert- a certain way, right?

And that could be- What is happening in their life or what is happening around them. And that came, uh, to me as an opportunity to explore uh, in the mental health space. Mm-hmm ... but broadly as a spectrum to think about wellbeing, think about happiness, satisfaction, all the way to when people are having struggles.

And then the last decade, I would say, um, was spent thinking about how that question could be answered as the platforms changed over time. But what became also apparent is that these platforms are not just monoliths, right? Mm. They influence, uh, our lives in positive and negative ways, and that led to the most recent iteration of my research, which is kind of around understanding what can we understand about our mental health by looking to these platforms and what influence and impact are these platforms having on our mental health, uh, whether they're helping our mental health- Mm-hmm whether they're worsening our mental health, and so on.

So that's kind of the bigger story here. Kevin: Absolutely. So let's just start back with the original social media work. What were you able to find just looking at people's transcripts of their interactions with things like Twitter that was predictive of their mental health outcomes?

Munmun: So obviously, there are reasons of mental health when people go to these platforms, right? Mm-hmm. When p- uh, when we have moments of vulnerability, when we, uh, feel- Mm ... uh, struggling in our lives, when we feel lonely.

But what was apparent from, uh, the data that we were looking at is that sometimes there are these implicit cues in the way that we behave on these platforms, in the way we express ourselves on these platforms that can hold a lot of information about our mental health. So to give you an example a particular aspect of our language is, you know, there are these content words, right, which are like different topics. Mm ... but there are all these other words in the English language that are called non-content words, or these are kind of words like the pronouns and the articles- Mm and the conjunctions and so on that we use- Mm ...

which we don't consciously manipulate. These are words that we need to construct our sentences or to express ourselves. Mm. And what we saw is that those non-content words, those language signals, held a lot of information about how we are feeling.

So people with mental health struggles often would use a lot of first-person pronouns. Kevin: Mm. Munmun: Singular pronouns, so I's and me's and myself's. And psychologically speaking, that happens when a lot of our focus is- is inward, uh, directed, right?

We are thinking about ourselves- Mm-hmm ... and we are talking about ourselves, and this was something that psychologists has ob- have observed to be the case in the speech of people with mental health struggles. So we, what we started to find is that those kinds of signals in the language could tell us a lot about how people are feeling, not just somebody explicitly saying that they're feeling depressed, which does- Mm ... happen, but also how they're saying that they are struggling in their life was the real signal there.

Kevin: And, uh, broadly speaking, to connect it up with AI, uh, in social media, i- i- you're saying something publicly, whereas if you're interacting with a chatbot then the AI is your interlocutor. Uh, w- do you see similar kinds of patterns in people's communication in those two contexts? Munmun: People certainly feel more candid in their interactions when they know that this is a platform that is, uh, private. And there are certain kinds of social media that are like that as well.

Mm-hmm. So we looked at uh, in the past, we have looked at public social media such as- Mm ... Twitter, but we have also looked at private conversations that happen on other types of social media, such as Instagram. Instagram, Facebook, and these platforms have the direct messaging mechanism where you can message another person or a group of people.

Mm. Mm. And these are inherently very private conversations. So we did see differences even at the time across social media where people are more publicly facing and when people are feeling that they are in a more private setting.

So obviously these are, these are information that people chose to share with us later on for research purposes. Mm-hmm. But that carries on to AI as well. When somebody is interacting with a bot or when somebody is, you know, searching on a search engine, that has a perception that it's a very private activity.

Kevin: Mm-hmm. Munmun: We see people being more disinhibited, more, uh, le- less guarded, um, more- Yeah ... candid in those conversations- Mm and talk about, uh, maybe things that they may not be sharing in a public platform. But public social media kind of gives us other sets of insights which you don't see sort of in these private AI, uh, interaction contexts.

Mm. For instance, how does a person engage with the larger community and s- how do they socialize with others? And those are a unique set of signals that you see in the public contras- context. So yeah, to your question, a lot of these behaviors do shift when we are talking about human-AI interaction.

Kevin: And obviously the, the big challenge with conversational AI systems is that on the one hand they can be a huge boon for mental health, that it's people who don't have access to therapy can now talk to someone. B- but on the other side, we've seen, uh, all these examples that, seems to cause psychoses or worsening mental health. So ju- just broadly speaking, you know, how do you evaluate those two sides in your work, and how do you think about that challenge? Munmun: It is truly, uh, a double-edged sword.

That's how I think about it. And none of these technologies are sort of all good or all bad- Mm-hmm ... whether it's social media or conversational AI. And what we need to do is to really identify where the points, where the benefits are.

Can we quantify them? Could we understand them in a more systematic, principled way? And the same goes for the harms and the risks. Hmm.

Where are these platforms uh, failing, uh, to you know, serve our mental health? And can we quantify, can we identify those situations? And then the question is this trade-off, right? And even for the same person, it's not a matter of AI works for some people and doesn't work for some people or harms some people.

We know from social media that even for the same person, the same technology can sometimes be helpful and can sometimes- Mm-hmm ... be harmful. So the path forward is, I would say it's a, it's a multi-stakeholder responsibility base, where we are kind of thinking about how different stakeholders in this ecosystem counterbalance that trade-off of benefits- Hmm and harms. Kevin: Say a little bit more what you mean by the different stakeholders.

Munmun: Yeah. So, you know, when we think about technologies like AI, we often think- Mm-hmm ... about the companies that build these technologies. Mm-hmm.

We think about the people who are using them. But that ecosystem is very rich. It's not just those two parties that are part of the system, but we have the broader society and context. We have other people in the person's life who is interacting with the conversational AI.

And we have, uh, governments and regulators and policymakers- ... as well. Uh, so what I meant to say is that, you know, this trade-off between these technologies being beneficial or harmful is something that is important to be investigated for each of those stakeholders. Mm-hmm.

And I would say that each of those stakeholders have a unique role to play in how they counterbalance this trade-off. Kevin: Mm-hmm. How much do we understand today about, uh, addressing that, that challenge you talked about earlier, that there are certain patterns in these interactions a- a- and these chatbots that, you know, will lead to more healthy, uh, results versus the ones that will push people towards depression and self-harm? Munmun: The understanding is evolving.

The good news is that it's evolving rapidly. The bad news is that it is evolving and- Mm-hmm ... we don't have a set understanding, okay, these are the benefits, this is where people are getting help, and these are the risks and harms where, you know, bad things are happening. That understanding is evolving because the technologies themselves are changing very rapidly.

We have a new version of, uh, a chatbot from some- Mm-hmm ... uh, platform or the other pretty much every month. Um- Kevin: Mm-hmm ... Munmun: and the capabilities of these tools are also changing.

But let's also not forget the way people use these platforms is also evolving, right? Kevin: Mm-hmm. Munmun: And the analogy I often use is a very old technology called platform called Google Flu Trends- Mm-hmm ... which Google released in the late 2000s, and then- Yep ...

it was kind of taken off in the early, uh, 2010s. That lesson has a lot of meaning and value and relevance to this moment as well. Google Flu Trends, which at some point was the poster child of big data, and it was able to predict influenza cases before the CDC numbers came out, um, started first underestimating and then overly overestimating these trends. Kevin: Mm-hmm.

Munmun: And one of the postmortems was that the way people were using Google changed. Yes. And this is a challenge for AI as well. The way people use AI is not fixed.

Not only are the AI tools themselves changing, but the way and the reasons why people use them are also changing. So this complicates our understanding of where the benefits are and where where the harms are. Kevin: Mm-hmm. Munmun: But here is, I think, a positive thing.

For something like social media, it took us a long time to acknowledge and recognize that there are real harms that could be happening, uh, to people. It took us almost a decade to be able to study those harms. Kevin: Mm-hmm. Munmun: I think for conversational AI and chatbots, this is already happening.

A lot of it is happening in academia, uh, but there are also nonprofits who are invested in that for specific applications of these platforms. And I'm so, uh... I feel very good to see that that attention is being- Mm-hmm ... given to conversational AI and chatbots early on in, in the life cycle of use of these technologies, rather than waiting for significant harm to happen later.

Kevin: No, absolutely. I agree with you completely, and I'm glad you mentioned Google Flu Trends. I've taught that for a number of times, and it's - in some ways it's even more, uh, complicated than you describe because there was, reflexivity involved that people read media coverage about Google Flu Trends and went to Google and said, "Oh, let me try this." And then that threw off the, you know, the, the dataset as well.

So, so no, it's a really great example. I guess the question is, given what you've described, that we're dealing with something where everything is, is moving and changing, a-and yet we have this big challenge how to mitigate the extent to which these chatbots are, you know, worsening mental health problems. You know, what are the, the research questions we need to answer or, or what are the things that stakeholders really need to focus on to be able to address that issue? Munmun: So I think we need some standards, right?

Mm-hmm. And we see that in, in the non-tech sectors- ... widely speaking, right? We see that in pharma, we see that in, uh, medical devices- where there are industry standards, and any product that is - or artifact that is developed- Mm ...

needs to meet. It's sort of like a minimum set of criteria. The standards is like you can always do better- Mm ... than that, but this is what you sh- your product should be doing at the minimum.

That is lacking in the, the AI space. Uh, so that is something that'll be needed, and that'll be particularly relevant from the tech company's perspective. Mm-hmm. Of course, this makes the assumption that the tech companies as a coalition would come forward and, and develop what those industry standards could look like.

Again, it happened, uh, for pharma and other, other- Mm-hmm uh, industries. So that is something that I would say is focused more on the tech company side. The other thing I would say is that, you know, tech companies, it'll be useful for them to acknowledge and recognize that wellbeing is an important- ... uh, criteria, even from their bottom line perspective.

Mm-hmm. Oftentimes that is not, uh, given as much importance compared to other facets of- Mm ... a product that might impact the bottom line or the next quarterly revenues, uh, or profits of a company. But you know, if, if the users are not feeling well, then people are gonna abandon the platform sooner or later.

So I would- Mm ... love for the companies to consider wellbeing as an important aspect of how they operate. Speaking of, uh, stakeholders such as, you know, our society or the people who use these platforms, I think we need AI literacy. Mm.

And that needs to start early on, I would argue in middle school or high school. And the example that I often cite is let's talk about programming, right? When a lot of, uh, the previous generation was growing up, we all said that, you know, programming is, is such an important skill set that we want kids to learn it from early on because, you know, it, it's not just about the actual activity of writing the code- Mm but it's also about helping develop analytical- Yeah ... thinking, logical ways to proceed and analyze situations and problems, and these are important life skills.

I think we need something like that for AI as well. Again, AI literacy is not just about how do I use the AI- But how does use of the AI impact our other aspects of life, critical thinking how we manage our well-being, what AI can and cannot do for us. And all of these are important things that I think will be necessary for the next generation to learn because the world that they are gonna see is gonna look very different. One in which AI is deeply enmeshed in everything that- Mm-hmm ...

they do or the way they lead their life. Mm. Um, and that means that, uh, you know, as caregivers, as parents- Mm-hmm ... as teachers you know, as, as college prof-professors, we all have something to do in that space and in how we help the next generation of people work with AI not abandon AI, not- Mm-hmm you know, totally give up or surrender to AI.

But what are good ways to, to, to use AI, you know? And mental health is a great example of that. Like, when is AI and, uh, a conversational AI good for mental health help, and when is it that- ... People need to defer to other people for getting- Mm-hmm getting help.

And then finally governments and policymakers. Mm-hmm. You know, we, we can turn to Europe, for instance, where we see that there is a lot of effort in a top-down fashion from - coming from policymakers and in terms of holding the technologies accountable to the harms that they might be causing to people. And while that is not happening in, in the United States at the moment, but this is- Right ...

something that will be needed, in my opinion- Mm-hmm ... going forward, along with those other approaches that I talked about. And again, any one of those is not gonna address the challenge that we are seeing or the moment that we are passing through. I think we need all of us to work together- Mm-hmm ...

and adopt different strategies. Kevin: Related specifically to the next generation, um, y- many of these cases, at least the, you know, the suicide cases where people are suing AI companies involve teenagers or, or young people. You know, how different is the, the mental health context of adolescents? What are your thoughts on, uh, the notions that we should restrict access to some of these technologies a- at certain age levels?

A- a- and more broadly, how do we confront that challenge? Munmun: Yeah. So I would mention that, uh, Australia is one of the countries- Mm-hmm ... who has adopted a social media ban, and, uh, many countries are hoping- Mm-hmm to follow s- suit.

Uh, so I am actually on the advisory board of the Australian government's- Mm-hmm ... e-safety panel. And I would say that, you know, we are waiting to see how that pans out, and that'll be- Mm-hmm ... a very valuable lesson- Mm-hmm ...

for other kinds of technologies on whether this kind of age-based restriction or other forms of ban- Can actually help. For instance, in the state of Georgia, a lot of the public school systems are, uh, implementing cell phone ban- bans. Mm-hmm. And this is happening in a number of other states- Mm-hmm ...

in the US as well. So again any - I feel any regulation or policy that is implemented like that, we need to pay very careful attention to whether implementation of that policy is actually having the intended effects. Um- Mm-hmm. And we n- I would say that we need to go at it, uh, with an open mind because at least the preliminary evidence shows that it is very complicated.

I'll give you the example of the cell phone bans in middle schools. Mm-hmm. And some of the emerging work says that, whether the cell phone ban is good or bad, it really depends upon what is it that you care about. If you care about whether the kids are gonna be bullied less online, you know, cell phone bans are not gonna help because, yes, they don't get bullied during the school hours.

They're gonna get bullied in the morning, later in the evening, and so on. So if that's the, the goal of the ban, then this is not a good policy. But if the goal is let's have the kids pay more attention to the teacher and socially interact with each other during school hours, then the cell phone ban makes a lot of sense. And it can have positive effects.

So as we think about these types of policies of bans or age verification and so on, we need to pay very careful attention to what is it that we are hoping to achieve with this policy. Because this kind of a blanket policy will not help us solve all of the 23 problems that we might be hoping to solve. Right. I wish that was the case.

It's gonna help resolve some issues and not gonna help resolve some others, and we need to bring that approach where we lean on the evidence, we lean on what the impacts of the implementation of the policy are and then go from there. Um, so yeah. Mm-hmm. So there is no golden answer, I would say- Yeah ...

to whether these policies are good or bad. Kevin: Yeah, and cellphone bans and social media bans are one thing that, as you say, are being explored, but now we've got chatbots and things like ChatGPT, which are not entertainment services. People are using them to do their schoolwork and to do information gathering, but also in some cases having these long conversations, uh, which may have a mental health effect. How does any of this relate to the generative AI situation, where now we've got these general purpose chatbots?

Munmun: That really complicates it. Yeah. Because if you look at the history of AI, for the last 30 or 40 years, we r- as a, as a society, we really excelled at building AI for specific purposes and Kevin: needs. Munmun: Right?

Or you, we, we want to we want to help humans evaluate- Mm. radiology reports- Mm-hmm ... for cancer patients. Let's build a custom AI- Yeah ...

for that, that, that only can read radiology reports, and we found that that can be pretty good. I mean, there are studies that show that the efficacy of these AI models- ... That can detect, let's say, cancer cells from a radiology report can work at par with human radiologists. But all of this changed with the generative AI paradigm, where we are talking about these general purpose AI, which flips the whole pyramid- Mm-hmm ...

that, you know, we are making these AI more and more niche and more and more specialized. Now we are saying that maybe it's an AI that is not gonna be amazing- Mm at everything, but it's gonna do reasonably well- ... in you know, a number of different tasks. The way this poses a problem is- Now these benefits and harms are very much task-dependent, or how- Mm-hmm ...

somebody's using these platforms depend on that. And mental health is a great example, right? I mean, none of these platforms were actually designed- Kevin: Mm-hmm ... Munmun: to, help a person with mental health struggles.

Although I would say that there are now companies who are trying to build therapy bots- Yeah ... and so on. So I'm not talking about those per se. I'm talking about the general purpose conversational AI tools, which can do a number of different things- Yeah from being your, being a, a kid's homework, homework buddy to, you know, being a, a soulmate, um- Mm-hmm ...

to, uh, to, uh, being a counselor during mental health struggles and everything in between. And it poses new types of challenges because now we don't know how somebody is gonna use the platform. Mm-hmm. And when we don't know how they're gonna use the platform, we don't know where the harms and benefits may lie.

So this is part of the larger issue that, you know, we, we are having sort of these general purpose monoliths, and we as users of these platforms have very little understanding of how it helps us. It, it caters to a variety of- Mm-hmm ... different tasks and needs and questions that we might have. And that lack of transparency and lack of understanding further complicates uh, how we navigate this issue.

Kevin: Mm-hmm. What is the research agenda for scholars like you in this area to understand uh, those mental health issues around these general purpose chatbots? Munmun: Yeah. So I would say that there are three different directions.

Uh, one where there, there is a lot of interest already, and I mentioned earlier that it's great that academics are working to- Mm-hmm uncover these harms and risks. Sure. And that needs to continue. As I said, these issues are evolving, these technologies are evolving, so we continue to invest.

It's not gonna be like day after tomorrow we would have figured out all the possible harms of these tools, and now we can, close that research program, move on to the next one. I wish that was the case, but it is not gonna be the case. So we continue - we need to continue to look at and probe that space, and there are a few different ways to go about it. There's theoretical work that is happening from researchers on proposing, let's say, a framework or a taxonomy of harms so that somebody, uh, coming into it can have a general sense- Mm-hmm of where these things land when somebody uses these tools.

Um, there's also work on what is known as red teaming, and red teaming- Mm-hmm ... is the concept of adversarially using these tools in a way that would, you know, uncover where the pitfalls are. Uh, so both of those things need further investment going forward. Uh, the second direction I would say is- A deeper engagement with the people.

As I said- Mm ... there are so many stakeholders in the space, and solutions to the harms are not gonna come by interacting or looking at any one stakeholder alone. We need to be able to understand where the challenges are, how these tools are being used, but also the users of these tools don't exist in a vacuum, right? Kevin: Mm.

Munmun: We know that, sometimes interactions with AI can lead to displacement of actual human connections. Um, we hear that, we read it in the news that somebody started to spend so much time interacting with the chatbot that they abandoned their family. So there are real impacts in our everyday lives as well, and I think more research is needed to look at what we can call the spillover effect. Mm-hmm.

So it's not just the harms that are happening in the interaction between the human and the AI, but what is happening beyond that in our society and our communities. And then the third, uh, research agenda I would say is about, okay, we, we uncover all these problems, but as researchers, we also like to provide s-solutions. Mm-hmm. And I think that is much needed here is okay, so let's say we do the red teaming, we identify where the harms are, so how do we move forward from there?

What needs to change in the design of these- Mm-hmm ... platforms, in the way that we build these AI tools that can, amplify the benefits and- Mm-hmm uh, minimize, if not mitigate, the harms? So kind of leading that path forward- ... and defining what that path would look like would go a long way.

And the hope is that enough research happening in this space would eventually make its way into the tech sector as well when- Mm-hmm ... when companies small and big see that, okay, there are paths forward that look pr-promising, uh, and hopefully that those would eventually get integrated into the next generation of AI that is built. Kevin: Mm-hmm. And what about, uh, tools that are built specifically to provide mental health support based on conversational AI?

What, what have you found in, in looking, uh, at how people interact with those? Munmun: Most of the tools that I have seen in that space are really putting an added layer on top of a general- purpose AI bot. And while it is an easy and quick way to uh, build something and bring help to people- Mm ... what we have to recognize that even at fundamentally Those tools are still that general purpose AI bot, which, you know, was not designed for the purposes of providing mental health support.

So we need a fundamental rethinking, a revamping of how we approach that. I don't think today's AI is ready to provide therapy, for instance- Mm-hmm ... uh, to people in lieu of actual human beings. But AI can be a way to provide or fill in support when human support may not be available.

Mm. So I feel like it - for those uh, actors who are building these tools, particularly for mental health support, do two things. One is fundamentally rethink where AI can actually be helpful, and I think that AI- Mm-hmm ... is best used in moments where other forms of human help are not available- Mm ...

not as a replacement for human help. And then the second thing is how can these models be, these foundation models be built ground up with principles from the mental health space, principles of- Mm psychotherapy, principles of how humans support each other. I often say that, you know, um, these tools are so good at problem-solving, right? These AI conversational- Mm ...

AI agents are good at problem-solving. But mental health often is not about problem-solving. Kevin: Yep. Munmun: Mental health help, if you talk to an expert like a counselor or- Mm therapist, uh, or peer supporter, they would say that my job is to give the person the tools so that they can navigate- Kevin: Mm ...

Munmun: their life challenges better. Mm-hmm. So my question for those actors who are building mental health AI is that can AI take on that ro-role? Instead of being- Sure ...

a problem solver, what would it mean for AI to provide the human being with the tools that they, that they need to empower themselves to better take care of their mental health? Kevin: Well, that's actually a great place to land on. Uh, that's the challenge going forward. Uh, Mona, thank you very much for being with me.

Munmun: Thank you so much for having me. Kevin: This has been The Road to Accountable AI. If you like what you're hearing, please give us a good review and check out my Substack for more insights on AI accountability. Thank you for listening.

If you wanna go deeper on AI governance, trust, and responsibility with me and other distinguished faculty of the world's top business school, sign up for the next cohort of Wharton's Strategies for Accountable AI online executive education program, featuring live interaction with faculty, expert interviews, and custom designed asynchronous content. Join fellow business leaders to learn valuable skills you can put to work in your organization. Visit execed.wharton.

upenn.edu/acai for full details. I hope to see you there.