The Exchanges

Q take a transcription of the conversation that they have with the patient and then have AI synthesize what they talked about and summarize it and put it into the systems that they have for electric medical records and then verify that. So they don't have to spend the time Writing those summaries up and can actually go and spend some more time with patients. So, what's the problem with that?

A There are so many problems with that. And the first thing I want to say is that you named the underlying problem when you talked about insurance requiring so much paperwork. So this is one of those situations where there's a real problem here. Um, it's not that doctors shouldn't be writing clinical notes. That is actually part of the care, but there is a lot of additional paperwork that is required because of the way insurance systems and especially the one in the United States are set up. And so we could work on solving that problem. And this is a case where the sort of the turn towards large language models, so-called generative AI as an approach to this is showing us the existence of an issue. Um, but that doesn't mean it is a good solution. So many problems. Um, one is writing the clinical note is actually part of the process of care. It is the doctor reflecting on what came out of that conversation with the patient and thinking it through, writing it down, plans for next treatment. That is not something that I want doctors to get out of the habit of doing as part of the care. Now they might feel like they don't have time for it. That's also a systemic issue. Secondly, these things are set up as like ambient listeners, which is a huge privacy issue. As soon as you've collected that data, it becomes sort of this like radioactive pile of danger. Thirdly, you've got the fact t…

AI assessment note: “There are so many problems with that. And the first thing I want to say”

Q fact that it's able to go and comb the internet for these events and then take into context some of the, um, the context that I've given it with these documents, I think is very impressive. And that's just one use case. So I was, I'm not asking it to be a lawyer. I'm kind of asking it to be what you said, an itinerary planner. What's wrong with that?

A Uh, so, I mean, first of all, you, you have these lovely documents from your friends, and I guess what you're saying is missing is whatever current events are. So they've given you some sort of like, these are general things to look for, but they haven't looked into what's going on right now. Um, what's wrong with that? You know, on several levels, um, what, what would we do in a prior age, like even pre-internet, right? The local newspapers would list current events. Here's what's going on. If you landed in a city, you would go find The local, probably local indie newspaper and, and look up the events page. And that system was based on a series of relationships within the community between the people putting on festivals and the newspaper writers, and it helps support probably the local news information ecosystem, which was a good thing. Um, but on top of that, uh, if something wasn't listed, you could think about why is this not listed? What's the relationship that's missing? Um, Your ChatGPT output is going to give you some nonsense, and you're right, this is a use case where you can verify whether this is real or not. Um, it is also likely going to miss some things, and the things that are not surfaced for you are not surfaced because of the complex set of biases that got rolled into the system, plus whatever the roll of the die was this time. Um, and anytime someone says, …

AI assessment note: “What's wrong with that? You know, on several levels”

Q listening, there would be a sense that There is an ability to go into various different disciplines and perform. And whether you say it's a magic trick or not, it's, it's clear that it can. And so what I'm, I guess I'm trying to get at is, do you, I mean, is there a way to measure that? Or do you think that that is in itself a wrong assertion?

A So yes, I think it's a wrong assertion. What ChatGPT can do is it can mimic human language use across many different domains. And so it can produce the form of a poem. It can produce the form of a travel itinerary. It can produce the form of a Wikipedia page on the history of some event. It is an extremely bad idea to use it if you actually have an information need. Setting aside the environmental impacts of using ChatGPT and setting aside the terrible labor practices behind it and the awful exploitation of data workers who have to look at the terrible outputs so that the consumer sees fewer of them. And by terrible outputs, I mean violence and, um, racism and all kinds of sort of psychologically harmful stuff.

AI assessment note: “So yes, I think it's a wrong assertion.”

Q But do we know that? I mean, you both wrote the book on this.

A Um, so we have, we certainly point to environmental impacts as a really important factor. It is not the main focus of the book. I would refer people, um, to reporting of people like Paris Marx over at Tech Won't Save Us, um, did a wonderful series called, uh, Data Vampires, looking at, I think there was, uh, stories in Spain and in Chile, um, and yeah, so this is, uh, you know, we are looking at the overall con, and the environmental impacts come in Um, because it is something we should always be thinking about, and also because it is very hidden, right? When you access these technologies, you're probably sitting, you know, looking at them through your mobile device or through your computer, and the compute and its environmental footprint and the noise and everything else is hidden from you in the immateriality of the cloud.

AI assessment note: “It is not the main focus of the book. I would refer people”