Muse

Group health insurance can challenge company budgets with large and unpredictable rate increases. Now, a new form of employer coverage called an ICHRA, or ICHRA, can help. ICHRAs make health plan expenses more predictable and affordable by tapping into a much larger risk pool. That means businesses know what to expect and what to budget. If you're tired of expensive surprises, plan on something predictable. Plan on an ICHRA. Learn more at ambetterhealth.com slash ICHRA.

When opportunity requires capital, be ready. Ledin's Bitcoin-backed loans give you liquidity across time zones 24-7. When timing matters, access to capital is critical. Ledin gives you liquidity so you can move quickly without disrupting your Bitcoin position. And Ledin has funded over $11 billion in loans since 2018 with a perfect track record of protecting client assets. Get a Ledin Bitcoin-backed loan today. Visit ledin.io.

Hey, folks. Did you know that Senator Chuck Schumer was booed at a New York City Pride parade? Whether you knew that or not may actually depend on where you get your news. According to Ground News, if you primarily read left-leaning or centrist news sources, you're dramatically less likely to have seen that story than if you mostly rely on conservative news sources. That's one of the reasons I use Ground News. Ground News, in addition to being the sponsor of this episode of the Lawfare Podcast, is an app that lets you jump out of your media bubbles and see the blind spots that exist in whatever media ecosystem you operate. Ground News doesn't just show you what the news is. It shows you who's reporting the news, how many news sources are covering it, and how coverage is distributed across the political spectrum, and it gives you insights into where your own blind spots might be based on the sources you actually read. The blind spot feed is one of my favorite features because it highlights stories that are receiving disproportionate attention from one side of the media ecosystem relative to others. I also appreciate that it doesn't stop at showing you who covered a story. Ground News also shows how many outlets reported the story, how coverage breaks down ideologically, who owns each publication, and the factuality and bias ratings for those outlets. For example, Ground News shows that there are 50 different sources reporting that Oman warned allies that ships going through the Straits of Hormuz might have to pay a fee. And the headlines are quite different depending on where you look. From right-leaning news sources, you get headlines like ships face voluntary charges under plan for post-war Hormuz. From more left-leaning outlets, you get headlines like Oman tells allies ships going through Hormuz may have to pay. And further, Brown News shows that only 19% of the news organizations covering the story are left-leaning, and only 12% are centrist. 69% of the news sources covering it are conservative. In other words, you're much less likely to have encountered that story if you primarily read left-leaning outlets than if you read conservative media. Ground news helps you jump out of your media bubbles and get closer to the ground truth. It's not a publisher, and it's much more than an aggregator. It's a combination of a rating system and a focused look at who is reporting what. I have to say I was skeptical of it at first, but I've come to appreciate it. Like Lawfare, Ground News is about bringing you high-quality information that you can figure out

what to do with. For me, the most valuable feature is simply discovering the stories I probably would have missed because of my own media habits. So check out Ground News and subscribe to the Advantage plan for 40% off, which gives you unlimited access to a better way to read the news. Visit groundnews.com slash law. That's groundnews.com slash law. One last time. It's really worth checking out groundnews.com slash law. Is this media fake is different than is this

media of me. And that's where I don't think the existing kind of report and attestation piece solves the kind of search problem or prevention of the content being made. It's the Lawfare Podcast. I'm Renee DiResta, Senior Editor at Lawfare, and I'm joined by Kate Klonick, also Senior Editor at Lawfare. And our guest today is Alyssa Red.

Ed Miles, an assistant professor of computer science at Georgetown University.

So I think there's a lot of attention that would need to be paid to, like, what qualifies something as a sufficient safeguard and how do we set people's expectations appropriately for how much that's going to protect them individually versus kind of mean something about mitigation in a specific product.

Today, we're talking about the people who create AI-generated sexual content and whether prominent technical proposals can actually prevent AI systems from generating exploitative content. So I wonder if you might want to start by telling us a little bit about the research that you've been doing. I know that most conversations about AI-generated imagery begin with a focus on detection, but you've been arguing that that really misses some central facts. I'd love to hear you just start by telling the listeners how you think about this problem.

So when it comes to AI generation of non-consensual intimate imagery, sometimes called revenge porn, one of the main harms is on the use of someone's likeness without their consent to create sexual content of them. And in particular, as a computer scientist, I was looking at different kind of claims that companies were starting to make. For example, like Dali has a claim about using advanced techniques to prevent photorealistic generations of real people. Civid AI, which is a website for hosting different kind of model objects that got found to have a lot that makes specific people, started making some claims that you could opt into preventing models from being shared that could generate your likeness. And as a computer scientist, that made me wonder, can we actually do that in practice? And what are the flaws that are going to kind of pop up if we try to automate that type of protection?

We actually just saw this from Meta. We saw some interesting decisions about not only opting in, but in fact, encouraging. I got a push notification from Meta AI telling me I could go create content of people just by adding their Instagram handle. And then it would go and it would generate content using the person's public Instagram account as the base layer. And it was a very interesting product decision, I thought, in light of the potential for abuse. I'm curious what you

thought about that call? Yeah, I share your concerns about that choice, particularly given that we've seen with Grok, there's been a lot of instances of nudification or at least sexualization of individuals by using a very similar affordance where you like at the individual's account or it's just kind of a reply to content that they've shared. And so when I saw the meta release, I was kind of immediately thinking about that kind of situation and was concerned that people may not expect that their content can be modified. And I think this goes beyond notification, right, even for various job or reputation concerns people may have. And actually, in particular, we're seeing people being asked to kind of set social media to public, right, for various kind of immigration or border reasons. And so that may put people in a really challenging position. The other thing I would say is, you know, even after, say, XAI has mentioned trying to implement new technical controls, we're still seeing many cases of notification. and in a report that we'll have coming out soon with the Center for Democracy and Technology, we took a look at all of the kind of technical ways that foundation model providers are trying to prevent this generation. And we really don't have reliable means of doing that right now. And so, you know, while I'm sure that medicine tent is not to allow for that, there's definitely a risk with a feature like this.

One of the things that I have followed in your work over the years is that you write a lot about how sometimes abuse isn't necessarily focused on the substance of the content, but whether or not the person consented to its creation or distribution, right? And that might mean that there is a nude image that the person actively made themselves and chose to upload voluntarily, or a person who is, as you know, not nude, but put into, you know, a potentially dicey image situation and that did not consent to that and doesn't have the same access to having it addressed because the image generator will in fact generate it. So can you tell us a little bit about how you see the consent problem and its differences from the content classification problem?

I think the idea with consent that we've seen even before use of AI for non-consensual intimate imagery was so large, we had done work with, as you said, victims who had not taken content but maybe had it made of them using secret cameras, other approaches, as well as folks who had taken content of themselves either for recreation or for... commercial purposes. And what we saw was for folks who are sharing this kind of content commercially, some people viewed the resharing of their content or now the editing of that content as like a commercial threat, like theft, like basically you've taken something that I make commercially and you're trying to make money off of it yourself. Whereas others saw it as both theft and a kind of sexual harm, privacy harm, victimization in the same way that folks who were not commercial actors would have felt about their content being disseminated or themselves being depicted. And I think that's one of the reasons why the idea of likeness or being identifiable as a real person and whether or not that person got the opportunity to consent is so compelling to me in this space. It's also compelling to me given kind of how we think about pornography in the past and people being able to provide consent forms and that being kind of a huge distinction or like point of process, at least in the US. So I think for both of those

reasons, it resonates with me. So just to be clear, NCII and why it's not called revenge porn is because the idea anymore, like that's kind of like, it's like, it's like a, you know, not a politically correct term among those who kind of study it anymore. And I've always understood that to be kind of true, because the idea of pornography is that one does it consensually, and inherently, like, pornography is kind of thought of as consensual. Whereas NCII, non-consensual image, intimate images, is essentially, like, inherently the opposite of that. And so it does. And also, because revenge is not some often we find it in revenge situations, but revenge isn't the only way that like non-consensual images show up, intimate images show up. So I just, is that correct? Is that kind of the definitions that we're operating under? Yeah, absolutely. Yeah. So I want to kind of pull out a string here, which I think is super interesting from a legal perspective, which is this question of focusing on consent. So from like a legal perspective, I think there is something really interesting about kind of this model that you bring up. One is kind of a theft of property, so to speak, like a theft of something that you can commercialize or something like that, like a content kind of theft. And then there is, you know, this privacy based regime. So those kind of like end up in different buckets in the law, I would say. But in both, there's like, an evidence issue, right, trying to show or demonstrate that consent has been given either explicitly or implicitly, and kind of to track that. One of the ways I know platforms have dealt with this in the non-consensual image context is to just believe the person who says this is my image. And I didn't consent to this. And so please take it down. Is that as feasible? Going forward? Did you do any kind of research on that? Do you see just kind of a, a believe it like in terms of just how platforms should react to this? Like, is that actually going to be an actively useful framework

going forward? Yeah, that's a great question. So I think there are a few issues here. So one is once the content is created, what do we do about it, right? And so if that content is shared on a platform that has a reporting stream, and people are able to report, then certainly kind of attestation is one approach. I think the one concern we sometimes have with attestation is that there have been cases of like widespread sort of activist-based reporting of say sex workers content for takedown. And so that becomes like a question of how you balance that. But typically there are other signals when that's happening where you may be able to kind of balance things out. So I think for content that is up, that's like a reasonable approach. I think the problem though becomes that this content often spreads to platforms that are specifically for hosting non-consensual imagery or are just not interested in doing takedowns, for example, thinking places like 4chan or a bit more hidden. The other issue that can come up is people discovering this content has been made in the first place. And that's a big issue for the AI generation of this kind of imagery is how are you going to go and detect that you're depicted? And that kind of comes back to this question of like, can computational approaches do detection of you or any real person in generated media in a reliable way? And that's a different question than the one we usually ask, which Renee brought up, which is like, is this media fake, is different than is this media of me? And that's where I don't think the existing kind of report and attestation piece solves the kind of search problem or prevention of the content being made. speaking, then, when we think about the technology for detecting consent, is that even, you know, it sounds even weird to say that, or how do you credential an image is in a way that is like machine readable, we can do this, of course, right, this is this has come up a bit. But we're seeing quite a lot of a lot of platforms struggling with labeling regimes with detection regimes, there's a lot of different things that they're being asked to build simultaneously, how should we think about essentially credentialing content as consented imagery that is not NCII?

Great question. So I think about this in those kind of two components. The first is how do I recognize that this image that has been generated or posted is an image of you? And how do I do that in a way that is privacy preserving, preferably? And then the second piece is how do I know whether that is an image that you consented to being created. So for the first piece of like, is this an image of you? One of the first things that may come to mind for computer scientists is facial recognition, which are the technologies when you go through TSA. Now they often want you to stand in front of a camera. They're using what's called one to N facial recognition, where they're searching for you in their database of faces, and essentially what they're doing is they're comparing the image of your face that they just took, doing a similarity score check against the other images in the database, and presumably your face or name should come up as the most similar one. When it comes to generated images, a few things change. The first thing is that the underlying distribution of pixels, so all the little points that makes up the picture, And that distribution is different in a generated image than in a photograph. And there actually has not been pretty much any research on how well facial recognition algorithms will identify individuals in generated photos. And the other thing that comes up is that facial recognition was originally created for these kinds of border type cases where you want to be like perfectly sure this is the person that you think it is. When it comes to something like NCII, I think the concept is a bit fuzzier, right, in terms of like, can a reasonable person recognize the individual in the image? And so how well something like facial recognition will map to human perception is an open question, especially because it depends like, is this you think it's you in the photo? Is it someone who knows you well in real life thinks it's you in the photo? Is it just like your online fans think it's you? All of those can add noise to this kind of machine process. So that's one aspect. I think in terms of attaching consent credentials to an image, certainly there are different approaches like we've seen for AI labeling to do kind of cryptographic watermarking that attaches content to an image. All of that relies on the platform where the content is being posted, keeping that metadata and respecting it. And so that is like an open question that one might have after the generation phase.

So I think this is really fascinating that you did all of this kind of research between the various different types of ways that we move from something like TSA. When you say one to end, like, I think that like, you know, most people are so creeped out by things like TSA, and it's incredible quick speed and it's incredible accuracy, mostly because they think that like, the TSA is running your face past a database of all known human beings and the planet. And it's not. It's like the flight list, like the manifest, right? Like it's not in any way that like kind of, which I think is actually kind of a super important point to make. And so I think that there's this really fascinating delta between our ability scientifically and through qualitative, I don't know, I guess you could say proof or mathematical proof, whether something is AI generated or whether it's real. But actually, that doesn't end up mattering for the vast majority of things. So, like, authenticity in both one sense is so crucial to certain types of, like, facial recognition and certain types of, like, of AI generation stuff. But it's actually not on point to, like, kind of this problem with non-consensual intimate images, because the damage is really just the idea that it's you, right? It's a reputational damage. It's a normative damage. It's kind of breaking, like, you know, being shamed or kind of, you know, embarrassed by something that you're caught on doing or that is revealed about you. How did you kind of capture that sociological component of this? I mean, you're a computer scientist. So I'm just really curious, you live in the mathematical proof, the authenticity proofs of these things. So how does one measure how if you you know, if you're designing a study, how does

and measure that other component? So one of the things that we do here is we often try to look at fields that measure human perception. So take psychology, for example. There's a well-studied task called the familiar facial recognition task that's used to diagnose things like propolisia. So it's been studied quite a lot. And basically the task is I'm going to show you an image. I'm going to ask you what is the name of the person in the image. And you're going to reply either with the name or with identifying details, like, oh, this is the person who played Wolverine in the most recent movie. And so that's an example of a task that we use to see, like, can someone recognize? And then we control for all sorts of confounds, things like, are you of the same gender and ethnicity as the person who we're asking you about? Are you familiar with them in real life or not, et cetera, et cetera. So we do that kind of work to capture human perception and compare it with computational metrics. But that doesn't get at the deeper part of what you're talking about, which is things like, even if this is a cartoon of me, is that something that would cause harm? And how do we think about that from like a recognition standpoint? Because certainly I can recognize a cartoon of Albert Einstein, even if it's not photorealistic. And so in those cases, we do one of two things. One is we create surveys that we call vignette surveys. So we basically create a scenario for someone to imagine themselves in. So for example, we would say, imagine that a stranger created a video showing you as a cartoon having sex. Would you find that acceptable or not acceptable? And then we have a rating scale, right? And survey methodologists like these kinds of scenarios because they kind of approximate people's feelings. They're not precisely real, but they're closer. That's one option. The other option is we interview those who run support organizations for victim survivors or work with them to speak with victim survivors themselves to understand case studies that are a bit more nuanced, like cases where there's cartoons, cases where things they're not quite as clear cut as maybe we think about in a kind of like photorealistic

nudification case. You've also done a lot of work looking at the motivations of the creators. You had a really fascinating paper where you talked about how people who create these images think about their role and what they're doing, particularly in cases where they make them for themselves and don't share them. And you have a lot of really interesting nuanced dynamics to your thinking about the creation process. I wonder if you could share a little bit about that with

the audience. Yeah, absolutely. So we started studying communities of folks who use AI to create sexual content because we were curious the extent to which those communities were creating norms that sort of separated out non-consensual creation. We were also curious how they thought about issues like accidentally creating NCII. For example, if a particular model has been really heavily trained on a particular celebrity individual, it's possible you would write sort of a generic prompt and get back someone who looks a whole lot like a celebrity who you know. And so we started going into these online communities, some of which have hundreds of thousands of members to understand how are people getting into this? What are they making? Is it abusive or not? How are they doing governance? What are their norms? And that helps us both see the type of content being created. It helps us understand what are the pipelines for creation? Because I think we all talk a lot about like notification websites, right? I upload an image, I get back one that's undressed. But there's a whole wide world out there of folks using what we call open weight models, things that have been uploaded on Hugging Face elsewhere. And they are either modifying them or using them just as they are to create their own content either locally or in the cloud. And I think before we did this project, the sort of prevailing wisdom was like, oh, you need some computer science background, maybe even some machine learning training to do this. But what we actually find in these communities is they're actually serving as like upskilling centers. So they're kind of training each other. If like a model is too big to run on someone's computer, then they're going to like compress it down so that they can run it. And in fact, one of our participants said, you know, I'm actually not interested in making sexual content. I have this other content I want to make, but the most active technical support I could find was in these communities for making sexual content. And of course, not all of that content is abusive, but of the 28 people we did interview across a couple of different communities,

three of them talked to us about making non-consensual intimate imagery, and several other people talked about getting requests for making it that they turned down. And the moderators who we discussed with, to greater or lesser degrees, were trying to moderate this type of behavior, but they were actually struggling a bit because people come to these communities for kind of an uncensored place for sexual expression, something they didn't feel like was easy to find, places to talk about sexual content creation or preferences. And so when there are kind of rules put in place, sometimes the moderators get pushback that's like, excuse me, you're not supposed to kind of yuck my yum. Don't tell me what to do. And so they actually said that they found two things useful to justify why they had rules. The first was the terms of service for the platform that they were hosting the community on. So the terms of service have some mention of non-consensual intimate imagery. And because people so want to be in the community, if they bring up, hey, the community could go down if you share this, or if you take these requests, that really helped. And the second was the new Take It Down Act. The idea that there might be legal consequences for sharing really motivated the moderators and was also a good

justification to the community. That's interesting that you bring up these notions of perceptions of censorship versus permissiveness within the community. I know there's also an emerging legal debate over whether generative AI outputs receive First Amendment protection, either through the rights of users to see content or the rights of developers to, you know, build models and, you know, sort of expressiveness of codes, a long running debate. But, you know, this is obviously unsettled law, we're starting to see it play out in a few different cases, there was the Garcia versus character technologies one. But right now, I think there's an interesting question. I'm not a constitutional lawyer. Kate can like jump in and pick this up. But one thing that I think about a lot is, does that conversation leave out the person who is being created? When we talk about the prompter's expressive interest, the model company's editorial or design interest, where does that leave the person whose likeness is being rendered without consent? Yeah, I guess from my

side, this is something I think about a lot, because in computer science, one of the main privacy threats that people address with computer science research is something called membership inference. And so the idea of this is like, there are some images used in the training data, can I attack the model and recreate the exact image from the training data, or can I extract it? And this is an important problem for various reasons, but I often talk to colleagues about my frustration that it is not the whole problem because my ability to reconstruct the exact source photo is maybe not my concern when it's a harm around my likeness. the issue is, can you reconstruct an identifiable version of me? And one of the challenges we've had is with membership inference, you can define this mathematically because like that training data object existed at some point. So we can like write some math about what we're trying to do with the likeness. We need some sort of mathematical definition of like, what is a likeness? And that's a much more, as we talked about a human perception kind of thing, like, can we achieve that? I'm not terribly convinced we can. And yet it's an important problem to work on to evaluate for, etc. So that's where I see this coming up. Okay, so I love this question.

I think that this is where kind of Alyssa's amazing research and kind of a lot of the work that you do, Renee, and some of the stuff that we've talked about, like kind of offline and off the podcast about deep fakes and the implications and what the legal answers to Gen AI is going to be. I think that this is kind of where like this is going to maybe be the four of where it's decided because I think that these are such obvious places where vulnerable individuals can really be exploited. But also it is the place where people who have the most economic stakes can be exploited. And the combination of those things motivates people. Well, yeah, this was I think

Meta pulled their weird tag in the Instagram person you want to make a picture of thing because one of the creator guilds protested, as I recall, right? Complained about it.

Yeah. So I don't want to kind of go all the way back to kind of, you know, to bore everyone in this pod who did not turn in for Kate Klonick's history of IP law and internet history. But this is essentially what you saw in kind of in my in my version of events, that this is what you saw with the early internet that we're kind of speed running through as an issue legal issue now for which is that you have property based rights being the most enforceable and the most economically viable rights that you can enforce in courts. And so you see them at the fore and you see huge interest groups going to bat and taking down the tech.

technologies or trying to take out the technologies that do this. And so obviously, I'm thinking of because this was like happening when I was growing up and a teenager. And so I was like, on both the side of like this, this group of lawsuits ending right as I got to law school and being a person who like downloaded illegally tons of stuff from Napster. Obviously, like the Napster Kazaa peer to peer sharing kind of issues where you had huge interest groups, like the RIA and the Motion Pictures Association of America kind of decide to bring suit because their actual content was getting ripped off. Now, that is a solely copyright kind of IP, like squarely an IP type of right, right? And that was the question there for a lot of people was like, do we change how we distribute IP rights and what we think of as IP and how we do this in the age of the internet? That's a little different when you have generative AI, which is both mathematically from like the authenticity point that like Alyssa was talking about before mathematically new material that happens to have a likeness like it is pixel by pixel a completely different image than a photo of Renee you know if you had a generated image of Renee that looked exactly like Renee looks right now in her Georgetown office like you know and then you had this image of Renee like that we asked the the machine to produce whatever model we wanted to use and if it was like almost one-to-one they would still be different images from a, from a, from a, from a, like, I guess, like a granular mathematical perspective. But, and this is kind of what I think is so interesting, I was getting it before, phenomenologically, the picture plays the same role. So sociologically, it plays the same role, it doesn't actually matter. And so the effects of the picture, the harm that it creates, right, doesn't end up totally mattering. And this is, I think, the thrust of essentially what the debate is going to be. Like, do we end up going down? And I mean, when we say like the law, does the law end up going down this really particularized route of like mathematical authenticity? Or is it going to go to kind of down this humanized route of like, how it categorizes the harm? And like, whether or not the generation of that harm exists through AI or something else, like we are going to recognize this harm. And that has different First Amendment valences. So if you tend to recognize generative AI, as like, or you were a proponent of recognizing Gen AI as having its own First Amendment rights, because it is generating something completely new and technically authentically different, you're just going to be able to block all a lot of not all of, but potentially block a lot of the regulation and potentially litigation, regulation and litigation, that is going to kind of be filed for like against, against some of these models and companies that are running the models for the outputs that they produce that are exactly the types of outputs that Alyssa's work is kind of talking about. If you decide to take a more kind of harm based approach, a more dignitary privacy based approach, where it doesn't really matter, like where the image comes from, just that

it exists, and it can be traced to the generation from like either a model or a human, then you have like a whole different kettle of fish and kind of a whole different set of legal solutions that you're going to put in place there. So I don't want to kind of take this over with a legal discussion. But I do think that this ends up being kind of the payout of a lot of Alyssa's work if we go to like put it to policy. And I think it's super interesting just because it reveals the limits of like what authenticity gets us and it reveals the power of things like likeness, right? Likeness being something that no one has been able to perfectly quantify or understand in either cognitive neuroscience or IP or whatever for years. I mean, like literally, you know, how trademark likeness is decided is through survey. It's through the exact type of research and kind of like qualitative and quantitative stuff that like Alyssa does or like, you know, social scientists do. And so I don't know, I just kind of I like bringing this up just because I think it it gives us kind of also context for like some of these ideas are new and new at scale and some of them are like taking a new twist on a really old discussion and a really classic discussion that we're kind of just getting to see a new valence of have you ever thought to

yourself I really should be doing something to protect myself from all those stalkers scammers and hackers out there, you know, you know you're vulnerable, but you've done all the obvious things. You know, you're not using short passwords. You're not repeating passwords, but you're not really sure what else to do. Maybe you've done all your two-step verification stuff. Let me tell you something simple you can do that'll make a big difference. Go to www.joindeleteme.com slash lawfare20 and enter the promo code lawfare20, you'll get 20% off delete.

me. What is delete me, you ask? Well, delete me removes your personal information that's being sold online from the internet. Here's what you do. You first of all, you recognize the reality that in the age of AI, we are all especially vulnerable to scammers who use your personal data, which they can buy, and it's floating around on the internet, and they use it against you. You get Delete.me, and they get rid of that. You ever Googled yourself and found out that you can find your own home address, phone number, and the names of your family members? Well, if you can do it to yourself, other people can do it to you. And if you don't find that unsettling, well, you should. Here's the good news. Delete.me can help. It was named the top pick for data removal services by Wirecutter. you can get an individual plan for a little over $8 a month. I use DeleteMe. I have used DeleteMe since before it started advertising with Lawfare. I actually pay for my own DeleteMe account myself. I know I could probably get a promo version because I read ad copy for them, but I don't. I actually use this product. I pay for this product, and I do it because I want to be able to tell you that completely honestly. Here you go. Take control of your data and keep your private life private by signing up for Delete Me. Now at a special discount for our listeners. Get 20% off your Delete Me plan when you go to www.joindeleteme.com slash lawfare20 and use the promo code LAWFARE20 at checkout. The only way to get 20% off is to go to www.joindeliteme.com slash lawfare20 and enter the code lawfare20 at checkout. That's www.joindeliteme.com slash lawfare20, code lawfare20. All right, I want to be straight with you. I did not start lawfare dreaming about payroll tax deadlines. I just didn't. I started it without thinking about running a business at all. I started it because I had things I wanted to say about law and national security, and my friends did too. And I thought, wouldn't it be fun to have, we called it a blog back then, wouldn't it be fun to have a blog where we could talk about national security and law? And not once in that process did I say, I want to file payroll tax stuff on time. And yet, every quarter, there they are sitting on our desk.

And if only Gusto had been there back then to handle that part quietly in the background so that I could stop dreading the calendar, that would have been awesome. Gusto is online payroll and benefits software built for small businesses. It's all-in-one, remote-friendly, and incredibly easy to use, so you can pay, hire, onboard, and support your team from anywhere. Now, I was just in Kenya. Would have been great to be able to do that stuff from there back in the day when I was still doing the payroll myself, which I'm not. You can save time with built-in automated tools, offer letters, onboarding documents, direct deposit, and more. I'm talking about automatic payroll tax filing, simple direct deposits, health benefits, commuter benefits, workers' comp, 401k, you name it. Gusto makes it simple and it has options for nearly every budget, which is why it is ranked number one in G2's highest satisfaction products list for 2026 and trusted by more than half a million small businesses. So try Gusto today at gusto.com slash lawfare and get three months free when you run your first payroll. That's three months of free payroll at gusto.com slash lawfare. One more time, gusto.com slash lawfare.

If you run a business or work with employers on benefits, you know the story. Group health insurance costs keep climbing, and the plans rarely fit everyone's needs. There's a newer model worth knowing about, called an ICRA, which lets employers set a contribution, and employees choose their own health plan based on their doctors' budget, and what actually works for them. Healthcare isn't a group decision, so why should coverage be? Get coverage you control. Find out if an ICRA is right for you at ambetterhealth.com. And Ledin has funded over $11 billion in loans since 2018 with a perfect track record of protecting client assets. Get a Ledin Bitcoin-backed loan today. Visit ledin.io.

We all belong outside. We're drawn to nature. It calls to us. Whether it's the recorded sounds of the ocean we doze off to or the succulents that adorn our homes, Nature makes all of our lives richer, calmer, and frankly, better. Despite all this, we often go about our busy lives removed from it. But the outdoors is closer than we realize. With AllTrails, you can discover trails nearby or trails worth traveling to and explore confidently with offline maps and on-trail navigation. Whether you're looking for a laid-back walk with family or something more adventurous to get your heart pumping. AllTrails gives you the tools you need to get out there and find your outside. Download the free app today and make the most of your summer with AllTrails.

I think this question of what should the regulated act be, I'm kind of curious to hear your thoughts on this, Alyssa. Is it the creation? Is it the distribution? Is it the implied threat maybe of putting a person into a very particular type of sexual scenario? Is it profiting from the ecosystem? Where do you see the lines around policy on this front? Or maybe regulatory laws, better term than policy for this one?

So in this space, I'm often kind of torn between my background in computer security and focus on kind of anti-censorship, protection of people's communications, and how we can best address harm. So I think certainly regulation that addresses sort of the pipeline that funds this would be most effective if done in a way that is careful about addressing non-consensual intimate imagery specifically, and that sexual content in general. And I think that's always kind of a tension, at least I've seen from doing work on the security side in this space is just when companies say react to a regulation by going really broad in their prohibitions. Sometimes we find that just kind of pushes everything to the corners of the internet where we have the least stability to kind of take control. So I always think about it in terms of a balance there. And because this is dual use technology, particularly that used to kind of create things, I think one of the biggest areas for computer scientists to do research is how do I identify when something's sort of purpose built for creation of non-consensual intimate imagery, whether that's a website with very clear kind of keywords and functionality or a model that's very clearly kind of trained or fine-tuned on particular individuals. And there are folks doing research in both of those directions. I think addressing those tools and both who owns and profits from them as well as who uses them is important. The other thing that I think about, maybe more so from corporate responsibility, I think I'll defer to you all on regulation, is kind of we have seen the use of mainstream platforms like Google, Facebook, et cetera, as single sign-on providers. So you click and then you can easily sign into the platform for various notification services. And when that happens, it gives this kind of like acceptability stamp to the platform, even if it's not the intent of the single sign-on providers. Same when we see those platforms like running ads or having applications in their app stores that can do this or are dedicated that way. And so I think there's a lot of kind of corporate governance points in terms of just like keeping an eye on where your brand is popping up and what you're promoting. And we've seen lip service for doing that. We've also seen some failures after the lip service. And I think a lot of those failures are because this is a dual use kind of thing. You have to really like test the actual application. Like you need a, we usually like generate an

image with a face covered. We try to do some double checks that it's not identifiable. And we use that to test what the system can do. And I think that often, that's kind of not what's happening. Just like the form is being read by whomever is reading the form or whatever is reading the form. And that's where we've seen a lot of failure. So I think whether that's a regulation thing that gets addressed, or just sort of a corporate responsibility thing,

that would be important. Yeah, so I want to actually kind of cut in because I know Renee might have thoughts on the same question. And she's also done a ton of work. I mean, Renee, you've also done like so much work on kind of how all these different systems are running, kind of capturing the idea of how they each respond to different types of prompts, how Grok is responding to prompts, how Meta is responding to these same types of prompts. I'm curious how your research, you would describe it as different or the same as some of the stuff that Alyssa has done and also if it has pointed you towards a different policy kind of suggestions because I do think that Alyssa's point which is like it kind of depends on like what it is exactly that the model or the actor or the whatever is doing like it is one thing if it is like a jerk ex-boyfriend using Grok to like to do this thing and it's another if you're talking about the model itself and then there's another thing if you're talking about kind of like literally some type of like six guys in a trench coat in russia deciding to kind of run a shop that does this for individuals off of grok right or something like to that extent um and so kind of i'm just like wondering like both of you have had kind of these qualitative conversations and interviews with people and done a lot of the qualitative experimental work yourself so anyways i just kind of we'll start with like a renee i'd love to hear that yeah it's a real big challenge i used

to think that corporate responsibility was was something where you would just never see this from large foundation models including grok because they would just prevent it and then we had that debacle with donut glaze right and on on elon's on elon's platform where for i think probably most of the country was aware of what happened with that but just to restate it in case or not. It was basically that people began to realize that spicy Grok would generate all sorts of images of people in very compromising situations. And with X, it's interesting because you could just ask Grok to generate the image right under the person. So this was an extraordinary vector for harassment. And many of those images got millions and millions of views. Ashley St. Clair, an influencer who also had a child with Elon Musk, is currently suing him. I don't think we have really seen that case move very much. I haven't seen any updates on it in a while. But, you know, she sued because of what happened. I think he's fighting about venue at the moment. This is where this, you know, beginning to realize that corporate responsibility isn't going to hold. You know, you just see these rather extraordinary things where like morals go out the window.

And I just want to say for listeners that the donut glaze thing is you would essentially ask, it was a way of routing around things that were. So I just want to say, like, because I do not think that most people are going to be for that are not very online are going to be familiar with this kind of absolutely horrid, specifically for the generation of child sexual explicit material, that like how bad this was, you could instead of asking for a pornographic, a specific type of like, thing to be all over somebody, you could ask for donut glaze to be all over someone, which was the approximate look and style of like, what that thing would be to create kind of a very explicit. image that could be taken into various contexts and just like, you know, having a child covered in this or anything else. It was very, it was a very kind of huge controversy in the world of people who follow this type of thing. Unfortunately, I don't think it actually kind of made the mainstream news at all. I mean, both because it's incredibly explicit and because I don't know, people tend to not care about these things at the margins. And like Elon Musk has such a high threshold for the thing like of all of the absurd things that he does. It's like half of them don't

even rank anymore. This one was interesting, because it did lead to immediate regulatory action in Europe, right? So you had a lot of immediate calls for, you know, various types of the regulators in Europe requested specific data from X, that sort of pushed it into the news a little bit more than I think it otherwise would have been, in part because you began to see coverage of Elon saying, this is censorship, you know, being asked to provide data on exploitation on his platform was censorship. Requests that, you know, people pointing out that maybe this is actually a terrible thing, you know, were recast as censorship. I actually wrote an article about this for Lawfare for anybody who wants to track it down. We'll put it in the show notes because then you can kind of see the specifics of this particular story. But I think the challenge on the regulatory front is not wanting to cross bounds into free expression, as we've talked about, you know, where, where are those lines, you know, you mentioned the child exploitation content, that's much more where I have done my work, not in adults and CII so much as, as where there is no consent, there is no consenting person in that, in that exchange, right? The child is either being exploited or an image that is potentially derived from models that have a perception or inadvertently were trained on that stuff. That's where you start to see that happen. So there are some very granular laws that come into play when the child content comes the focus of the thing that are a little bit different than the potential that it might be voluntary content creation and uploading. So I think that's where, for me, a little bit of my work has been focused much more on the former. I think I want to talk about the technological barriers to creation, though, because this was the other thing, right? If you have corporate responsibility and the corporate responsibility relates to them putting guardrails on their model, there is this dynamic where people will go and do things with open source models. And now with Cloud Code, it will just tell you how to set up an open source model on your computer. This used to be something where it was a little bit more gated by competence, and now anybody can do it. So each theoretical bound related to ethics or user competence, those have all been sort of eroded. So I'm curious, Alyssa, when we think about this creation stack or this distribution stack or the payment stack for sites that are making money off of this, how do you think about the different roles that each of them play and what would you like to see as policies for that stack?

So I think of this as kind of a landscape, right? And as you said, we have sort of the tip of the iceberg is like the notification websites and services. I go to a website, I upload an image, I click some buttons, I get something back. I think those are in some ways the easiest to some degree. We take down the website. It's pretty clear what the website is. Now, is it going to pop back up a million times? Yes. And so that gets to what underpins that website and what underpins that website as far as we can tell from the research we've done in the communities that I mentioned where some people were building these kind of tools, as well as looking at like how-to guides that are hosted across the web is that they're using, as you said, an open weight model. They have kind of a system prompt and they've built this such that people can put in whatever they want, but they've kind of specialized it to perform well at notification. And so one of the things we find in the report that I mentioned we'll have coming out is there is no mention of open weight model providers doing any kind of monitoring of downstream use of their services. And in fact, If you look at Stability AI, which builds stable diffusion, which is one of the open weight models that's used very heavily on the website called Civit AI that I mentioned that has been used for modifications of people, you'll see that I believe they reported zero reports to NCMEC during a particular one-year period. During the same period, Open AI, for example, had far, far more reports.

NCMEC being the center that runs basically the database for photo DNA and other things that automatically use hashes to take down child sexual abuse material and things like that.

Yes. And so, you know, as NCMEC themselves sort of says, like the number of reports maybe tells you more about how well you're monitoring than it does about how much abuse is happening. And when researchers have done research, they've found several different open rate models being used quite prevalently and in ways that are observable because they're kind of publicly on the web. So that's kind of one piece of the puzzle. The other piece of the puzzle that you're getting at, Renee, is can we do anything in the model itself, like such that the model couldn't be used in this way? And what we find is there's like three different approaches you could take at the model level technically right now. None of them work very well. So one set of approaches, people have proposed like, what if I remove certain kinds of content from the training data? will that prevent that idea? So let's say I take all the images of children out of the training data. Can the model now not make a child plus sexual content, otherwise known as child sexual abuse material? If you could do this perfectly, then it might help in a closed weight model. So in one that you can't modify, but even if you could do it perfectly, it wouldn't help in an open weight because you can just reintroduce the concept and we can't do it perfectly. So in our own research, we find it kind of makes it go from three prompts worth of difficult to 12 prompts worth of difficult in order to generate just because you still have a few examples.

There is a bunch of research on ways to kind of make this harder, do anti-tampering, kind of prevent fine-tuning. None of it's ready for deployment. We don't see any platforms using it. And so most of the protections that we see from foundation model providers are for the products that they build themselves on their foundation models. And they're really those like input-output filters, like let's try to check that your prompt is okay. Let's try to check that your output is okay. And as we've discussed, those are circumventable. They're also using AI models to do that filtering and AI models are unreliable. So right now it's a very kind of gappy ecosystem. The last thing I wanted to mention was we've talked about kind of like the EU response in terms of regulating. And I know there's been this proposed update to the AI Risk Act in terms of kind of saying you can't have a model that can do this unless you've deployed certain safeguards. And I think one of the things we found in our research is you hear a lot about the safeguards that foundation model providers have implemented, but we find very little transparency information on how well those safeguards are evaluated. And even when we interview foundation model providers, we find a lot of kind of nebulousness about the kind of threat model that they're trying to address. So are they only worried about direct generation of an image? Or as you mentioned, are they worried about code being generated that can go and be used to make the image? Are they worried about instructions? Are they worried about API users, et cetera? All of that's a little fuzzy. The answers to questions we've had about likenesses and sort of what fits the definition of NCII have been fuzzy. So I think there's a lot of attention that would need to be paid to like what qualifies something as a sufficient safeguard? And how do we set people's expectations appropriately for how much that's going to protect them individually versus kind of mean something about mitigation

in a specific product? Yeah, so this is actually kind of getting at something that I want to ask both of you, which is that like, to kind of bucket a little bit the different ways that we've talked about this. Tell me if for both of you, if this seems right, or this seems wrong. Like, I feel like we're kind of talking around this idea that there's different ways of attacking this problem it's kind of like the post hoc kind of legal reaction litigation type of way to kind of create a post hoc reaction and kind of a tort based like kind of solution set for an individual that is harmed by this type of stuff and ideally kind of like an disincentive to do it in the future for other people for tort feasors who might do this type of thing again to other people there's obviously we didn't talk about this but criminal liability you could create around that to a certain extent, like the NCII in state kind of laws that have been created over the last few years. And then there's, I think you just said it really well, Alyssa, there is like the and Renee kind of gestured at it when she was talking about like the model based versus the open source versus the guardrails versus the regulatory scheme, like world of things that we could open up. There's like, how do you get at this problem when one information is free, you can kind of pass all of this along, these models are free. Like, is there it seems like whack a mole on steroids, You just can't possibly kind of do this. And then there has been, you know, for better or worse, all of this call for transparency around these things, which lets out from a security standpoint, a lot of the secrets that we're like, learning and refining to like, make these products and models safer. So one of those speaking of the donut glaze kind of thing to go back to that was something that I had heard about a year and a half ago when I was interviewing people at OpenAI about like their how they're what their red teaming process looked like. And that was a popular knowledge of the time. And I remember being like, wow, that's horrifying. And also like, oh, that's not going to stay a secret just to open AI's red team for a very long time. And of course, it didn't. And so I'm kind of just like, very interested from both of you going forward, you have things like red teams, which for those unfamiliar, our red teams are essentially like, specialty people who are either in specialty areas, like you're either in nuclear development or poison control, or you're in like, you know, some version of, you know, I don't know, some version of harm prevention. And you get asked by the models to essentially run a bunch of bad actor questions, the red hat, like kind of idea, like you put on a bad actor hat, and you test the machine with as much stuff as you can throw on it. And that is kind of then guard rails and safety mechanisms are in theory brought in by the company. Like that is a corporate responsibility idea that we could put into into regulation and force them to have that and be responsive to it. But that doesn't get rid of the whack-a-mole problem. And maybe, you know, keeps going. So like, Renee, you asked this question of Alyssa, but I want to ask it of you.

And then like Alyssa, I kind of want to see if you just agree with that framework that I presented from your research. Renee, do you think that the, where do you think that like the choke point of this is? Is it all of the above? I think there's different voluntary things that the different actors can do immediately. Red teaming, by the way, it's not clear for outsiders what the legal frameworks are, right? There's certain things that, you know, you can't do and reasonably you can't do. You know, when Meta makes an announcement about its thing and you want to know if you can, you know, if it will generate for under 18 accounts, that's not something that you can really do ethically as an outside tester, right? So these questions around what is the formal legal framework for authorized AI safety research? What should it be for the people within the companies? What should it be outside of the companies for maybe specifically designated individuals? I think it's very hard to know how to do that because just saying I was red teaming is not a legal defense. As far as the responsibilities in the chain, so first of all, I don't have very much faith that open source communities are not – There are these subsets that are actively there for just this sort of content, right? That is the motivation. So with that, then I think you look at questions around distribution. I've seen, you know, law enforcement go after certain types of monetization sites. Sometimes that's hard because they live outside of the country. Bellingcat did a really amazing investigation into one of the sites, and the name is escaping me right now, maybe Alyssa remembers it, but sort of trying to track down where this operator was and then trying to see what sort of legal rules might apply to that person where they are and how to think about that issue. There's the hosting and cloud services have something of a role to play here, but then people get very uncomfortable about the idea that they're scanning. And again, this doesn't adequately address issues of consent and whether a person has those images up in their drive because it is theirs, they made them, they're an adult, and this is fine. So you do have a whole lot of different component parts here. And a lot of it is going to be this patchwork of, there will be, I think, the mainstream companies that will adhere to laws and be good corporate citizens. And then you'll have these sort of niche outsiders that are going to continue to keep that market active.

Yeah, no, I do agree, Renee. And one thing that I might add, and I think this relates to the kind of, you know, should we require red teaming as well as to the continued proliferation is red teaming can vary in quality and it can vary in like the extent to which it covers the threat models, right? Part of why we do the research with perpetrators is so that we can make sure when we're doing tests that we're representing how those people are actually doing things. Sometimes that's not happening. And so I think the question kind of becomes if we find that we can't currently safeguard a particular system or the odds of harm are X out of Y, how do we sort of trade off what we want to release or allow kind of under like an EU AI Risk Act type concept versus what we're able to protect? Like if we're only able to protect non-person generations of video, how does that kind of change what we want to release? And I don't think I alone have the answer, but it becomes kind of a societal question. And so I think the thing I worry about is we're often looking for like a technical bandaid where there just may not be one or there isn't one yet. And that becomes kind of a question of like, do we try to not let new stuff out? Is that feasible? Do we try to put more resources into deterrence messaging, primary prevention, doing like social norms shaping? Those are things that I worry about.

I'd love to maybe close with something like, do you think there is one intervention that is technically possible now, but largely neglected by companies that you would like to see implemented?

Yes. Tell us about it. I would love to see influenced more widely is deterrence messaging. So this is something there's like three studies out there looking at how deterrence messaging can, which are basically messages that try to say, hey, the request you have appears to be for content that might be illegal to share, let's say, under the Take It Down Act. These have been used for child sexual abuse material before. there's been research on using them for non-consensual intimate imagery. The research shows a lot of promise when they're deployed. However, none of the seven foundation model providers we look at at and none of the ones we interviewed were doing deterrence messaging for ncii two had it on the roadmap but nobody else um did and i think there's a lot of research to be done in terms of not just kind of deterrence messages for generation but also like for bystanders how do we think about that within these communities where people are making sexual content that's non-abusive how do we think about it but those kind of social norms shaping efforts um can be very promising yeah and I should maybe be clear. When I say deterrence messaging, I mean, when you attempt to generate content on a particular, say, foundation models website, or what have you not necessarily when you're sharing content in encrypted messaging, I would not prefer that. So I think what this

conversation has really made clear is that AI generated sexual abuse is not just a problem of bad content slipping through imperfect filters. It's really a problem of systems that are built without consent at the center, trying to tack it on now, legal rules that often intervene after the harm is done, and then safety regimes that put an awful lot of burden on the target as opposed to the creator sometimes. So I think you've really brought out a lot of the very nuanced, challenging issues here. I think maybe some people who are new to the conversation have always thought that, oh, well, those just ban it, and then it'll stop. So I appreciate you walking through the nuance with us. I think the questions are not just whether a model can generate this material, but who is responsible for thinking about potential guardrails, what kind of research is necessary to test the risk, and then how we protect expression without turning content creation into a shield for exploitation. So I want to thank you for joining us. And you have a CDT report that you mentioned. We will put that in the show notes. And yeah, really appreciate you being here.

Thank you to you both for a great conversation.

The Lawfare Podcast is produced by the Lawfare Institute. If you want to support the show and listen ad-free, you can become a Lawfare material supporter at lawfaremedia.org support. Supporters also get access to special events and other bonus content we don't share anywhere else. If you enjoy the podcast, please rate and review us wherever you listen. It really does help. And be sure to check out our other shows, including Rational Security, Allies, The Aftermath, and Escalation, our latest Locker Presents podcast series about the war in Ukraine. You can also find all of our written work at LockerMedia.org. The podcast is edited by Jen Pater. Our theme song is from Alibi Music. And as always, thanks for listening.

Hi, this is Elise Hu from TED Talks Daily, and this episode is brought to you by AmBetter Health. For many employers and brokers, group health insurance means unpredictable costs and plans that try to fit everyone but end up fitting no one. A model called ICRA is changing that. Employers set a fixed contribution. Employees pick the health plan that works for them, their family, their doctors, their budget. Predictable for the business, personal for the people. That's a better kind of coverage. Get coverage you control. Find out if an ICRA is right for you at ambetterhealth.com.