Transcript
Bruna Pereira: I'm Bruna. I'm a software engineer at DoorDash. Today, I will take you on a story about a system that we built to solve a real safety problem in production, and when it worked, we threw it all away to build something even more powerful. By the end, I want to leave you with two things, how we built SafeChat at DoorDash, and one architectural pattern that you can take home and use at almost any AI use case that you have. I've been a software engineer at DoorDash for one-and-a-half years now. I lead the trust and safety engineering team from our hub in Sao Paulo, Brazil. Before that, I spent three years building my own startup. My background is mostly on FinTech, building highly scalable systems.
The DoorDash Marketplace (Safety-First)
DoorDash is a marketplace. We have an app that consumers can order food from. We have the Dashers that deliver the food. We have the merchants that prepare the food. These three groups, they can meet in different parts of the journey. They can talk to each other in chat. They can call each other during active deliveries, and of course they can meet in person. For DoorDash, two things matter equally for us, that people are safe and that people feel safe in the platform. For us, both metrics are equal product metrics for us. When we talk about safety, a meaningful amount of the safety incidents that we see on the platform are related to verbal abuse. These groups, since they meet in chat or in voice, it doesn't matter. We need to act right away when we sense that there's something unsafe happening, especially because different from a social media platform, we don't get to build strong relationships between these parties.
These relationships, they last from 40 minutes to 6 minutes max. We must act right away when we see something unsafe happening. Talking about scale, we have over 4 million messages exchanged in chat every day. Over 400,000 calls are exchanged between consumers and Dashers during the deliveries. Over 200,000 images are exchanged in chat or even SMS. For chat specifically, our goal is to make sure that every message that reaches the destination is classified as safe by us. This is what leverages the difficulty of this system, because look at the amount of messages that we have. For chat, we only have a fraction of a second to decide if this message is safe or not. Otherwise, we would be disturbing the chat experience. When we first talked about it, the first response, especially from business folks, was to just throw an LLM on the problem. Just call an LLM, ask if the message is safe or not.
If it's safe, deliver it. If it's not, block it. Yes, it would have worked theoretically, but in practice with the volume that we have, if you have used LLMs before in production context, you should know that latency of these LLM calls, they can vary a lot. For our use case, we have like 2 to 10 seconds in average of latency for each of these calls. Also, imagine what is the size of the bill of calling the LLM 4 million times a day. It would just not work.
Building SafeChat
The question became, how do we be smart about the usage of LLMs? The first thing that we did is we tried to understand our data. We spent a couple of months understanding what unsafety means in our context, and also, what is the percentage of messages that is actually unsafe. Because we know that most messages are unsafe, but how many messages? The point was not to start making decisions here. The point was just to learn from the data that we have. We instrumented the chat. We used a free moderation API available on the market. We asynchronously called this API to understand, what's the category of unsafe messages that we have here? This is the boring step that everyone wants to skip. This is the conversation that I have a lot back at work. Let's spend a couple of months understanding our data before we act, because it doesn't matter how cheap or expensive the model is.
If you don't know your data, you won't be able to get anything out of it. When we ran this analysis, we confirmed what we already knew. Most of the messages are safe, but we actually put a number on it. Only a small single-digit percent of messages were unsafe. This shaped our entire architecture, because it told us that we could build an aggressive cheap layer that was right almost all of the time.
Then we built a small classifier. It is an ML model trained on the data that we collected in the previous step. It had three jobs. It had to be fast and respond in less than 100 milliseconds at 90%. It had to be cheap, so no per-call costs, and only the cost of our infrastructure. It had to be good at one thing, identifying what is obviously safe. This layer is not a final judge. This layer is, just imagine that you are at the U.S. airport and you have the metal detector. A metal detector does not say that you are threatened. It just says that you are worth a closer look. For the messages that this model that we trained in-house is not able to classify as safe, then we call an LLM, which is less than 10% of the messages. Here's the design choice that matters.
When we call the LLM, we don't ask it if the message is safe or not. Instead, we ask it to classify or to score across multiple axes. I will tell you in a couple of slides why this is important. What we ask is depending on the context, in our context, we ask how threatening this message is, or how profane this message is, or how sexual this message is. This is the layer where the model gets to be smart, because you only ask it the hard questions.
Talking about the overall architecture here, a message comes in and we strip out all of the noise. We remove the empty messages, the image attachments, because those go to a different pipeline. We remove the common pleasantries-like things. Then it goes to this first layer that is the small model, the classifier. If the message is safe, we just ship it. If it's not, it goes to the layer 2 which is the LLM. The LLM classifies across multiple axes. With that result, we can take a graduated action. This is the shape that I'll show you through the entire talk. The cheap layer that goes most of the messages, the expensive layer, and then action. Each layer is doing the job it's good at, at the right cost. That's the important part. This is the slide that I want you to remember if you ever build your moderation system.
Do not ask the LLM for a Boolean. Instead, ask the LLM for a score. The reason for that is because a Boolean is a flag and a score is a knob. With a score, you can take graduated actions depending on the score that you received. You can add new categories later without having to go back and recreate everything. You can move the thresholds if you think that it's too high or too low. Also, it plays at what LLMs are good at. Because imagine if I throw a sentence here to you and say, is this message safe or unsafe? Probably your answers will be all different, because what is unsafe in your context and how would we classify something as unsafe? Instead, understand what unsafety means and ask the LLM to score across this axis. The results will be way more stable.
This is why the score is important as well, because we must act according to the severity of the message. For us in our context, low-severity content means like a swearing. If you swear, we just censor the message and let it go through. If it's a mid-severity message, for example, you insulted someone, we can block the message entirely and not let it go through. If it's a high-severity message, for example a threaten, we can block the message and offer the affected party to cancel the order without paying anything. If it's a very high severity content, we just also cancel the order, block the message, warn the offender, and remove the affected party completely from this loop. This letter is only possible because we have those scores. If we didn't have it, we wouldn't be able to differentiate what is rude and what is actually dangerous. I talked about chat, but for voice and image, the structure is pretty much the same.
We have the same scoring engine underneath. Some things are different. For image, for example, instead of that cheap internal layer, we have a commercial vision API that is already good enough at identifying violence, nudity, and unsafe images. We use that as the cheap layer. For voice, it's tricky, because for chat, we can read the message and decide if we ship it or not. For voice, at the moment that we transcribed the message and analyzed it, the words were already heard by the recipient. There's no way that we can avoid the message from being delivered. The only thing that we can do differently is to act. We can hang up the call as soon as we identify that there's something unsafe there, and we can, of course, offer the order to be canceled or cancel the order ourselves. The engine is pretty much the same.
The Result
After we did that, we measured how many incidents we had driven by verbal abuse before and after we implemented what we call SafeChat. We saw that we had roughly a 50% reduction in those incidents. This number is not like a model accuracy improvement that we had. This is real reduction in human harm. This is the number that we can use to justify the months that we spent understanding our data to build something powerful enough to help our customers.
A Content-Agnostic Moderation Platform - Building Blocks
This is the moment that I would finalize the talk and ship the product, and the team would get promoted and everyone goes home. What happened was that we threw it all away. Not the learnings that we had. Not the model that we trained. Not the data that we collected. What we threw away was the system, the SafeChat system. I'll tell you why we did that. When we started rolling out SafeChat, other people at DoorDash and from other teams, they were like, "I like what you're doing. I want to do it myself as well for my use case." People started asking us, can we moderate profile pictures for Dashers and consumers? Can we moderate name at signup? Can we moderate food reviews, for example? Even for cases that are not related to safety at all, like can we identify fraud in chat and in phone calls?
We noticed that if we started to implement each of these asks, we would be rebuilding everything over and over from scratch. We noticed that what they actually wanted was not SafeChat, was not the system that we built. What they wanted was the pattern that we created. That cheap, then expensive, then graduated action. That is what they wanted. We thought, why don't we transform it into a platform that anyone can use for any use case? That's what we built. We built a content-agnostic moderation platform. The idea here is that we don't need to know what your content means. We don't need to know your business logic. What we know, what to do, is how to log decisions, how to integrate with different model providers, how to change steps with conditions between them. The teams, they bring the meaning and the platform orchestrates that. That's the idea.
What we promised to the teams is that they can do that without writing any code. They can go to the platform in a UI. They can configure it and they can take the action that they want. That's the flexibility story here. We did that. It doesn't matter if you have a content guideline or a fraud use case or a safety use case. It doesn't matter for us. The platform, it has some building blocks that make it possible, and I will show you what they are.
We have three kinds of models inside of this platform as of now. We have what we call internal models that are models that we can train, fine-tune, and deploy in our own servers. We have external models, which are models that we can just sign a contract and use it from a vendor. We have an external prompt. An external prompt is a prompt that you can write to use in any LLM from any vendor that we have on the market. For internal models, they are trained and hosted internally. We have an ML platform that helps us to use any model available on the market, use our own labeled data. If we have labeled data, we can use that to train these models. We can deploy that in our own infrastructure. These models, they are served under an API that we can use inside our platform. The idea is that they are cheap and fast.
When we are training it, we have a predefined input and output schema that when we defined it, everyone, every client can use following this input and output schema. The small classifier that we built for SafeChat, it's actually an internal model now. We have the external models. The idea here is not to reinvent anything that is already available on the market. For example, image safety is something that is already solved. We don't need to build it all over again. The idea here is that the platform does not rebuild stuff just because. We have it available. We have a contract with them. We integrate once in our platform and any client is free to use. We just have different API keys so we can trace billing accordingly. We have the external prompts. The external prompts are the most flexible ones. We use an LLM gateway to integrate with pretty much any model from any vendor available on the market.
If you are more interested in knowing how we built this LLM gateway, there is a talk from DoorDash folks about how we built the LLM gateway at DoorDash. The idea here is that this gateway sits between the platform and all the models from all providers available outside. The clients can choose what models they are targeting. They can choose the input schemas and the output schemas for each prompt that they have. We can use the features that are available in this LLM gateway to declare fallback and retry strategies without writing any code again. We use the fallback and retry to, for example, if you've already used LLM models, you must know that sometimes they just are not responding. The providers are all down or the models are not working as expected. If you have created a prompt that targets a model that is not available, is there any other prompt that you can use as a fallback while this is not back healthy? Also, you define what is the output schema that you want from that LLM call. You define that JSON. Sometimes the LLMs, they just don't respond to that. You can define how many times you want to retry before returning a structured error to the client.
Composing Moderation Agents
Those are the building blocks. With these building blocks, you can compose into what we call moderation agents. A moderation agent is a pipeline or a workflow that you can compose the steps. Here is an example of an internal model that you can choose a condition. This condition is going to use whatever is returned from this internal model. With this condition, you can go to one step or another. In this case, it goes to an external vendor or an LLM prompt. You can change it, however, and you can take an action in the end. The action is the responsibility of the client. The client decides. At this point, the client has the output of each step that they have in their moderation agent. With that, they can take any action that they have, or no if needed. Talking about condition, the way that we express conditions is, again, entirely from the UI.
You can write and express any conditions that you want, simple or complex, using the output from the previous model. This is actually the pipeline from SafeChat. It's super simple. It has an internal model. If the unsafe label that is returned from this internal model is greater than 0.5, then it goes to the LLM prompt. The LLM prompt is going to respond with a set of categories and the score of each of them. It goes to an action. If this is not greater than 0.5, it just goes to an action. Here I know that we don't take any action. Actually, we censor the message if there is any swearing in it. You can express any condition between any steps and you can create your moderation agent. Your moderation agent, it can be of two different types. It can be a synchronous moderation agent, which means that it's an HTTP call.
The connection is held open while the moderation agent is being executed. We execute all the steps that we have in the agent. We use synchronous only when we want to gate a decision. For example, in chat, we want to be able to block the message if the message is unsafe. There is a downside of using asynchronous because you need to cap the latency of each of your steps according to the overall latency of your call. We try to avoid that as much as possible, and we prefer to use the async version of the moderation agent. You can choose whatever when you are creating. For async moderation, we receive a request of moderation agent execution. We acknowledge that you asked that, and we run the entire agent in the background. When we finish it, we publish a message to a Kafka topic that the client is subscribed to, and they can react to it when the message comes. It's way more flexible. It helps you to not cap so much the latency of the execution, and you can add more complex steps and more steps actually inside the moderation agent.
One other feature that we have on the moderation platform is the backtesting. This is interesting because sometimes when we create a prompt we are like, that's going to work for sure but we are not that sure. We test it against one, two, three examples but it's not enough to make sure that it actually works in production. Also imagine an agent as complex as I showed before, you probably don't know what's going to come out of it. The idea is that you can use the backtesting functionality from the platform to build your agent. If you have some historical set of data, you can test your agent against this set of data, and you can even test it against a single step if you want to. A human can go there and classify the results of each step or each agent as correct or incorrect, and in some cases we can even label it as true positive, true negative, false positive, false negative. By the end we use that information that the human input to calculate some metrics to understand if it's good enough to go to production or if we need to fine-tune it even harder. This is what makes the test it before trust it a real built-in workflow in our platform.
Lessons Learned
I have three lessons here that I want to share with you. One of them is, if you have a hot path with high volume and you want to use an LLM for it, put a cheap model in front of it. Usually, the 90% of content that you are sure that should not reach the LLM, it can be caught by a cheap model that you can train at home. It takes some time. You need to learn from your data. That part that the LLM does for you, you need to do it yourself first. Don't skip that part. Resist that. If someone asks you, yes, but let's add the LLM just because we don't have, no way. Because the economics, they only work if you ask only the hard questions to the LLM. When using LLM, do not ask for labels. Do not ask for, is it true or false?
Ask it for a severity, ask it for a score. Also don't ask it for a 5-digit score because the LLMs are not calculating anything, they are reasoning. Ask it for a score in a simple way that the LLM can answer that. Know when to throw a system away. If you build something and you see that it's good, and people are asking for their version of it, maybe the system that you built is not the asset that you have. Maybe the pattern is, and you can transform it into something that everyone can use. I noticed that today it's so cheap to create code that people just go for it. Yes, we don't need to reuse because we can build our own because it's that cheap. It's cheap to create code, it's not cheap to maintain the code.
Summary
This is the whole talk in one shape. We have this cheap filter that we built, learning the data that we have. We have the smart judge using LLMs. We have the graduated action. All of that on top of a platform that we can use. Internal models, models that we deploy and serve from our own infrastructure. We have vendors, contracts, APIs that we can use and provide to any client. We have the LLMs. The LLM gateway is actually a very important piece of this platform. All of that with declarative configuration, so no need to code for that. It's already set in a way that you just need to select the pieces in the UI and create your own pipeline. The backtesting is super important as well. It's terrible to go blindly with some prompt to production to see what happens when you throw it there. I'm pretty sure that if you have a moderation problem at home, if you use that pattern, you'll be able to solve it in an easy way.
Questions and Answers
Participant 1: You mentioned that the internal models were cheap. Can you give us a sense of how cheap? Because I imagine there's resources that goes into maintaining and training those models internally too. How cheap is it compared to lower tier models from the vendor itself?
Bruna Pereira: How cheap it is depends on the volume that you have. Because the cheap is in a sense that we only need to deploy it to our servers. It depends on the infrastructure that you have. The point is that you do not pay per call. You pay just to deploy it. The training part, yes, that has some costs that you do it only once. Actually, for the SafeChat, for example, we are in the 9th version. We trained it nine times. It's like a one-time cost for 4 million calls a day. That's how we calculate cheap and expensive. It depends a lot on the use case and on your infrastructure.
Participant 2: Did you come up with any way to quantify the backtesting, like how much is enough or we need more kind of scenario?
Bruna Pereira: That's hard. I'll give you an example that we just used, that is safety versus fraud. For safety, the unsafe messages, they are clearly unsafe. The internal model cannot always identify what is safe or not. For fraud, for example, there are things that are pretty fraud-y, but then they are not in the end. Because they say, yes, I'm going to pay you when you get here. They want just to give some extra tips to the Dasher because the Dasher is doing a favor. Is it fraud or not? It depends on how much of a gray area your use case is. Usually, what we do is we do not want to like test over a thousand examples, because we need a human to go over that manually and classify what is true and what is not true. If you have 100,000, you won't be able to do it. A thousand is a number that we actually trust. People stop to do that, and care about that. Because we tried more and people would just answer with anything, so it reflects in the scores that we have. If I had to throw a number, I would say a thousand is a good number.
Participant 3: Was the cheap model ever super deterministic, like just an array of swear words before it got more complicated than that? Then, do you also score cheap model instead of just having an airport security, yes, no, this is safe, this is not safe?
Bruna Pereira: Yes, the cheap model is scored also. It has like a 0 to 1, how unsafe and how safe it is. Yes, we did have some problems with that, because when we ran our first round, we tried to just collect some unsafe messages that we had. When the message comes to us, it's already bad enough. Because if it's like, not that bad, no one is going to complain. The first round of training that we had, it was like a bunch of really bad messages and a bunch of ok messages. The model was a bit of almost binary. Then we ran a second round of it, we used this moderation API that we have available on the market, which is super slow. That's why it does not work for us. It takes seconds to return something. Then it could help us with data that is, yes, this is super unsafe, this is kind of unsafe, and this is safe.
It helped us to return gradual and more reasonable results from the model. Definitely the model is not a true and false. It's also a score. It's hard also to define what is the score that you say that with this score, you should go to the second layer. Because if it says yes, then 85% chance of it being an unsafe message, does it go to a second layer or not? This is something that you need to learn from your data and move the thresholds when you need. With this, the ML platform team, the data engineers, the machine learning engineers, actually, they helped us a lot with that.
Participant 4: Does your platform also help with automating the building of new internal models to fit other use cases like for the cheap and fast step?
Bruna Pereira: No, not right now. Actually, the training and retraining and fine-tuning part is something that is totally handled by the ML platform team. We rely on their platform to do that. We do not need to do it ourselves. This is a good separation of concern because there is totally knowledge, as I mentioned here, that the ML engineers they have and we don't. We do not want to have this overlap. The moderation platform is only a client of this model that is already trained.
Participant 5: You mentioned that you retrain the model a couple times, the cheap one. How do you decide?
Bruna Pereira: First, we did random analysis on the data. Also, these unsafe cases, they create incidents and they go to agents, and the agents feed us back with the information of, yes, this message came to the customer and it was not caught by the model. Then when we note that, we try to understand what's the pattern that we missed and in which step? It was in layer 1 or 2? I'll give you an example. We had a retraining. The last time that I retrained, I retrained it on abbreviations. It doesn't matter if the LLM understands the abbreviation because it never reached there because the first layer says, of course, it's safe. There is nothing unsafe here. I had to find a meaningful amount of examples of unsafe messages that use abbreviations and then run another set of training in this first layer. With the volume that we have, we cannot see all the messages that go through. Usually, the flow reaches a safety person. They come to us and we try to identify what is this pattern that we missed.
Participant 5: When you go for the smart piece of the platform, you're connecting the external model. What's the fallback if this becomes too slow, for instance, because you mentioned that you need low latency on the messages? What's the fallback?
Bruna Pereira: Actually, one thing that I didn't mention here is that the fallback can be any kind of model. We do have this internal model, but sometimes our servers fail. We have a fallback with this moderation layer that is an external vendor. We use it while we don't recover our servers. It's slower, but it's better than nothing. The fallback here can be any. You can use, as a fallback, the LLM, for example. We wouldn't use for chat specifically because the volume is too high. You can add as a fallback, any sort of model. If you have a vendor that is not working properly, maybe you can use an LLM just while the vendor is recovering their API, and then you can return to use the vendor.
Participant 6: Just talking a little bit about the backtesting, you talked about a little bit the human need to actually produce that. Is that data that already existed or is that data that had to be calculated as you began this project? Did you have a whole set of unsafe messages already established or is that something you had to review old data and find those as part of this process?
Bruna Pereira: When we started, we had only a few data because we didn't have it instrumented, the chat. We didn't have any data to train on. What we did first is that we found the safety cases that were created due to verbal abuse. From that, we could find examples of unsafe messages. We instrumented the chat. When we started, we spent like one-and-a-half months only collecting data. From that time on, we had the data that we needed to retrain it whenever we wanted. Some things we missed. For example, the abbreviation part we missed because we don't get to look at all the messages. Sometimes we learn stuff when a human comes to us and says that something is wrong, and then we can find the pattern and integrate that pattern in the prompts or in the models that we have. One example that we had is like, we started with some set of categories and then we realized that most of the complaints that reached a human were related to disrespect and not, that's unsafety. We added a disrespect category in our LLM to identify how disrespectful this message is, for example. Yes, you don't do it once and then it works forever. You need to keep looping and improving it.
Participant 7: As you started solving the more nuanced problem of fraud, did you also find you had to start tackling prompt injection or other emergent problems?
Bruna Pereira: I think we were already smart about this problem. Especially the LLM gateway, it also helps us with blocking or not allowing this kind of stuff. Yes, we do have some ways to handle that. Even before fraud use cases, we already had stuff like that, definitely.
See more presentations with transcripts
