AI moderation platform is becoming not just a filter, but a part of the decision-making architecture. In the case of DoorDash, this was critical because the system had to operate on millions of messages per day without disrupting the user experience.
The problem began where most architectures hit the cost of error. In a marketplace chat, it is necessary to quickly separate safe messages from dangerous ones; otherwise, both safety and the sense of security in the product decline. DoorDash dealt with millions of messages per day, and the solution had to be found in fractions of a second. Under such load, a direct call to LLM seemed logical only on paper: latency was too high, and the cost was too great.
The team chose not a “single smart layer,” but a two-step scheme. First, a cheap internal classifier filters out obviously safe messages. Only those cases where the model is uncertain are sent to the LLM pipeline. This is a pragmatic compromise: LLM is applied selectively, not to the entire stream. This architecture reduces costs and maintains latency where it is critical for the chat experience.
What proved decisive was how exactly LLM is used. It is not asked whether a message is safe or not in a Boolean format. Instead, the model receives scoring across multiple axes: threat, profanity, sexuality, and other contexts depending on the scenario. This is an important engineering detail. Boolean fixes the system at a rigid fork, while scoring provides a knob to adjust thresholds and take graduated actions.
The implementation was built around this principle. The stream was cleaned of noise: empty messages, image attachments, and simple pleasantries. Then the message passed through the internal classifier. If it was clearly safe, the system allowed it to proceed. If not, the LLM was engaged to evaluate the content across several axes. After that, the system could choose an action based on severity: from censoring the message to blocking, canceling an order, or warning the user. The same general approach was used for voice and image, but with different input mechanisms. For images, the cheap layer was implemented through a commercial vision API, while for voice, some constraints were fundamental: the words had already been heard, so the system could only react, not prevent the delivery of the utterance.
The outcome was practical rather than decorative. After implementing SafeChat, the team observed approximately a 50% reduction in incidents related to verbal abuse. Importantly, this is not a metric of model quality in a vacuum, but a reduction of real harm to users. However, the story did not end there. When other teams began to adopt the solution, it became clear that the point system for a single use case did not scale well as a platform. Essentially, the team needed not a separate SafeChat, but a reusable architectural pattern for moderation across multiple surfaces.
This case illustrates how an AI moderation platform should evolve in production. First, the team studies the data and only then chooses the form of the model. Next, it separates the cheap layer and the expensive layer. After that, it introduces scoring instead of a binary response. This approach does not eliminate complexity but makes it manageable. And it is in this, rather than in grand promises, that a working architecture typically resides.