BYOC Logs addresses the challenge of log management at a petabyte scale without sacrificing observability. This is crucial when the growth of telemetry begins to overwhelm traditional self-hosted solutions.
The problem manifests gradually. As cloud-native systems and AI workloads grow, the volume of logs increases non-linearly. Each service and container generates a stream of telemetry that needs to be stored, analyzed, and secured. In self-hosted log management, this leads to fragmentation: data is scattered across tools, scaling requires data redistribution, and operations become a constant rebalancing act. Additionally, there are requirements for data storage and residency, where some logs cannot be moved outside the region. As a result, teams lose balance between full visibility and control over their data.
The solution is built as a hybrid model: BYOC Logs (Bring Your Own Cloud) retains storage in the user’s infrastructure while maintaining integration with the Datadog SaaS platform. This is a compromise between control and convenience. The key architectural idea is the separation of compute and storage. Logs are stored in object storage, while the compute layer scales independently. This approach eliminates the need to move data when adding nodes. The trade-off here is clear: reliance on object storage as a single layer of storage, but in return, predictable scaling and reduced operational costs.
The implementation relies on several key components. Indexing occurs by writing “splits” directly to object storage. Metadata is tracked through a centralized metastore, which makes data available almost immediately. The search layer is stateless: the coordinator receives a request, finds the necessary splits, and distributes the load among search nodes. Each node reads data directly from object storage, without local index storage. This simplifies operations and enhances fault tolerance. Additionally, Observability Pipelines are used for normalizing, enriching, and filtering logs before indexing, which reduces costs and improves data consistency.
A separate layer is data correlation. BYOC Logs is integrated with metrics and traces (APM), allowing incident analysis within a single interface. In a scenario with API degradation, logs, traces, and metrics can be viewed simultaneously without switching tools. An AI layer is added: the agent can form hypotheses and perform searches through natural language queries. Support for MCP (Model Context Protocol) is also claimed, enabling the connection of external AI agents for analyzing observability data without custom integrations.
In terms of results, the system addresses several bottlenecks. Scaling occurs without data redistribution. Storage becomes cheaper due to object storage and compression. Management is centralized through a single UI. However, precise performance metrics or savings in original data are not provided. One can only speak of reduced operational complexity and improved observability integrity.
It is also worth noting the security use case. Logs from firewalls, CDNs, and VPCs traditionally create high-volume loads. In a classic scheme, this is split between SIEM and archives, increasing complexity. BYOC Logs consolidates these streams, allowing for the storage of large volumes and applying enrichment (e.g., through GeoIP or threat intelligence) before indexing. This accelerates investigations due to structured data and event correlation.
In conclusion, BYOC Logs represents an evolutionary advancement in log management architecture. It shifts the heavy lifting of storage to a cheap and scalable layer, leaving analytics and correlation on the platform side. This approach is particularly relevant for hybrid and distributed systems, where control over data and a unified observability surface must coexist.