Data Engineering on ThecoreGrid focuses on building scalable, reliable, and efficient data platforms for modern highload systems.
We cover architecture and operation of data pipelines, batch and stream processing, data modeling, and storage systems designed for performance and consistency. Topics include distributed processing frameworks, real-time ingestion, ETL/ELT patterns, schema evolution, and data quality management in production environments. We analyze trade-offs between latency, throughput, and cost, as well as failure handling, observability, and governance in large-scale data systems. Content is based on real-world BigTech practices, including incident post-mortems, platform design decisions, and lessons from operating data infrastructure at scale. Instead of introductory tutorials, we provide deep technical insights into building and maintaining data platforms that support critical business workloads. The tag is aimed at data engineers, platform teams, backend engineers, and architects responsible for robust and scalable data ecosystems.
Transitioning from SSH to REST-based job submission changes the behavior of the data pipeline at the architectural level. This is about manageability, fault tolerance, and resource control. The problem does not manifest immediately — until the system hits a scale limit. In this case, over 700 jobs were executed via SSH to EMR clusters. This … Read more
API design and data architecture: how to avoid system degradation, choose the right approach, and maintain consistency during scaling
Hive federation in a data warehouse: how to move from a monolith to a distributed architecture without downtime or loss of data consistency.
Topology-preserving compression without sacrificing speed: how EXaCTz achieves GB/s throughput while preserving the contour tree and extremum graph.
LLM evaluation at scale on Apache Spark: how the distributed architecture, caching, and statistical validation of models are structured.
How an ML pipeline based on Amazon SageMaker accelerates training and reduces labeling costs in edge robots and distributed systems
How LLM agents automate building-grid co-simulation through DAG and multi-agent orchestration, reducing errors and complexity in pipelines.
How Knowledge Graph and LangExtract enhance data extraction accuracy and traceability in Total Airport Management systems –>
Tansu proposes rebuilding the Kafka model: removing state from the brokers and delegating reliability to external storage. This changes the system’s behavior under load and simplifies the operational model. The problem manifests at the operational level. A classic Kafka broker is a stateful component: replication, leader elections, persistent state, long uptime. Such nodes are hard … Read more
Controls: ← → to move, ↑ to rotate, ↓ to drop.
Mobile: use buttons below.