Spotify described Random Access Parquet (RAP) as a way to perform low-latency point queries directly on the data lake. This is important where analytics already reside in Parquet, but online services and AI applications require individual records without copying them to operational databases.
The problem in this architecture is familiar to many teams that have grown their data lake to the level of a shared platform. Data lakes work well for scans, analytics, and ML pipelines, but start to degrade on point queries. The reason is not in a single layer, but in the chain as a whole: query planning, metadata traversal, and file searching add latency, even if the object storage itself already responds in milliseconds.
Spotify specifically points out the scale of this compromise. The company’s online data is stored in Bigtable, while exabytes reside in a Google Cloud Storage-based data lake. Duplicating all of this into separate serving databases becomes costly. Therefore, RAP chooses a different path: not to copy data, but to add an external index on top of Apache Parquet. This is a pragmatic choice with a clear trade-off. The platform gains fast access to keys but retains the complexity of managing indexes and layout optimizations.
The essence of the solution is that the external index links lookup keys, such as user IDs, to specific Parquet files and row positions. Instead of scanning thousands of files, the query first goes through the index and then performs a targeted ranged read in object storage. For immutable Parquet files, this is especially important: data is not rewritten, and the index is built as a separate layer. Spotify states that as new data is written to Apache Iceberg tables, the builder creates append-only index fragments without affecting the original files.
At the implementation level, RAP is complemented by storage optimizations. Data is sorted by lookup key to reduce the number of files that need to be touched. Related records are grouped together. Value columns are interleaved so that multiple attributes can be read in one contiguous read. Covering indexes are also used, which in some queries allow for not reading Parquet files at all. Here we see a typical engineering compromise: the size of files and the index structure slightly increase, but the system pays for this with fewer storage operations. For some queries, this boils down to a single ranged read of a few kilobytes.
Spotify also describes secondary indexes. They are necessary when access is not only by a single primary key. Such indexes are maintained at the serving layer and allow for the addition of new access paths without changing data pipelines. Hash-based indexes are suitable for exact lookups. Sorted indexes are suitable for range queries. Additionally, Z ordering and Hilbert curves are mentioned as ways to improve data locality for secondary dimensions. This does not negate the overall complexity but expands the set of scenarios where the same Parquet dataset can serve both analytical scans and interactive point queries.
In an engineering sense, RAP addresses not only the latency issue. It reduces the gap between analytical storage and operational access. This is particularly relevant for online services, notebooks, AI agents, and ML workloads, which increasingly want to read the same data but with different access models. Spotify effectively demonstrates how open data lake technologies can be brought to a more operational mode without breaking the underlying storage format and without building a second copy of the platform alongside.
At the same time, it is important not to overestimate the universality of the approach. RAP does not make the data lake a complete replacement for all serving databases. It improves point access through indexing and layout optimizations but requires discipline in data organization and index maintenance. This is not magic, but a careful engineering overlay on an already existing storage stack. This is why the approach appears convincing to architects: it does not negate limitations but shifts them to a more manageable part of the system.