Spotify's RAP: A New Way to Cache Point Queries on the Data Lake

Why Point Queries on Data Lakes Are Slow

Data lakes have become the central repository for analytics and AI workloads. But if you want to fetch a single record by key—say, a user's profile or a transaction—the experience can be painfully slow. Distributed query engines like Trino and BigQuery are built for analytical scans, not key-based lookups. They'll happily chew through millions of rows to compute an aggregate, but ask them to find one row and they'll still scan a lot of metadata and plan a heavy query.

That overhead adds up. Even with cloud object storage offering millisecond-level access to individual bytes, the planning, metadata traversal, and file discovery can take seconds. Spotify, for instance, stores petabytes of online data in Bigtable and exabytes in a data lake on Google Cloud Storage. Copying subsets of that lake into a separate operational database for every new use case is expensive and operationally messy.

Introducing Random Access Parquet (RAP)

Spotify's answer is Random Access Parquet, or RAP for short. RAP adds an external index layer on top of Apache Parquet files. That index maps query keys—like user IDs—directly to the Parquet file and the row position within it. Instead of scanning thousands of files, a query first resolves the key through the index, then issues a targeted range read on the object store.

The index is built incrementally as new data lands in Apache Iceberg tables, and it doesn't touch the immutable Parquet files. That's a key design choice: the same dataset can now serve analytical processing, machine learning pipelines, notebooks, AI agents, and latency-sensitive online apps without maintaining duplicate storage systems.

Caching vs. Indexing: What RAP Actually Does

You might wonder if this is just caching by another name. Not quite. Caching usually stores copies of hot data in a faster layer, like Redis or an in-memory cache. RAP doesn't store the data itself—it stores pointers to where the data lives in the object store. The index is more like a lookup table that avoids the expensive part of a point query: figuring out which files to read.

But there's a caching angle too. The index acts as a form of metadata cache, pre-computing the mapping from keys to file locations. That's similar to how a cache reduces repeated computation. And because the index is external, it can be scaled independently of the data, which is a load-balancing win. You can add more index replicas to handle more query traffic without moving data around.

Storage Layout Optimizations That Reduce Reads

Spotify also introduced several storage layout tricks to cut down on the number of storage operations. For example, they sort data by the query key to reduce the number of files that need to be accessed. They group related records together so a single read can fetch several at once. They interleave value columns so that multiple attributes can be retrieved in one continuous read. And they support covering indexes, which allow some queries to be answered without touching the Parquet files at all.

These techniques trade a modest increase in file or index size for a big reduction in storage operations. In some cases, a point query can be satisfied with a single range read of just a few kilobytes. That's a huge improvement over scanning entire partitions or even multiple files.

Secondary Indexes Without Rewriting Data

RAP also supports secondary indexes, which let you query across different dimensions—say, buyer ID or seller ID—without rewriting the Parquet files. Hash-based indexes handle exact-match queries, while sorted indexes support range queries. The secondary indexes are managed at the service layer, so you can add new access paths without changing the data pipeline. The same Parquet dataset continues to serve analytical scans and interactive point queries.

To further improve data locality for secondary query dimensions, Spotify mentions techniques like Z-ordering and Hilbert curves. These are well-known in the geospatial world but can be applied to any multi-dimensional key. They help ensure that records that are often queried together are stored close together, which reduces the number of reads needed.

The Bigger Picture: Open Data Lakes Beyond Analytics

Spotify's move is part of a broader industry push to make data lakes work for more than just batch analytics. Google Cloud recently described a lakehouse architecture based on Apache Iceberg that aims to support AI applications while reducing data duplication. RAP takes a different approach: it adds a dedicated external index layer optimized for point queries, while staying compatible with existing Parquet files and Iceberg tables.

Data engineers have taken notice. Andrew Lamb sees RAP as an example of extending open data formats to support interactive workloads. In a LinkedIn discussion, Vikas Singh pointed out that as cloud object storage gets faster, the bottleneck for point queries shifts to query planning and metadata access—and RAP reduces that overhead by precomputing indexes.

What This Means for Caching and Load Balancing

If you're building systems that need both heavy analytics and fast online lookups, RAP offers a promising pattern. Instead of maintaining a separate operational database that duplicates data, you can add an index layer on top of your data lake. That index acts as a lightweight cache for the metadata needed to locate records, and it can be scaled horizontally to distribute query load.

The trade-offs are worth noting. The index itself needs storage and maintenance. And for workloads with very high QPS and ultra-low latency, you might still need a dedicated cache like Redis. But for many use cases, RAP could eliminate the need for a separate datastore, simplifying your architecture and reducing costs.

Spotify's approach also highlights the importance of separating compute from data. The index is a separate service, so you can scale it independently of both the data lake and the query engine. That's a classic load-balancing technique: distribute the work across multiple index nodes, and you can handle more concurrent point queries without overwhelming the object store.

Of course, RAP is still early. Spotify hasn't open-sourced it yet, and there are open questions about how it performs at the extreme scale of exabytes. But the ideas are solid: precompute indexes to avoid metadata overhead, optimize storage layout for point access, and let the same dataset serve both analytical and operational workloads. That's a future where data lakes truly become the single source of truth for everything.