In data-driven companies, we often assume the biggest challenge is lacking data.
But at Spotify, the real problem was the opposite: they had too much data — yet no one could find it.
To solve this, Spotify built Lexikon — an internal data discovery and catalog platform designed to help data scientists quickly find, understand, and connect with the people behind the data.
Even after launch, data discovery remained one of the most complained-about pain points. Instead of treating it as a simple search problem, the Spotify team approached Data Discovery as a real product challenge.
This article breaks down Spotify's journey from a product manager's perspective (especially useful for Data PMs and AI PMs) — covering problem diagnosis, user intent frameworks, key design decisions, and lessons that remain highly relevant in the age of AI Agents and RAG.
In 2016, as Spotify migrated to Google Cloud Platform (GCP), the number of datasets in BigQuery exploded. At the same time, the company dramatically increased hiring of data scientists and analysts.
This led to classic problems:
In 2017, Spotify launched the first version of Lexikon. It gained solid adoption (~75% of data scientists used it regularly, ~550 monthly active users). However, follow-up surveys revealed that data discovery was still a major bottleneck.
Instead of declaring victory, the team dug deeper into user behavior.
Spotify applied its world-class music discovery expertise to data discovery, distinguishing between two modes:
Users don't know exactly what they're looking for — common for new hires or those starting new projects.
Solution: Complete redesign of Lexikon's homepage with personalized recommendations:
Interestingly, complex NLP or recommendation models weren't necessary. Simple heuristics based on usage statistics performed extremely well. The homepage recommendations now drive ~20% of table traffic.
💡 PM Insight: Before jumping to advanced AI, validate whether simple usage signals are already sufficient.
Users have a clear target — e.g., a specific table, schema field, project, or colleague's dataset.
Solutions:
Example: Searching "track_uri" leads to a dedicated schema field page showing all popular tables containing it. These entity pages are used by ~44% of monthly active users.
💡 PM Insight: Great search products don't force everything into one box — they understand intent and create multiple discovery paths.
Spotify initially followed a classic codification strategy (turning knowledge into documents). They soon realized that when people couldn't find data, they still asked colleagues.
Solutions:
💡 PM Insight: Don't fight human connection — design tools that enhance it and build a true Knowledge Network.
Many users still got stuck after finding a dataset. Spotify added critical "last mile" features:
💡PM Insight: The last mile often determines whether a product is truly adopted.
In the age of AI Agents and RAG (Retrieval-Augmented Generation), enterprise knowledge systems must evolve beyond passive retrieval.
Traditional RAG is mostly passive: Query → Retrieve → Generate. Agentic RAG introduces real agency — enabling agents to break down tasks, perform multi-turn retrieval, validate information, and connect with both data and people when needed.
Spotify's Lexikon offers a powerful blueprint for the future. Successful knowledge platforms will need to excel at three core capabilities:
When these three elements come together, knowledge systems transform from simple data catalogs into true decision infrastructure — helping both humans and AI make faster, more reliable decisions.
If this article helped you, I'd really appreciate your claps 👏 and follow — it encourages me to keep writing more PM deep dives.
What's your experience with data discovery at work? Drop a comment below — I'd love to hear your thoughts! 😊
This article reinterprets Spotify's 2020 Engineering Blog post through a product management lens: How We Improved Data Discovery for Data Scientists at Spotify.