UX, Data & Research

Probing a music recommender system.

Part 1: Serendipity and Recommender Systems

Full paper on serendipity and recommender systems.

In this short paper, I explore the interplay between recommender systems and serendipity in digital music discovery, particularly on platforms like Spotify and YouTube. While recommender systems are designed to suggest content based on user preferences and behavior, they sometimes create unexpected encounters with new music—raising the question of whether these experiences can be considered true serendipity.
The paper highlights the tension between personalization and discovery in digital music platforms. While recommendation systems enhance user experience by providing relevant content, they may also limit exposure to diverse music. Understanding how these systems balance predictability and surprise is key to improving digital serendipity in music discovery.

Key Points

Serendipity in Digital Discovery
- Traditionally associated with chance, serendipity can also be structured by digital systems.
- McCay-Peet and Toms define serendipity as a process involving a prepared mind, an act of noticing, chance, and a fortuitous outcome.
- In music discovery, encountering an unexpected but enjoyable song can create this type of serendipitous experience.
From iPod Shuffle to AI-Powered Recommendations
- Earlier methods of digital music discovery, such as the iPod Shuffle, relied purely on randomness.
- Randomness can create serendipity but may be constrained by the size and diversity of a user’s library.
- Modern recommendation systems attempt to balance structure and surprise, sometimes neutralizing serendipity in the process.
Challenges and Opportunities for Serendipitous Discovery
- Algorithms optimize for user retention, often reinforcing existing tastes rather than introducing unexpected content.
- Some recommendation strategies introduce randomness or contextual factors (mood, location, time of day) to foster discovery.
- Designing systems that promote serendipity without overwhelming users remains an open challenge.

Part 2: Spotify and Recommender Systems.

Full essay on Spotify and recommender systems.

Spotify's recommender system plays a crucial role in shaping user engagement, artist visibility, and overall listening behavior. The platform’s algorithmic curation, powered by collaborative filtering and deep learning models, personalizes playlists such as Discover Weekly and Release Radar.
The paper highlights how recommendation mechanisms influence users' interactions with new and familiar music while also affecting how artists gain exposure. One key challenge is the potential reinforcement of mainstream trends, where highly streamed artists benefit from more frequent recommendations, possibly at the expense of emerging or niche musicians.
It also discusses the balance between engagement optimization and user autonomy. While Spotify effectively retains users through personalized suggestions, its system can sometimes limit exploration beyond algorithm-driven recommendations. The distinction between editorial playlists (curated by humans) and algorithmic playlists also raises questions about visibility biases. Addressing these issues could involve transparency measures in Spotify’s recommendation process, as well as user-centric design features that encourage greater serendipity in music discovery.
Lastly, the study explored how Spotify’s recommender system differs from traditional music browsing methods (e.g., radio, physical media, and early online stores). Rather than focusing purely on genre-based recommendations, Spotify prioritizes mood and activity-based playlists, offering users highly personalized listening experiences. This shift allows for finer-grained curation, tailoring playlists to specific moments, moods, or behaviors rather than just musical style. While this level of personalization enhances engagement, it raises concerns about filter bubbles, where users may be confined to a narrow set of musical experiences.

Key Points:

Spotify’s recommender system relies on collaborative filtering and deep learning to generate personalized playlists.
Mainstream bias: High-stream artists receive more recommendations, potentially limiting exposure for smaller artists.
User autonomy vs. algorithmic control: While recommendations drive engagement, they may restrict organic exploration.
Editorial vs. algorithmic curation: Human-curated playlists differ from algorithmic ones in their impact on discovery.
Potential improvements: Transparency in recommendation logic and design strategies that encourage serendipitous discovery.
Spotify’s Recommender System:
(1) Moves beyond genre-based discovery to mood and activity-based curation.
(2) Uses listening history and audio features to fine-tune recommendations.
(3) Encourages engagement but may limit exploration outside algorithmic suggestions.

The paper also included a quantitative component and examined Spotify’s Top 50 Artists of 2020 playlist using the platform’s audio features (available via Spotify’s API) to assess song diversity, the role of genre classification, and the impact of recommender systems on music discovery. The analysis revealed that, while tracks varied in terms of musical attributes like Energy, Valence, Loudness, and Acousticness, they overwhelmingly belonged to just three genres—Pop, Hip-Hop, and Reggaeton. Notably, no instrumental tracks were present, highlighting a potential bias in popular music consumption on the platform.

Key Findings

Musical Diversity in Top Tracks
- Songs varied in Energy, Valence, and Tempo, but all belonged to a limited set of genres.
- No instrumental tracks were present, suggesting a bias in popular streaming habits.
- Uniform Time Signature: All songs in the dataset had a 4/4 time signature.
- Track Duration: The average song duration was 3 minutes and 30 seconds, reflecting historical music industry trends rooted in vinyl record limitations.
Spotify’s Audio Features Issues

Tempo Detection Issues: The average tempo of tracks was 118 BPM. However, Spotify’s tempo detection algorithm misclassified some songs, assigning double their actual BPM due to rhythmic subdivisions.
Low Acousticness Scores: The dataset had an average acousticness score of 0.218, indicating a predominance of electronic and synthesized sounds over acoustic instruments. Some high-ranking tracks with electric instrumentation received high acousticness scores, reaising questions about the accuracy of Spotify’s detection algorithm.
Liveness Anomalies: The average liveness score was 0.15, indicating that most tracks were studio recordings. However, Roddy Ricch’s The Box was an outlier with a liveness score of 0.79, despite not being a live recording.

Future Research & Open Questions
How do external industry factors (e.g., music labels, promotions) influence which artists are most streamed?
Do recommender systems truly broaden musical horizons or reinforce mainstream dominance?
How have Spotify’s audio features evolved over time, and can they be used for historical analyses of musical trends?