Building an AI Music Recommendation System: An Idea I Decided Not to Pursue

Introduction#

I wanted to build an AI-powered music recommendation system that could recommend songs based on individual listening preferences.

The idea was straightforward: train a machine learning model using historical music data, learn patterns in user preferences, and use those patterns to recommend songs that users might enjoy.

However, I had a more ambitious requirement.

I didn't want to build a system that simply recommended songs it had already seen during training. I wanted it to discover newly released songs and recommend them to users based on their individual preferences.

The system needed to work for multiple users, scale beyond a small demonstration, and potentially become a publicly available application.

Most importantly, I wanted to build something I could confidently publish on GitHub, showcase on my résumé, and potentially develop into a research project.

Before writing any code, I decided to investigate the data requirements and licensing constraints.

That investigation changed my decision.

1. The Problem I Wanted to Solve#

Traditional recommendation systems can struggle when they encounter new items.

For example, a music recommendation model trained on historical listening data might learn that a particular user enjoys energetic Tamil film music.

When a new song is released, the model should ideally be able to evaluate that song and determine whether it might appeal to the user.

This introduces a problem commonly known as the cold-start problem.

In particular, a newly released song has no historical interactions. A collaborative-filtering model cannot directly learn from interactions that do not yet exist.

I wanted to explore whether combining musical characteristics with listening behaviour could address this limitation.

The proposed system would aim to:

• Learn from historical user listening behaviour.
• Understand numerical representations of songs.
• Generate personalised recommendations for different users.
• Recommend songs released after the model's training period.
• Incorporate feedback to improve future recommendations.

The challenge was not simply recommending similar songs. It was learning individual preferences while remaining capable of recommending previously unseen music.

2. The Proposed Approach: Hybrid Recommendation#

I planned to investigate a hybrid recommendation architecture combining content-based filtering and collaborative filtering.

2.1 Content-based filtering#

Content-based filtering recommends items based on their characteristics.

For music, these characteristics could include:

• Tempo.
• Energy.
• Danceability.
• Acousticness.
• Genre.
• Artist information.
• Audio-derived embeddings.

A song could be represented as a numerical feature vector:

where each $f_j$ represents a feature of song $i$.

For example, a song might have numerical values representing its tempo, energy, and acousticness.

The model could compare these characteristics with the preferences learned from a user's listening history.

One advantage of this approach is that a previously unseen song can potentially be recommended if its features are available.

However, musical similarity alone does not necessarily predict personal enjoyment.

Two songs can have similar acoustic characteristics while appealing to different audiences.

2.2 Collaborative filtering#

Collaborative filtering learns from interactions between users and items.

A simplified dataset might contain:

The interactions could be used to learn patterns in listening behaviour.

Possible approaches include:

• Matrix factorisation.
• Implicit-feedback recommendation.
• Neural collaborative filtering.
• Two-tower recommendation models.

Collaborative filtering can identify relationships that are difficult to capture using explicit musical features alone.

For example, users with similar listening histories might enjoy the same unfamiliar artist.

However, a conventional collaborative-filtering model can struggle with newly released songs because those songs have no interaction history.

2.3 Combining both approaches#

The proposed hybrid system would combine information about songs with information about user preferences.

Conceptually, the architecture would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
                  HISTORICAL DATA
                        |
             +----------+----------+
             |                     |
             v                     v
       Song Features        User Interactions
             |                     |
             v                     v
     Content-Based Model    Collaborative Model
             |                     |
             +----------+----------+
                        |
                        v
                  Hybrid Ranker
                        |
                        v
              Personalised Recommendations

The objective would be to use the strengths of both approaches.

Content-based features could help the system recommend songs without historical interactions, while collaborative filtering could capture patterns in user behaviour.

The hybrid approach would be a hypothesis to evaluate, not an assumption that combining models automatically improves performance.

3. The Most Important Requirement: Training Data#

The more I investigated the project, the clearer it became that data availability was the central challenge.

The model required two complementary datasets.

3.1 Song metadata and musical features#

The first dataset would describe the songs themselves.

Potential fields included:

• Song identifiers.
• Artist identifiers.
• Genre labels.
• Release dates.
• Numerical audio features.
• Audio-derived representations.

This information would allow the content-based component to evaluate songs that had not appeared in the original training dataset.

However, obtaining a song's title, artist, and release date would not automatically provide the numerical audio features needed by the model.

We would need a suitable, legally usable source for those features.

3.2 User listening behaviour#

The second dataset would contain user-song interactions.

Potential signals included:

• Listening events.
• Play counts.
• Likes and dislikes.
• Repeated listening.
• Skips, where available.

These interactions would allow the model to learn from the behaviour of multiple users.

A large dataset of song metadata without user interactions would not be sufficient to train a conventional collaborative-filtering model.

Likewise, listening histories without useful song features would make it more difficult to recommend genuinely new songs.

The two datasets would need to be compatible, with a reliable way to associate their song identifiers.

3.3 Why publicly available data was not enough#

Finding a dataset online does not automatically establish that it can be used for every purpose.

Before using a dataset, I needed to establish:

1. Whether the data could legally be collected and stored.
2. Whether its licence permitted machine learning training.
3. Whether the trained model could be published or distributed.
4. Whether the data could be used in a public demonstration.
5. Whether the intended use complied with the relevant platform terms.

These questions mattered because I wanted the project to be suitable for public release, rather than merely producing a local experiment.

I did not want to build the entire system around a data source that might prevent me from publishing or demonstrating the final result.

4. Investigating Potential Data Sources#

I considered several possible sources.

4.1 Spotify#

Spotify was an obvious candidate because the original idea involved music recommendations.

However, Spotify's developer policies restrict using Spotify Platform or Spotify Content to train machine learning or AI models.

Official documentation:

https://developer.spotify.com/documentation/web-api/reference/search

This made Spotify unsuitable as the assumed training-data source for the proposed model.

Spotify could still be considered for certain integration purposes if permitted by its current policies, but that would need to be evaluated separately.

4.2 YouTube#

YouTube was another possibility because it provides access to music videos and information about published content.

Its API could potentially help identify newly published videos and retrieve metadata.

However, access to video metadata does not automatically grant permission to download audio, extract musical features, or train an AI model using that content.

YouTube's developer policies and applicable content terms would need to be examined before building a training pipeline around it.

Official documentation:

https://developers.google.com/youtube/terms/developer-policies

I therefore decided not to assume that YouTube would solve the data problem.

4.3 Last.fm#

Last.fm appeared promising because its API provides music-related information and listening-history functionality.

It could potentially help us investigate user-song interactions.

However, the API's terms impose conditions on data use, and research or academic projects may require additional clarification.

I identified several questions that needed answers:

• Could listening-history data obtained through the API be used to train a recommendation model?
• What storage restrictions would apply?
• Could the trained model be published?
• Could the model be demonstrated through a public application?
• Would commercialisation require a separate agreement?

Official documentation:

https://www.last.fm/api/tos

I considered contacting Last.fm to request clarification before using its API.

However, until the permitted use was clear, I could not confidently treat it as the foundation of the project.

4.4 Free Music Archive#

I also considered the Free Music Archive research dataset.

It provides music-related metadata and genre information that could be useful for content-based recommendation experiments.

Official repository:

https://github.com/mdeff/fma

However, it does not by itself solve the requirement for large-scale user listening histories.

The licences of individual audio tracks also need to be checked separately from the licence governing the dataset's metadata.

It could be useful for a different project or a carefully scoped experiment, but it did not immediately provide everything required for the complete system.

5. The Proposed New-Release Pipeline#

An additional challenge was ensuring that the system could recommend songs released after training.

I did not want the model to depend exclusively on a fixed catalogue of songs.

The proposed pipeline would have three distinct components.

Component 1: New-release discovery#

A discovery service would identify newly released songs through an authorised source.

It would collect the available metadata and identify each candidate song.

Component 2: Feature processing#

The system would obtain the permitted musical features for each new song.

Those features would be transformed into the representation expected by the trained model.

Component 3: Recommendation and ranking#

The recommendation engine would evaluate the new songs for individual users.

It would produce a ranked list based on predicted relevance.

The pipeline would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
New Releases
     |
     v
Discover Song Metadata
     |
     v
Obtain Available Musical Features
     |
     v
Generate Song Representations
     |
     v
Score Songs for Each User
     |
     v
Rank Recommendations
     |
     v
Personalised Song Lists

The model would not necessarily need to be retrained whenever a new song appeared.

However, it would still need access to the new song's features, and its ability to generalise to unseen songs would need to be tested.

This architecture could address the new-song cold-start problem, but it would not eliminate every cold-start issue.

For example, a new user with no listening history would still require an onboarding process, contextual information, or another way to estimate their preferences.

6. How I Would Have Evaluated the Model#

If I had proceeded, I would have started with simple baselines before attempting a more complicated neural network.

The evaluation would have compared:

1. Content-based recommendations.
2. Collaborative-filtering recommendations.
3. A hybrid recommendation model.

I would have used a chronological train-test split where appropriate, ensuring that the evaluation did not inadvertently use future interactions to predict past preferences.

Potential evaluation metrics included:

• Precision@K.
• Recall@K.
• NDCG@K.
• Recommendation coverage.
• Performance on newly released songs.
• Performance on songs with no training interactions.

The goal would have been to establish whether the hybrid model actually improved recommendation quality.

A model that simply recommends popular songs or repeatedly suggests familiar artists would not necessarily provide meaningful personalisation.

I also wanted to evaluate whether the system could discover relevant music outside a user's established listening habits.

7. Why I Decided Not to Proceed#

Ultimately, I decided to pause the project before implementation.

The main reason was not that music recommendation was an uninteresting machine learning problem.

Quite the opposite: it presented an interesting combination of recommendation systems, feature engineering, collaborative learning, and cold-start challenges.

The problem was that I could not yet establish a suitable data foundation that met all the requirements of the intended project.

I wanted a dataset that could support both musical feature analysis and learning from user listening behaviour.

I also wanted to be confident that I could use the data for model training, publish the implementation, and demonstrate the resulting system without violating the applicable licences or platform policies.

The sources I investigated did not immediately provide a complete solution.

I could have proceeded with an isolated content-based prototype, but that would have changed the scope of the project.

Instead of building the personalised, new-release recommendation system I originally envisioned, I would have been building a narrower demonstration.

That might still have been useful, but it was not the project I wanted to commit to at that point.

I decided that it was better to pause than to build an ambitious system on top of an uncertain data foundation.

8. Lessons Learned#

This exploration reinforced several important lessons about applied machine learning.

Data availability is part of system design#

Choosing a model before understanding the data can lead to an architecture that cannot be implemented as intended.

Data collection, licensing, storage, and access restrictions should be considered during project selection, not after implementation.

A public dataset is not automatically unrestricted#

A dataset may be publicly accessible while imposing restrictions on redistribution, commercial use, or particular forms of processing.

The permissions needed depend on the dataset, its licence, the intended application, and any applicable platform terms.

New-item recommendation requires deliberate design#

A model trained exclusively on user-item interactions can struggle with items that have no interaction history.

Musical features or other item representations can help, but their availability and permitted use must be established.

A good project needs a realistic evaluation plan#

It is not enough to build a model that produces a list of recommendations.

A credible project should compare against baselines and evaluate whether its recommendations are relevant, diverse, and useful for the intended users.

Research feasibility matters#

An ambitious idea is only a good project choice when there is a practical path to implementing and evaluating it.

Understanding the constraints early can save substantial development time.

Conclusion#

The original idea was to build a hybrid AI music recommendation system capable of learning from user listening behaviour and recommending newly released songs.

The proposed architecture was technically plausible.

However, the data requirements and uncertainties around permitted data use made it difficult to establish a reliable path to the intended implementation.

Rather than proceed without resolving those issues, I decided to pause the project and explore alternatives with a clearer data foundation.

The broader lesson is simple:

A successful AI project begins not only with an interesting model, but also with data that can be obtained, used, and evaluated responsibly.