Back to projects

Emotion & Genre Classifier

2024

PythonMLHuggingFace TransformersMusic2EmotionGenius API

2-person ML project · music mood & genre from audio + lyrics

Overview

A teammate and I built an ML pipeline that takes a single audio file and spits out the song’s mood, dominant emotion, and genre. We wanted to use both the sound and the lyrics—so the system fetches lyrics (via Genius), runs emotion models on the audio and the text, and does zero-shot genre prediction. One function, classify_song(), ties it all together: you drop in a file and get back mood tags, valence–arousal scores, and a genre label.

Figuring out how to wire the pieces together was the interesting part. We had to decide where each model lived in the flow, how to normalize the outputs, and how to handle missing or messy lyrics. I focused a lot on making the pipeline clear and easy to extend so we could swap or add models without breaking everything.

It was a small team and we both had a hand in the architecture and the code. Getting the first end-to-end run to work—from raw audio to a coherent set of labels—felt like we’d actually built something you could hand to someone and say, “here, try it.”

Contributions

Key Features

  1. 1.
    Multimodal Pipeline: Audio emotion recognition, lyric emotion classification, zero-shot genre prediction
  2. 2.
    Unified classify_song(): Single entry point: mood tags, valence-arousal, dominant emotion, genre from one audio file
  3. 3.
    Lyric Integration: Automated lyric retrieval and emotion classification via Genius API

Design Decisions & Reflections

01 / Information Density

It was tough to choose what to measure. We had too many candidate signals, and a large song set plus a wide genre list meant every weak metric multiplied false positives and false negatives. A lot of time went into experimenting, digging through results, and cutting outliers so the pipeline would not get sidetracked too much. Attention and evaluation budget were finite.

02 / Choice of Metrics

Most mood systems lean on valence–arousal (and denser jargon I had to learn on the job). We wanted another angle: lyrics, with sentiment analysis helping the deduction. Seeing how words overlapped across genres was fascinating — and also the trap. Plain sentiment flattened a lot of nuance; a more complex and richer lyric model would have been more beneficial. Still, trying the lyric path taught us what the audio-only baseline was missing.

03 / What I'd Do Differently

I’d start with a narrower focus. For example, balancing R&B, pop, and funk was harder than it looked. They had similar emotional color, blurry lyric boundaries, and no clean separators. We burned cycles just simply tunnel visioning on how to force them apart before accepting that one track can sit in multiple genres. A tighter scope earlier would have left more room for depth instead of boundary-policing.

Implementation