The decision was the metric, not the model

Predicting mature ratings for streaming titles from metadata alone, and why I shipped the least accurate of three models.

Charlene Athena · Data Science capstone, Universitas Ciputra, June 2026 · source code

01The problem

A streaming catalogue needs every title labelled mature or not, because that label drives parental controls. Labelling by hand is slow and inconsistent. The question: can we predict the label from the metadata a title already has when it is added (type, release year, duration, genre, country), before anyone has reviewed it?

Before touching a model, I asked which mistake is worse. A family film wrongly flagged as mature costs a human a minute of review. A mature title wrongly marked safe can end up in a child’s feed. Those errors are not equal, so accuracy was the wrong thing to optimise. The number that matters is recall on mature titles: of all the mature titles, how many do we catch?

02The data

The raw data was messy in ways that matter:

• Values in the wrong column. Three rows had a duration stored in the rating field. I moved them back rather than dropping them silently.
• Unusable labels. Titles rated NR or UR (not rated) can’t be used as ground truth, so they were removed.
• Multi-value fields. country and listed_in (genres) held comma-separated lists. I reduced each to a primary value, grouped categories with fewer than 10 titles into “Other”, and parsed durations into numbers (minutes for films, seasons for shows).
• Missing values. Director was missing for 29.9% of titles, country and cast for 9.4%. Numeric gaps got the median; categorical gaps got “Unknown”.

03Three models, one trade-off

I trained Logistic Regression (with balanced class weights), k-nearest neighbours (k = 15) and Gaussian Naive Bayes, and evaluated them on 1,744 held-out titles, 802 of them mature. Pick a model to see what it does to those 802 titles.

k-NN wins on accuracy (70.4%) and ROC-AUC (0.789). If I had optimised the default metric, I would have shipped it. But it lets about 263 mature titles through. Naive Bayes is the least accurate model, yet it misses only 116. It pays for that with more false alarms: 509 safe titles sent to a human for a quick check.

I chose Naive Bayes (best F1, 0.687; recall 0.855) and wrote the trade-off down explicitly. In practice, this model is a triage tool that decides which titles a human reviews first, not an automatic labeller. Framed that way, false alarms cost review time, and missed titles cost trust, which is the right way round.

04Shipping it

I turned the notebook into a Streamlit app with three views: Prediction (enter a title’s metadata and get a verdict with a probability), Training data (the distributions behind the model) and Model performance (the comparison above). The app reuses the notebook’s exact preprocessing, so the deployed predictions match the model I evaluated. Users can switch models and see the trade-off themselves.

05What it can’t do, and what I’d try next

• The features are weak. Every numeric feature correlates with the label at |r| ≤ 0.13. Metadata only goes so far; the ceiling here is the data, not the algorithm.
• I used the default decision threshold. The next step would be to tune the threshold to a recall target the business agrees on (for example “catch 95% of mature titles”) and report the review workload that costs.
• Text would help. Title descriptions almost certainly carry more signal than genre and country. That would be the first new feature source I would test.

The lesson I took: the most consequential decision in the project happened before any model was trained, when I asked which mistake mattered more.