Automatic Clickbait Detection: The Features That Didn’t Work
I think we all know what clickbait is, but to define it for this project, clickbait is something designed to attract clicks by holding back any meaningful information. For my dissertation project, I built and compared machine learning models that try to detect it automatically, then tested whether showing readers a warning label changes which articles they choose to open. The models were trained on the Webis Clickbait Corpus 2017, about 38,500 tweets from US news publishers, each judged as clickbait or not by human annotators. The user study put the best model’s scores in front of 19 university students. The plan I submitted at the start and the thing I eventually built turned out to be two different projects.
TL;DR: I built models that try to spot clickbait, found that the usual tells (question marks, capitals, exclamation marks) do very little on their own, and tested warning labels on 19 students. The labels didn’t clearly change what they chose to open.
Initially, I wanted to scrape headlines from various major news outlets, assemble my own collection of data, and run it through different Recurrent Neural Network (RNN) architectures. I wanted to measure click-through rate, time spent reading, and an upvote or downvote at the end of every article. Almost none of that survived.
The custom corpus went first, for two reasons: time, and label credibility. Building it meant sitting down and personally scoring every headline on a 1–5 clickbait scale, and there is no reason anyone should take my labels seriously. Webis 2017 contains 38,517 posts from 27 US publishers, each judged by five annotators on Amazon Mechanical Turk (a crowd-work platform), and years of published work benchmarked against it. Using it meant my numbers could be placed next to somebody else’s. That was worth more than owning the data.
The baseline that should worry you
Webis 2017 leans about three to one toward non-clickbait: 14,777 against 4,761 in the training split.
Before running anything real, I trained a model on nothing at all. A constant vector (a model that always answers non-clickbait) with no features, text or signal. It scored 0.76 accuracy.
Everything after that had to be read against that number. A model can be right about three quarters of the time on this dataset while having learned nothing, because guessing “non-clickbait” is a reasonable strategy when three quarters of the corpus is non-clickbait. Macro-F1 (a score that treats both classes equally, so ignoring the rarer one is penalised) on that empty baseline was 0.43, and that is the figure that actually separates a working model from a lucky one. Every result below is macro-averaged for this reason.
The features I assumed mattered
I engineered eight features from each headline: character count, word count, whether it opened with who/what/when/why/where/how, adjective count, exclamation marks, question marks, slang terms, and uppercase ratio. Then I trained models on restricted subsets to see which linguistic dimension was carrying the weight.
Question-based features scored 0.18 macro-F1. Below the empty baseline, with recall on the clickbait class close to zero. The rhetorical question is one of the most commonly assumed markers of clickbait, and on 38,000 real posts it was worse than useless on its own. They appear too rarely to classify anything.
Style features, the exclamation marks and capitals and slang, reached 0.47. Barely above a model trained on nothing.
The best shallow feature was headline length. Count-based features alone hit 0.60. How many characters the thing is.
Dropping one of two text fields
Webis gives you two text fields: postText, the tweet as it was posted, and targetTitle, the original article headline. I started with both. Combining them was the strongest feature-engineered configuration I had at 0.62, before dropping postText entirely.
The reasoning was about the reader rather than the model. postText varies with platform constraints and whoever ran the social account that day. targetTitle is the thing a person actually sees before deciding to click. A detector trained on the field the reader never encounters first is solving a slightly different problem than the one I claimed to be solving. My study interface shows article headlines rather than tweets, so the title field was the closer match to what participants would see. Giving up two points of macro-F1 to model the real decision seemed like the right trade, and it made the later A/B study (two groups, each seeing a different version) logical.
TF-IDF vectors (a way of turning text into numbers by weighting distinctive words) from that one field lifted Logistic Regression to 0.72, which none of my eight hand-built features did. Rebalancing the training set through undersampling, repeated across five random seeds so I could report a standard deviation rather than a single hopeful run, held the best classical models around 0.73–0.74. A 1D CNN on tokenised titles reached 0.76 on test.
The danger of a good validation score
Fine-tuning RoBERTa-base (a pretrained language model) on titles alone produced 0.796 validation macro-F1 and 0.689 on test. That gap is the most honest thing in the results.
I then added the handcrafted features back, one group at a time, to see whether any of them added something the transformer had missed. Count features helped by boosting test macro-F1 to 0.706 whereas style features did not. And the variant with everything stacked on top scored the highest validation figure in the entire study, 0.807, then fell to 0.680 on test. The best looking model I trained turned out to be one of the weakest on test.
The confusion matrices explain what the count variant actually bought. Title-only RoBERTa caught 1,686 of 4,515 clickbait headlines in the test set while falsely flagging 1,324. Adding length features caught 3,331 and falsely flagged 3,495. It roughly doubled detection and increased false alarms more than two and a half times. Over-flagging risks teaching readers to ignore the badge, and under-flagging risks making it decorative.
Putting it in front of people
I wanted to make an interface to test the model. Three rounds of ten headlines each (thirty in all, drawn evenly from clickbait and non-clickbait), with five picks per round.
The control group saw plain text. The experimental group saw the same headlines with a coloured badge carrying a band and a percentage: Unlikely (up to 0.30), Likely (up to 0.60), Highly Likely (up to 0.75) and Clickbait (above 0.75). Across the thirty headlines the bands fell 18 / 4 / 4 / 4, which meant most badges were reassuring and only a few were warnings. Selections were logged with timestamps and group assignment.
I had 19 students complete it. Ten in control and nine in experimental. University guidelines restricted recruitment to students at the university. The study collected no personal data and participants were identified only by an anonymous ID.
The control group’s median was 8 clickbait picks out of 15, with several participants hitting 10. The experimental group’s median dropped to 7 and the spread widened. Two people in the experimental group picked 11 and 12.
Those two saw the warnings and clicked the flagged headlines anyway, more than anyone in the control group did. With 19 participants I cannot tell you whether that is reactance (pushing back against being told what to do), curiosity, distrust of the model or two individuals having a normal day. It is the finding I would most want to chase with a real sample, because a labelling system that advertises the most manipulative headlines to a subset of readers is doing the opposite of its job.
The technical result is that RoBERTa with headline length is the best transformer configuration I produced, around 0.71 macro-F1, well short of the high-90s figures that appear in papers using small single-domain corpora. The more useful result is that most of what people believe identifies clickbait, the punctuation and the capitals and the rhetorical questions, does almost no classification work at scale, and that showing someone a probability score is not the same as changing their mind.
What I would do differently
If I were to revisit my project, the evaluation would have to come from one script. Every number in this write-up should be emitted by the code into a single results file, and the write-up should read from that file rather than from numbers copied across at different points in the project. I would tune the decision threshold rather than accepting 0.5. The more useful question is where to sit on the precision-recall curve. I would also test on an out-of-domain set, since models fine-tuned on Webis may not hold up outside it. On the study side, I would run label vs. label instead of labels vs. nothing. Comparing a badged interface to a bare one only tells you that badges do something. The two people who clicked more when warned suggest the more useful experiment is whether a percentage, a band name, or a plain icon produces different behaviour.