A Cure for Hacker News Overload

Hacker News is the unofficial home page for the tech startup/entrepreneur community. It’s what most of us check first thing in the morning before the NYT, the WSJ, TechCrunch or any other news source. Like many other popular news feeds, however, the sheer volume of stories makes it difficult to keep up.

One strategy for consuming Hacker News is to periodically check the top stories on the home page since that’s where most of the high-scoring articles reside. This works okay if you’re always in front of the computer, but if you’re away, you miss out. It’s also terribly inefficient to repeatedly check HN, because you have to rescan stories you’ve already seen.

A different approach is to get Hacker News via RSS. This is nice because it frees you from having to check the HN site all the time, and you can catch up on articles whenever you have time. But there’s a fatal problem with the default Hacker News RSS feed: it contains way too many low-scoring stories-hundreds each day, it seems like. This renders it a huge time waster.

Score thresholds

Here’s a robust cure for Hacker News overload: use score thresholds. This solves the problem of too many stories by cutting out low-scoring noise while still leaving behind those links interesting enough to have been upvoted by the HN community.

I’ve created a handful of Twitter and RSS feeds that apply score thresholds to Hacker News. Stories enter the feeds in near real-time as soon as they reach X points on the main HN site.

Hacker News stories reaching 20 points  @newsyc20  RSS feed
Hacker News stories reaching 50 points  @newsyc50  RSS feed
Hacker News stories reaching 100 points  @newsyc100  RSS feed
Hacker News stories reaching 150 points  @newsyc150  RSS feed
Hacker News stories reaching 300 points  @newsyc300  RSS feed
Hacker News stories reaching 500 points  @newsyc500  RSS feed

Here’s an example tweet from @newsyc20:

Advantages over existing feeds

How does @newsyc20, for example, differ from established Twitter bots like @newsycombinator and @hackernewsbot?

@newsyc20 includes links to the story itself and the Hacker News comment page for that story. Other feeds only link to the story. Having a link to the comments page is crucial because reading the HN discussion thread about an article is sometimes better than the article itself.

@newsyc20 preserves the actual story URL if it fits inside a tweet. This makes it easier to decide if a story is click-worthy. Compare with @newsycombinator, which applies bit.ly shortening to all URLs regardless of length.

@newsyc20 has a well-defined selection criteria: 20 points or above. It’s unclear how the other bots do their filtering.

@newsyc20 attributes stories to the original submitter on Hacker News if that user has a Twitter account listed in their HN profile. (Just for fun.)

The newsycNN feeds have different score thresholds to choose from, so if you feel like @newsyc20 has too much volume, you can switch to @newsyc50, @newsyc100 or @newsyc150 — or even @newsyc300 or @newsyc500.

Implementation

Hacker note: these feeds are generated by a Python script run every 5 minutes with cron. The program uses BeautifulSoup to scrape the Hacker News front page, Tweepy to send out out tweets, and sqlite3 to keep track of which stories have already been tweeted.

Feedback

I welcome any feedback or suggestions for ways to improve the Hacker News feeds. Please leave them in the comments below, email me or reach out to me on Twitter.