Every now and then someone makes the argument that all really good games eventually do well, otherwise “if there are really good games that don’t sell, how come I never come across one?”
That argument is always refuted by the statement that of course you don’t come across games that are obscure, by definition. Which is itself then easy to refute by saying yeah, but surely you’d still come across one occasionally if you went looking.
Let’s go looking.
—
First, let’s take all the games on Steam.[1]
Then let’s sort them by Evan Miller’s recommended weighted method from here. That is, a lower bound of Wilson score confidence interval for a Bernoulli parameter. Basically, look at the score, but also consider how confident you can be in that score – i.e. how many reviews it has in total.
Now we have quite a nice list of the best games on Steam:
[Click for a top 250 with working links. Also note that you’ll probably see the occasional DLC in the full list that I didn’t manage to filter out.]
—
Next we want to fish out any “hidden gems”: Games that people really like but that aren’t very popular. Here’s a basic formula I made up:
- Take the weighted score above.
- Leave games with less than 150 reviews[2] per year alone. Rate of user reviews is a decent gauge of popularity.
- If the game gets more than 150 reviews on Steam per year, start penalising its score. I slowly rolled the score off linearly, penalising it 0.0002 per additional review.
Lastly, I added a release date limit of 1 January 2008. That one is a bit of a hack that shouldn’t really be needed, but the problem is I’m basing “reviews per year” on the stated release date, but some games came to Steam recently yet list their original, earlier release date. This meant that without the date cutoff, re-released games from the 90s ended up looking like the most hiddenest gems ever.
Some older games that have been recently re-released on Steam, and have used their Steam release date as their release date, have ended up on the lists. I decided to leave them in since I would have had to go through manually to find them all. Technically their data is correct, for Steam.
Putting all that together, here’s a list of top Hidden Gems on Steam:
[Click for a top 250 with working links.]
There’s a lot of obscure stuff, a few cult games that you might have heard of but didn’t buy, obviously (the Blackwell games, Escape Goat), and Bejeweled 3? Apparently Bejeweled just doesn’t sell on Steam. Although it actually has over 900 reviews, it’s just that that’s only 144 per year – it’s pretty old too. You could choose to cap the total number of reviews in a Hidden Gems formula to avoid games like this.
I think it’s interesting that the top Hidden Gems here tend to be games in less popular genres, or VR games requiring uncommon hardware. There are three theories I can think of for that. One theory is that great games in a popular genre always tend to sell better than great games in obscure genres, keeping them off the Hidden Gems list. The second is the exact opposite: That great games in a popular genre tend to sell worse due to the flooded market (apart from a few big successes), where having less reviews would push them down in the weighted lists below the niche games. My final theory is that games in popular genres may have more critical reviewers, since the players have a lot more games to choose from. In less popular genres, or in VR, people may be happy to be getting a decent game at all!
There’s a lot more investigation that could be done here. In a more complete Hidden Gem finding system, you’d probably want to be able to view a list per genre, per tag, or based off what else you’ve played. That way you could view the complete list if you wanted to explore whole new genres you might like, but you could also easily find little-known games in your favourite genre.
—
There are also various ways you could decide what a Hidden Gem is exactly. In my system above, since it’s based on the weighted score, a game with 95% positive over 150 reviews will still be considered “better” than a game with 95% positive over 100 reviews. Only after 150/yr does it start trending down.
Maybe you’d rather treat all games that fit in the “Hidden Gem” range equally. Here they are with the same rules as before (plus one extra – see below), using an unweighted score:
[Click for a top 250 with working links.]
—
Looping back to the initial statement of this article, how many games are out there on Steam that have great reviews but little exposure?
The answer is A LOT.
For instance one game I knew of already was NeonXSZ. It’s a 6DOF shooter made by one person, with an 89% positive score on Steam and 63 reviews. It’s always made me a bit sad that it got so little exposure while being so good, but it’s telling that in the weighted hidden gems calculation above, it’s #1147.
There are over one thousand near-unknown games on Steam, in all sorts of genres, that are subjectively better than NeonXSZ.
It might be nice if Steam had a way to easily find lesser-known but well-reviewed games like this, so more people could try them and see if they deserved a boost. This is the category of broken dreams. Lars Doucet does some of this sort of theorising and a bit of my knowledge of this stuff comes from his work.
One thing I will note, is that while there are games in popular genres that fall into the Hidden Gems category, so it’s not a guaranteed ticket out of obscurity, it definitely seems more likely for games in less popular genres to get in there. If I had the data to split games by genre without manually going through them all, it would be interesting to see how much less likely a highly-rated FPS is to languish in obscurity versus say a visual novel. In fact it could be that popular genres aren’t any less likely to end up in obscurity, but instead they have harsher critics who prevent them from being highly rated enough to make the top of the list! There are plenty of hidden non-gems.
Popular genres have a bigger audience but they’re also more flooded with games to choose from. Wuppo is near the top of the Hidden Gems list and it’s a platformer, one of the most popular genres of all. But there are so many platformers.
—
Finally, a couple of bonus lists.
[Click for a top 250 with working links. Taking the weighted hidden gems formula and making it much more strict on review counts.]
After these there are many more games with great scores but even fewer reviews. Eventually the review counts become too low to get any real meaning from the scores.
[Click for a top 250 with working links. Same as Top Games but reversed – more reviews and worse overall scores are better.]
And a bonus chart:
[Click for full size. Showing overall user review score vs. total review count for all games on Steam. There is at least some general correlation between quality and popularity. X-axis is logarithmic.]
[1] As of 2017-05-09. I couldn’t find any information from Valve about what they think re scraping data from Steam. I wrote a little tool to grab some basic details of each game on Steam and I only ran it once, and I’m not sharing the full data set here. I hope that’s simple enough to be okay.
[2] Review counts used here are the ones Steam counts in its overall rating, which at the time of writing is only Steam reviews where the game was purchased directly on Steam. Reviews don’t count for users who just activated a key on Steam – this is a measure by Valve to prevent shady developers gaming the system.
Takeaway:
Play Wuppo
Hehe 🙂
Good article. You should do a similar article but with lowest review scores.
Would it be possible for you to link the actual spreadsheets? I use excel to track my steam games and it’s easier for me to cross reference the lists with games that I own if I can get ahold of the actual data instead of an image.
I’ve just updated the click-through lists to be HTML tables instead of images. So you should now be able to Select All on those and paste into a spreadsheet.
Thank you so much, I really appreciate it!
That was a really interesting article, and I think it quite decisively refutes the idea that really good games always end up selling well. (It also gave me a bunch more games to check out.)
I wanted to comment a bit on “The House in Fata Morgana”, which was at the top of both the weighted and unweighted hidden gems lists. I’ve played it, and I think it’s a pretty quintessential example of a Hidden Gem, in a way that might give some insight as to what sorts of games end up with great reviews but little exposure. It’s a visual novel, and so in a niche/less popular genre to begin with- but on top of that, it’s a very atypical visual novel. It’s pretty much devoid of the usual tropes of the genre, and the art style is distinctly different from the norm. Basically, it’s in a niche genre, but it’s also not the sort of thing that most fans of that niche are looking for, and that’s pretty obvious just from the screenshots. At the same time, I don’t think it ever had much of a chance of finding an audience outside of that niche- it’s still a visual novel (which probably have an inherently somewhat limited appeal), and it still has enough of the anime look that I don’t think it would stand out to someone who wasn’t already a fan of the genre. It got almost no attention from the more general-interest gaming review sites, at any rate.
In short, when it comes to finding an audience, “The House in Fata Morgana” was in a really bad position right out of the starting gate- and yet, IMO, it’s a masterpiece. It just might be the best example of storytelling through the medium of a computer game I’ve ever experienced. At the least, it’s easily the best visual novel I’ve ever played/read (whichever is the right term there). All that is subjective, of course, but nevertheless it seems to be a pretty common reaction from those who’ve played it. I think that’s how it ends up at the top of both those lists- not many have discovered it, but those who have tend to be passionate enough about it that they’re more likely than average to write a review, and they tend to be glowing.
I suspect that for a number of the Hidden Gems, something like this may be going on. Basically, a combination of (a) niche genre; (b) something about them which leads to difficulty finding an audience even within that niche; and (c) something else about them which is good enough to find a small but devoted audience among those who take the chance on them. The “Really Hidden Gems” list, in particular, seems to have a fair number of games like this- “Dream Quest”, which you mentioned, looks like a perfect example of this. Ultra-low-budget graphics which I’m pretty sure would lead a lot of people to not even consider trying it, but gameplay that’s apparently good enough to get a rave review from Richard Garfield. (I might pick that one up during the Summer Sale, actually.)
Thanks for your detailed comments on this. There definitely seems to be a prevalence of those sorts of niche games right at the top. Then as you go down a bit, more popular genres start to appear more often. Hence I think a combination of a full list like in this article, plus separate lists filtered by genre or tag, would probably be the best implementation. That way you can check out everything if you want to, but can also optionally avoid the “niche genre” effect.
Well, if you know Fata Morgana by Werner Herzog — and it’s honestly possible to assume the game’s author(s) named their work after that –, originality and genuineness in the game shall not come as a surprise.
You could use this to make a dedicated site of “diamonds in the rough” games on Steam. I bet it would get some nice traffic.
Tetrobot and Co. is criminally overlooked.
868-HACK is extremely niche is all.
Agree on Militia, as well.
Waves² is just in real early access, a bit under made.
HyperRogue is free for the version before last.
A lot of them are just very very niche products.
Wow, No Man’s Sky really stands out in that last diagram; get so many people to buy your game, but don’t deliver on promises/ don’t work against the false hype and that’s what will happen.
Adding some more support for The House in Fata Morgana here, it’s an absolutely incredible story and a shining example of what the medium of visual novels can deliver. Since it avoids the usual elements of the genre (cute girls, waifu-oriented story, fan service) it’s also a good choice as an introduction to the medium, and the great art and especially great soundtrack help too. I really cannot recommend it enough.
I own, and highly recommend, three of the games in the top 25 of the Really Hidden Gems list.
SculptrVR is, of course, a VR game, so that limits the audience. It’s not all that popular even among those who own VR systems
I play Tetrobot and 868-Hack on my iPad.
Hi when do you think wiseguys will be done? 🙂
I do work on WiseGuys but release isn’t really my call. Bill Ford is the WiseGuys master.
This is great! I just picked up Wuppo and it’s really adorable, now to go digging through the rest.
I was thinking about the cutoff date for hidden gems, would it instead make sense to just set a release date floor of 25th November 2013, ie the date Steam user reviews launched? It might unfairly penalise some older games that people have reviewed retrospectively, but on the other hand a lot of that sort of activity implies a popular game anyway, or at least something of a cult classic.
And it would help push out your Bejewelleds and Monkey Islands that probably snuck in from releasing long before the review system launched.
There are definitely some superb games in there, always happy to cheerlead for things like Redirection, The Sea Will Claim Everything and The Last Door
That sounds like a good idea. Although it would exclude some appropriate indie stuff: Blackwell Deception is one example.
Sorry I didn’t describe that very well, floor as in treat everything older than 25/11/2013 as having that release date. Means that older games can still be included, but don’t get treated as having many years more reviews than they actually have. Blackwell Deception would still have <150/year with that date so would still be included.
Although Deception is probably an example of skew due to being released before reviews existed, compounded by being the middle of a series, so fans going back would probably just review the first one: Note Epiphany has more reviews despite being its direct narrative sequel.
Ah yeah, that should work, as long as you cut out reviews older than than date as well (or else some older games would seem much more popular). Unfortunately it’d be hard to get review counts over a specific date range out of Steam.
Thank you! I think this would make a great Steam curated list.
What about putting out an updated version of this list as time goes by? New undiscovered games will come out and it may be important to help discoverability with this system.
You might be interested in http://steam250.com/hidden_gems.html
You should revisit this in 2018. I’d be interested in seeing how things changed.
This new site (no affiliation with me, but they were indirectly inspired by my post here) doesn’t use exactly the same algorithm as I did, but does use a very similar one, and it updates automatically all the time: http://steam250.com/hidden_gems