Kickscraper

Hi, long time no blog! I’m terribly bored – so might as well post something useful.

So for anyone who knows me, I’m a Kickstarter whore. As of this posting, I’ve backed about 51 projects and am still constantly searching for the next interesting and unique thing I could back/support/get.

Kickstarter is basically a crowdfunding site. If you want to produce something but have no budget, you submit your proposal to the site and the community can choose to pledge some money to fund your project – usually in exchange for getting the you’re offering for themselves – at heavily discounted costs. People who pledge only get charged when the funding goal is reached. If it doesn’t make it, then no harm, no foul. So in that sense it’s a very risk free way of getting cool stuff.

I guess it’s the whole uniqueness of the products that really appeal to me. While I do appreciate good consumer products, 1 And while a lot of Kickstarter products end up commercialized over time I guess the prospect of being one of the few people (and in some cases the only one) who has some awesome doodad you can’t find anywhere else.

The Problem

Navigating the site can be a pain though. The stuff I like in Kickstarter are from 2 specific categories: Design and Technology. The problem there is that those two categories tend to be very broad. In Design alone, there are 6 subcategories. Technology is even bigger with 15 subcategories

How is this an issue? In the Design category, I really only am interested in Product Design – basically because those are the stuff that have physical tangible results you can own after the funding period. In Technology on the other hand, I’m interested in Camera Equipment, DIY ELectronics, Gadgets, Hardware and Wearables

It’s an issue because Kickstarter’s filtering algorithm doesn’t allow you to filter results from multiple select categories. So that means I have to drill down to 6 different pages (all from the home screen) just to check out the stuff I want. Ain’t no one got time fo dat!

Seriously, it’s so tedious to do that it kinda discourages people checking the site 2 Which could be a good thing if you want to protect yourself from the wallet vampire that is Kickstarter. The trouble with that is because you tend to miss out on the “early bird” offers. Early bird rewards are common practice to give a project the needed boost in hype by offering a limited amount of “slots” to get the product with much more added value – whether by extra discounts, or eariler release times, etc. – which serves as an incentive to be one for the first ones to back the project and start the ball rolling.

Of course, there’s also the concern that you end up missing a project entirely, like I did with the Ember Modular Urban Backack it would be very difficult to buy it after the campaign is done (you’ll have to wait IF and when they actually end up commercializing the product. 3 So right now, I’m actually stalking the producers – begging them to sell me a unit even after the campaign has ended LOL

The Solution

Thankfully, Kickstarter sort of has a public API – so to address my need, I’ve decided to create a scraper/aggregator to consolidate all the subcategories into one listing.

It’s very simple in what it does, and is limited by the [maddening] limitations of the KS API itself. Like I said you cannot create a filter in the site that will show you multiple selected categories – you can only select parent categories. Naturally I had set to ALL as in absolutely all projects from all categories.

The problem with that is as of this posting, 8.7k live projects so it would be madness to manually scroll through all that especially since Kickstarter has a 20 project per page limitation on every listing request.

So what I did was pretty simple. Based on the JSON response we take the total_hits key and divide it by 20. This will give us roughly how many individual pages we’ll have to load (about 436 as of this post). Since we’re loading ALL projects regardless of category, I’m going to do the filtering via JS which is simple enough to do. That way if it doesn’t fall under the categories I’m interested in, it skips it.

The only real issue at this point is to automate the loading. Since we have no idea when or how many of the “valid” projects would appear in a page – sometimes we’d have to go through a bunch of pages before actually hitting a project that falls under the category I set. So I had to find a way to load those in the background without requiring user interaction. I ended up using my trusty proxy script to cache the results of each page for 12 hours – so we only slam the KS servers at most twice in a day and avoid getting banned. During the times we do have to refresh, we have no choice but to just have it fire requests at random intervals between 0-3 seconds (so KS won’t interpret the 400+ sequential requests as some sort of attempted DOS attack)

The result works like a charm. It is a bit slow since we are limited by forced request delays (whether via KS or my proxy) but it sure as hell beats going to the site. So now I just load up the page, have some breakfast, and check it out when everything has loaded – all in one page.

I also added some extra filters such as displaying those ending in a certain amount of time, or only those that have been funded, or those that show promise (60% funded or more). It, for me, truly is a better way of accessing Kickstarter – and hopefully I wouldn’t miss a good project ever again 🙂

You can try it here

Notes

Notes
1 And while a lot of Kickstarter products end up commercialized over time
2 Which could be a good thing if you want to protect yourself from the wallet vampire that is Kickstarter
3 So right now, I’m actually stalking the producers – begging them to sell me a unit even after the campaign has ended LOL

Have a say

This site uses Akismet to reduce spam. Learn how your comment data is processed.