{"id":1123,"date":"2014-08-20T20:02:42","date_gmt":"2014-08-20T12:02:42","guid":{"rendered":"http:\/\/www.nargalzius.com\/blog\/?p=1123"},"modified":"2014-08-20T20:02:42","modified_gmt":"2014-08-20T12:02:42","slug":"kickscraper","status":"publish","type":"post","link":"http:\/\/nargalzius.com\/blog\/archives\/2014\/08\/20\/kickscraper","title":{"rendered":"Kickscraper"},"content":{"rendered":"<p>Hi, long time no blog! I&#8217;m terribly bored &#8211; so might as well post something useful.<\/p>\n<p>So for anyone who knows me, I&#8217;m a <a href=\"http:\/\/www.kickstarter.com\">Kickstarter<\/a> whore. As of this posting, I&#8217;ve backed about <a href=\"https:\/\/www.kickstarter.com\/profile\/nargalzius\">51 projects<\/a> and am still constantly searching for the next interesting and unique thing I could back\/support\/get.<\/p>\n<p><a href=\"http:\/\/www.kickstarter.com\">Kickstarter<\/a> is basically a crowdfunding site. If you want to produce something but have no budget, you submit your proposal to the site and the community can choose to pledge some money to fund your project &#8211; usually in exchange for getting the you&#8217;re offering for themselves &#8211; at heavily discounted costs. People who pledge only get charged when the funding goal is reached. If it doesn&#8217;t make it, then no harm, no foul. So in that sense it&#8217;s a very risk free way of getting cool stuff.<\/p>\n<p>I guess it&#8217;s the whole uniqueness of the products that really appeal to me. While I do appreciate good consumer products, <span class=\"footnote_referrer\"><a role=\"button\" tabindex=\"0\" onclick=\"footnote_moveToReference_1123_1('footnote_plugin_reference_1123_1_1');\" onkeypress=\"footnote_moveToReference_1123_1('footnote_plugin_reference_1123_1_1');\" ><sup id=\"footnote_plugin_tooltip_1123_1_1\" class=\"footnote_plugin_tooltip_text\">1 <\/sup><\/a><span id=\"footnote_plugin_tooltip_text_1123_1_1\" class=\"footnote_tooltip\">And while a lot of Kickstarter products end up commercialized over time <\/span><\/span><script type=\"text\/javascript\"> jQuery('#footnote_plugin_tooltip_1123_1_1').tooltip({ tip: '#footnote_plugin_tooltip_text_1123_1_1', tipClass: 'footnote_tooltip', effect: 'fade', predelay: 0, fadeInSpeed: 200, delay: 400, fadeOutSpeed: 200, position: 'top right', relative: true, offset: [10, 10], });<\/script> I guess the prospect of being one of the few people (and in some cases the only one) who has some awesome doodad you can&#8217;t find anywhere else.<!--more--><!-- \/\/ --><\/p>\n<h1>The Problem<\/h1>\n<p>Navigating the site can be a pain though. The stuff I <em>like<\/em> in Kickstarter are from 2 specific categories: <strong>Design<\/strong> and <strong>Technology.<\/strong> The problem there is that those two categories tend to be very broad. In <em>Design<\/em> alone, there are <a href=\"https:\/\/www.kickstarter.com\/discover\/categories\/design?ref=discover_index\">6 subcategories<\/a>. <em>Technology<\/em> is even bigger with <a href=\"https:\/\/www.kickstarter.com\/discover\/categories\/technology?ref=discover_index\">15 subcategories<\/a><\/p>\n<p>How is this an issue? In the <em>Design<\/em> category, I really only am interested in <strong>Product Design<\/strong> &#8211; basically because those are the stuff that have physical tangible results you can own after the funding period. In <em>Technology<\/em> on the other hand, I&#8217;m interested in <strong>Camera Equipment, DIY ELectronics, Gadgets, Hardware<\/strong> and <strong>Wearables<\/strong><\/p>\n<p>It&#8217;s an issue because Kickstarter&#8217;s filtering algorithm doesn&#8217;t allow you to filter results from multiple <em>select<\/em> categories. So that means I have to drill down to 6 different pages (all from the home screen) just to check out the stuff I want. <em>Ain&#8217;t no one got time fo dat!<\/em><\/p>\n<p>Seriously, it&#8217;s so tedious to do that it kinda discourages people checking the site <span class=\"footnote_referrer\"><a role=\"button\" tabindex=\"0\" onclick=\"footnote_moveToReference_1123_1('footnote_plugin_reference_1123_1_2');\" onkeypress=\"footnote_moveToReference_1123_1('footnote_plugin_reference_1123_1_2');\" ><sup id=\"footnote_plugin_tooltip_1123_1_2\" class=\"footnote_plugin_tooltip_text\">2 <\/sup><\/a><span id=\"footnote_plugin_tooltip_text_1123_1_2\" class=\"footnote_tooltip\">Which could be a good thing if you want to protect yourself from the wallet vampire that is Kickstarter<\/span><\/span><script type=\"text\/javascript\"> jQuery('#footnote_plugin_tooltip_1123_1_2').tooltip({ tip: '#footnote_plugin_tooltip_text_1123_1_2', tipClass: 'footnote_tooltip', effect: 'fade', predelay: 0, fadeInSpeed: 200, delay: 400, fadeOutSpeed: 200, position: 'top right', relative: true, offset: [10, 10], });<\/script>. The trouble with <em>that<\/em> is because you tend to miss out on the &#8220;early bird&#8221; offers. Early bird rewards are common practice to give a project the needed boost in hype by offering a limited amount of &#8220;slots&#8221; to get the product with much more added value &#8211; whether by extra discounts, or eariler release times, etc. &#8211; which serves as an incentive to be one for the first ones to back the project and start the ball rolling.<\/p>\n<p>Of course, there&#8217;s also the concern that you end up missing a project <em>entirely,<\/em> like I did with the <a href=\"https:\/\/www.kickstarter.com\/projects\/842978788\/ember-equipment-modular-urban-pack\">Ember Modular Urban Backack<\/a> it would be very difficult to buy it after the campaign is done (you&#8217;ll have to wait IF and when they actually end up commercializing the product. <span class=\"footnote_referrer\"><a role=\"button\" tabindex=\"0\" onclick=\"footnote_moveToReference_1123_1('footnote_plugin_reference_1123_1_3');\" onkeypress=\"footnote_moveToReference_1123_1('footnote_plugin_reference_1123_1_3');\" ><sup id=\"footnote_plugin_tooltip_1123_1_3\" class=\"footnote_plugin_tooltip_text\">3 <\/sup><\/a><span id=\"footnote_plugin_tooltip_text_1123_1_3\" class=\"footnote_tooltip\">So right now, I&#8217;m actually stalking the producers &#8211; begging them to sell me a unit even after the campaign has ended LOL<\/span><\/span><script type=\"text\/javascript\"> jQuery('#footnote_plugin_tooltip_1123_1_3').tooltip({ tip: '#footnote_plugin_tooltip_text_1123_1_3', tipClass: 'footnote_tooltip', effect: 'fade', predelay: 0, fadeInSpeed: 200, delay: 400, fadeOutSpeed: 200, position: 'top right', relative: true, offset: [10, 10], });<\/script><\/p>\n<h1>The Solution<\/h1>\n<p>Thankfully, Kickstarter sort of has a public API &#8211; so to address my need, I&#8217;ve decided to create a <a href=\"http:\/\/nargalzius.com\/ks\/\">scraper\/aggregator<\/a> to consolidate all the subcategories into one listing. <\/p>\n<p>It&#8217;s very simple in what it does, and is limited by the [maddening] limitations of the KS API itself. Like I said you cannot create a filter in the site that will show you multiple selected categories &#8211; you can only select parent categories. Naturally I had set to ALL as in absolutely all projects from all categories.<\/p>\n<p>The problem with that is as of this posting, <a href=\"https:\/\/www.kickstarter.com\/discover\/categories\/all?ref=discover_index&amp;#38;format=json&amp;#38;state=live\">8.7k <em>live<\/em> projects<\/a> so it would be madness to manually scroll through all that especially since Kickstarter has a 20 project per page limitation on every listing request.<\/p>\n<p>So what I did was pretty simple. Based on the <a href=\"https:\/\/www.kickstarter.com\/discover\/categories\/all?ref=discover_index&amp;#38;format=json&amp;#38;state=live\">JSON response<\/a> we take the total_hits key and divide it by 20. This will give us roughly how many individual pages we&#8217;ll have to load (about 436 as of this post). Since we&#8217;re loading ALL projects regardless of category, I&#8217;m going to do the filtering via JS which is simple enough to do. That way if it doesn&#8217;t fall under the categories I&#8217;m interested in, it skips it.<\/p>\n<p>The only real issue at this point is to automate the loading. Since we have no idea when or how many of the &#8220;valid&#8221; projects would appear in a page &#8211; sometimes we&#8217;d have to go through a bunch of pages before actually hitting a project that falls under the category I set. So I had to find a way to load those in the background without requiring user interaction. I ended up using my trusty <a href=\"http:\/\/www.nargalzius.com\/blog\/archives\/2013\/04\/27\/universal-proxy\">proxy script<\/a> to cache the results of each page for 12 hours &#8211; so we only slam the KS servers at most twice in a day and avoid getting banned. During the times we do have to refresh, we have no choice but to just have it fire requests at random intervals between 0-3 seconds (so KS won&#8217;t interpret the 400+ sequential requests as some sort of attempted DOS attack)<\/p>\n<p>The result works like a charm. It is a bit slow since we <em>are<\/em> limited by forced request delays (whether via KS or my proxy) but it sure as hell beats going to the site. So now I just load up the page, have some breakfast, and check it out when everything has loaded &#8211; all in one page.<\/p>\n<p>I also added some extra filters such as displaying those ending in a certain amount of time, or only those that have been funded, or those that show promise (60% funded or more). It, for me, truly is a better way of accessing Kickstarter &#8211; and hopefully I wouldn&#8217;t miss a good project ever again \ud83d\ude42<\/p>\n<p>You can try it <a href=\"http:\/\/www.nargalzius.com\/ks\">here<\/a><\/p><div class=\"speaker-mute footnotes_reference_container\"> <div class=\"footnote_container_prepare\"><p><span role=\"button\" tabindex=\"0\" class=\"footnote_reference_container_label pointer\" onclick=\"footnote_expand_collapse_reference_container_1123_1();\">Notes<\/span><span role=\"button\" tabindex=\"0\" class=\"footnote_reference_container_collapse_button\" style=\"display: none;\" onclick=\"footnote_expand_collapse_reference_container_1123_1();\">[<a id=\"footnote_reference_container_collapse_button_1123_1\">+<\/a>]<\/span><\/p><\/div> <div id=\"footnote_references_container_1123_1\" style=\"\"><table class=\"footnotes_table footnote-reference-container\"><caption class=\"accessibility\">Notes<\/caption> <tbody> \r\n\r\n<tr class=\"footnotes_plugin_reference_row\"> <th scope=\"row\" class=\"footnote_plugin_index_combi pointer\"  onclick=\"footnote_moveToAnchor_1123_1('footnote_plugin_tooltip_1123_1_1');\"><a id=\"footnote_plugin_reference_1123_1_1\" class=\"footnote_backlink\"><span class=\"footnote_index_arrow\">&#8673;<\/span>1<\/a><\/th> <td class=\"footnote_plugin_text\">And while a lot of Kickstarter products end up commercialized over time <\/td><\/tr>\r\n\r\n<tr class=\"footnotes_plugin_reference_row\"> <th scope=\"row\" class=\"footnote_plugin_index_combi pointer\"  onclick=\"footnote_moveToAnchor_1123_1('footnote_plugin_tooltip_1123_1_2');\"><a id=\"footnote_plugin_reference_1123_1_2\" class=\"footnote_backlink\"><span class=\"footnote_index_arrow\">&#8673;<\/span>2<\/a><\/th> <td class=\"footnote_plugin_text\">Which could be a good thing if you want to protect yourself from the wallet vampire that is Kickstarter<\/td><\/tr>\r\n\r\n<tr class=\"footnotes_plugin_reference_row\"> <th scope=\"row\" class=\"footnote_plugin_index_combi pointer\"  onclick=\"footnote_moveToAnchor_1123_1('footnote_plugin_tooltip_1123_1_3');\"><a id=\"footnote_plugin_reference_1123_1_3\" class=\"footnote_backlink\"><span class=\"footnote_index_arrow\">&#8673;<\/span>3<\/a><\/th> <td class=\"footnote_plugin_text\">So right now, I&#8217;m actually stalking the producers &#8211; begging them to sell me a unit even after the campaign has ended LOL<\/td><\/tr>\r\n\r\n <\/tbody> <\/table> <\/div><\/div><script type=\"text\/javascript\"> function footnote_expand_reference_container_1123_1() { jQuery('#footnote_references_container_1123_1').show(); jQuery('#footnote_reference_container_collapse_button_1123_1').text('\u2212'); } function footnote_collapse_reference_container_1123_1() { jQuery('#footnote_references_container_1123_1').hide(); jQuery('#footnote_reference_container_collapse_button_1123_1').text('+'); } function footnote_expand_collapse_reference_container_1123_1() { if (jQuery('#footnote_references_container_1123_1').is(':hidden')) { footnote_expand_reference_container_1123_1(); } else { footnote_collapse_reference_container_1123_1(); } } function footnote_moveToReference_1123_1(p_str_TargetID) { footnote_expand_reference_container_1123_1(); var l_obj_Target = jQuery('#' + p_str_TargetID); if (l_obj_Target.length) { jQuery( 'html, body' ).delay( 0 ); jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight * 0.2 }, 380); } } function footnote_moveToAnchor_1123_1(p_str_TargetID) { footnote_expand_reference_container_1123_1(); var l_obj_Target = jQuery('#' + p_str_TargetID); if (l_obj_Target.length) { jQuery( 'html, body' ).delay( 0 ); jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight * 0.2 }, 380); } }<\/script>","protected":false},"excerpt":{"rendered":"<p>Hi, long time no blog! I&#8217;m terribly bored &#8211; so might as well post something useful. So for anyone who knows me, I&#8217;m a Kickstarter whore. As of this posting, I&#8217;ve backed about 51 projects and am still constantly searching for the next interesting and unique thing I could back\/support\/get. Kickstarter is basically a crowdfunding &hellip; <p class=\"link-more\"><a href=\"http:\/\/nargalzius.com\/blog\/archives\/2014\/08\/20\/kickscraper\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Kickscraper&#8221;<\/span><\/a><\/p><\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[13],"tags":[],"class_list":["post-1123","post","type-post","status-publish","format-standard","hentry","category-technology"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack-related-posts":[],"_links":{"self":[{"href":"http:\/\/nargalzius.com\/blog\/wp-json\/wp\/v2\/posts\/1123","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/nargalzius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/nargalzius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/nargalzius.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/nargalzius.com\/blog\/wp-json\/wp\/v2\/comments?post=1123"}],"version-history":[{"count":0,"href":"http:\/\/nargalzius.com\/blog\/wp-json\/wp\/v2\/posts\/1123\/revisions"}],"wp:attachment":[{"href":"http:\/\/nargalzius.com\/blog\/wp-json\/wp\/v2\/media?parent=1123"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/nargalzius.com\/blog\/wp-json\/wp\/v2\/categories?post=1123"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/nargalzius.com\/blog\/wp-json\/wp\/v2\/tags?post=1123"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}