Hello guys, today we’re going to talk of another minor minor (minor) project that is coming to life thanks to another big project of mine that I should have already released but due to my slowness I still not be ready to show it (yes, I’m talking of the same project I was talking in this other post). Anyway let’s talk about what this module I developed does and how.
So my goal was to search in node for italian torrents and the only good italian torrent search engine that I know is ilcorsaronero.info, it’s not so much different than KickAss, The Pirate Bay etc.
Since they doesn’t have any kind of API, I needed to scrape these information (here if you don’t know what’s this technique) and when we talk about “scraping” only two libraries come to the mind of a node developer: cheeriojs and request. The first one is, in two words, a server side jquery implementation and second is used to make requests and get the html of a webpage… they make the art of web scraping a child’s play.
So let’s see the most important piece of code of this library just to show what I did technically (a lot of lines has been deleted down here) :
I defined a “scrape” function within, with request, I requested the “html” of a webpage passed through parameter. After I passed as parameter to cheerio the webpage in order to use the classical jquery selector to get attributes, texts and everything else we need. You see two requests in the code: the first is the search page, and for every item resulted from the search it will request for another link (due to the structure of ilcorsaronero that doesn’t expose the magnet links in the search page). And at the end there is a callback call (of course the code is async and a callback is required to get the scraped results).
I hope someone will appreciate this and if you do something great with this library make me know about it through comments or privately on social network!
Stay awesome guys :D