more than 35 milion rows
The order to gather the data of tens of millions of users of the Allegro auctioning site. The client most of all required us to deliver a particular number of comments, information whether the given user is a private individual or represents a business, and their e-mail address from the “About me” page.
To complete this project we built a temporary structure including tens of servers located in different cities of Poland managed from a dedicated web application installed on a central device. Once the system was initiated, the entire operation took about a week to complete.