Automation of data collection and uploading the products to the online clothes shop

Customer challenges

The customer sells goods from suppliers located in multiple countries through their online store. One of their primary suppliers has a product catalog with up to a million SKUs. Unfortunately, this supplier does not have its own API for exchanging data with their customers (online stores). Working with such vast volumes of data manually is impossible, and it was of utmost importance for our customer to obtain accurate information regarding the goods, their prices, and product availability.

The main details of the project

• Prices and discounts for products change frequently on the supplier’s end
• The availability and sizes of the goods from the supplier change regularly.
• It is crucial to receive the most recent data promptly.
• The supplier’s website displays different products and prices based on the country from which the user accesses the site.
• The supplier’s website has protections in place to prevent a scraper from frequently accessing pages from the same IP address.
• Depending on the brand’s country of origin, the supplier offers different clothing and footwear with various sizing charts.

Project stages

• We developed a scraper that works with a minimum of calls to the supplier’s site, this allows us not to load the site each time and at the same time get all the necessary information.
• Since prices and availability of products change frequently, the customer needs to update the data as frequently as on the supplier’s side. We have configured our scraping infrastructure solution in such a way that we receive data about the entire catalogue and spend a minimum of time on this task. The data collection time is calculated so that the customer has fresh data by the time he updates his catalogue.
• Since the supplier’s website is a multilingual store with visitors from different countries, the prices and availability of goods vary depending on the location of the visitor. We have set up data scraping from the customer’s desired geographic location.
• To ensure that the site does not block data collection, and we could meet the deadline in which to provide updated prices, we have configured scraper work so that it does not load the supplier’s site. Also we used an IP server rotator, it helped us avoid blocking the scraper.
• One of the problems faced by the customer was the availability of different sizes of clothes and shoes. If we would leave everything as it is, users would have to find the right product on their own using the comparison tables. It would be inconvenient and could lead to errors and as a result to product returns and refunds. It was necessary to bring the size grids of hundreds of thousands of goods to the sizes of clothes and shoes to one universal format that would be convenient to all the users. To solve this problem, we developed a program that converts the dimensions and size grids into understandable format to all the users.
• The collected data is transmitted to the customer in their data storage system in the required format.