Success Story – Real-Time Pricing for E-Commerce
This case study shows how DxMinds met the data needs of a research and analytics client. We created a customized data scraper to gather e-commerce data fields in real time.
The client, from the Research & Analytics industry, needed a highly customized e-commerce scraper to get real-time e-commerce data feeds.
Real-Time Pricing with E-Commerce Data Feeds
Client Information
Data Research and Analytics Services for Retail and E-Commerce
Challenges in E-Commerce Data
Our client, a research and analytics business, needed reliable and accurate e-commerce data for their projects.
They wanted easy access to detailed product lists from specific categories, including specifications and pricing. Previously, their in-house data team manually collected data from various web sources, but the results were not satisfactory for the effort required.
DxMinds Enterprise Crawling helps aggregate data from thousands of web sources, enabling big data enterprises to transform data into actionable insights. The aims to be one of the largest data-sourcing companies with its cloud-based automated data harvesting ecosystem. Started in February 2012, DxMinds has achieved 200% growth year on year since inception and now operates over 13,500 sq. ft. with a team of 200+ resources across two delivery centers in Ahmedabad, India.
The client provided us with a list of sources to scrape, the data points needed, and how often they needed the data. Our team set up crawlers to gather the necessary e-commerce data from the specified websites. With our custom Data-as-a-Service (DaaS) model, we manage the entire extraction process, including setting up crawlers, servers, proxies, extraction, monitoring, quality assurance, timely delivery, and continuous maintenance to handle structural changes in the websites.
The client wanted the data in CSV format and uploaded it to their S3 servers. We completed the setup in a few days, and the crawlers started delivering data immediately.
DxMinds Solution
Setting up the Crawler: We initially set up the crawler to automatically scrape product pricing and other necessary data fields from predefined categories on a daily basis.
Data Template: Using the schema provided by the customer, we created a template for structuring the data.
Data Delivery: The final data was delivered daily in XML format through a Data API, with no manual involvement needed from either side.
Each record in the dataset included the product’s name, price, availability, long and short descriptions, image URLs, dimensions, category, SKU, brand, resource, and the source URLs from which the data was fetched.
DxMinds Enterprise Crawling Advantages
- We managed any changes on the resource websites, so our clients didn’t have to worry about it.
- Any updates to the plan were completed as requested.
- Faster data turnaround improved our client’s market capabilities and services.
- We can add new categories based on changing needs.
- The client’s data team became more productive and could focus on other projects, helping them expand into new business areas.
- Data quality improved significantly without extra time investment from our team.
- The value added from this project was about 50 times the cost.
