1. modern shopping sites are built haphazardly without structure;
2. a typical page load time is 10-20 seconds for many (most?) shops.
1. matters because you cannot build a simple crawler. This is because sometimes, the brand is in the title, the description is in the brand, and the gender is in the quantity, which itself is in the photo as a pixellated image...
You must load the entire page and figure out where the attributes are in a smart way. You can do this today with headless chrome, in the old days with Selenium, etc.
... which brings us to 2. A mainstream, regionally-dominating shop, particularly any that adopt the marketplace model (Amazon, Taobao, etc.) may have as many as 20 million products. Given 2., good luck refreshing this price list in a sane time, even if you crack the attribute extraction.
If by some miraculous feat you solve these issues you now have to product match an inventory of billions of listings, most of which are misspelt, missing attributes (brand, etc.), and use photos that even a human could not match by sight. Think how hard it is to find exactly what you want on eBay...
The only way is to get the data directly from the retailers. Assuming they will give it to you, you end up with the same problem of feed poisoning that you have with marketing.
Oh one thing you could do: Create a huge community of users who share their purchase data and other data.and then build the model through that died so if you want to participate and save you need to be part of the panel
Funny, I had a lot of overlap in this kind of smart data quantizing when I was trying to build a venue page crawler that would intelligently mete out live music event dates, times and locations. I realized about 20% into the project I'd need some form of machine learning and then bailed.
Here's the real problem:
1. modern shopping sites are built haphazardly without structure;
2. a typical page load time is 10-20 seconds for many (most?) shops.
1. matters because you cannot build a simple crawler. This is because sometimes, the brand is in the title, the description is in the brand, and the gender is in the quantity, which itself is in the photo as a pixellated image...
You must load the entire page and figure out where the attributes are in a smart way. You can do this today with headless chrome, in the old days with Selenium, etc.
... which brings us to 2. A mainstream, regionally-dominating shop, particularly any that adopt the marketplace model (Amazon, Taobao, etc.) may have as many as 20 million products. Given 2., good luck refreshing this price list in a sane time, even if you crack the attribute extraction.
If by some miraculous feat you solve these issues you now have to product match an inventory of billions of listings, most of which are misspelt, missing attributes (brand, etc.), and use photos that even a human could not match by sight. Think how hard it is to find exactly what you want on eBay...
The only way is to get the data directly from the retailers. Assuming they will give it to you, you end up with the same problem of feed poisoning that you have with marketing.