Business based on web scraping
Discussion
A friend has approached me for advice as I have a technical background. They have a business plan that is essentially a comparison website solely dependent on scraping data from other websites.
While it's technically possible, I'm not sure about the legal aspects of this, and also whether or not there could be a way to completely prevent scraping? I know of ways round most blocks (rotating proxies to get round the IP being blocked, using tools like Selenium to handle sites where most of the code is Javascript etc).
Does anyone have any experience in this area and whether or not this is likely to be a viable business?
Thanks.
While it's technically possible, I'm not sure about the legal aspects of this, and also whether or not there could be a way to completely prevent scraping? I know of ways round most blocks (rotating proxies to get round the IP being blocked, using tools like Selenium to handle sites where most of the code is Javascript etc).
Does anyone have any experience in this area and whether or not this is likely to be a viable business?
Thanks.
A friend of mine runs multiple web page scrapers for his business expenses - eg scraping the Dartford Crossing account statement, petrol statement, credit card statements from various banks etc. They're all written based on Selenium which makes them nice and scriptable and adaptable which is essential - the scrapers can be very flaky as soon as eg. a bank puts a temporary additional message or promotion page after login and before you get to the statement page for example. Upside is that banks in the same group all have the same site structure so for example as soon as you have a scraper working for Nat West it pretty much works straight out of the box for RBS (until one of the aforementioned changes).
I wouldn't want to build something like that for an actual front line business though. Aside from the flakiness of the scrapers it "feels" dodgy as hell and likely breaches their terms and conditions. If the sites wanted to share their data to consolidators programmatically, they'd offer an API to do it.
I wouldn't want to build something like that for an actual front line business though. Aside from the flakiness of the scrapers it "feels" dodgy as hell and likely breaches their terms and conditions. If the sites wanted to share their data to consolidators programmatically, they'd offer an API to do it.
The idea is to scrape things from the websites and compare them to their competitors, with a view to picking up a referral fee if a user clicks through to the website. Much like the likes of MoneySupermarket etc do, although I appreciate companies would be more likely to do a deal with the likes of MoneySupermarket given how big their user base is.
There are no APIs available, that would definitely have been preferable but they just don't exist.
There are no APIs available, that would definitely have been preferable but they just don't exist.
XJ75 said:
The idea is to scrape things from the websites and compare them to their competitors, with a view to picking up a referral fee if a user clicks through to the website. Much like the likes of MoneySupermarket etc do, although I appreciate companies would be more likely to do a deal with the likes of MoneySupermarket given how big their user base is.
There are no APIs available, that would definitely have been preferable but they just don't exist.
Do the companies have the choice as to whether or not they are included? There is a whole minefield around who owns that data...There are no APIs available, that would definitely have been preferable but they just don't exist.
XJ75 said:
The idea is to scrape things from the websites and compare them to their competitors, with a view to picking up a referral fee if a user clicks through to the website. Much like the likes of MoneySupermarket etc do, although I appreciate companies would be more likely to do a deal with the likes of MoneySupermarket given how big their user base is.
There are no APIs available, that would definitely have been preferable but they just don't exist.
Scraping to get a bit of affiliate rev has been done to death. There are no APIs available, that would definitely have been preferable but they just don't exist.
I wouldn't jump into this unless it's just a side hustle.
Pay for a 3rd party product to do the scraping for you until you can see the business model working
Dromedary66 said:
Would require so much work to keep on top of vs the correct method of using the APIs.
Agree with this. Anywhere with information worth having will publish an API and probably charge for it. Just pay the subscription, if that means the business isn't viable then it never was.Also debatable legally as to whether it's unauthorised access if it's against the site T&C. Wouldn't base a business on it myself.
Absolutely, it’s fairly easy to script.
I have coded a few of them previously for specific tasks. I do agree with the above posts though, it’s a bit cheeky. Because the scraper works like a regular website visitor, there isn’t much the website owner can do to stop you.
The other problem you are faced with is if the website changes the controls on the page, you have to adjust your scraper script.
Obviously, API or Webservices are the correct way of doing this. We have written some large systems using webservices to compare and crunch car prices, a bit like MoneySupermarket. Using webservices, we can allow many simultaneous users to compare prices at the same time.
I have coded a few of them previously for specific tasks. I do agree with the above posts though, it’s a bit cheeky. Because the scraper works like a regular website visitor, there isn’t much the website owner can do to stop you.
The other problem you are faced with is if the website changes the controls on the page, you have to adjust your scraper script.
Obviously, API or Webservices are the correct way of doing this. We have written some large systems using webservices to compare and crunch car prices, a bit like MoneySupermarket. Using webservices, we can allow many simultaneous users to compare prices at the same time.
XJ75 said:
The idea is to scrape things from the websites and compare them to their competitors, with a view to picking up a referral fee if a user clicks through to the website.
Most companies who offer referral fees or affiliate programmes also offer a feed of their products. They may be in different formats (XML, CSV etc) so need transformation, but you shouldn't need to scrape content directly from their sites.Another way to tap in to company's data is that many platforms have an export function, for providing feeds to things like Google Shopping. If he is prepared to go the official route (which is ultimately will have to if he wants to earn referral fees) you could utilise those existing data feeds.
If done properly then it can be ethical. What I mean is I've known people to just hammer a site constantly several times a day without caring about the impact they're having on the site.
I do it myself, not for comparisons or anything just for data collection for a long term project. The difficulty would be in making your scrapes look kosher, and being able to rebuild the scrape when the website changes some HTML. So the error reporting needs to be spot on otherwise your data will get old very quickly without being updated.
I do it myself, not for comparisons or anything just for data collection for a long term project. The difficulty would be in making your scrapes look kosher, and being able to rebuild the scrape when the website changes some HTML. So the error reporting needs to be spot on otherwise your data will get old very quickly without being updated.
Gassing Station | Business | Top of Page | What's New | My Stuff