Data gathering/web scraping/API

[technodup]

If I need to find a quicker way of giving data to someone what are my options?

I've used import.io before and it's great but only for a results page, I need the next level...

For example if I search Ebay for Mondeo I'll get a list of cars for sale. Putting that web address in will output all the auction titles, end times and prices as listed. But let's say I want information from each individual auction page as you click into it, the feedback score for example, how can I get that without doing it manually?

TIA

[andyb28]

Not sure about quick ways, but we wrote a screenscraper for one of our customers that wanted to get some info out of a governing body website for their industry.

There are a number of languages you could use to do it in. We used .Net for ours, but you could probably do it quicker with Perl or Python.

[mr_spock]

Can't you use the eBay API for that?

[technodup]

mr_spock said:

Can't you use the eBay API for that?

I was just using Ebay as an example. AFAIK the sites I want don't have an API, and even if they did I wouldn't know where to start.

I think I'll get someone to do it manually.

[essayer]

Ask the site owners to publish an API allowing you to receive the content.

Don't just steal it without considering the legalities..

[nyt]

check out http://aignes.com and http://imacros.net

both useful in different ways.

Both will download a page only when it changes. You can then process the HTML using whatever tool you like.
If I remember correctly, you can pick out individual elements to save and avoid all coding.

Your use case may be more complex but these were used to monitor over 1,000 websites in a previous job.

ETA: Websites change all the time. Scraping data isn't a do-once-and-forget exercise unfortunately.

Excel does a pretty good job of reading websites. Data->Import from web if I recall correctly. Loads the text from the webpage into cells.

Edited by nyt on Tuesday 29th March 18:43

[technodup]

Thanks all. It's all a bit beyond my computer literacy.

Good thing is it seems possible, so it's a case of finding out what works and at what cost.

[48k]

We've written a few different screen scrapers for a client, allows him to compile expenses by grabbing things like his online credit card statements, Dartford crossing data and so on, and imports the data in to a database. There's a few different techniques for doing it depending on the target platform and the requirements but more often than not we use Selenium to drive automated inputs in to a web browser and capture the response. Feel free to PM me some details if you'd like us to give you a quote for something similar.

[cat with a hat]

Lots and lots of options but it depends on a number of things:
Reliability/accuracy
Level of automation
Speed
Cost
Technical ability/willing to learn or willing to outsource

How many pages are we talking?
How automated do you need it?
How quickly and regularly?
How much were you intending to spend?
Do you intend to change what it crawls/extracts regularly?
What format do you need it in/output?
How reliable/ resilient does it need to be?
Want to do comparison of the crawls? Or need insights?


There is a full range..

[bishbash]

This may help http://scrapinghub.com/