How do aggregators work?
Author
Discussion

zoidberg

Original Poster:

8,893 posts

313 months

Wednesday 11th July 2007
quotequote all
How does a site like Kelkoo actually work - I mean in terms of technology?

User types in a search query -> Kelkoo does some stuff -> User sees summary from loads of different websites

What happens in the middle bit? What technology actually allows Kelkoo to perform (remote?) searches on other sites?

Cheers

Plotloss

67,280 posts

301 months

Wednesday 11th July 2007
quotequote all
Kelkoo is updated by organisations that participate in it, just like PriceRunner and Froogle.

Froogle does try to spider and aggregate but the best results are from feeding it.

zoidberg

Original Poster:

8,893 posts

313 months

Wednesday 11th July 2007
quotequote all
Do you mean that Kelkoo locally stores a copy of all the partner data, so only has to search within its own DBs?

In terms of spidering, how does that work?

Plotloss

67,280 posts

301 months

Wednesday 11th July 2007
quotequote all
You submit todays prices of your portfolio to Kelkoo and Kelkoo works off that data - yes.

With regards to spidering it works like the Google text spider but is tuned to look for product names, descriptions and costs and then assumes that its a product. Its hit and miss which is why there is the feed upload mechanism.

zoidberg

Original Poster:

8,893 posts

313 months

Wednesday 11th July 2007
quotequote all
(Sorry to labour the point).

How do spiders work though? What language are they written in? Are they hard to "make"?

dilbert

7,741 posts

262 months

Wednesday 11th July 2007
quotequote all
zoidberg said:
(Sorry to labour the point).

How do spiders work though? What language are they written in? Are they hard to "make"?
http://en.wikipedia.org/wiki/Web_crawler 

slapmatt

1,132 posts

253 months

Wednesday 11th July 2007
quotequote all
E-commerce sites submit XML feeds of their products, in a format specified by the comparison site. The comparisn site then loads the data into their own databases and effectively search on that. Froogle blew the whole market open by allowing sites to submit their XML feeds for free. AFAIK, Froogle doesn't "crawl" for feeds, you have to submit to them.