Bots, Spiders, databases and reporting

[mondeoman]

Does anyone know anything about bots/spiders?

I've got a commercial project that I could use some professional help with......

email me if you're interested

[mondeoman]

bump

[BliarOut]

Are you looking for bot traps and stopping slurpers etc? I have a limited bit of knowledge on these at home and I'll dig out the link if you're interested.

[Plotloss]

No direct experience but there is a crawler/spider class in .NET called Charlotte.

I suppose anyone familiar with either vb.NET or C# could utilise it...

[mondeoman]

I need something to trawl the net and give me a daily report on changes/additions to pages that meet specific criteria.

First I need to know if its feasible really, but I cant think of a reason why it isn't.

[Plotloss]

Yes it is technically possible.

Its easy if you know the direct name of the page you are accessing and its static HTML

If the page is live its more tricky but still feasible.

[dilbert]

What sort of budget are you looking at?

I know of a number of companies already in that business. You have to be pretty careful though. If you buy their advertised products, you'll probably not get what you want. On the other hand if you specify what you want you'll probably not afford it.

Do you have a specification of any kind?

[mondeoman]

Plotloss said:

Yes it is technically possible. Its easy if you know the direct name of the page you are accessing and its static HTML If the page is live its more tricky but still feasible.

YHM