Download an entire website

[miniman]

Anyone know of a bit of software that will start at the homepage of a website, and then automatically follow all the links within the site, and save each page as it finds them??? Sure there used to be something out there, the name "snake" springs to mind for some reason.

[billb]

seems to be lots if u google eg:

www.surfoffline.com/

[agent006]

I use a program called WinHTTRACK which is very good. Free AFAIK too.

[zumbruk]

The definitive program for doing this is "wget".

The GNU version is at; www.gnu.org/software/wget/wget.html

There's a Windows port at; www.interlog.com/~tcharron/wgetwin.html

[SlvrCrb]

unless there's a robots exclusion for the site e.g

User-agent: WebCopy
Disallow: /


type www.xyzurl.com/robots.txt and see what it says under the user-agents, most sites don't bother...but I do

[zumbruk]

robots.txt is advisory, not compulsory. You can ignore it without any (technical) problems.

[miniman]

billb said:

seems to be lots if u google eg: www.surfoffline.com/

Spot on - many thanks