Listing all pages on a website?

[tigger1]

Is it possible to do this? Or all files? Would be useful when browsing, as some sites aren't overly helpful about divulging whats actually on them...IYSWIM!

ta

[zaktoo]

Some sites have a "Site Map" link which should do almost what you're after - depending on what exactly it is you are after by wanting to see all the pages on the site of course...

Otherwise, try google: site:<domain name> <some keyword that would likely be on all pages>

[Silent1]

Just type in google "site:www.sitename.com" sans quotes and it will list all pages except those disallowed by robots.txt and private pages IIRC

i'm just not sure about the private pages bit

[zaktoo]

Silent1 said:

Just type in google "site:www.sitename.com" sans quotes and it will list all pages except those disallowed by robots.txt and private pages IIRC

i'm just not sure about the private pages bit

If they're password-protected they won't show (beyond the part where the password is asked for if you see what I mean), and I'm almost certin that the site: term needs to be followed by a search term as well.

Ciao

Zak

[Silent1]

zaktoo said:

Silent1 said:

Just type in google "site:www.sitename.com" sans quotes and it will list all pages except those disallowed by robots.txt and private pages IIRC

i'm just not sure about the private pages bit

If they're password-protected they won't show (beyond the part where the password is asked for if you see what I mean), and I'm almost certin that the site: term needs to be followed by a search term as well.

Ciao

Zak

nope see this link

just the website address

[zaktoo]

Silent1 said:

zaktoo said:

Silent1 said:

Just type in google "site:www.sitename.com" sans quotes and it will list all pages except those disallowed by robots.txt and private pages IIRC

i'm just not sure about the private pages bit

If they're password-protected they won't show (beyond the part where the password is asked for if you see what I mean), and I'm almost certin that the site: term needs to be followed by a search term as well.

Ciao

Zak

nope see this link

just the website address

Cool, it was bothersome having to try and guess a word likely to be on most of the pages. It definitely used to require a keyword as well.

[Silent1]

yeah i think it changed about a year ago, it certainly used to require a word, then it went down to a wildcard and now it's automatic

[slapmatt]

Silent1 said:

Just type in google "site:www.sitename.com" sans quotes and it will list all pages except those disallowed by robots.txt and private pages IIRC

i'm just not sure about the private pages bit

That query will only return pages that Google knows about - so if a site is not well indexed, or content has been added recently, it might not show up.

There are several sites that will produce an XML-style sitemap from a URL. This site is very good. It depends that you are trying to do with it.

[shocker-z]

Most bad link checkers should show you..

www.siteowner.com/badlinks.cfm

Regards
Liam

[tigger1]

Thanks guys, I'll havea play with this tonight and see if it does what I want it to.

[ThePassenger]

As well as the google route, one thing to try if is grabbing the sitemap from a domain. Often http://<domain>/sitemap.xml or sitemap.xml.gz will grab the raw feed for google and you can find all the interesting stuff.

[arcturus]

I've used Website Extractor to do this in the past.

[shocker-z]

If the page is unsecure then off a linux shell you can run wget -m www.example.com which mirrors the entire site but alot of servers protect against this now.


Liam