Welcome to WebmasterWorld Guest from

Forum Moderators: bakedjake

Message Too Old, No Replies

Any Good Search Engines That Don't Respect robots.txt ?

Searching for information on a site blocked by robots.txt

1:50 pm on Jun 6, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 10, 2006
votes: 0

This may be asking too much, but are there any good search engines that ignore the disallow command in robots.txt?

I want to search the contents of a public website that has excluded part of its content with disallow in the robots file. Their internal site-search is appalling, so I'm hoping that there is a half-decent - yet non-spammy - search engine that will allow me to find some content on the site.