Welcome to WebmasterWorld Guest from

Forum Moderators: bakedjake

Message Too Old, No Replies

Any Good Search Engines That Don't Respect robots.txt ?

Searching for information on a site blocked by robots.txt

1:50 pm on Jun 6, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 10, 2006
votes: 0

This may be asking too much, but are there any good search engines that ignore the disallow command in robots.txt?

I want to search the contents of a public website that has excluded part of its content with disallow in the robots file. Their internal site-search is appalling, so I'm hoping that there is a half-decent - yet non-spammy - search engine that will allow me to find some content on the site.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members