Welcome to WebmasterWorld Guest from 54.160.131.144

Forum Moderators: bakedjake

Message Too Old, No Replies

Any Good Search Engines That Don't Respect robots.txt ?

Searching for information on a site blocked by robots.txt

     

bouncybunny

1:50 pm on Jun 6, 2009 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



This may be asking too much, but are there any good search engines that ignore the disallow command in robots.txt?

I want to search the contents of a public website that has excluded part of its content with disallow in the robots file. Their internal site-search is appalling, so I'm hoping that there is a half-decent - yet non-spammy - search engine that will allow me to find some content on the site.