| Any Good Search Engines That Don't Respect robots.txt ?
Searching for information on a site blocked by robots.txt |
bouncybunny
msg:3927824 | 1:50 pm on Jun 6, 2009 (gmt 0) |
This may be asking too much, but are there any good search engines that ignore the disallow command in robots.txt? I want to search the contents of a public website that has excluded part of its content with disallow in the robots file. Their internal site-search is appalling, so I'm hoping that there is a half-decent - yet non-spammy - search engine that will allow me to find some content on the site.
|
|
