homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Alternative Search Engines
Forum Library, Charter, Moderators: bakedjake

Alternative Search Engines Forum

Any Good Search Engines That Don't Respect robots.txt ?
Searching for information on a site blocked by robots.txt

WebmasterWorld Senior Member 5+ Year Member

Msg#: 3927822 posted 1:50 pm on Jun 6, 2009 (gmt 0)

This may be asking too much, but are there any good search engines that ignore the disallow command in robots.txt?

I want to search the contents of a public website that has excluded part of its content with disallow in the robots file. Their internal site-search is appalling, so I'm hoping that there is a half-decent - yet non-spammy - search engine that will allow me to find some content on the site.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Alternative Search Engines
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved