homepage Welcome to WebmasterWorld Guest from 54.198.148.191
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / Search Engines / Alternative Search Engines
Forum Library, Charter, Moderators: bakedjake

Alternative Search Engines Forum

    
Any Good Search Engines That Don't Respect robots.txt ?
Searching for information on a site blocked by robots.txt
bouncybunny




msg:3927824
 1:50 pm on Jun 6, 2009 (gmt 0)

This may be asking too much, but are there any good search engines that ignore the disallow command in robots.txt?

I want to search the contents of a public website that has excluded part of its content with disallow in the robots file. Their internal site-search is appalling, so I'm hoping that there is a half-decent - yet non-spammy - search engine that will allow me to find some content on the site.

 

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Alternative Search Engines
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved