homepage Welcome to WebmasterWorld Guest from 107.20.25.215
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Spiders?!
Help me pls
TakeTown




msg:408406
 6:04 pm on Mar 30, 2001 (gmt 0)

Well I know spiders are used by search engines...so will the spiders work correctly without a spider.txt? I have a site and I submitted to a bunch of search engines and there spiders will coming soon....anythng i need to do?
Please reply to admin@taketownrecords.com

 

Froggyman




msg:408407
 11:27 pm on Mar 30, 2001 (gmt 0)

Don't worry, the spiders.txt is used to EXCLUDE robots and only if the robot obeys the Robot Exclusion Standard. In other words, your site is wide open for any spider to rummage through at will.

Froggyman




msg:408408
 5:08 pm on Mar 31, 2001 (gmt 0)

Whoops! Thats robots.txt, see how you confused me:)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved