Welcome to WebmasterWorld Guest from 54.145.53.251

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Spiders?!

Help me pls

     

TakeTown

6:04 pm on Mar 30, 2001 (gmt 0)

Inactive Member
Account Expired

 
 


Well I know spiders are used by search engines...so will the spiders work correctly without a spider.txt? I have a site and I submitted to a bunch of search engines and there spiders will coming soon....anythng i need to do?
Please reply to admin@taketownrecords.com
11:27 pm on Mar 30, 2001 (gmt 0)

Full Member

joined:Jan 24, 2001
posts:230
votes: 0


Don't worry, the spiders.txt is used to EXCLUDE robots and only if the robot obeys the Robot Exclusion Standard. In other words, your site is wide open for any spider to rummage through at will.
5:08 pm on Mar 31, 2001 (gmt 0)

Full Member

joined:Jan 24, 2001
posts:230
votes: 0


Whoops! Thats robots.txt, see how you confused me:)