Welcome to WebmasterWorld Guest from 54.144.79.200

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Spiders?!

Help me pls

     

TakeTown

6:04 pm on Mar 30, 2001 (gmt 0)

10+ Year Member



Well I know spiders are used by search engines...so will the spiders work correctly without a spider.txt? I have a site and I submitted to a bunch of search engines and there spiders will coming soon....anythng i need to do?
Please reply to admin@taketownrecords.com

Froggyman

11:27 pm on Mar 30, 2001 (gmt 0)



Don't worry, the spiders.txt is used to EXCLUDE robots and only if the robot obeys the Robot Exclusion Standard. In other words, your site is wide open for any spider to rummage through at will.

Froggyman

5:08 pm on Mar 31, 2001 (gmt 0)



Whoops! Thats robots.txt, see how you confused me:)