Forum Moderators: open

Message Too Old, No Replies

Spiders?!

Help me pls

         

TakeTown

6:04 pm on Mar 30, 2001 (gmt 0)



Well I know spiders are used by search engines...so will the spiders work correctly without a spider.txt? I have a site and I submitted to a bunch of search engines and there spiders will coming soon....anythng i need to do?
Please reply to admin@taketownrecords.com

Froggyman

11:27 pm on Mar 30, 2001 (gmt 0)



Don't worry, the spiders.txt is used to EXCLUDE robots and only if the robot obeys the Robot Exclusion Standard. In other words, your site is wide open for any spider to rummage through at will.

Froggyman

5:08 pm on Mar 31, 2001 (gmt 0)



Whoops! Thats robots.txt, see how you confused me:)