Welcome to WebmasterWorld Guest from 54.226.60.235

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Spiders?!

Help me pls

     

TakeTown

6:04 pm on Mar 30, 2001 (gmt 0)



Well I know spiders are used by search engines...so will the spiders work correctly without a spider.txt? I have a site and I submitted to a bunch of search engines and there spiders will coming soon....anythng i need to do?
Please reply to admin@taketownrecords.com

Froggyman

11:27 pm on Mar 30, 2001 (gmt 0)



Don't worry, the spiders.txt is used to EXCLUDE robots and only if the robot obeys the Robot Exclusion Standard. In other words, your site is wide open for any spider to rummage through at will.

Froggyman

5:08 pm on Mar 31, 2001 (gmt 0)



Whoops! Thats robots.txt, see how you confused me:)
 

Featured Threads

Hot Threads This Week

Hot Threads This Month