Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies


Help me pls



6:04 pm on Mar 30, 2001 (gmt 0)

Well I know spiders are used by search engines...so will the spiders work correctly without a spider.txt? I have a site and I submitted to a bunch of search engines and there spiders will coming soon....anythng i need to do?
Please reply to admin@taketownrecords.com


11:27 pm on Mar 30, 2001 (gmt 0)

Don't worry, the spiders.txt is used to EXCLUDE robots and only if the robot obeys the Robot Exclusion Standard. In other words, your site is wide open for any spider to rummage through at will.


5:08 pm on Mar 31, 2001 (gmt 0)

Whoops! Thats robots.txt, see how you confused me:)

Featured Threads

Hot Threads This Week

Hot Threads This Month