Welcome to WebmasterWorld Guest from 54.205.96.97

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Validation Error
4 decdim 12:12 am Apr 11, 2004
New to robots.txt and how spider find robots
9 jazzvn 11:49 pm Apr 10, 2004
Will the following robots.txt disallow all robots BUT Google?
3 georgec 1:15 pm Apr 9, 2004
Bot blocking
best way to accomplish
2 IONWeb 7:18 am Apr 8, 2004
new to robots
is this right?
6 sara2 5:38 pm Apr 5, 2004
Hiding robot.txt
How to prevent robot.txt from beeing read by visitors
7 Spazio1999 1:00 pm Apr 3, 2004
Googlebot GET /robots.txt HTTP/1.0" 404
Googlebot is the only bot getting 404
1 eesnard 10:52 pm Apr 2, 2004
mozDex is spidering
FYI on the user agent
2 ByronM 9:18 pm Apr 1, 2004
WebmasterWorld disallowing Mozilla, Why?
If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
5 bufferzone 10:59 pm Mar 31, 2004
What does "root of domain" mean & what is 404 error?
root of domain 404 error
2 motherhen 8:14 am Mar 31, 2004
Will this robots.txt do it?
3 rover 5:10 am Mar 30, 2004
disallow: /*
robot.txt newbie, please help
2 emm0uk 11:15 pm Mar 29, 2004
robots.txt blocking foreign search engines
google robots.txt block foreign engines
1 jer61375 4:49 pm Mar 29, 2004
Allowing only search engines to crawl forums
Only want search engines in my forums
7 Psycho111 8:40 pm Mar 27, 2004
does a search engine care if a robots.txt file is 403 instead of 404
4 kahuna 6:20 pm Mar 27, 2004
robots.txt and multiple domain names
Using a Virtual Server...
1 nerolabs 7:45 pm Mar 26, 2004
google images ignoring disallow
2 apollo 10:43 pm Mar 25, 2004
Validation Error
Ran the robots.txt validation checker and recieved nothing but errors.
3 jseamless 5:06 pm Mar 25, 2004
Why use Robots - Again?
What are the benefits of using Robots.
4 oltc 4:48 pm Mar 25, 2004
How can I block Cyveillancebot(63.148.99.****)
It doesn't look like a useful bot...
6 itisgene 4:43 pm Mar 25, 2004
Yahoo-MMCrawler
1 simondev3 8:08 pm Mar 24, 2004
Referral log spamming: How use robots.txt to keep them out?
10 Dan_Vendel 12:36 pm Mar 24, 2004
robots.txt elementary question
3 Vespasian 5:29 am Mar 24, 2004
How to identify search engine robots?
What's the best way for ASP developers to identify robots?
1 vieth 3:48 pm Mar 23, 2004
Robot text for dissallow asp pages indexed
2 DivinoStore 8:06 am Mar 21, 2004

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month