Welcome to WebmasterWorld Guest from 54.159.64.172

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Hiding robot.txt
How to prevent robot.txt from beeing read by visitors
7 Spazio1999 1:00 pm Apr 3, 2004
Googlebot GET /robots.txt HTTP/1.0" 404
Googlebot is the only bot getting 404
1 eesnard 10:52 pm Apr 2, 2004
mozDex is spidering
FYI on the user agent
2 ByronM 9:18 pm Apr 1, 2004
WebmasterWorld disallowing Mozilla, Why?
If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
5 bufferzone 10:59 pm Mar 31, 2004
What does "root of domain" mean & what is 404 error?
root of domain 404 error
2 motherhen 8:14 am Mar 31, 2004
Will this robots.txt do it?
3 rover 5:10 am Mar 30, 2004
disallow: /*
robot.txt newbie, please help
2 emm0uk 11:15 pm Mar 29, 2004
robots.txt blocking foreign search engines
google robots.txt block foreign engines
1 jer61375 4:49 pm Mar 29, 2004
Allowing only search engines to crawl forums
Only want search engines in my forums
7 Psycho111 8:40 pm Mar 27, 2004
does a search engine care if a robots.txt file is 403 instead of 404
4 kahuna 6:20 pm Mar 27, 2004
robots.txt and multiple domain names
Using a Virtual Server...
1 nerolabs 7:45 pm Mar 26, 2004
google images ignoring disallow
2 apollo 10:43 pm Mar 25, 2004
Validation Error
Ran the robots.txt validation checker and recieved nothing but errors.
3 jseamless 5:06 pm Mar 25, 2004
Why use Robots - Again?
What are the benefits of using Robots.
4 oltc 4:48 pm Mar 25, 2004
How can I block Cyveillancebot(63.148.99.****)
It doesn't look like a useful bot...
6 itisgene 4:43 pm Mar 25, 2004
Yahoo-MMCrawler
1 simondev3 8:08 pm Mar 24, 2004
Referral log spamming: How use robots.txt to keep them out?
10 Dan_Vendel 12:36 pm Mar 24, 2004
robots.txt elementary question
3 Vespasian 5:29 am Mar 24, 2004
How to identify search engine robots?
What's the best way for ASP developers to identify robots?
1 vieth 3:48 pm Mar 23, 2004
Robot text for dissallow asp pages indexed
2 DivinoStore 8:06 am Mar 21, 2004
Anyone recognize this agent?
Can't find any info about it anywhere
2 martyt 4:15 pm Mar 20, 2004
A new bot?
Anyone else seeing this?
4 walrus 6:48 am Mar 19, 2004
Can't prevent php files from being searched
Is this possible
1 Jack_mcs 1:58 pm Mar 18, 2004
newbie robots.txt and forum Q
how to write it
6 buksida 3:27 am Mar 18, 2004
Robot.txt File
What is it's importance?
7 Enigmatic 8:52 pm Mar 16, 2004