Welcome to WebmasterWorld Guest from 50.17.5.36

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
robots.txt validator warning suggestion
Missing newlines can cause problems, but in some cases can be detected.
2 mortenf 4:42 pm Apr 25, 2003
Dangers of the robots.txt file
A warning to the amateurs like myself
3 joeuz 6:37 am Apr 23, 2003
Robots.txt
Stopping robots visiting single dynamic page.
2 donovana 2:02 am Apr 22, 2003
Do search engines cache robots.txt?
10 trillianjedi 11:18 pm Apr 16, 2003
Site hacked and robots.txt placed, taken off google.
robots.txt, hacked, taken off google
3 Mohammed 10:43 am Apr 15, 2003
robots.txt
Disallow other URLs for one site.
3 spinweb 6:09 pm Apr 11, 2003
Question about simple robots.txt file
13 nadsab 12:56 pm Apr 10, 2003
Robots.txt
Is this valid?
5 Alternative_Future 9:52 am Apr 3, 2003
Get a Robot
Get the robot file
7 Glenn 2:45 pm Apr 2, 2003
robots.txt question
2 deltron23 9:01 pm Mar 28, 2003
External Javascript and bots and robots.txt?
2 JeremyL 6:07 pm Mar 12, 2003
Robot.txt and Templates
The <meta>Robot tag in templates, with specific disallow in robot.txt
5 juniperwasting 4:12 pm Mar 11, 2003
My Robots.txt Screw Up.
case study in how to kill well performing serps in 1 easy step
5 webdevsf 1:45 am Mar 8, 2003
How do I apply the robots.txt?
Seems important and need to know more!
3 askjoe 6:24 pm Mar 7, 2003
I don't want to be listed yet
Is disabling anonomas access enough?
8 currington 9:53 am Mar 5, 2003
empty robots.txt
yes or no
6 HuhuFruFru 8:32 am Mar 5, 2003
question about an empty robots.txt
5 HuhuFruFru 8:36 pm Mar 4, 2003
logs and robots.txt
7 tyrojds 7:42 pm Mar 4, 2003
Why robots.txt if you want everything indexed?
Does it even matter?
9 Larry_C 3:49 pm Mar 3, 2003
Robots.txt question
excluding every folder expect one
5 JVB_Mktg 7:54 pm Mar 2, 2003
Logs show a code 206 with robots.txt
code 206 - partial content
2 nativenewyorker 11:03 pm Mar 1, 2003
robots.txt syntax
spaces or slashes?
3 keyplyr 5:14 pm Feb 27, 2003
Strange Incident with robots.txt
error code 401
3 neh2008 4:57 pm Feb 17, 2003
robots.txt to block bots access to images
got to save bandwidth
4 stavs 3:00 pm Feb 14, 2003
ROBOTS.TXT Location
ROBOTS.TXT Location
4 neh2008 5:27 pm Feb 11, 2003