Welcome to WebmasterWorld Guest from 54.205.209.95

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Using Robot.txt to Block Spidering of Link pages
Is this a common underhanded practice?
12 jdancing 4:26 pm May 8, 2003
robots.txt
should i even bother if i want the whole site indexed?
9 madeonmoon 4:37 pm May 7, 2003
robots.txt
used for site with 2 urls
3 spinweb 2:11 am May 7, 2003
robots.txt validator warning suggestion
Missing newlines can cause problems, but in some cases can be detected.
2 mortenf 4:42 pm Apr 25, 2003
Dangers of the robots.txt file
A warning to the amateurs like myself
3 joeuz 6:37 am Apr 23, 2003
Robots.txt
Stopping robots visiting single dynamic page.
2 donovana 2:02 am Apr 22, 2003
Do search engines cache robots.txt?
10 trillianjedi 11:18 pm Apr 16, 2003
Site hacked and robots.txt placed, taken off google.
robots.txt, hacked, taken off google
3 Mohammed 10:43 am Apr 15, 2003
robots.txt
Disallow other URLs for one site.
3 spinweb 6:09 pm Apr 11, 2003
Question about simple robots.txt file
13 nadsab 12:56 pm Apr 10, 2003
Robots.txt
Is this valid?
5 Alternative_Future 9:52 am Apr 3, 2003
Get a Robot
Get the robot file
7 Glenn 2:45 pm Apr 2, 2003
robots.txt question
2 deltron23 9:01 pm Mar 28, 2003
External Javascript and bots and robots.txt?
2 JeremyL 6:07 pm Mar 12, 2003
Robot.txt and Templates
The <meta>Robot tag in templates, with specific disallow in robot.txt
5 juniperwasting 4:12 pm Mar 11, 2003
My Robots.txt Screw Up.
case study in how to kill well performing serps in 1 easy step
5 webdevsf 1:45 am Mar 8, 2003
How do I apply the robots.txt?
Seems important and need to know more!
3 askjoe 6:24 pm Mar 7, 2003
I don't want to be listed yet
Is disabling anonomas access enough?
8 currington 9:53 am Mar 5, 2003
empty robots.txt
yes or no
6 HuhuFruFru 8:32 am Mar 5, 2003
question about an empty robots.txt
5 HuhuFruFru 8:36 pm Mar 4, 2003
logs and robots.txt
7 tyrojds 7:42 pm Mar 4, 2003
Why robots.txt if you want everything indexed?
Does it even matter?
9 Larry_C 3:49 pm Mar 3, 2003
Robots.txt question
excluding every folder expect one
5 JVB_Mktg 7:54 pm Mar 2, 2003
Logs show a code 206 with robots.txt
code 206 - partial content
2 nativenewyorker 11:03 pm Mar 1, 2003
robots.txt syntax
spaces or slashes?
3 keyplyr 5:14 pm Feb 27, 2003