Welcome to WebmasterWorld Guest from 22.214.171.124
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|How often should a robot check the robots.txt file?
Your opinions please
|8||jrobbio||2:25 am May 18, 2003|
|Using Robot.txt to Block Spidering of Link pages
Is this a common underhanded practice?
|12||jdancing||4:26 pm May 8, 2003|
should i even bother if i want the whole site indexed?
|9||madeonmoon||4:37 pm May 7, 2003|
used for site with 2 urls
|3||spinweb||2:11 am May 7, 2003|
|robots.txt validator warning suggestion
Missing newlines can cause problems, but in some cases can be detected.
|2||mortenf||4:42 pm Apr 25, 2003|
|Dangers of the robots.txt file
A warning to the amateurs like myself
|3||joeuz||6:37 am Apr 23, 2003|
Stopping robots visiting single dynamic page.
|2||donovana||2:02 am Apr 22, 2003|
|Do search engines cache robots.txt?
||10||trillianjedi||11:18 pm Apr 16, 2003|
|Site hacked and robots.txt placed, taken off google.
robots.txt, hacked, taken off google
|3||Mohammed||10:43 am Apr 15, 2003|
Disallow other URLs for one site.
|3||spinweb||6:09 pm Apr 11, 2003|
|Question about simple robots.txt file
||13||nadsab||12:56 pm Apr 10, 2003|
Is this valid?
|5||Alternative_Future||9:52 am Apr 3, 2003|
|Get a Robot
Get the robot file
|7||Glenn||2:45 pm Apr 2, 2003|
||2||deltron23||9:01 pm Mar 28, 2003|
||2||JeremyL||6:07 pm Mar 12, 2003|
|Robot.txt and Templates
The <meta>Robot tag in templates, with specific disallow in robot.txt
|5||juniperwasting||4:12 pm Mar 11, 2003|
|My Robots.txt Screw Up.
case study in how to kill well performing serps in 1 easy step
|5||webdevsf||1:45 am Mar 8, 2003|
|How do I apply the robots.txt?
Seems important and need to know more!
|3||askjoe||6:24 pm Mar 7, 2003|
|I don't want to be listed yet
Is disabling anonomas access enough?
|8||currington||9:53 am Mar 5, 2003|
yes or no
|6||HuhuFruFru||8:32 am Mar 5, 2003|
|question about an empty robots.txt
||5||HuhuFruFru||8:36 pm Mar 4, 2003|
|logs and robots.txt
||7||tyrojds||7:42 pm Mar 4, 2003|
|Why robots.txt if you want everything indexed?
Does it even matter?
|9||Larry_C||3:49 pm Mar 3, 2003|
excluding every folder expect one
|5||JVB_Mktg||7:54 pm Mar 2, 2003|
|Logs show a code 206 with robots.txt
code 206 - partial content
|2||nativenewyorker||11:03 pm Mar 1, 2003|