Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
||3||ktd121||3:14 am Jul 4, 2003|
|meta name = robot
What does that do?
|7||aroye||5:57 pm Jun 30, 2003|
|Scooter 3.3 ignores robots.txt?
this is ridiculous
|6||berli||2:39 am Jun 27, 2003|
Help please mine is missing?
|3||peteredmonds||8:37 pm Jun 25, 2003|
|2||zoobie||6:52 pm Jun 19, 2003|
What should it be set at?
|4||kimderella86||6:22 pm Jun 18, 2003|
|Robots.txt exclusion for dynamic page
||2||rogerd||9:22 pm Jun 17, 2003|
|Robots.txt & How your site is spidered
Positives of Robots.txt file aside from disallowing bots
|4||floridadesigns||11:31 pm Jun 14, 2003|
|Why should I obey robots.txt
|22||PWalker||1:01 am Jun 13, 2003|
|Php robots.txt parser script
does that exitst? if not regex help needed.
|5||ikbenhet1||2:57 pm Jun 9, 2003|
|files requested by robots.txt
i would like to find out about the files requested by robots
|8||andrea_edwards||7:53 pm Jun 8, 2003|
|Toolbar and Robots.txt
||8||rintrah||8:41 pm Jun 5, 2003|
|Robots.txt - files in folders
||3||rintrah||9:43 pm Jun 4, 2003|
|6||copongcopong||9:11 pm Jun 4, 2003|
|Robots.txt - What's the Point?
||7||rintrah||6:59 pm May 30, 2003|
|Pragma & Cache-Control
Are these Safe with Search Engines
|9||stuart||8:32 pm May 27, 2003|
|Quick robots.txt question
Do I just need to put the file name
|2||bateman_ap||8:21 pm May 24, 2003|
|Using Allow: / in robots.txt
Google says to use it?
|5||werty||3:28 am May 20, 2003|
|How often should a robot check the robots.txt file?
Your opinions please
|8||jrobbio||2:25 am May 18, 2003|
|Using Robot.txt to Block Spidering of Link pages
Is this a common underhanded practice?
|12||jdancing||4:26 pm May 8, 2003|
should i even bother if i want the whole site indexed?
|9||madeonmoon||4:37 pm May 7, 2003|
used for site with 2 urls
|3||spinweb||2:11 am May 7, 2003|
|robots.txt validator warning suggestion
Missing newlines can cause problems, but in some cases can be detected.
|2||mortenf||4:42 pm Apr 25, 2003|
|Dangers of the robots.txt file
A warning to the amateurs like myself
|3||joeuz||6:37 am Apr 23, 2003|
Stopping robots visiting single dynamic page.
|2||donovana||2:02 am Apr 22, 2003|