Welcome to WebmasterWorld Guest from 18.104.22.168
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|the wrong way to do robots.txt
kinda funny, whitehouse.gov robots file
|13||amznVibe||7:42 pm Nov 10, 2003|
Is it worth blocking spambots with robots.txt
|4||WyrdoManx||11:52 pm Nov 8, 2003|
|Does content type matter?
is text/html okay
|2||kevinsl||8:27 am Nov 8, 2003|
|IE "Synchronize Offline while idle" is killing me
|3||Helice||12:46 am Nov 6, 2003|
|Bots ignoring my robots.txt
In place for a week, validates, but no use
|12||roddy||7:10 am Nov 5, 2003|
|Blocking a spider from a directory
||3||JVB_Mktg||4:30 am Nov 5, 2003|
To use... or not to use? That is the question
|6||panic||1:29 am Nov 5, 2003|
We're sorry, this robots.txt does NOT validate.
|2||midge||12:48 pm Nov 3, 2003|
|Robot.txt - What is it, and why use it
||4||laffe||4:36 am Nov 2, 2003|
How to block a directory.
|6||jdancing||7:52 pm Oct 31, 2003|
|Robots.txt during development
Anyone advise on this?
|2||stuntdubl||3:30 am Oct 30, 2003|
||3||cgj5204||4:40 am Oct 29, 2003|
|Restrict Specific Domain Names from Being Indexed
Using the robots.txt file on Google to restrict certain domains...
|3||jmh1970||1:36 am Oct 29, 2003|
|New Forum : Robots.txt
||1||Brett_Tabke||9:24 pm Oct 28, 2003|
|Put your robots.txt on a diet
Three easy ways to reduce bloat in robots.txt
|20||jdMorgan||8:36 pm Oct 28, 2003|
|Blocking image file indexing without robots.txt
Can this be done?
|2||Winedark||8:08 pm Oct 28, 2003|
How long can a robots.txt be?
|4||WyrdoManx||9:47 pm Oct 22, 2003|
|Robots.txt - Am I Missing Somthing?
Query strings and general syntax is suspected...
|13||Nick_W||3:40 pm Oct 21, 2003|
|How to correctly set robots.txt for a subdomain?
|5||PatrickDeese||6:36 pm Oct 15, 2003|
|Robots.txt for Query Strings
Can I stop bots from following links with query stings
|10||Nick_W||6:08 pm Oct 15, 2003|
Crawling no longer allowed
|12||Yidaki||5:38 pm Oct 14, 2003|
|Robots.txt Question :)
Question regarding how to use robots txt
|8||Prana||9:33 pm Sep 27, 2003|
3 domains in root
|9||Alternative_Future||11:33 pm Sep 23, 2003|
|Robots.txt disallow: /index.php?
then /index.php?param=example still allowed?
|5||Yidaki||12:17 pm Sep 21, 2003|
|Robots.txt exclusion question
Robots.txt exclude all but one directory?
|4||olof_bailey||8:52 pm Sep 20, 2003|