Welcome to WebmasterWorld Guest from 22.214.171.124
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
||5||payday||11:26 am Nov 22, 2003|
|Importance of Hotbot
||5||slobizman||9:08 am Nov 20, 2003|
|How long until robots.txt works?
||11||slobizman||3:23 pm Nov 18, 2003|
crawl, spider, and robot.
|2||synergy||4:10 am Nov 16, 2003|
|Would crawlers crawl this?
Usiing the correct robots.txt technique
|4||ulysee||3:02 pm Nov 15, 2003|
billing for requested data
|14||plumsauce||7:47 pm Nov 14, 2003|
|Pros and cons of having
a robots.txt file?
|3||ideavirus||5:32 pm Nov 13, 2003|
|Does anyone know what this is?
I am thinking virus infected machine but unsure
|3||Crowbar||4:28 pm Nov 12, 2003|
|Blocking .log file with robots.txt
Is it necessary, and syntax...?
|7||Robert_Charlton||6:59 am Nov 11, 2003|
|How to make a page *inaccessible* to searches
without using password protection?
|12||halfandhalf||8:04 pm Nov 10, 2003|
|the wrong way to do robots.txt
kinda funny, whitehouse.gov robots file
|13||amznVibe||7:42 pm Nov 10, 2003|
Is it worth blocking spambots with robots.txt
|4||WyrdoManx||11:52 pm Nov 8, 2003|
|Does content type matter?
is text/html okay
|2||kevinsl||8:27 am Nov 8, 2003|
|IE "Synchronize Offline while idle" is killing me
|3||Helice||12:46 am Nov 6, 2003|
|Bots ignoring my robots.txt
In place for a week, validates, but no use
|12||roddy||7:10 am Nov 5, 2003|
|Blocking a spider from a directory
||3||JVB_Mktg||4:30 am Nov 5, 2003|
To use... or not to use? That is the question
|6||panic||1:29 am Nov 5, 2003|
We're sorry, this robots.txt does NOT validate.
|2||midge||12:48 pm Nov 3, 2003|
|Robot.txt - What is it, and why use it
||4||laffe||4:36 am Nov 2, 2003|
How to block a directory.
|6||jdancing||7:52 pm Oct 31, 2003|
|Robots.txt during development
Anyone advise on this?
|2||stuntdubl||3:30 am Oct 30, 2003|
||3||cgj5204||4:40 am Oct 29, 2003|
|Restrict Specific Domain Names from Being Indexed
Using the robots.txt file on Google to restrict certain domains...
|3||jmh1970||1:36 am Oct 29, 2003|
|New Forum : Robots.txt
||1||Brett_Tabke||9:24 pm Oct 28, 2003|
|Put your robots.txt on a diet
Three easy ways to reduce bloat in robots.txt
|20||jdMorgan||8:36 pm Oct 28, 2003|