Welcome to WebmasterWorld Guest from 22.214.171.124
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|META Robots no index tag
||2||BMach||2:36 am Apr 14, 2004|
|Using wild card in robots
Can you add * to ip?
|2||walrus||5:37 pm Apr 13, 2004|
|How to ban inktomisearch.com's bots?
I have been hit hard with so many requests by this.
|3||itisgene||5:24 pm Apr 11, 2004|
||4||decdim||12:12 am Apr 11, 2004|
|New to robots.txt and how spider find robots
||9||jazzvn||11:49 pm Apr 10, 2004|
|Will the following robots.txt disallow all robots BUT Google?
||3||georgec||1:15 pm Apr 9, 2004|
best way to accomplish
|2||IONWeb||7:18 am Apr 8, 2004|
|new to robots
is this right?
|6||sara2||5:38 pm Apr 5, 2004|
How to prevent robot.txt from beeing read by visitors
|7||Spazio1999||1:00 pm Apr 3, 2004|
|Googlebot GET /robots.txt HTTP/1.0" 404
Googlebot is the only bot getting 404
|1||eesnard||10:52 pm Apr 2, 2004|
|mozDex is spidering
FYI on the user agent
|2||ByronM||9:18 pm Apr 1, 2004|
|WebmasterWorld disallowing Mozilla, Why?
If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
|5||bufferzone||10:59 pm Mar 31, 2004|
|What does "root of domain" mean & what is 404 error?
root of domain 404 error
|2||motherhen||8:14 am Mar 31, 2004|
|Will this robots.txt do it?
||3||rover||5:10 am Mar 30, 2004|
robot.txt newbie, please help
|2||emm0uk||11:15 pm Mar 29, 2004|
|robots.txt blocking foreign search engines
google robots.txt block foreign engines
|1||jer61375||4:49 pm Mar 29, 2004|
|Allowing only search engines to crawl forums
Only want search engines in my forums
|7||Psycho111||8:40 pm Mar 27, 2004|
|does a search engine care if a robots.txt file is 403 instead of 404
||4||kahuna||6:20 pm Mar 27, 2004|
|robots.txt and multiple domain names
Using a Virtual Server...
|1||nerolabs||7:45 pm Mar 26, 2004|
|google images ignoring disallow
||2||apollo||10:43 pm Mar 25, 2004|
Ran the robots.txt validation checker and recieved nothing but errors.
|3||jseamless||5:06 pm Mar 25, 2004|
|Why use Robots - Again?
What are the benefits of using Robots.
|4||oltc||4:48 pm Mar 25, 2004|
|How can I block Cyveillancebot(63.148.99.****)
It doesn't look like a useful bot...
|6||itisgene||4:43 pm Mar 25, 2004|
||1||simondev3||8:08 pm Mar 24, 2004|
|Referral log spamming: How use robots.txt to keep them out?
||10||Dan_Vendel||12:36 pm Mar 24, 2004|