Welcome to WebmasterWorld Guest from 188.8.131.52
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|proper way of changing site
||2||jmdb71||5:11 pm May 1, 2004|
|How to get spider to visit parts of site more frequently
Is this possible?
|5||Pimpernel||10:35 am Apr 30, 2004|
||2||zyshen||5:19 pm Apr 27, 2004|
|Google ignoring robots.txt
I'm seeing lots of my 'Disallowed' pages in their index
|4||Amygdala||4:54 am Apr 24, 2004|
|Strange bot keep on visiting my site and don't obey my robots.txt file
What to do?
|2||Compworld||8:01 pm Apr 22, 2004|
|revisit after so many days
||2||tommygunner||3:21 am Apr 21, 2004|
|index.php vs index.html
Will This Work i Wonder
|2||yummycarp||11:28 pm Apr 20, 2004|
|Banning a script using robots.txt
how do do this
|4||Imaster||9:51 pm Apr 20, 2004|
|Robots.txt - Why?
||3||warlordbb||2:08 pm Apr 20, 2004|
|Can IMDB do this?
charge per unwanted request
|3||jrap||10:19 pm Apr 19, 2004|
Google keeps crawling page
|2||anxvariety||2:27 pm Apr 19, 2004|
|Froogle wants my images!
||3||seadog||5:59 am Apr 14, 2004|
|META Robots no index tag
||2||BMach||2:36 am Apr 14, 2004|
|Using wild card in robots
Can you add * to ip?
|2||walrus||5:37 pm Apr 13, 2004|
|How to ban inktomisearch.com's bots?
I have been hit hard with so many requests by this.
|3||itisgene||5:24 pm Apr 11, 2004|
||4||decdim||12:12 am Apr 11, 2004|
|New to robots.txt and how spider find robots
||9||jazzvn||11:49 pm Apr 10, 2004|
|Will the following robots.txt disallow all robots BUT Google?
||3||georgec||1:15 pm Apr 9, 2004|
best way to accomplish
|2||IONWeb||7:18 am Apr 8, 2004|
|new to robots
is this right?
|6||sara2||5:38 pm Apr 5, 2004|
How to prevent robot.txt from beeing read by visitors
|7||Spazio1999||1:00 pm Apr 3, 2004|
|Googlebot GET /robots.txt HTTP/1.0" 404
Googlebot is the only bot getting 404
|1||eesnard||10:52 pm Apr 2, 2004|
|mozDex is spidering
FYI on the user agent
|2||ByronM||9:18 pm Apr 1, 2004|
|WebmasterWorld disallowing Mozilla, Why?
If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
|5||bufferzone||10:59 pm Mar 31, 2004|
|What does "root of domain" mean & what is 404 error?
root of domain 404 error
|2||motherhen||8:14 am Mar 31, 2004|