Welcome to WebmasterWorld Guest from 54.221.119.45

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
proper way of changing site
2 jmdb71 5:11 pm May 1, 2004
How to get spider to visit parts of site more frequently
Is this possible?
5 Pimpernel 10:35 am Apr 30, 2004
robots.txt missing?
2 zyshen 5:19 pm Apr 27, 2004
Google ignoring robots.txt
I'm seeing lots of my 'Disallowed' pages in their index
4 Amygdala 4:54 am Apr 24, 2004
Strange bot keep on visiting my site and don't obey my robots.txt file
What to do?
2 Compworld 8:01 pm Apr 22, 2004
revisit after so many days
2 tommygunner 3:21 am Apr 21, 2004
index.php vs index.html
Will This Work i Wonder
2 yummycarp 11:28 pm Apr 20, 2004
Banning a script using robots.txt
how do do this
4 Imaster 9:51 pm Apr 20, 2004
Robots.txt - Why?
3 warlordbb 2:08 pm Apr 20, 2004
Can IMDB do this?
charge per unwanted request
3 jrap 10:19 pm Apr 19, 2004
Robots.txt exclusion
Google keeps crawling page
2 anxvariety 2:27 pm Apr 19, 2004
Froogle wants my images!
3 seadog 5:59 am Apr 14, 2004
META Robots no index tag
2 BMach 2:36 am Apr 14, 2004
Using wild card in robots
Can you add * to ip?
2 walrus 5:37 pm Apr 13, 2004
How to ban inktomisearch.com's bots?
I have been hit hard with so many requests by this.
3 itisgene 5:24 pm Apr 11, 2004
Validation Error
4 decdim 12:12 am Apr 11, 2004
New to robots.txt and how spider find robots
9 jazzvn 11:49 pm Apr 10, 2004
Will the following robots.txt disallow all robots BUT Google?
3 georgec 1:15 pm Apr 9, 2004
Bot blocking
best way to accomplish
2 IONWeb 7:18 am Apr 8, 2004
new to robots
is this right?
6 sara2 5:38 pm Apr 5, 2004
Hiding robot.txt
How to prevent robot.txt from beeing read by visitors
7 Spazio1999 1:00 pm Apr 3, 2004
Googlebot GET /robots.txt HTTP/1.0" 404
Googlebot is the only bot getting 404
1 eesnard 10:52 pm Apr 2, 2004
mozDex is spidering
FYI on the user agent
2 ByronM 9:18 pm Apr 1, 2004
WebmasterWorld disallowing Mozilla, Why?
If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
5 bufferzone 10:59 pm Mar 31, 2004
What does "root of domain" mean & what is 404 error?
root of domain 404 error
2 motherhen 8:14 am Mar 31, 2004

Featured Threads

Hot Threads This Week

Hot Threads This Month