Welcome to WebmasterWorld Guest from 54.144.79.200

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Strange bot keep on visiting my site and don't obey my robots.txt file
What to do?
2 Compworld 8:01 pm Apr 22, 2004
revisit after so many days
2 tommygunner 3:21 am Apr 21, 2004
index.php vs index.html
Will This Work i Wonder
2 yummycarp 11:28 pm Apr 20, 2004
Banning a script using robots.txt
how do do this
4 Imaster 9:51 pm Apr 20, 2004
Robots.txt - Why?
3 warlordbb 2:08 pm Apr 20, 2004
Can IMDB do this?
charge per unwanted request
3 jrap 10:19 pm Apr 19, 2004
Robots.txt exclusion
Google keeps crawling page
2 anxvariety 2:27 pm Apr 19, 2004
Froogle wants my images!
3 seadog 5:59 am Apr 14, 2004
META Robots no index tag
2 BMach 2:36 am Apr 14, 2004
Using wild card in robots
Can you add * to ip?
2 walrus 5:37 pm Apr 13, 2004
How to ban inktomisearch.com's bots?
I have been hit hard with so many requests by this.
3 itisgene 5:24 pm Apr 11, 2004
Validation Error
4 decdim 12:12 am Apr 11, 2004
New to robots.txt and how spider find robots
9 jazzvn 11:49 pm Apr 10, 2004
Will the following robots.txt disallow all robots BUT Google?
3 georgec 1:15 pm Apr 9, 2004
Bot blocking
best way to accomplish
2 IONWeb 7:18 am Apr 8, 2004
new to robots
is this right?
6 sara2 5:38 pm Apr 5, 2004
Hiding robot.txt
How to prevent robot.txt from beeing read by visitors
7 Spazio1999 1:00 pm Apr 3, 2004
Googlebot GET /robots.txt HTTP/1.0" 404
Googlebot is the only bot getting 404
1 eesnard 10:52 pm Apr 2, 2004
mozDex is spidering
FYI on the user agent
2 ByronM 9:18 pm Apr 1, 2004
WebmasterWorld disallowing Mozilla, Why?
If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
5 bufferzone 10:59 pm Mar 31, 2004
What does "root of domain" mean & what is 404 error?
root of domain 404 error
2 motherhen 8:14 am Mar 31, 2004
Will this robots.txt do it?
3 rover 5:10 am Mar 30, 2004
disallow: /*
robot.txt newbie, please help
2 emm0uk 11:15 pm Mar 29, 2004
robots.txt blocking foreign search engines
google robots.txt block foreign engines
1 jer61375 4:49 pm Mar 29, 2004
Allowing only search engines to crawl forums
Only want search engines in my forums
7 Psycho111 8:40 pm Mar 27, 2004

Featured Threads

Hot Threads This Week

Hot Threads This Month