homepage Welcome to WebmasterWorld Guest from 54.163.89.8
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconGoogle ignoring robots.txt
  I'm seeing lots of my 'Disallowed' pages in their index
4Amygdala 4:54 am on Apr 24, 2004 (gmt 0)
newpost thread iconStrange bot keep on visiting my site and don't obey my robots.txt file
  What to do?
2Compworld 8:01 pm on Apr 22, 2004 (gmt 0)
newpost thread iconrevisit after so many days
 
2tommygunner 3:21 am on Apr 21, 2004 (gmt 0)
newpost thread iconindex.php vs index.html
  Will This Work i Wonder
2yummycarp 11:28 pm on Apr 20, 2004 (gmt 0)
newpost thread iconBanning a script using robots.txt
  how do do this
4Imaster 9:51 pm on Apr 20, 2004 (gmt 0)
newpost thread iconRobots.txt - Why?
 
3warlordbb 2:08 pm on Apr 20, 2004 (gmt 0)
newpost thread iconCan IMDB do this?
  charge per unwanted request
3jrap 10:19 pm on Apr 19, 2004 (gmt 0)
newpost thread iconRobots.txt exclusion
  Google keeps crawling page
2anxvariety 2:27 pm on Apr 19, 2004 (gmt 0)
newpost thread iconFroogle wants my images!
 
3seadog 5:59 am on Apr 14, 2004 (gmt 0)
newpost thread iconMETA Robots no index tag
 
2BMach 2:36 am on Apr 14, 2004 (gmt 0)
newpost thread iconUsing wild card in robots
  Can you add * to ip?
2walrus 5:37 pm on Apr 13, 2004 (gmt 0)
newpost thread iconHow to ban inktomisearch.com's bots?
  I have been hit hard with so many requests by this.
3itisgene 5:24 pm on Apr 11, 2004 (gmt 0)
newpost thread iconValidation Error
 
4decdim 12:12 am on Apr 11, 2004 (gmt 0)
newpost thread iconNew to robots.txt and how spider find robots
 
9jazzvn 11:49 pm on Apr 10, 2004 (gmt 0)
newpost thread iconWill the following robots.txt disallow all robots BUT Google?
 
3georgec 1:15 pm on Apr 9, 2004 (gmt 0)
newpost thread iconBot blocking
  best way to accomplish
2IONWeb 7:18 am on Apr 8, 2004 (gmt 0)
newpost thread iconnew to robots
  is this right?
6sara2 5:38 pm on Apr 5, 2004 (gmt 0)
newpost thread iconHiding robot.txt
  How to prevent robot.txt from beeing read by visitors
7Spazio1999 1:00 pm on Apr 3, 2004 (gmt 0)
newpost thread iconGooglebot GET /robots.txt HTTP/1.0" 404
  Googlebot is the only bot getting 404
1eesnard 10:52 pm on Apr 2, 2004 (gmt 0)
newpost thread iconmozDex is spidering
  FYI on the user agent
2ByronM 9:18 pm on Apr 1, 2004 (gmt 0)
newpost thread iconWebmasterWorld disallowing Mozilla, Why?
  If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
5bufferzone 10:59 pm on Mar 31, 2004 (gmt 0)
newpost thread iconWhat does "root of domain" mean & what is 404 error?
  root of domain 404 error
2motherhen 8:14 am on Mar 31, 2004 (gmt 0)
newpost thread iconWill this robots.txt do it?
 
3rover 5:10 am on Mar 30, 2004 (gmt 0)
newpost thread icondisallow: /*
  robot.txt newbie, please help
2emm0uk 11:15 pm on Mar 29, 2004 (gmt 0)
newpost thread iconrobots.txt blocking foreign search engines
  google robots.txt block foreign engines
1jer61375 4:49 pm on Mar 29, 2004 (gmt 0)
Displaying Page 58 of 72<< Back 1 ... 48 49 50 51 52 53 54 55 56 57 [ 58 ] 59 60 61 62 63 64 65 66 67 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved