homepage Welcome to WebmasterWorld Guest from 54.237.184.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconRobots.txt - Why?
 
3warlordbb 2:08 pm on Apr 20, 2004 (gmt 0)
newpost thread iconCan IMDB do this?
  charge per unwanted request
3jrap 10:19 pm on Apr 19, 2004 (gmt 0)
newpost thread iconRobots.txt exclusion
  Google keeps crawling page
2anxvariety 2:27 pm on Apr 19, 2004 (gmt 0)
newpost thread iconFroogle wants my images!
 
3seadog 5:59 am on Apr 14, 2004 (gmt 0)
newpost thread iconMETA Robots no index tag
 
2BMach 2:36 am on Apr 14, 2004 (gmt 0)
newpost thread iconUsing wild card in robots
  Can you add * to ip?
2walrus 5:37 pm on Apr 13, 2004 (gmt 0)
newpost thread iconHow to ban inktomisearch.com's bots?
  I have been hit hard with so many requests by this.
3itisgene 5:24 pm on Apr 11, 2004 (gmt 0)
newpost thread iconValidation Error
 
4decdim 12:12 am on Apr 11, 2004 (gmt 0)
newpost thread iconNew to robots.txt and how spider find robots
 
9jazzvn 11:49 pm on Apr 10, 2004 (gmt 0)
newpost thread iconWill the following robots.txt disallow all robots BUT Google?
 
3georgec 1:15 pm on Apr 9, 2004 (gmt 0)
newpost thread iconBot blocking
  best way to accomplish
2IONWeb 7:18 am on Apr 8, 2004 (gmt 0)
newpost thread iconnew to robots
  is this right?
6sara2 5:38 pm on Apr 5, 2004 (gmt 0)
newpost thread iconHiding robot.txt
  How to prevent robot.txt from beeing read by visitors
7Spazio1999 1:00 pm on Apr 3, 2004 (gmt 0)
newpost thread iconGooglebot GET /robots.txt HTTP/1.0" 404
  Googlebot is the only bot getting 404
1eesnard 10:52 pm on Apr 2, 2004 (gmt 0)
newpost thread iconmozDex is spidering
  FYI on the user agent
2ByronM 9:18 pm on Apr 1, 2004 (gmt 0)
newpost thread iconWebmasterWorld disallowing Mozilla, Why?
  If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
5bufferzone 10:59 pm on Mar 31, 2004 (gmt 0)
newpost thread iconWhat does "root of domain" mean & what is 404 error?
  root of domain 404 error
2motherhen 8:14 am on Mar 31, 2004 (gmt 0)
newpost thread iconWill this robots.txt do it?
 
3rover 5:10 am on Mar 30, 2004 (gmt 0)
newpost thread icondisallow: /*
  robot.txt newbie, please help
2emm0uk 11:15 pm on Mar 29, 2004 (gmt 0)
newpost thread iconrobots.txt blocking foreign search engines
  google robots.txt block foreign engines
1jer61375 4:49 pm on Mar 29, 2004 (gmt 0)
newpost thread iconAllowing only search engines to crawl forums
  Only want search engines in my forums
7Psycho111 8:40 pm on Mar 27, 2004 (gmt 0)
newpost thread icondoes a search engine care if a robots.txt file is 403 instead of 404
 
4kahuna 6:20 pm on Mar 27, 2004 (gmt 0)
newpost thread iconrobots.txt and multiple domain names
  Using a Virtual Server...
1nerolabs 7:45 pm on Mar 26, 2004 (gmt 0)
newpost thread icongoogle images ignoring disallow
 
2apollo 10:43 pm on Mar 25, 2004 (gmt 0)
newpost thread iconValidation Error
  Ran the robots.txt validation checker and recieved nothing but errors.
3jseamless 5:06 pm on Mar 25, 2004 (gmt 0)
Displaying Page 58 of 72<< Back 1 ... 48 49 50 51 52 53 54 55 56 57 [ 58 ] 59 60 61 62 63 64 65 66 67 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved