homepage Welcome to WebmasterWorld Guest from 54.197.215.146
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconValidation Error
 
4decdim 12:12 am on Apr 11, 2004 (gmt 0)
newpost thread iconNew to robots.txt and how spider find robots
 
9jazzvn 11:49 pm on Apr 10, 2004 (gmt 0)
newpost thread iconWill the following robots.txt disallow all robots BUT Google?
 
3georgec 1:15 pm on Apr 9, 2004 (gmt 0)
newpost thread iconBot blocking
  best way to accomplish
2IONWeb 7:18 am on Apr 8, 2004 (gmt 0)
newpost thread iconnew to robots
  is this right?
6sara2 5:38 pm on Apr 5, 2004 (gmt 0)
newpost thread iconHiding robot.txt
  How to prevent robot.txt from beeing read by visitors
7Spazio1999 1:00 pm on Apr 3, 2004 (gmt 0)
newpost thread iconGooglebot GET /robots.txt HTTP/1.0" 404
  Googlebot is the only bot getting 404
1eesnard 10:52 pm on Apr 2, 2004 (gmt 0)
newpost thread iconmozDex is spidering
  FYI on the user agent
2ByronM 9:18 pm on Apr 1, 2004 (gmt 0)
newpost thread iconWebmasterWorld disallowing Mozilla, Why?
  If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
5bufferzone 10:59 pm on Mar 31, 2004 (gmt 0)
newpost thread iconWhat does "root of domain" mean & what is 404 error?
  root of domain 404 error
2motherhen 8:14 am on Mar 31, 2004 (gmt 0)
newpost thread iconWill this robots.txt do it?
 
3rover 5:10 am on Mar 30, 2004 (gmt 0)
newpost thread icondisallow: /*
  robot.txt newbie, please help
2emm0uk 11:15 pm on Mar 29, 2004 (gmt 0)
newpost thread iconrobots.txt blocking foreign search engines
  google robots.txt block foreign engines
1jer61375 4:49 pm on Mar 29, 2004 (gmt 0)
newpost thread iconAllowing only search engines to crawl forums
  Only want search engines in my forums
7Psycho111 8:40 pm on Mar 27, 2004 (gmt 0)
newpost thread icondoes a search engine care if a robots.txt file is 403 instead of 404
 
4kahuna 6:20 pm on Mar 27, 2004 (gmt 0)
newpost thread iconrobots.txt and multiple domain names
  Using a Virtual Server...
1nerolabs 7:45 pm on Mar 26, 2004 (gmt 0)
newpost thread icongoogle images ignoring disallow
 
2apollo 10:43 pm on Mar 25, 2004 (gmt 0)
newpost thread iconValidation Error
  Ran the robots.txt validation checker and recieved nothing but errors.
3jseamless 5:06 pm on Mar 25, 2004 (gmt 0)
newpost thread iconWhy use Robots - Again?
  What are the benefits of using Robots.
4oltc 4:48 pm on Mar 25, 2004 (gmt 0)
newpost thread iconHow can I block Cyveillancebot(63.148.99.****)
  It doesn't look like a useful bot...
6itisgene 4:43 pm on Mar 25, 2004 (gmt 0)
newpost thread iconYahoo-MMCrawler
 
1simondev3 8:08 pm on Mar 24, 2004 (gmt 0)
newpost thread iconReferral log spamming: How use robots.txt to keep them out?
 
10Dan_Vendel 12:36 pm on Mar 24, 2004 (gmt 0)
newpost thread iconrobots.txt elementary question
 
3Vespasian 5:29 am on Mar 24, 2004 (gmt 0)
newpost thread iconHow to identify search engine robots?
  What's the best way for ASP developers to identify robots?
1vieth 3:48 pm on Mar 23, 2004 (gmt 0)
newpost thread iconRobot text for dissallow asp pages indexed
 
2DivinoStore 8:06 am on Mar 21, 2004 (gmt 0)
Displaying Page 58 of 71<< Back 1 ... 48 49 50 51 52 53 54 55 56 57 [ 58 ] 59 60 61 62 63 64 65 66 67 ... 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved