homepage Welcome to WebmasterWorld Guest from 54.167.182.81
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconGooglebot GET /robots.txt HTTP/1.0" 404
  Googlebot is the only bot getting 404
1eesnard 10:52 pm on Apr 2, 2004 (gmt 0)
newpost thread iconmozDex is spidering
  FYI on the user agent
2ByronM 9:18 pm on Apr 1, 2004 (gmt 0)
newpost thread iconWebmasterWorld disallowing Mozilla, Why?
  If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
5bufferzone 10:59 pm on Mar 31, 2004 (gmt 0)
newpost thread iconWhat does "root of domain" mean & what is 404 error?
  root of domain 404 error
2motherhen 8:14 am on Mar 31, 2004 (gmt 0)
newpost thread iconWill this robots.txt do it?
 
3rover 5:10 am on Mar 30, 2004 (gmt 0)
newpost thread icondisallow: /*
  robot.txt newbie, please help
2emm0uk 11:15 pm on Mar 29, 2004 (gmt 0)
newpost thread iconrobots.txt blocking foreign search engines
  google robots.txt block foreign engines
1jer61375 4:49 pm on Mar 29, 2004 (gmt 0)
newpost thread iconAllowing only search engines to crawl forums
  Only want search engines in my forums
7Psycho111 8:40 pm on Mar 27, 2004 (gmt 0)
newpost thread icondoes a search engine care if a robots.txt file is 403 instead of 404
 
4kahuna 6:20 pm on Mar 27, 2004 (gmt 0)
newpost thread iconrobots.txt and multiple domain names
  Using a Virtual Server...
1nerolabs 7:45 pm on Mar 26, 2004 (gmt 0)
newpost thread icongoogle images ignoring disallow
 
2apollo 10:43 pm on Mar 25, 2004 (gmt 0)
newpost thread iconValidation Error
  Ran the robots.txt validation checker and recieved nothing but errors.
3jseamless 5:06 pm on Mar 25, 2004 (gmt 0)
newpost thread iconWhy use Robots - Again?
  What are the benefits of using Robots.
4oltc 4:48 pm on Mar 25, 2004 (gmt 0)
newpost thread iconHow can I block Cyveillancebot(63.148.99.****)
  It doesn't look like a useful bot...
6itisgene 4:43 pm on Mar 25, 2004 (gmt 0)
newpost thread iconYahoo-MMCrawler
 
1simondev3 8:08 pm on Mar 24, 2004 (gmt 0)
newpost thread iconReferral log spamming: How use robots.txt to keep them out?
 
10Dan_Vendel 12:36 pm on Mar 24, 2004 (gmt 0)
newpost thread iconrobots.txt elementary question
 
3Vespasian 5:29 am on Mar 24, 2004 (gmt 0)
newpost thread iconHow to identify search engine robots?
  What's the best way for ASP developers to identify robots?
1vieth 3:48 pm on Mar 23, 2004 (gmt 0)
newpost thread iconRobot text for dissallow asp pages indexed
 
2DivinoStore 8:06 am on Mar 21, 2004 (gmt 0)
newpost thread iconAnyone recognize this agent?
  Can't find any info about it anywhere
2martyt 4:15 pm on Mar 20, 2004 (gmt 0)
newpost thread iconA new bot?
  Anyone else seeing this?
4walrus 6:48 am on Mar 19, 2004 (gmt 0)
newpost thread iconCan't prevent php files from being searched
  Is this possible
1Jack_mcs 1:58 pm on Mar 18, 2004 (gmt 0)
newpost thread iconnewbie robots.txt and forum Q
  how to write it
6buksida 3:27 am on Mar 18, 2004 (gmt 0)
newpost thread iconRobot.txt File
  What is it's importance?
7Enigmatic 8:52 pm on Mar 16, 2004 (gmt 0)
newpost thread iconQuepasaCreep Agent
  where is it from?
3barbeta 8:17 pm on Mar 16, 2004 (gmt 0)
Displaying Page 59 of 72<< Back 1 ... 49 50 51 52 53 54 55 56 57 58 [ 59 ] 60 61 62 63 64 65 66 67 68 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved