homepage Welcome to WebmasterWorld Guest from 54.226.168.96
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconnew to robots
  is this right?
6sara2 5:38 pm on Apr 5, 2004 (gmt 0)
newpost thread iconHiding robot.txt
  How to prevent robot.txt from beeing read by visitors
7Spazio1999 1:00 pm on Apr 3, 2004 (gmt 0)
newpost thread iconGooglebot GET /robots.txt HTTP/1.0" 404
  Googlebot is the only bot getting 404
1eesnard 10:52 pm on Apr 2, 2004 (gmt 0)
newpost thread iconmozDex is spidering
  FYI on the user agent
2ByronM 9:18 pm on Apr 1, 2004 (gmt 0)
newpost thread iconWebmasterWorld disallowing Mozilla, Why?
  If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
5bufferzone 10:59 pm on Mar 31, 2004 (gmt 0)
newpost thread iconWhat does "root of domain" mean & what is 404 error?
  root of domain 404 error
2motherhen 8:14 am on Mar 31, 2004 (gmt 0)
newpost thread iconWill this robots.txt do it?
 
3rover 5:10 am on Mar 30, 2004 (gmt 0)
newpost thread icondisallow: /*
  robot.txt newbie, please help
2emm0uk 11:15 pm on Mar 29, 2004 (gmt 0)
newpost thread iconrobots.txt blocking foreign search engines
  google robots.txt block foreign engines
1jer61375 4:49 pm on Mar 29, 2004 (gmt 0)
newpost thread iconAllowing only search engines to crawl forums
  Only want search engines in my forums
7Psycho111 8:40 pm on Mar 27, 2004 (gmt 0)
newpost thread icondoes a search engine care if a robots.txt file is 403 instead of 404
 
4kahuna 6:20 pm on Mar 27, 2004 (gmt 0)
newpost thread iconrobots.txt and multiple domain names
  Using a Virtual Server...
1nerolabs 7:45 pm on Mar 26, 2004 (gmt 0)
newpost thread icongoogle images ignoring disallow
 
2apollo 10:43 pm on Mar 25, 2004 (gmt 0)
newpost thread iconValidation Error
  Ran the robots.txt validation checker and recieved nothing but errors.
3jseamless 5:06 pm on Mar 25, 2004 (gmt 0)
newpost thread iconWhy use Robots - Again?
  What are the benefits of using Robots.
4oltc 4:48 pm on Mar 25, 2004 (gmt 0)
newpost thread iconHow can I block Cyveillancebot(63.148.99.****)
  It doesn't look like a useful bot...
6itisgene 4:43 pm on Mar 25, 2004 (gmt 0)
newpost thread iconYahoo-MMCrawler
 
1simondev3 8:08 pm on Mar 24, 2004 (gmt 0)
newpost thread iconReferral log spamming: How use robots.txt to keep them out?
 
10Dan_Vendel 12:36 pm on Mar 24, 2004 (gmt 0)
newpost thread iconrobots.txt elementary question
 
3Vespasian 5:29 am on Mar 24, 2004 (gmt 0)
newpost thread iconHow to identify search engine robots?
  What's the best way for ASP developers to identify robots?
1vieth 3:48 pm on Mar 23, 2004 (gmt 0)
newpost thread iconRobot text for dissallow asp pages indexed
 
2DivinoStore 8:06 am on Mar 21, 2004 (gmt 0)
newpost thread iconAnyone recognize this agent?
  Can't find any info about it anywhere
2martyt 4:15 pm on Mar 20, 2004 (gmt 0)
newpost thread iconA new bot?
  Anyone else seeing this?
4walrus 6:48 am on Mar 19, 2004 (gmt 0)
newpost thread iconCan't prevent php files from being searched
  Is this possible
1Jack_mcs 1:58 pm on Mar 18, 2004 (gmt 0)
newpost thread iconnewbie robots.txt and forum Q
  how to write it
6buksida 3:27 am on Mar 18, 2004 (gmt 0)
Displaying Page 58 of 71<< Back 1 ... 48 49 50 51 52 53 54 55 56 57 [ 58 ] 59 60 61 62 63 64 65 66 67 ... 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved