homepage Welcome to WebmasterWorld Guest from 54.166.54.215
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconBlocking just one parameter via robots.txt
  Is this even possible?
2werty 7:31 pm on Aug 18, 2004 (gmt 0)
newpost thread iconProblem with spiderbots scrounging for robot.txt
 
3Mr_N 6:49 am on Aug 10, 2004 (gmt 0)
newpost thread iconi have a subdomain I dont want indexed
  is robots.txt the answer?
4chrisandsarah 1:21 pm on Aug 8, 2004 (gmt 0)
newpost thread iconrobots.txt appearing in my log files
  But I have n't got one there!
5uk_webber 2:59 pm on Aug 5, 2004 (gmt 0)
newpost thread iconTo Robots.txt or not
  Do we use both?
4obsos 11:20 pm on Aug 1, 2004 (gmt 0)
newpost thread iconmsn search violating robots.txt!
  `disallowed` images appear in preview on result page
26DoppyNL 4:15 pm on Aug 1, 2004 (gmt 0)
newpost thread iconRobots text file or noindex,nofollow
  Whats better?
4mthorpe 4:18 am on Jul 31, 2004 (gmt 0)
newpost thread icongoogle allow - disallow all others
 
6jimmy19 7:20 am on Jul 30, 2004 (gmt 0)
newpost thread iconMSN and Yahoo both missing robots.txt
 
4bppilot 3:26 pm on Jul 29, 2004 (gmt 0)
newpost thread iconYahoo robots.txt problem
  what redirect?
4yowza 9:53 pm on Jul 27, 2004 (gmt 0)
newpost thread iconGoogle listing disallowed pages
  this normal?
2sonik 6:44 pm on Jul 22, 2004 (gmt 0)
newpost thread iconrobots.txt for different language sites
  can use charset tag to allow/disallow?
4a2ztranslate 11:51 pm on Jul 20, 2004 (gmt 0)
newpost thread iconabout subdomain and robots
  should i copy robots to all subdomains?
2webon 3:33 am on Jul 18, 2004 (gmt 0)
newpost thread iconrobots.txt question
 
3motorcycle 5:30 pm on Jul 17, 2004 (gmt 0)
newpost thread iconDisallowing Robot's Effect on Sub-Domain
  Will it disallow the sub domains as well
2bppilot 5:53 am on Jul 14, 2004 (gmt 0)
newpost thread iconWhy is Google following my exclusions?
  Robots.txt file says dont go and Google does!
5anxvariety 5:49 am on Jul 14, 2004 (gmt 0)
newpost thread iconWhy are my .swf files still being indexed by Googlebot?
 
3ichthyous 6:26 pm on Jul 13, 2004 (gmt 0)
newpost thread iconexcluding dynamic pages in all search engines
  excluding dynamic pages in robots.txt
3toddmc04 2:49 pm on Jul 13, 2004 (gmt 0)
newpost thread iconGeneral Guide to Robots.txt
  is there one here :)
4rj87uk 7:51 pm on Jul 7, 2004 (gmt 0)
newpost thread icondomain.com vs www.domain.com
  different sites
9HitProf 4:01 pm on Jul 7, 2004 (gmt 0)
newpost thread iconDo i need the txt file?
  if i want to allow everyone
4markbiz 12:59 am on Jul 5, 2004 (gmt 0)
newpost thread iconI'm not understanding the different makes of bots
  Can I stop some, while allowing others?
2tomequin 10:11 pm on Jul 1, 2004 (gmt 0)
newpost thread iconYahoo! Slurp - robots.txt
  Not obeyng robots.txt
5EarWig 1:59 pm on Jun 29, 2004 (gmt 0)
newpost thread iconMaking robots?
 
4lifedeuce 1:22 pm on Jun 29, 2004 (gmt 0)
newpost thread iconDisallow pages in a subdirectory
  Basic help please
3pjuk 8:40 pm on Jun 24, 2004 (gmt 0)
Displaying Page 56 of 72<< Back 1 ... 46 47 48 49 50 51 52 53 54 55 [ 56 ] 57 58 59 60 61 62 63 64 65 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved