homepage Welcome to WebmasterWorld Guest from 54.237.78.165
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconrobots.txt used in court case
 
2transactiongeek 1:27 pm on Feb 3, 2004 (gmt 0)
newpost thread iconwhat is the "KAIST AITrc Crawler" bot?
  it ignored my "robots.txt" file and got caught in my bot trap
4stapel 12:01 pm on Feb 3, 2004 (gmt 0)
newpost thread iconIs it illegal to ignore a robots.txt?
  Could I sue google?
2transactiongeek 6:44 am on Feb 3, 2004 (gmt 0)
newpost thread iconrobots metatag
  I have noindex,follow but the page is indexed
2nevetS 4:41 am on Feb 3, 2004 (gmt 0)
newpost thread iconsave and upload robots.txt
 
4helenp 8:27 pm on Feb 2, 2004 (gmt 0)
newpost thread iconIs this robots.txt blocking anybody
 
8nalavanje 2:20 pm on Feb 2, 2004 (gmt 0)
newpost thread iconIs this enough
 
5dougie 11:20 am on Feb 2, 2004 (gmt 0)
newpost thread iconHow Do I Code My Robots.txt File To Disallow Subdomians?
  Is it Disallow: /apple.domain.com or Disallow: /apple/
2Christyl_Stevens 9:34 am on Feb 2, 2004 (gmt 0)
newpost thread iconasterias/2.0
  Looks for streaming media files...
6pendanticist 11:43 pm on Feb 1, 2004 (gmt 0)
newpost thread iconwill all robots crawl the sites without robot.txt?
  robot crawl, robot.txt
2johnlim 3:08 am on Feb 1, 2004 (gmt 0)
newpost thread iconWhat will Inktomi do in this scenario?
 
18flobaby 6:02 am on Jan 31, 2004 (gmt 0)
newpost thread iconGoogle spidering past secure server
  Robots.txt file to boot the bot?
2xbase234 12:52 am on Jan 31, 2004 (gmt 0)
newpost thread iconLogs keeping showing 404 error
  I have a robots file and it looks ok
3eesnard 12:03 am on Jan 30, 2004 (gmt 0)
newpost thread iconneed robots.txt for pages that are not linked?
  robots.txt
5rld3 5:09 pm on Jan 29, 2004 (gmt 0)
newpost thread iconSEO Robot.txt tag and robot meta tags... need advice
  Help about robot.txt and robot meta tags
3jamieabbott 2:09 pm on Jan 28, 2004 (gmt 0)
newpost thread iconRobots.txt returning "test"
  Host has really messed me up
4GodLikeLotus 10:37 am on Jan 28, 2004 (gmt 0)
newpost thread iconNeed quick help
  Prevent robots from clicking on PPC links
2hostlead 11:11 am on Jan 27, 2004 (gmt 0)
newpost thread iconhow to let robot crawl the links at a page but not index this page?
  index, robot
2johnlim 11:39 pm on Jan 26, 2004 (gmt 0)
newpost thread iconWildcard use in robots.txt
  Need to disallow spidering of graphics files
2Probably 4:58 am on Jan 26, 2004 (gmt 0)
newpost thread iconrobots.txt and htaccess
  .htaccess circumvents to site root
4stefan009 5:35 pm on Jan 24, 2004 (gmt 0)
newpost thread iconAltavista Scooter eats up too much trafic
 
2maxidrom11 5:28 am on Jan 24, 2004 (gmt 0)
newpost thread iconDoes every site need a robots.txt?
  I have one, but I'm not sure I need it
2MatthewHSE 4:00 pm on Jan 23, 2004 (gmt 0)
newpost thread iconcreating robots.txt
 
5artdog 12:21 pm on Jan 23, 2004 (gmt 0)
newpost thread iconMy page has never been indexed
 
2kknusa 6:45 am on Jan 23, 2004 (gmt 0)
newpost thread iconSubdomains
 
2BBB_Guy 6:38 am on Jan 23, 2004 (gmt 0)
Displaying Page 62 of 72<< Back 1 ... 52 53 54 55 56 57 58 59 60 61 [ 62 ] 63 64 65 66 67 68 69 70 71 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved