homepage Welcome to WebmasterWorld Guest from 54.161.176.90
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconIs this bad robots.txt
 
1scoobydooaustralia 8:02 pm on Dec 8, 2006 (gmt 0)
newpost thread iconduplicate pages .aspx?id=
  Not sure what to do next
2schatzm 9:01 pm on Dec 6, 2006 (gmt 0)
newpost thread iconIs this a bot?
  Unwanted visitor to my website
2brancook 8:57 pm on Dec 6, 2006 (gmt 0)
newpost thread iconTwo domains mirror each other
  Is it possible to a command to a robots.txt file to apply to just one?
4Jeremy_H 9:08 pm on Dec 5, 2006 (gmt 0)
newpost thread iconDo I need to disallow files that aren't linked to?
  for example, 'included' PHP & javascript pages
9sssweb 3:34 pm on Dec 4, 2006 (gmt 0)
newpost thread iconremoving permanent redirect - honoring content in page
  removing permanent redirect
2thu_household 2:56 pm on Dec 4, 2006 (gmt 0)
newpost thread iconDo we need robots at all? New member here
  Confused as to whether I need to add robots txt
5plenusvita 9:23 pm on Dec 2, 2006 (gmt 0)
newpost thread iconBlock all spiders, but only allow well known spiders?
 
2foxfox 3:19 pm on Dec 2, 2006 (gmt 0)
newpost thread iconmsnbot visits robots.txt and exits
 
7pemba76 12:13 pm on Nov 28, 2006 (gmt 0)
newpost thread iconUpload of incorrect robots.txt
  What a screw up - any permanent damage?
4TinkyWinky 10:12 am on Nov 28, 2006 (gmt 0)
newpost thread iconDoes excluding Googlebot create a penalty?
  If other bots are allowed, is this a red flag to Google
2AndyA 9:26 pm on Nov 27, 2006 (gmt 0)
newpost thread iconwhat to put into robots.txt?
 
11Hitman3266 5:09 pm on Nov 27, 2006 (gmt 0)
newpost thread iconHow long does it take for SEs to learn about a robots'txt
 
4bouncybunny 12:04 pm on Nov 20, 2006 (gmt 0)
newpost thread iconYahoo! Slurp Now Supports Wildcards in robots.txt
 
17OldWolf 2:54 am on Nov 20, 2006 (gmt 0)
newpost thread iconRobots.txt and password-protection
 
3hwy2hel 10:56 pm on Nov 16, 2006 (gmt 0)
newpost thread iconYahoo robots txt
  nedd to disallow all urls with?
2cheaperholidays 4:54 pm on Nov 16, 2006 (gmt 0)
newpost thread iconrobots.txt? OR .htaccess?
  disallowing or protecting directories
2johnjoyce 4:27 pm on Nov 16, 2006 (gmt 0)
newpost thread iconWildcards in robots.txt for forum software
 
6Asia_Expat 1:40 am on Nov 14, 2006 (gmt 0)
newpost thread iconrobots.txt syntax
  '$' and '?'
2sssweb 8:40 pm on Nov 11, 2006 (gmt 0)
newpost thread iconrobots.txt help
 
2miracle 7:16 pm on Nov 8, 2006 (gmt 0)
newpost thread iconLink Checker / Site map generator
  One that observes robots.txt and meta tags
5fish_eye 5:33 am on Nov 7, 2006 (gmt 0)
newpost thread iconDisallow by ISP
  Where to get a list of ISPs and the countries they are in?
1youfoundjake 7:13 pm on Nov 3, 2006 (gmt 0)
newpost thread iconMade the mistake of putting sub folders I want allowed
  disallow everthing but?
2Sunnyvale 7:07 pm on Oct 31, 2006 (gmt 0)
newpost thread iconGoogle remove url tool for supplemental urls a waste of time?
  Should I bother removing duplicate supplementals?
3Tonerman 4:20 pm on Oct 30, 2006 (gmt 0)
newpost thread iconallow major engines, block the fleas
  and to add further control
5youfoundjake 5:16 pm on Oct 26, 2006 (gmt 0)
Displaying Page 33 of 72<< Back 1 ... 23 24 25 26 27 28 29 30 31 32 [ 33 ] 34 35 36 37 38 39 40 41 42 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved