homepage Welcome to WebmasterWorld Guest from 54.163.72.86
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconMultiple Domains Issue
 
3kdobson99 4:58 am on Jan 4, 2007 (gmt 0)
newpost thread iconDo spiders follow links from pages with the noindex tag?
  As opposed to links from robots files?
4bouncybunny 6:59 am on Jan 2, 2007 (gmt 0)
newpost thread iconDirecting Spiders to specific Servers
 
2TheSeoGuy 1:50 pm on Dec 29, 2006 (gmt 0)
newpost thread iconI had robots.txt blocking my phpbb folders which are now deleted
  Should I remove that line from robots.txt now?
2cmendla 4:16 pm on Dec 24, 2006 (gmt 0)
newpost thread iconDisallowing Indexed files in Google
 
5leo11877 2:01 pm on Dec 23, 2006 (gmt 0)
newpost thread iconDisallowing by date.
  Is it possible to keep some page and disallow new ones...?
3bouncybunny 8:31 am on Dec 20, 2006 (gmt 0)
newpost thread iconWorried about profile links
 
4Asia_Expat 3:53 pm on Dec 19, 2006 (gmt 0)
newpost thread iconIs it possible to exclude a URL by specifying an anchor?
  Excluding #...?
3bouncybunny 12:40 am on Dec 17, 2006 (gmt 0)
newpost thread iconRemoving Pages - Does Robots.txt Help?
 
3akreider 4:17 am on Dec 16, 2006 (gmt 0)
newpost thread iconAllowing only some pages for ia archive
 
5sosoo 8:20 am on Dec 13, 2006 (gmt 0)
newpost thread iconMediapartners-Google fellow the robots.txt?
 
2AlphaWu 1:41 pm on Dec 12, 2006 (gmt 0)
newpost thread iconmy robots file
  need help
3experienced 12:26 am on Dec 12, 2006 (gmt 0)
newpost thread iconDisagreement between Google and a validator
  Robots.txt Checker says it's so, Google robots anayler says the other way
6Gusgsm 9:49 am on Dec 11, 2006 (gmt 0)
newpost thread iconGooglebot tripping bad bot script
 
5RobertRogers 5:29 pm on Dec 10, 2006 (gmt 0)
newpost thread iconIs this bad robots.txt
 
1scoobydooaustralia 8:02 pm on Dec 8, 2006 (gmt 0)
newpost thread iconduplicate pages .aspx?id=
  Not sure what to do next
2schatzm 9:01 pm on Dec 6, 2006 (gmt 0)
newpost thread iconIs this a bot?
  Unwanted visitor to my website
2brancook 8:57 pm on Dec 6, 2006 (gmt 0)
newpost thread iconTwo domains mirror each other
  Is it possible to a command to a robots.txt file to apply to just one?
4Jeremy_H 9:08 pm on Dec 5, 2006 (gmt 0)
newpost thread iconDo I need to disallow files that aren't linked to?
  for example, 'included' PHP & javascript pages
9sssweb 3:34 pm on Dec 4, 2006 (gmt 0)
newpost thread iconremoving permanent redirect - honoring content in page
  removing permanent redirect
2thu_household 2:56 pm on Dec 4, 2006 (gmt 0)
newpost thread iconDo we need robots at all? New member here
  Confused as to whether I need to add robots txt
5plenusvita 9:23 pm on Dec 2, 2006 (gmt 0)
newpost thread iconBlock all spiders, but only allow well known spiders?
 
2foxfox 3:19 pm on Dec 2, 2006 (gmt 0)
newpost thread iconmsnbot visits robots.txt and exits
 
7pemba76 12:13 pm on Nov 28, 2006 (gmt 0)
newpost thread iconUpload of incorrect robots.txt
  What a screw up - any permanent damage?
4TinkyWinky 10:12 am on Nov 28, 2006 (gmt 0)
newpost thread iconDoes excluding Googlebot create a penalty?
  If other bots are allowed, is this a red flag to Google
2AndyA 9:26 pm on Nov 27, 2006 (gmt 0)
Displaying Page 32 of 72<< Back 1 ... 22 23 24 25 26 27 28 29 30 31 [ 32 ] 33 34 35 36 37 38 39 40 41 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved