homepage Welcome to WebmasterWorld Guest from 54.166.173.147
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconDirecting Spiders to specific Servers
 
2TheSeoGuy 1:50 pm on Dec 29, 2006 (gmt 0)
newpost thread iconI had robots.txt blocking my phpbb folders which are now deleted
  Should I remove that line from robots.txt now?
2cmendla 4:16 pm on Dec 24, 2006 (gmt 0)
newpost thread iconDisallowing Indexed files in Google
 
5leo11877 2:01 pm on Dec 23, 2006 (gmt 0)
newpost thread iconDisallowing by date.
  Is it possible to keep some page and disallow new ones...?
3bouncybunny 8:31 am on Dec 20, 2006 (gmt 0)
newpost thread iconWorried about profile links
 
4Asia_Expat 3:53 pm on Dec 19, 2006 (gmt 0)
newpost thread iconIs it possible to exclude a URL by specifying an anchor?
  Excluding #...?
3bouncybunny 12:40 am on Dec 17, 2006 (gmt 0)
newpost thread iconRemoving Pages - Does Robots.txt Help?
 
3akreider 4:17 am on Dec 16, 2006 (gmt 0)
newpost thread iconAllowing only some pages for ia archive
 
5sosoo 8:20 am on Dec 13, 2006 (gmt 0)
newpost thread iconMediapartners-Google fellow the robots.txt?
 
2AlphaWu 1:41 pm on Dec 12, 2006 (gmt 0)
newpost thread iconmy robots file
  need help
3experienced 12:26 am on Dec 12, 2006 (gmt 0)
newpost thread iconDisagreement between Google and a validator
  Robots.txt Checker says it's so, Google robots anayler says the other way
6Gusgsm 9:49 am on Dec 11, 2006 (gmt 0)
newpost thread iconGooglebot tripping bad bot script
 
5RobertRogers 5:29 pm on Dec 10, 2006 (gmt 0)
newpost thread iconIs this bad robots.txt
 
1scoobydooaustralia 8:02 pm on Dec 8, 2006 (gmt 0)
newpost thread iconduplicate pages .aspx?id=
  Not sure what to do next
2schatzm 9:01 pm on Dec 6, 2006 (gmt 0)
newpost thread iconIs this a bot?
  Unwanted visitor to my website
2brancook 8:57 pm on Dec 6, 2006 (gmt 0)
newpost thread iconTwo domains mirror each other
  Is it possible to a command to a robots.txt file to apply to just one?
4Jeremy_H 9:08 pm on Dec 5, 2006 (gmt 0)
newpost thread iconDo I need to disallow files that aren't linked to?
  for example, 'included' PHP & javascript pages
9sssweb 3:34 pm on Dec 4, 2006 (gmt 0)
newpost thread iconremoving permanent redirect - honoring content in page
  removing permanent redirect
2thu_household 2:56 pm on Dec 4, 2006 (gmt 0)
newpost thread iconDo we need robots at all? New member here
  Confused as to whether I need to add robots txt
5plenusvita 9:23 pm on Dec 2, 2006 (gmt 0)
newpost thread iconBlock all spiders, but only allow well known spiders?
 
2foxfox 3:19 pm on Dec 2, 2006 (gmt 0)
newpost thread iconmsnbot visits robots.txt and exits
 
7pemba76 12:13 pm on Nov 28, 2006 (gmt 0)
newpost thread iconUpload of incorrect robots.txt
  What a screw up - any permanent damage?
4TinkyWinky 10:12 am on Nov 28, 2006 (gmt 0)
newpost thread iconDoes excluding Googlebot create a penalty?
  If other bots are allowed, is this a red flag to Google
2AndyA 9:26 pm on Nov 27, 2006 (gmt 0)
newpost thread iconwhat to put into robots.txt?
 
11Hitman3266 5:09 pm on Nov 27, 2006 (gmt 0)
newpost thread iconHow long does it take for SEs to learn about a robots'txt
 
4bouncybunny 12:04 pm on Nov 20, 2006 (gmt 0)
Displaying Page 32 of 71<< Back 1 ... 22 23 24 25 26 27 28 29 30 31 [ 32 ] 33 34 35 36 37 38 39 40 41 ... 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved