homepage Welcome to WebmasterWorld Guest from 54.211.182.241
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconNimbleCrawler is Bad Bot, well worth banning.
  NimbleCrawler is Bad Bot. Never reads robots.txt file.
10d_mot 1:24 am on Feb 23, 2006 (gmt 0)
newpost thread iconIn the era of increasing numbers of bad bots is robots.txt irrelevant?[2] ( 1 2 )
  Why bother? Why not go the other way and white-list with controls?
34Webwork 9:21 pm on Feb 21, 2006 (gmt 0)
newpost thread iconDisallow a directory, but allow one of its subdirectories
 
2Nutter 9:48 am on Feb 21, 2006 (gmt 0)
newpost thread iconrobots.txt disallow
  How do I ensure re-direct pages are not duplicated?
1robinantill 11:05 pm on Feb 18, 2006 (gmt 0)
newpost thread iconComplicated Robots.txt / htaccess Question
  maybe
1fraggle 9:23 pm on Feb 18, 2006 (gmt 0)
newpost thread iconrobots.txt - subdomains and domain
  where to place
3Rose_1171 8:06 am on Feb 18, 2006 (gmt 0)
newpost thread iconHttp: vs Https:
  What are the best ways to stop crawlers?
3thaedge 7:54 am on Feb 18, 2006 (gmt 0)
newpost thread iconDoes robots.txt work per directory or per domain?
 
3zver0 6:40 pm on Feb 16, 2006 (gmt 0)
newpost thread iconDo bots confuse .htm and .html files in robots.txt?
 
2subgenius 6:08 pm on Feb 16, 2006 (gmt 0)
newpost thread iconrobots.txt Advertising Methods
  The new trend...
3pageoneresults 4:49 pm on Feb 16, 2006 (gmt 0)
newpost thread iconBanning all subdirectories in a directory without the files?
 
5zver0 1:28 pm on Feb 16, 2006 (gmt 0)
newpost thread iconrenewed site not being indexed - old pages in serp cache
  how to fix robots.txt mistake
4Ring 10:47 am on Feb 15, 2006 (gmt 0)
newpost thread iconWhy doesn't this validate?
  Help please
4Vimes 11:04 am on Feb 14, 2006 (gmt 0)
newpost thread iconTripAdvisorBot?
 
1jexx 9:02 pm on Feb 13, 2006 (gmt 0)
newpost thread iconhow to disallow shopping cart
  rank is dropping since we set it up
2Lorel 7:39 am on Feb 12, 2006 (gmt 0)
newpost thread iconRobots.txt help
  Ok, maybe I did not understand how it works
15followgreg 9:20 pm on Feb 10, 2006 (gmt 0)
newpost thread icon? and * in robots.txt
  What stands a questionmark in the robots.txt for?
2solandre 5:50 pm on Feb 9, 2006 (gmt 0)
newpost thread iconMeta tags and Robots
 
5Buster42 4:48 pm on Feb 9, 2006 (gmt 0)
newpost thread iconCharacter Limit
  Has anyone head of a character limit in robots.txt?
7bumpaw 8:39 am on Feb 9, 2006 (gmt 0)
newpost thread iconIs this Robots.txt OK?
  Please check this robot.txt file
9mzconstruction 11:06 pm on Feb 3, 2006 (gmt 0)
newpost thread iconrobots.txt and SSL
 
3AlpacaAnnie 8:24 am on Feb 3, 2006 (gmt 0)
newpost thread iconAre different User-agent entries needed for versions of similar robot
 
3Ceyx 2:29 am on Feb 1, 2006 (gmt 0)
newpost thread iconRobots list service?
  Looking for a service provider or subscription service
2xbase234 1:41 am on Feb 1, 2006 (gmt 0)
newpost thread iconHow to disallow URLs with parameters?
  "/page-name.php?" or "/page-name.php"
11iProgram 11:34 pm on Jan 31, 2006 (gmt 0)
newpost thread iconThis OK?
 
3TonysDesigns 10:13 pm on Jan 30, 2006 (gmt 0)
Displaying Page 39 of 72<< Back 1 ... 29 30 31 32 33 34 35 36 37 38 [ 39 ] 40 41 42 43 44 45 46 47 48 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved