homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Library
Forum: Sitemaps, Meta Data, and robots.txt
Displaying Topics 1 - 40 (9 total) Sorted by: Date-Last-Post, Direction: forward
#Subjectsort orderDate
1: new post   Cloaking Your robots.txt
Oct 28, 2003
2: new post   SonicWALL Firewall Blocks Spiders
"Wondering why your website isn't being spidered? Your firewall may now be part of the issue."
May 3, 2005
3: new post   In the era of increasing numbers of bad bots is robots.txt irrelevant?
"I just read through a number of posts in an attempt to understand what, if anything, can be done about bad bots."
Feb 20, 2006
4: new post   Yahoo! Slurp Now Supports Wildcards in robots.txt
You can now use '*' in robots directives for Yahoo! Slurp to wildcard match a sequence of characters in your URL. You can use this symbol in any part of the URL string you provide in the robots directive.
Nov 6, 2006
5: new post   Why Should I Have A Robots.txt File?
Dec 31, 2006
6: new post   When Robots.txt And Meta Robots Collide
"WebmasterWorld members clarify the priority and use of meta name="robots" content="ALL" & robots.txt"
Mar 8, 2007
7: new post   Add Sitemap To robots.txt For Autodiscovery
"...you can now find your sitemaps in a uniform way across all participating engines. To do this, simply add the following line to your robots.txt..."
Apr 13, 2007
8: new post   Proposal for robots.txt To Have Greater Flexibility
"The desire for greater control over how search engines index and display Web sites is driving an effort by leading news organizations and other publishers to revise a 13-year-old technology for restricting access."
Nov 29, 2007
9: new post   Google's Current Specifications for Robots Directives
A very cool reference. Google's exact technical specification for handling robots.txt, robots meta tags and x-robots directives
Nov 24, 2010
#Subjectsort orderDate
open in new window=open in new window

1 Next >>
Home / Forums Index / Library
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved