homepage Welcome to WebmasterWorld Guest from 54.205.106.111
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Allow: ?
mihomes




msg:4369352
 11:24 pm on Sep 30, 2011 (gmt 0)

Noticed this the other day... for the longest time my robots.txt followed the format of :

User-agent: *
Disallow: /test/
Allow: /
Sitemap: [test.com...]

which is consistent with what Google shows you in their webmaster tools excluding the sitemap field.

Now, on robotstxt.org I noticed they state that there is no 'allow' field and the only requirement is atleast one 'disallow'. So in other words the above example would be changed to :

User-agent: *
Disallow: /test/
Sitemap: [test.com...]

What is everyones thoughts on this? In theory it sounds like they accomplish the same exact thing, however, is the 'Allow: /' necessary... or on the flipside... would it hurt to have it?

Would like to hear opinions on this.

 

g1smd




msg:4369361
 12:10 am on Oct 1, 2011 (gmt 0)

Some searchengine bots understand "Allow". Most bots do not.

I believe you need a blank line before the sitmap line.

Use example.com in this forum to suppress the auto-linking function. See RFC 2606.

mihomes




msg:4369371
 12:50 am on Oct 1, 2011 (gmt 0)

"Some searchengine bots understand "Allow". Most bots do not."

- Yes, I realize that. That wasn't really what I was asking.

You DO NOT need a line before the 'Sitemap:' for anyone reading this.

lucy24




msg:4369374
 12:57 am on Oct 1, 2011 (gmt 0)

which is consistent with what Google shows you in their webmaster tools excluding the sitemap field.

It must be buried in WMT somewhere, because I'm pretty sure that's where I learned you can put the sitemap info in your robots.txt. (Look! I have a sitemap in the exact location and with the exact name you would have looked for if I hadn't said anything!)

I was going to find the specific place and quote them, but g### flew into a rage because I wouldn't give them my cell phone number-- an easy decision to make, since I don't have a cell phone-- and now they are treating me to the perpetual-motion spinning thingie. I can sign out, but if I sign in they just pick up where they left off.

:: sigh ::

On second thought, maybe it was bing.

:: search, search ::

Nope, can't find it there. (But am very intrigued by the notation "URLs listed below are currently blocked by your site’s robots.txt. URLs marked with ! are considered by Bing to be important or significant for your site. Please ensure these are not blocked unintentionally." They put this up even if you don't actually have anything marked with a !)

Awright, g###, time's up. Ten minutes is not a computer glitch, it's the sulks.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved