homepage Welcome to WebmasterWorld Guest from 54.81.170.186
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
To "Allow" or not to "Allow" in robots.txt
Can it force a deep crawl?
webtamers




msg:1525813
 9:29 pm on Nov 5, 2000 (gmt 0)

Hi Folks,

Does anyone really understand the "Allow" in a robots.txt file?

Here's a link to an IETF Internet draft on webcrawler
that explains the Allow directive:

[info.webcrawler.com...]

Whe I first ran into this it was on a site about promotion that claimed it would force some robots to crawl your whole site. Any truth to that?

Here's an example of what they say to do:

User-agent: *
Disallow:
Allow: /

Any feedback would be appreciated...

Thanks,

 

tedster




msg:1525814
 12:34 am on Nov 6, 2000 (gmt 0)

webtamers,

Here's the way I see it.

There isn't any way for a site to force a particular behavior from any spider. The companies who operate the spiders program them according to their own wishes. Some spiders do not ever seem to check a robots.txt file at all (Googlebot for instance). There certainly is no way to force a robot to crawl your whole site, and there wasn't any four years ago when this short paper was written.

This paper looks like either notes or a proposal. It's not a report on actual existing standards ... and it's four years old. There are updates from this author on the webcrawler site for 1997 and 1998, then they stop.

Whatever this paper is or was, it is just not the way things actually work. Stick with "disallow".

webtamers




msg:1525815
 8:03 am on Nov 6, 2000 (gmt 0)

Dear Tedster,

Thank you very much! Common sense tells me you are right about not being able to force the robots to do anything. I think I was overcomplicating the whole thing. The "promotion" info I read may have been utter nonesense.

I've been studying this whole promotion thing voratiously, and have been able to get some very good results, but after obsessing over every bit of advice from WPG, and trying to make the "perfect" page, I'm starting to get it that you just give the engines what they want instead of agonizing over perfect prominence or whatever. I've even decided to quit using words in alt tags and comments, or anything else that could resemble spam. I want these pages to stick!

Thanks again,

tedster




msg:1525816
 9:53 am on Nov 6, 2000 (gmt 0)

Sounds like you are evolving a very sane approach. I know that after a period of experimenting with various "tricks", I returned to an increased focus on writing good content. My client sites are better off for it.

Traffic is not equivalent to ranking, and conversions are not the same as traffic. I feel like I am keeping my eye on the ball a lot more -- and that ball is actual paying customers for my clients.

As in your case, my WPG experience provided a good foundation and a jump in awareness that will always be useful. But I too have stopped obsessing.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved