Forum Moderators: open

Message Too Old, No Replies

Gain by disallowing Googlebot in robots.txt

a site I exchange links with disallows google - why?

         

decstar

11:06 pm on Jun 5, 2003 (gmt 0)

10+ Year Member



I exchanged links with a relatively similar site last week when I went to look at googles cache of the links page (to be sure they weren't hiding outgoing links) I discovered that there was no cache. A quick look at their robots.txt shows this:

# FULL access (Alta Vista)
User-agent: Scooter
Disallow:

# FULL access (Excite)
User-agent: ArchitextSpider
Disallow:

# FULL access (Google)
User-agent: Googlebot
Disallow:

# FULL access (Inktomi)
User-agent: Slurp
Disallow:

# PARTIAL access (All Spiders)
# NO access (All Spiders)
User-agent: *
Disallow:

Is it possible that they are adding new exchange links but that google isn't seeing them as it is keeping the old crawl of the site on file. That would give them the benefit of the incoming links, but not the downside of the outgoing exchanged links.

Any thoughts?

pageoneresults

11:10 pm on Jun 5, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



# FULL access (Google)
User-agent: Googlebot
Disallow:

They are allowing Googlebot. All of those lines that have the # FULL access are allowed. If there was a forward slash in the Disallow field like this...

User-agent: Googlebot
Disallow: /

Then they would be disallowing Googlebot.

Netizen

11:12 pm on Jun 5, 2003 (gmt 0)

10+ Year Member



If the Disallow is blank then that allows full access. Take a look here [robotstxt.org] for a full description.

decstar

8:04 am on Jun 6, 2003 (gmt 0)

10+ Year Member



Thanks