Welcome to WebmasterWorld Guest from 54.198.78.121
Forum Moderators: bakedjake
Is there any other submissions to Excite that have been getting the spider to visit?
c
That is huge. From LookSmart's Robots.txt file [looksmart.com]:
User-agent: Googlebot
Disallow:
From a log file I just looked up: "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
So if you're correct, good old google is indeed ignoring robots.txt. Unless they have a special deal with LookSmart that we all don't know about.
From: the Robots Exclusion Standard [robotstxt.org]:
To allow all robots complete access:
User-agent: *
Disallow:
Oops. (And here I was thinking I was on to something, too bad I know realize it smells like manure :)
Until recently I've never really dealt personally with robots.txt. (note to self, star means ban, and nothing means do whatever...) Gotcha.