Spider visiting if you pay Looksmart?

ArchitextSpider has visited 3 domains in the past 24 hrs. Not one of the more active spiders.

c

oops! forgot to add,
the sites are not paid sites

I've gotten visits from Architext, but no listing in Excite. First submitted in October...

Most of the spider visits only requested robots.txt and logged a hit on the base directory. Didn't seem to actually 'spider' anything.

ArchitextSpider occasionally visits sites that have been in their database for years. It so happens that these are the only ones that I have paid for at LookSmart (but spidering was already ongoing). No new sites have been spidered, but I have not paid for any new ones yet at LookSmart.

Add google to the list of spiders who will visit a site after it gets a listing in looksmart.

If you're right about that, then google doesn't necessarily obey robots.txt.

That is huge. From LookSmart's Robots.txt file [looksmart.com]:
User-agent: Googlebot
Disallow:

From a log file I just looked up: "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

So if you're correct, good old google is indeed ignoring robots.txt. Unless they have a special deal with LookSmart that we all don't know about.

I have thought for awhile now that Google will scoot over to Looksmart to get newly listed pages. This is why I list the page first in Looksmart before anything else.

If there were a star there jeremy, that would ban Google. No star in the disallow field means it is ok. (I think there was another thread around here where I misspoke about that).

From: the Robots Exclusion Standard [robotstxt.org]:
To allow all robots complete access:
User-agent: *
Disallow:

(smacks head, looks really foolish).

Oops. (And here I was thinking I was on to something, too bad I know realize it smells like manure :)

Until recently I've never really dealt personally with robots.txt. (note to self, star means ban, and nothing means do whatever...) Gotcha.