Spider visiting if you pay Looksmart?

[jlara]

I have seen the architect spider only visit one domain of 700 in the last six months. It just happens to be the only domain I paid for a listing in Looksmart with.

Is there any other submissions to Excite that have been getting the spider to visit?

[cirelle]

ArchitextSpider has visited 3 domains in the past 24 hrs. Not one of the more active spiders.

c

[cirelle]

oops! forgot to add,
the sites are not paid sites

[mivox]

I've gotten visits from Architext, but no listing in Excite. First submitted in October...

Most of the spider visits only requested robots.txt and logged a hit on the base directory. Didn't seem to actually 'spider' anything.

[budterm]

ArchitextSpider occasionally visits sites that have been in their database for years. It so happens that these are the only ones that I have paid for at LookSmart (but spidering was already ongoing). No new sites have been spidered, but I have not paid for any new ones yet at LookSmart.

[littleman]

Add google to the list of spiders who will visit a site after it gets a listing in looksmart.

[jeremy goodrich]

If you're right about that, then google doesn't necessarily obey robots.txt.

That is huge. From LookSmart's Robots.txt file [looksmart.com]:
User-agent: Googlebot
Disallow:

From a log file I just looked up: "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

So if you're correct, good old google is indeed ignoring robots.txt. Unless they have a special deal with LookSmart that we all don't know about.

[ihelpyou]

I have thought for awhile now that Google will scoot over to Looksmart to get newly listed pages. This is why I list the page first in Looksmart before anything else.

[Brett_Tabke]

If there were a star there jeremy, that would ban Google. No star in the disallow field means it is ok. (I think there was another thread around here where I misspoke about that).

From: the Robots Exclusion Standard [robotstxt.org]:
To allow all robots complete access:
User-agent: *
Disallow:

[jeremy goodrich]

(smacks head, looks really foolish).

Oops. (And here I was thinking I was on to something, too bad I know realize it smells like manure :)

Until recently I've never really dealt personally with robots.txt. (note to self, star means ban, and nothing means do whatever...) Gotcha.