| 5:52 pm on Dec 8, 2000 (gmt 0)|
ArchitextSpider has visited 3 domains in the past 24 hrs. Not one of the more active spiders.
| 8:46 pm on Dec 8, 2000 (gmt 0)|
oops! forgot to add,
the sites are not paid sites
| 9:58 pm on Dec 11, 2000 (gmt 0)|
I've gotten visits from Architext, but no listing in Excite. First submitted in October...
Most of the spider visits only requested robots.txt and logged a hit on the base directory. Didn't seem to actually 'spider' anything.
| 2:08 am on Dec 12, 2000 (gmt 0)|
ArchitextSpider occasionally visits sites that have been in their database for years. It so happens that these are the only ones that I have paid for at LookSmart (but spidering was already ongoing). No new sites have been spidered, but I have not paid for any new ones yet at LookSmart.
| 1:54 am on Jun 10, 2001 (gmt 0)|
Add google to the list of spiders who will visit a site after it gets a listing in looksmart.
| 12:57 pm on Jun 10, 2001 (gmt 0)|
If you're right about that, then google doesn't necessarily obey robots.txt.
That is huge. From LookSmart's Robots.txt file [looksmart.com]:
From a log file I just looked up: "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
So if you're correct, good old google is indeed ignoring robots.txt. Unless they have a special deal with LookSmart that we all don't know about.
| 2:06 pm on Jun 10, 2001 (gmt 0)|
I have thought for awhile now that Google will scoot over to Looksmart to get newly listed pages. This is why I list the page first in Looksmart before anything else.
| 12:45 pm on Jun 11, 2001 (gmt 0)|
If there were a star there jeremy, that would ban Google. No star in the disallow field means it is ok. (I think there was another thread around here where I misspoke about that).
| 12:57 pm on Jun 11, 2001 (gmt 0)|
(smacks head, looks really foolish).
Oops. (And here I was thinking I was on to something, too bad I know realize it smells like manure :)
Until recently I've never really dealt personally with robots.txt. (note to self, star means ban, and nothing means do whatever...) Gotcha.