Welcome to WebmasterWorld Guest from 22.214.171.124 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Spider visiting if you pay Looksmart? jlara
I have seen the architect spider only visit one domain of 700 in the last six months. It just happens to be the only domain I paid for a listing in Looksmart with.
Is there any other submissions to Excite that have been getting the spider to visit?
ArchitextSpider has visited 3 domains in the past 24 hrs. Not one of the more active spiders.
oops! forgot to add, the sites are not paid sites mivox
I've gotten visits from Architext, but no listing in Excite. First submitted in October...
Most of the spider visits only requested robots.txt and logged a hit on the base directory. Didn't seem to actually 'spider' anything.
ArchitextSpider occasionally visits sites that have been in their database for years. It so happens that these are the only ones that I have paid for at LookSmart (but spidering was already ongoing). No new sites have been spidered, but I have not paid for any new ones yet at LookSmart. littleman
Add google to the list of spiders who will visit a site after it gets a listing in looksmart. jeremy goodrich
If you're right about that, then google doesn't necessarily obey robots.txt.
That is huge. From
LookSmart's Robots.txt file [ looksmart.com]: User-agent: Googlebot Disallow:
From a log file I just looked up: "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
So if you're correct, good old google is indeed ignoring robots.txt. Unless they have a special deal with LookSmart that we all don't know about.
I have thought for awhile now that Google will scoot over to Looksmart to get newly listed pages. This is why I list the page first in Looksmart before anything else. Brett_Tabke
If there were a star there jeremy, that would ban Google. No star in the disallow field means it is ok. (I think there was another thread around here where I misspoke about that).
(smacks head, looks really foolish).
Oops. (And here I was thinking I was on to something, too bad I know realize it smells like manure :)
Until recently I've never really dealt personally with robots.txt. (note to self, star means ban, and nothing means do whatever...) Gotcha.