Welcome to WebmasterWorld Guest from 107.20.54.98

Forum Moderators: bakedjake

Message Too Old, No Replies

Spider visiting if you pay Looksmart?

     
7:51 pm on Dec 7, 2000 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 1, 2000
posts:111
votes: 0


I have seen the architect spider only visit one domain of 700 in the last six months. It just happens to be the only domain I paid for a listing in Looksmart with.

Is there any other submissions to Excite that have been getting the spider to visit?

cirelle

5:52 pm on Dec 8, 2000 (gmt 0)

Inactive Member
Account Expired

 
 


ArchitextSpider has visited 3 domains in the past 24 hrs. Not one of the more active spiders.

c

cirelle

8:46 pm on Dec 8, 2000 (gmt 0)

Inactive Member
Account Expired

 
 


oops! forgot to add,
the sites are not paid sites
9:58 pm on Dec 11, 2000 (gmt 0)

Senior Member

WebmasterWorld Senior Member mivox is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 6, 2000
posts:3928
votes: 0


I've gotten visits from Architext, but no listing in Excite. First submitted in October...

Most of the spider visits only requested robots.txt and logged a hit on the base directory. Didn't seem to actually 'spider' anything.

2:08 am on Dec 12, 2000 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 25, 2000
posts:174
votes: 0


ArchitextSpider occasionally visits sites that have been in their database for years. It so happens that these are the only ones that I have paid for at LookSmart (but spidering was already ongoing). No new sites have been spidered, but I have not paid for any new ones yet at LookSmart.
1:54 am on June 10, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


Add google to the list of spiders who will visit a site after it gets a listing in looksmart.
12:57 pm on June 10, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 4, 2000
posts:3468
votes: 0


If you're right about that, then google doesn't necessarily obey robots.txt.

That is huge. From LookSmart's Robots.txt file [looksmart.com]:
User-agent: Googlebot
Disallow:

From a log file I just looked up: "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

So if you're correct, good old google is indeed ignoring robots.txt. Unless they have a special deal with LookSmart that we all don't know about.

2:06 pm on June 10, 2001 (gmt 0)

Full Member

10+ Year Member

joined:July 12, 2000
posts:265
votes: 0


I have thought for awhile now that Google will scoot over to Looksmart to get newly listed pages. This is why I list the page first in Looksmart before anything else.
12:45 pm on June 11, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


If there were a star there jeremy, that would ban Google. No star in the disallow field means it is ok. (I think there was another thread around here where I misspoke about that).


From: the Robots Exclusion Standard [robotstxt.org]:
To allow all robots complete access:
User-agent: *
Disallow:
12:57 pm on June 11, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 4, 2000
posts:3468
votes: 0


(smacks head, looks really foolish).

Oops. (And here I was thinking I was on to something, too bad I know realize it smells like manure :)

Until recently I've never really dealt personally with robots.txt. (note to self, star means ban, and nothing means do whatever...) Gotcha.