Forum Moderators: open

Message Too Old, No Replies

tracerlock

spidering 2 pages every 15 minutes

         

bobriggs

3:23 pm on Jun 1, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



209.61.182.37
tracerlock.com
UA: libwww-perl/5.47.

I know that this service checks AV for new submissions based on keywords or kw phrases, and then emails the user the new sites.

It has started spidering my site, taking only 2 pages which I recently submitted. I thought this thing only checked AV, not the sites themselves.

Anybody know what it's doing or have you seen similar activity?

If it's checking for 404's, then why does it keep coming back every 15 or 30 minutes?

skirril

11:58 pm on Jun 4, 2001 (gmt 0)

10+ Year Member



Could it check for uptime as well?

After reading the blurb at [tracerlock.com...] I think its a good candidate to block, for it allows the competition to monitor your site without being detected, and if said competition was locked out, it could still get past the lock with said utility.

Does it honor robots.txt?

bobriggs

3:09 am on Jun 5, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I thought about blocking them.

And I suppose that it would really just be the competition and not normally a future customer. So possibly I might lose a few hits from the good guys.

If the site is well indexed, the comp probably will find out about it anyway.

Doesn't even look at robots.txt. I can block with htaccess, it only uses one IP (so far)

And I subscribed after I posted this and yesterday I got two email reports from them on new submissions. I can see the need for spidering now because it does some kind of summary and doesn't give just the URL.

bobriggs

12:27 am on Oct 1, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Just received this (email) for anyone interested in tracerlock:

A few days ago, some search engines began blocking a lot of free search-engine-monitoring services from connecting to their Web sites. As you've probably noticed, this has reduced the number of matches that TracerLock has been reporting for your search terms. (Of the search engines that are not blocked, there are still problems, because some of them have gone into a "slump" where they go out searching for new pages but add very few new pages to their database. For example, if you go to Alta Vista Advanced Search at [altavista.com...] and search for all new matches for "books" from 22/9/2001 to 22/9/2001, only 20 matches are listed -- so for more specific queries, very few matches at all will be found.)

If you are a former user of the Informant that was brought over to TracerLock as a result of the merger -- these problems have nothing to do with the Informant merger, as it is a problem that affects any company that does search engine monitoring...