homepage Welcome to WebmasterWorld Guest from 54.234.228.64
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
Link Program Bots
disallow?
keyplyr




msg:902164
 8:21 am on Mar 10, 2002 (gmt 0)

Saw that LinksManager crawled my site a few days ago. Then I started to see site referrers from LinksManager.com. Then today I received an email from a domain, using this service for their links pages, demanding I reciprocate a link back to them... nice try :-)

Of course I ignored the email and added the bot to my disallow list, but my concern is that my site will be seen (by Google possibly) as participating in link farm tactics.

 

starec




msg:902165
 9:12 am on Mar 10, 2002 (gmt 0)

Your site is not participating since it has no links to "bad neighborhood" (this term is almost as catchy as the (in)famous Google dance:))

The most likely scenario they will remove your link soon (otherwise their tactic would not work).

keyplyr




msg:902166
 6:12 am on Mar 14, 2002 (gmt 0)

LinkManager apparently does not respect the robots.txt disallow. Shame on them!

CNVI




msg:902167
 11:01 pm on Mar 18, 2002 (gmt 0)

Hello,

My name is Joel Lesser and I am the owner and CEO of Creative NetVentures, Inc, parent company of LinksManager.com.

I was notified that LinksManager and our LinkChecker bot was being discussed here in this forum so wanted to take a moment and reply to the concerns addressed here.

LinksManager.com is a patent pending application service that manages links pages for hundreds of websites that are powered by LinksManager. We are not a link farm or an automated service that pushes links to our customer's accounts. The majority of an account's links are submitted by webmasters requesting inclusion on a website's links pages powered by LinksManager.

LinksManager powers links pages for all sorts of sites such as http://www.1stforparties.com and http://www.localmusicdirectory.com and
large portals such as http://gamblinglinks.com.

When we designed LinksManager back in the late 90's, we incorporated a LinkChecker bot into the system so that when a link is added to a LinksManager powered website, the administrator of that account is notified when the link is being reciprocated. The only way to check for reciprocation is to run a link checker bot through the site looking for reciprocation.

The link checkers only browse text and does not download high bandwidth files such as graphics or rich media. The checkers are intentionally programmed not to put excessive load onto any one site. For more information about our bot: http://linksmanager.com/linkchecker.html.

It is true that the LinksManager link checker currently ignores robots.txt. We have struggled with this in the past.. if the checker ignores robots.txt, then reciprocation can be verified. If the link checker skips sites that have robots.txt, it's impossible for the LinksManager LinkChecker bot to do it's job effectively.

We are happy to add your domain to our ignore file upon request.

As robots.txt becomes more popular with webmasters, we are reanalyzing how reciprocation can be reported without having to spider the reciprocating site without the use of the LinkChecker bot. This takes time and an alternative solution will hopefully be forthcoming in the not too distant future. Comments such as yours on this forum help us realize how important this is to webmasters and I appreciate you discussing it here so we can be aware of your concerns.

Kindest Regards,
Joel Lesser

MarkHutch




msg:902168
 11:10 pm on Mar 18, 2002 (gmt 0)

Hello Joel and thanks for explaining how your service works. None of our sites are part of any referral link program, but you still crawl our sites. I went to your website several months ago and found out that you do not honor robots.txt. Can any member of your service request a site be crawled, even if that site is not part of a referral program? If so, what type of information are you providing to your members about sites you crawl for members? Something about this just doesn't feel right to me, but maybe I'm just not seeing the full picture of what's going on.

(edited by: MarkHutch at 6:55 pm (utc) on Mar. 26, 2002)

Key_Master




msg:902169
 11:18 pm on Mar 18, 2002 (gmt 0)

>>>If the link checker skips sites that have robots.txt, it's impossible for the LinksManager LinkChecker bot to do it's job effectively.

Actually, it **would** be doing it's job most effectively. Otherwise, what's the use for robots.txt? That's what it's there for- to ban unwelcome web robots.

LinksManager does obey .htaccess though.

Brett_Tabke




msg:902170
 11:21 pm on Mar 18, 2002 (gmt 0)

What is the exact agent name used?

Key_Master




msg:902171
 11:26 pm on Mar 18, 2002 (gmt 0)

Here's one user agent:

LinksManager.com (http://linksmanager.com/linkchecker.html)

CNVI




msg:902172
 2:18 am on Mar 19, 2002 (gmt 0)

http://linksmanager.com/linkchecker.html is the proper name of the agent so that users who see this in their log reports can learn more about the checker and request to be added to the ignore file if needed.

Mark asked "Can any member of your service request a site be crawled?" .. the answer is yes however we don't make it easy for a webmaster to import thousands of links and then request that they all be checked for reciprocation.

If your site is not swapping links with a LinksManager powered site, simply send your domain name(s) to checker@linksmanager.com to be added to the ignore file.

If you have any other specific questions, let me know and I'll be happy to answer.

Regards,
Joel Lesser

Woz




msg:902173
 2:37 am on Mar 19, 2002 (gmt 0)

Hey Joel, thanks for dropping in.

Following from your posts, I guess the next question is, can your site be slated for checking by one of your members even though you may not be aware they are linking to you?

And following up from that, assuming you see the linksmanager bot in your logs, which implies someone is linking to you, is it possible to request a list of the site/s linking to you to enable direct dialog before deciding on whether to ask to be ignored?

And finally, if this is a link tracking system, are the hits tracked through linksmanager as well and if so how do they appear in logs?

Again, thanks for helping out with useful info,

Onya
Woz

Brett_Tabke




msg:902174
 2:43 am on Mar 19, 2002 (gmt 0)

Well, since this thread has turned into something other than the original post, please take any further questions up with Joel on his site.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved