Welcome to WebmasterWorld Guest from 54.211.82.105

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Link Program Bots

disallow?

     
8:21 am on Mar 10, 2002 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:7028
votes: 178


Saw that LinksManager crawled my site a few days ago. Then I started to see site referrers from LinksManager.com. Then today I received an email from a domain, using this service for their links pages, demanding I reciprocate a link back to them... nice try :-)

Of course I ignored the email and added the bot to my disallow list, but my concern is that my site will be seen (by Google possibly) as participating in link farm tactics.

9:12 am on Mar 10, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 17, 2001
posts:409
votes: 0


Your site is not participating since it has no links to "bad neighborhood" (this term is almost as catchy as the (in)famous Google dance:))

The most likely scenario they will remove your link soon (otherwise their tactic would not work).

6:12 am on Mar 14, 2002 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:7028
votes: 178


LinkManager apparently does not respect the robots.txt disallow. Shame on them!
11:01 pm on Mar 18, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 13, 2004
posts:411
votes: 0


Hello,

My name is Joel Lesser and I am the owner and CEO of Creative NetVentures, Inc, parent company of LinksManager.com.

I was notified that LinksManager and our LinkChecker bot was being discussed here in this forum so wanted to take a moment and reply to the concerns addressed here.

LinksManager.com is a patent pending application service that manages links pages for hundreds of websites that are powered by LinksManager. We are not a link farm or an automated service that pushes links to our customer's accounts. The majority of an account's links are submitted by webmasters requesting inclusion on a website's links pages powered by LinksManager.

LinksManager powers links pages for all sorts of sites such as http://www.1stforparties.com and http://www.localmusicdirectory.com and
large portals such as http://gamblinglinks.com.

When we designed LinksManager back in the late 90's, we incorporated a LinkChecker bot into the system so that when a link is added to a LinksManager powered website, the administrator of that account is notified when the link is being reciprocated. The only way to check for reciprocation is to run a link checker bot through the site looking for reciprocation.

The link checkers only browse text and does not download high bandwidth files such as graphics or rich media. The checkers are intentionally programmed not to put excessive load onto any one site. For more information about our bot: http://linksmanager.com/linkchecker.html.

It is true that the LinksManager link checker currently ignores robots.txt. We have struggled with this in the past.. if the checker ignores robots.txt, then reciprocation can be verified. If the link checker skips sites that have robots.txt, it's impossible for the LinksManager LinkChecker bot to do it's job effectively.

We are happy to add your domain to our ignore file upon request.

As robots.txt becomes more popular with webmasters, we are reanalyzing how reciprocation can be reported without having to spider the reciprocating site without the use of the LinkChecker bot. This takes time and an alternative solution will hopefully be forthcoming in the not too distant future. Comments such as yours on this forum help us realize how important this is to webmasters and I appreciate you discussing it here so we can be aware of your concerns.

Kindest Regards,
Joel Lesser

11:10 pm on Mar 18, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 6, 2002
posts:742
votes: 0


Hello Joel and thanks for explaining how your service works. None of our sites are part of any referral link program, but you still crawl our sites. I went to your website several months ago and found out that you do not honor robots.txt. Can any member of your service request a site be crawled, even if that site is not part of a referral program? If so, what type of information are you providing to your members about sites you crawl for members? Something about this just doesn't feel right to me, but maybe I'm just not seeing the full picture of what's going on.

(edited by: MarkHutch at 6:55 pm (utc) on Mar. 26, 2002)

11:18 pm on Mar 18, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2001
posts:1472
votes: 0


>>>If the link checker skips sites that have robots.txt, it's impossible for the LinksManager LinkChecker bot to do it's job effectively.

Actually, it **would** be doing it's job most effectively. Otherwise, what's the use for robots.txt? That's what it's there for- to ban unwelcome web robots.

LinksManager does obey .htaccess though.

11:21 pm on Mar 18, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38066
votes: 15


What is the exact agent name used?
11:26 pm on Mar 18, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2001
posts:1472
votes: 0


Here's one user agent:

LinksManager.com (http://linksmanager.com/linkchecker.html)

2:18 am on Mar 19, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 13, 2004
posts:411
votes: 0


http://linksmanager.com/linkchecker.html is the proper name of the agent so that users who see this in their log reports can learn more about the checker and request to be added to the ignore file if needed.

Mark asked "Can any member of your service request a site be crawled?" .. the answer is yes however we don't make it easy for a webmaster to import thousands of links and then request that they all be checked for reciprocation.

If your site is not swapping links with a LinksManager powered site, simply send your domain name(s) to checker@linksmanager.com to be added to the ignore file.

If you have any other specific questions, let me know and I'll be happy to answer.

Regards,
Joel Lesser

Woz

2:37 am on Mar 19, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 13, 2000
posts:4823
votes: 0


Hey Joel, thanks for dropping in.

Following from your posts, I guess the next question is, can your site be slated for checking by one of your members even though you may not be aware they are linking to you?

And following up from that, assuming you see the linksmanager bot in your logs, which implies someone is linking to you, is it possible to request a list of the site/s linking to you to enable direct dialog before deciding on whether to ask to be ignored?

And finally, if this is a link tracking system, are the hits tracked through linksmanager as well and if so how do they appear in logs?

Again, thanks for helping out with useful info,

Onya
Woz

2:43 am on Mar 19, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38066
votes: 15


Well, since this thread has turned into something other than the original post, please take any further questions up with Joel on his site.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members