Forum Moderators: open

Message Too Old, No Replies

Anyone allowing AhrefsBot?

         

born2run

5:41 pm on Jun 9, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Here is the ahrefs.com bot UA:

Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)

I did some research here, and so for now I've allowed the bot as I run ads. However it's hitting my site at around 50,000 per week.

Is it wise to put delays in the robots.txt or even just block it completely? Please advise. Thanks!

keyplyr

6:49 pm on Jun 9, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I allow AhrefsBot access to all pages and some other data they need to properly represent my pages to their advertising clients.

They sell marketing data to Adwords and other ad publishers. I publish Adsense so it is in my interest.

I do not allow AhrefsBot to scrape images, PDFs, audio files, video files and other data I feel is unnecessary to that process.

Many bot runners have more than one agenda. Its important to descriminate this.

Related: [webmasterworld.com...]

topr8

10:06 pm on Jun 9, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



very full answer from keyplyr as usual

i have a few sites that are a free for all - basically test sites in a way.
but mostly i block it because mostly i sell products not ads.

born2run

1:43 am on Jun 10, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



As usual thanks keyplyr. I'll allow ahrefs bot for now.

lucy24

3:05 am on Jun 10, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



They’ve been in the “no skin off my nose” bin for a few years. I think at one time I did block and/or disallow them, but that was a long time ago.