Welcome to WebmasterWorld Guest from 54.91.16.95

Forum Moderators: open

Any list of bad bots to block on site?

     
9:15 pm on Sep 28, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts:701
votes: 8


Hi is there any advise or list of bad bots to block for website? Please advise thanks!
9:26 pm on Sept 28, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:10393
votes: 597


Hi born2run,

Again, all that info can be found in the Search Engine Spider & User Agent ID Forum [webmasterworld.com]

There is no "list of bad bots" since that so-called list would be different for each website. What may be "bad" for you might be "good" for me.

I would recommend never taking anyone's advice on what to block or allow. Do the research and make that decision yourself.

There is no quick fix to website security. You need to do the leg work yourself. Blocking without full understanding of what you're blocking could have a significant negative affect on your website.

Read here: Blocking Methods [webmasterworld.com]
10:34 pm on Sept 28, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts:701
votes: 8


Thanks! I'm analysing my logs and it seems the most active bots are stuff like SemrushBot and AhrefsBot :-(
11:14 pm on Sept 28, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:10393
votes: 597


Semrush - [webmasterworld.com...]

AhrefsBot - [webmasterworld.com...]


Note: If you publish ads, it may be to your benefit to allow bots that offer products about marketing data or advertising. These products might be used by advertisers who bid for spots on your pages. If you block those bots, the advertisers may never know those ad spots are available.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members