Hi, Masters! Working on prevision of production website at dedicated hosting environment. Want collaborate on some preliminary steps in particular [bot mashing]. Can you share your ideas on this practice. Thank you!
not2easy
4:45 pm on Dec 5, 2019 (gmt 0)
Hi rivsrush, if you mean blocking unwanted bot traffic, that is an ongoing topic around here. Step one is to determine what you want to block. Examining your raw access logs helps you with that. Then you can use the past many years of accumulated information in the Server Farms [webmasterworld.com] threads.
As for UA blocking, those are generally brought to our attention by a thread for each UA that includes the information and experience of those of us who were/are interested in that UA.
Unfortunately, because of its nature it is a self help project. I don't even block the same IPs on every domain - at least until they visit. That makes passing on some list a waste of time. See what is unwanted for your sites, then block those with the data here available via our Search tools. On desktop, that is found in the upper right corner of each page and you can select which SE results you want. On mobile it's in the compact menu (the hamburger thing).
What to block is up to you; how to do that offers options and of course folks are here to point the way.
rivsrush
7:19 pm on Dec 5, 2019 (gmt 0)
not2easy, very appreciate you Input. I am not familiar with abbreviation at "As for UA blocking..." for yours above post, can you put more light on this. Thank you! P.S. The tools link you mentioned above response with page contains server error output.
not2easy
8:04 pm on Dec 5, 2019 (gmt 0)
The "Search tools" I mentioned above was our Search, not the Tools. There are multiple options in the search, those options were what I was referring to. For example, specific kinds of IP searches return different formats for results. I have found the results for Google (in our site search) show more relevant results. I like DDG but they don't seem get the point for this kind of search.
UA is short for User Agent, this is the name a robot calls itself. Googlebot is one of those UAs. Other examples are "DotBot/1.1" and "Daum/4.1" so you get the idea.
