Welcome to WebmasterWorld Guest from 22.214.171.124
[edited by: wilderness at 1:34 pm (utc) on Feb 21, 2018]
UA s and their behavior are documented for information, nothing more. Each webmaster may then determine whether that agent is of benefit to their specific interests or notOne of the functions of this forum is to provide a service to those interested in this information.
I'm sorry for the inconvenience here! DotBot will essentially crawl every link which we can reach/follow.This of course avoided the specific issue I asked them about, e.g. the sudden image file requests.
Moz is an Inbound Marketing company that creates analytics software for online marketers.Whether they are still in the marketing business is unknown.
rogerbot/1.0 (http://www.moz.com/dp/rogerbot, email@example.com)
rogerbot ... only goes after robots.txt and rss feedsThey get information from RSS feeds. The only time I ever see them is when they're following-up a link from (a different site's) RSS feed. That is, they don't send a referer, but I know where they're getting the name of the requested page.