Welcome to WebmasterWorld Guest from 54.82.93.116

Forum Moderators: Ocean10000 & keyplyr

Robot's User Agent String

What webmasters want

     
9:34 pm on Oct 14, 2018 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 890


To Bot Runners

If you are using a robot, bot, spider, crawler, etc to gather data from websites, it is highly recommended (and proper netiquette) to inform the site owner of certain information.

An acceptable User Agent (UA) string would look something like this:

Example_User_Agent/1.0 (+https://www.example.com/robot_info.html)

When creating your UA string, it is important to include a link to your robot info page. To verify ownership, this information page should be hosted on your TLD, the same domain as you do business at.

The robot info page should explicitly describe:
Who you are?
Does your bot support robots.txt?
What UA should be used in robots.txt?
What is your verified crawl IP range(s)?
What data do you want at our websites?
What you will do with the data you retrieve?
Why we should allow you access to our property? How will it benefit the content owner?

If these things are not clearly presented to site owners, you risk being blocked by default.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members