Welcome to WebmasterWorld Guest from 184.72.177.182

Forum Moderators: open

Message Too Old, No Replies

Cloaking bots list of Ip and agents

anyone?

     
2:27 am on Apr 22, 2008 (gmt 0)

Junior Member

5+ Year Member

joined:Dec 16, 2007
posts:117
votes: 0


Anyone knows where to find a list of ips and agents that are
used by bots?

I read the original post (Cloaking : Webmaster World Knowledge base excerpt.) here, but the links are dead.

8:40 pm on Apr 22, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


There are a number of resources for IP addresses and user agents of search engine spiders. You could try searching for "search engine spider ip addresses" at your favorite search engine.
2:37 am on Apr 23, 2008 (gmt 0)

Junior Member

5+ Year Member

joined:Dec 16, 2007
posts:117
votes: 0


thank you, i did find quite a few pages, just thought may be you could recommend your preferred site for that (without breaking the forum rules of course)
9:39 pm on Apr 30, 2008 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14624
votes: 88


There are a lot of lists out there but some of the lists also use legacy IPs now utilized in other ways, such as mobile proxies, etc.

When you get a list, run reverse DNS on the list and make sure all of the IPs clearly identify themselves as the crawler such as "crawl-*.googlebot.com." so you don't accidentally let obsolete IPs see the wrong pages.