homepage Welcome to WebmasterWorld Guest from 107.22.25.119
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library : Charter : Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Baidu's IPs - Which Are Legit?
incrediBILL




msg:4475769		 3:05 am on Jul 15, 2012 (gmt 0)

Most of these IPs resolve to Baidu's reverse DNS like baiduspider-180-76-5-180.crawl.baidu.com but many don't as well. Currently I only authorize the ones that do full trip DNS validation.

Anyone know if they're real Baidu IPs or not?

USER AGENT: "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
IP: 119.63.196.10
IP: 119.63.196.102
IP: 119.63.196.103
IP: 119.63.196.104
IP: 119.63.196.105
IP: 119.63.196.106
IP: 119.63.196.107
IP: 119.63.196.108
IP: 119.63.196.109
IP: 119.63.196.11
IP: 119.63.196.110
IP: 119.63.196.111
IP: 119.63.196.112
IP: 119.63.196.113
IP: 119.63.196.114
IP: 119.63.196.115
IP: 119.63.196.116
IP: 119.63.196.117
IP: 119.63.196.119
IP: 119.63.196.12
IP: 119.63.196.120
IP: 119.63.196.13
IP: 119.63.196.15
IP: 119.63.196.16
IP: 119.63.196.17
IP: 119.63.196.18
IP: 119.63.196.19
IP: 119.63.196.20
IP: 119.63.196.21
IP: 119.63.196.22
IP: 119.63.196.23
IP: 119.63.196.24
IP: 119.63.196.25
IP: 119.63.196.26
IP: 119.63.196.27
IP: 119.63.196.39
IP: 119.63.196.40
IP: 119.63.196.41
IP: 119.63.196.42
IP: 119.63.196.43
IP: 119.63.196.45
IP: 119.63.196.47
IP: 119.63.196.48
IP: 119.63.196.50
IP: 119.63.196.51
IP: 119.63.196.52
IP: 119.63.196.53
IP: 119.63.196.55
IP: 119.63.196.56
IP: 119.63.196.57
IP: 119.63.196.73
IP: 119.63.196.74
IP: 119.63.196.75
IP: 119.63.196.76
IP: 119.63.196.77
IP: 119.63.196.78
IP: 119.63.196.79
IP: 119.63.196.80
IP: 119.63.196.81
IP: 119.63.196.82
IP: 119.63.196.83
IP: 119.63.196.84
IP: 119.63.196.85
IP: 119.63.196.86
IP: 119.63.196.88
IP: 119.63.196.89
IP: 123.125.71.100
IP: 123.125.71.101
IP: 123.125.71.102
IP: 123.125.71.103
IP: 123.125.71.104
IP: 123.125.71.105
IP: 123.125.71.106
IP: 123.125.71.107
IP: 123.125.71.108
IP: 123.125.71.109
IP: 123.125.71.110
IP: 123.125.71.111
IP: 123.125.71.112
IP: 123.125.71.113
IP: 123.125.71.114
IP: 123.125.71.115
IP: 123.125.71.116
IP: 123.125.71.117
IP: 123.125.71.70
IP: 123.125.71.71
IP: 123.125.71.81
IP: 123.125.71.83
IP: 123.125.71.94
IP: 123.125.71.95
IP: 123.125.71.96
IP: 123.125.71.97
IP: 123.125.71.98
IP: 123.125.71.99
IP: 180.76.5.100
IP: 180.76.5.103
IP: 180.76.5.111
IP: 180.76.5.137
IP: 180.76.5.141
IP: 180.76.5.142
IP: 180.76.5.145
IP: 180.76.5.146
IP: 180.76.5.147
IP: 180.76.5.148
IP: 180.76.5.150
IP: 180.76.5.151
IP: 180.76.5.155
IP: 180.76.5.159
IP: 180.76.5.161
IP: 180.76.5.162
IP: 180.76.5.166
IP: 180.76.5.167
IP: 180.76.5.170
IP: 180.76.5.171
IP: 180.76.5.172
IP: 180.76.5.176
IP: 180.76.5.177
IP: 180.76.5.178
IP: 180.76.5.179
IP: 180.76.5.180
IP: 180.76.5.181
IP: 180.76.5.182
IP: 180.76.5.183
IP: 180.76.5.185
IP: 180.76.5.187
IP: 180.76.5.189
IP: 180.76.5.190
IP: 180.76.5.192
IP: 180.76.5.193
IP: 180.76.5.194
IP: 180.76.5.195
IP: 180.76.5.196
IP: 180.76.5.197
IP: 180.76.5.49
IP: 180.76.5.51
IP: 180.76.5.54
IP: 180.76.5.55
IP: 180.76.5.57
IP: 180.76.5.58
IP: 180.76.5.59
IP: 180.76.5.62
IP: 180.76.5.63
IP: 180.76.5.64
IP: 180.76.5.65
IP: 180.76.5.67
IP: 180.76.5.87
IP: 180.76.5.88
IP: 180.76.5.90
IP: 180.76.5.92
IP: 180.76.5.93
IP: 180.76.5.94
IP: 180.76.5.95
IP: 180.76.5.96
IP: 180.76.5.97
IP: 180.76.5.98
IP: 180.76.5.99
IP: 180.76.6.21
IP: 180.76.6.212
IP: 180.76.6.213
IP: 180.76.6.222
IP: 180.76.6.223
IP: 180.76.6.224
IP: 180.76.6.231
IP: 180.76.6.28
IP: 180.76.6.29
IP: 180.76.6.36
IP: 180.76.6.37
IP: 199.36.73.116
IP: 220.181.108.100
IP: 220.181.108.107
IP: 220.181.108.110
IP: 220.181.108.81
IP: 220.181.108.84
IP: 220.181.108.93
IP: 220.181.108.95
IP: 220.181.108.97
IP: 222.186.24.59

USER AGENT: "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
IP: 67.159.56.162
IP: 72.11.144.119
IP: 119.63.196.10
IP: 119.63.196.103
IP: 119.63.196.105
IP: 119.63.196.106
IP: 119.63.196.11
IP: 119.63.196.113
IP: 119.63.196.116
IP: 119.63.196.117
IP: 119.63.196.118
IP: 119.63.196.120
IP: 119.63.196.124
IP: 119.63.196.125
IP: 119.63.196.13
IP: 119.63.196.16
IP: 119.63.196.19
IP: 119.63.196.20
IP: 119.63.196.23
IP: 119.63.196.25
IP: 119.63.196.26
IP: 119.63.196.27
IP: 119.63.196.28
IP: 119.63.196.30
IP: 119.63.196.31
IP: 119.63.196.32
IP: 119.63.196.38
IP: 119.63.196.40
IP: 119.63.196.42
IP: 119.63.196.43
IP: 119.63.196.44
IP: 119.63.196.45
IP: 119.63.196.47
IP: 119.63.196.53
IP: 119.63.196.54
IP: 119.63.196.57
IP: 119.63.196.58
IP: 119.63.196.60
IP: 119.63.196.61
IP: 119.63.196.76
IP: 119.63.196.78
IP: 119.63.196.80
IP: 119.63.196.82
IP: 119.63.196.88
IP: 119.63.196.89
IP: 119.63.196.9
IP: 119.63.196.93
IP: 119.63.196.94
IP: 119.63.196.96
IP: 123.125.67.164
IP: 123.125.71.100
IP: 123.125.71.101
IP: 123.125.71.102
IP: 123.125.71.103
IP: 123.125.71.104
IP: 123.125.71.105
IP: 123.125.71.106
IP: 123.125.71.107
IP: 123.125.71.108
IP: 123.125.71.109
IP: 123.125.71.110
IP: 123.125.71.111
IP: 123.125.71.112
IP: 123.125.71.113
IP: 123.125.71.114
IP: 123.125.71.115
IP: 123.125.71.116
IP: 123.125.71.117
IP: 123.125.71.12
IP: 123.125.71.13
IP: 123.125.71.14
IP: 123.125.71.15
IP: 123.125.71.16
IP: 123.125.71.17
IP: 123.125.71.18
IP: 123.125.71.19
IP: 123.125.71.20
IP: 123.125.71.21
IP: 123.125.71.22
IP: 123.125.71.23
IP: 123.125.71.24
IP: 123.125.71.25
IP: 123.125.71.26
IP: 123.125.71.27
IP: 123.125.71.28
IP: 123.125.71.29
IP: 123.125.71.30
IP: 123.125.71.31
IP: 123.125.71.32
IP: 123.125.71.33
IP: 123.125.71.34
IP: 123.125.71.35
IP: 123.125.71.36
IP: 123.125.71.38
IP: 123.125.71.39
IP: 123.125.71.40
IP: 123.125.71.41
IP: 123.125.71.42
IP: 123.125.71.43
IP: 123.125.71.44
IP: 123.125.71.45
IP: 123.125.71.46
IP: 123.125.71.47
IP: 123.125.71.48
IP: 123.125.71.49
IP: 123.125.71.50
IP: 123.125.71.51
IP: 123.125.71.52
IP: 123.125.71.53
IP: 123.125.71.54
IP: 123.125.71.55
IP: 123.125.71.56
IP: 123.125.71.57
IP: 123.125.71.58
IP: 123.125.71.59
IP: 123.125.71.60
IP: 123.125.71.69
IP: 123.125.71.70
IP: 123.125.71.71
IP: 123.125.71.72
IP: 123.125.71.73
IP: 123.125.71.74
IP: 123.125.71.75
IP: 123.125.71.76
IP: 123.125.71.77
IP: 123.125.71.78
IP: 123.125.71.79
IP: 123.125.71.80
IP: 123.125.71.81
IP: 123.125.71.82
IP: 123.125.71.83
IP: 123.125.71.84
IP: 123.125.71.85
IP: 123.125.71.86
IP: 123.125.71.87
IP: 123.125.71.88
IP: 123.125.71.89
IP: 123.125.71.90
IP: 123.125.71.91
IP: 123.125.71.92
IP: 123.125.71.94
IP: 123.125.71.95
IP: 123.125.71.96
IP: 123.125.71.97
IP: 123.125.71.98
IP: 123.125.71.99
IP: 125.39.78.168
IP: 125.39.78.171
IP: 125.39.78.173
IP: 125.39.78.174
IP: 125.39.78.177
IP: 125.39.78.179
IP: 125.39.78.181
IP: 125.39.78.183
IP: 125.39.78.185
IP: 125.39.78.187
IP: 125.39.78.188
IP: 125.39.78.189
IP: 125.90.93.141
IP: 173.236.136.101
IP: 180.76.5.100
IP: 180.76.5.101
IP: 180.76.5.103
IP: 180.76.5.107
IP: 180.76.5.110
IP: 180.76.5.111
IP: 180.76.5.113
IP: 180.76.5.136
IP: 180.76.5.137
IP: 180.76.5.138
IP: 180.76.5.139
IP: 180.76.5.140
IP: 180.76.5.141
IP: 180.76.5.142
IP: 180.76.5.143
IP: 180.76.5.144
IP: 180.76.5.145
IP: 180.76.5.146
IP: 180.76.5.147
IP: 180.76.5.148
IP: 180.76.5.149
IP: 180.76.5.150
IP: 180.76.5.151
IP: 180.76.5.153
IP: 180.76.5.154
IP: 180.76.5.155
IP: 180.76.5.156
IP: 180.76.5.157
IP: 180.76.5.158
IP: 180.76.5.159
IP: 180.76.5.160
IP: 180.76.5.161
IP: 180.76.5.162
IP: 180.76.5.163
IP: 180.76.5.164
IP: 180.76.5.165
IP: 180.76.5.166
IP: 180.76.5.167
IP: 180.76.5.168
IP: 180.76.5.169
IP: 180.76.5.170
IP: 180.76.5.171
IP: 180.76.5.172
IP: 180.76.5.173
IP: 180.76.5.175
IP: 180.76.5.176
IP: 180.76.5.177
IP: 180.76.5.178
IP: 180.76.5.179
IP: 180.76.5.180
IP: 180.76.5.181
IP: 180.76.5.182
IP: 180.76.5.183
IP: 180.76.5.184
IP: 180.76.5.185
IP: 180.76.5.186
IP: 180.76.5.187
IP: 180.76.5.188
IP: 180.76.5.189
IP: 180.76.5.190
IP: 180.76.5.191
IP: 180.76.5.192
IP: 180.76.5.193
IP: 180.76.5.194
IP: 180.76.5.195
IP: 180.76.5.196
IP: 180.76.5.197
IP: 180.76.5.48
IP: 180.76.5.49
IP: 180.76.5.50
IP: 180.76.5.51
IP: 180.76.5.52
IP: 180.76.5.53
IP: 180.76.5.54
IP: 180.76.5.55
IP: 180.76.5.56
IP: 180.76.5.57
IP: 180.76.5.58
IP: 180.76.5.59
IP: 180.76.5.60
IP: 180.76.5.61
IP: 180.76.5.62
IP: 180.76.5.63
IP: 180.76.5.64
IP: 180.76.5.65
IP: 180.76.5.66
IP: 180.76.5.67
IP: 180.76.5.87
IP: 180.76.5.88
IP: 180.76.5.89
IP: 180.76.5.90
IP: 180.76.5.91
IP: 180.76.5.92
IP: 180.76.5.93
IP: 180.76.5.94
IP: 180.76.5.95
IP: 180.76.5.96
IP: 180.76.5.97
IP: 180.76.5.98
IP: 180.76.5.99
IP: 180.76.6.20
IP: 180.76.6.21
IP: 180.76.6.211
IP: 180.76.6.212
IP: 180.76.6.213
IP: 180.76.6.222
IP: 180.76.6.223
IP: 180.76.6.224
IP: 180.76.6.225
IP: 180.76.6.227
IP: 180.76.6.230
IP: 180.76.6.231
IP: 180.76.6.232
IP: 180.76.6.233
IP: 180.76.6.26
IP: 180.76.6.28
IP: 180.76.6.29
IP: 180.76.6.35
IP: 180.76.6.36
IP: 180.76.6.37
IP: 204.45.133.74
IP: 220.181.108.165
IP: 220.181.108.166
IP: 220.181.108.167
IP: 220.181.108.168
IP: 220.181.108.169
IP: 220.181.108.170
IP: 220.181.108.171
IP: 220.181.108.172
IP: 220.181.108.173
IP: 220.181.108.174
IP: 220.181.108.175
IP: 220.181.108.176
IP: 220.181.108.177
IP: 220.181.108.178
IP: 220.181.108.179
IP: 220.181.108.180
IP: 220.181.108.181
IP: 220.181.108.182
IP: 220.181.108.183
IP: 220.181.108.184
IP: 220.181.108.185
IP: 220.181.108.186
IP: 220.181.108.187
IP: 220.181.108.79

USER AGENT: "Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)"
IP: 222.76.212.176

 

keyplyr




msg:4475786		 5:03 am on Jul 15, 2012 (gmt 0)



When I *did* allow it, the only authentic ranges I found were:

119\.63\.19[2-9]\.
180\.76\.
123\.125\.71\.
220\.181\.
dstiles




msg:4475903		 8:14 pm on Jul 15, 2012 (gmt 0)

I only permit JP Baidu - although I suspect the results are shared across all baidu-operating countries. Reason behind this is: I need traffic from JP but do not particularly want it from CN.

Really 204.45.133.74? That's FDC Server in USA.

I have "company" ranges listed as:

119.63.192.0 - 119.63.199.255 (JP)
180.76.0.0 - 180.76.255.255 (CN)

Permitted bots (I may be behind on these lists):

China:
61.135.169.32 - 61.135.169.32
61.135.190.1 - 61.135.190.254
123.125.66.0 - 123.125.66.255
123.125.71.0 - 123.125.71.255
180.76.5.0 - 180.76.6.255
220.181.7.0 - 220.181.7.255
220.181.108.0 - 220.181.108.255

Japan:
119.63.192.128 - 119.63.192.254
119.63.193.0 - 119.63.193.255
119.63.196.1 - 119.63.196.127
119.63.198.0 - 119.63.198.255
119.63.199.103 - 119.63.199.103
tangor




msg:4476059		 2:00 pm on Jul 16, 2012 (gmt 0)

I deny the Chinese version, otherwise, Baidu has (so far) honored robots.txt, taking that, (some ips more aggressive than others) but that's all they take.
rowan194




msg:4485015		 12:38 pm on Aug 15, 2012 (gmt 0)

I've had a lot of problems with Baidu, so much so that I wrote a script that firewalls any c class that loads with a Baidu user-agent. Not a great long term solution, as anyone knowing this could perform a simple DoS - load a single page with a faked Baidu referer and the 256 IPs around you are quickly blocked - but I'd had it with them hitting my sites. It's the only time I've had to firewall a major crawler, rather than just blocking it with robots.txt (which doesn't seem to work.)

An interesting side effect is that crawlers purporting to be Baiduspider get blocked too. :)
wilderness




msg:4485054		 2:21 pm on Aug 15, 2012 (gmt 0)

rather than just blocking it with robots.txt (which doesn't seem to work.)


FWIW, robots.txt doesn't block or deny anything, rather, robots.txt is a request to compliant bots to honor your wishes.

htaccess on the other hand, is fully capable of denying access to a variety of visitors, and utilizing a variety of methods and/or criteria.
rowan194




msg:4485077		 3:28 pm on Aug 15, 2012 (gmt 0)

I understand that robots.txt doesn't block a site. What I meant was that Baidu don't seem to respect my requests in that file for them to not go anywhere on my site. Thus, I moved towards blocking at the IP level.
Igal Zeifman




msg:4485388		 9:25 am on Aug 16, 2012 (gmt 0)

Hi,
You can verify Baidu Spider IPs by using "Check IP" function in Botopedia.org.
It will also provide you with all legit, user-agent data for this and other bots.
dstiles




msg:4485584		 6:58 pm on Aug 16, 2012 (gmt 0)

Thanks. Looks to be a useful site if it really has all IPs for any given bot.
not2easy




msg:4535715		 5:55 pm on Jan 13, 2013 (gmt 0)

I ended up here looking for info on the 180.76.5.nnn range because of about two dozen requests for "robots.txt" from UA: Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2
The same UA comes in from 202.46.53.82 and 202.46.62.95 (both 403s)
I have not seen Baidu anywhere for a long time, but this UA ONLY requests robots.txt so maybe it is part of a tag team.(?)
blend27




msg:4535732		 7:10 pm on Jan 13, 2013 (gmt 0)

off topic. sorry.
You can verify Baidu Spider IPs by using "Check IP" function in *otopedia.org.

Ever heard of 404, well, This is it.?
Great Idea, but get you contact form fixed, first. This should not be a place to promote affiliates.
blend27




msg:4535733		 7:21 pm on Jan 13, 2013 (gmt 0)

Same as dstiles. Japan OK, CN blocked. Have a client that does over 60% of her retail business true JP, lots of traffic from there.
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved