Forum Moderators: open

Message Too Old, No Replies

Do search engines bots crawl proxy servers data?

         

Ezer

11:03 am on Jan 22, 2009 (gmt 0)

10+ Year Member



Hope this is a right place to post the question.

Do SE bots crawl our sites directly or from proxy servers?

Really like to know.

Thanks

Regards,

incrediBILL

1:34 pm on Jan 23, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Hi Ezer,

Not sure exactly what you're asking but many SEs may use an internal proxy server for their crawling activity otherwise we'd probably be tracking thousands of unique IPs.

Additionally, there have been problems in the past with what's called "proxy hijacking" where someone operating a proxy server put up vast pages of links redirected through their proxy server so that Google or others will crawl through that server and then attribute the page content to the proxy site to bring them visitors.

Does either of the above answer what you're looking for?

Ezer

1:40 pm on Jan 25, 2009 (gmt 0)

10+ Year Member



I appreciate your reply.

Your answers are the very reasonable.

Does it mean when we search SEs data centers actually may also come from some where proxy servers?

Thanks,

Regards,