Would they affect AdSense?

'They': large number of hosts like lj512118.crawl.yahoo.net

[Jo555]

I found that large number of hosts like lj512118.crawl.yahoo.net are accessing to my website. Their ip's all fall into 74.6.X.X.
Are they really yahoo crawlers? What are they doing? Why is yahoo using so large number of ip's for crawling? At least a few hundreds to my website within an hour.

Most important question is that would they affect AdSense?

[Hobbs]

>Are they really yahoo crawlers?
Yes

>What are they doing?
They are indexing your pages just like Google does, except those usually send you less traffic (see below)

>Why is yahoo using so large number of ip's for crawling?
Because they are not very smart.

Added:
>would they affect AdSense
No, they don't load Java ads, they are getting only the text on your pages.

[edited by: Hobbs at 12:01 pm (utc) on July 14, 2007]

[sailorjwd]

If they are crawlers, hopefully they won't be Java enabled and adsense won't know about them.

[incrediBILL]

Even if they did run javascript it's likely that Google would block them.

[SEOPTI]

What are they doing?

They are crawling your pages, NOT indexing them.

[trader]

I have similar problems with Yahoo crawlers indexing my sites far too frquently and to make it worse they often use different IP's. I have seen hundreds if visits in a few weeks from Yahoo and dozens of their IP addresses. That compares to only a few visits from Google in the same time span and not using multiple IP's.

My question is since Yahoo visits so excessively do my traffic stats programs count them as unique visitors and therfeore inflate my stats? With some of my lower traffic or new websites it sometimes appears the majority of its traffic comes from the Yahoo crawlers.

Why does Yahoo do that anyway? Seems like a waste of resources and bandwith.

P.S. What is the difference between a crawl and indexing?

[incrediBILL]

to make it worse they often use different IP's

That's just standard operating procedure for ALL search engines as the crawlers are distributed across multiple servers and multiple IPs.

More IPs doesn't make it worse, it's just how it works normally.

[Hobbs]

Bill,
Google comes from a consistent IP
Yahoo comes from many PLUS Yahoo China now too
MSN is all over the place with IPs

[incrediBILL]

Google comes from a consistent IP

Um no, Google has a LOT of IP's, it doesn't just crawl from a single IP.

Even with Yahoo and MSN crawling from China it's still a consistent block of IPs.

[edited by: incrediBILL at 7:21 am (utc) on July 15, 2007]