Welcome to WebmasterWorld Guest from 35.173.48.224

Forum Moderators: martinibuster

Message Too Old, No Replies

Would they affect AdSense?

'They': large number of hosts like lj512118.crawl.yahoo.net

     
11:28 am on Jul 14, 2007 (gmt 0)

New User

10+ Year Member

joined:Nov 14, 2006
posts:36
votes: 0


I found that large number of hosts like lj512118.crawl.yahoo.net are accessing to my website. Their ip's all fall into 74.6.X.X.
Are they really yahoo crawlers? What are they doing? Why is yahoo using so large number of ip's for crawling? At least a few hundreds to my website within an hour.

Most important question is that would they affect AdSense?

11:59 am on July 14, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member hobbs is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 19, 2004
posts:3056
votes: 5


>Are they really yahoo crawlers?
Yes

>What are they doing?
They are indexing your pages just like Google does, except those usually send you less traffic (see below)

>Why is yahoo using so large number of ip's for crawling?
Because they are not very smart.

Added:
>would they affect AdSense
No, they don't load Java ads, they are getting only the text on your pages.

[edited by: Hobbs at 12:01 pm (utc) on July 14, 2007]

12:00 pm on July 14, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 1, 2004
posts:1987
votes: 0


If they are crawlers, hopefully they won't be Java enabled and adsense won't know about them.
9:36 pm on July 14, 2007 (gmt 0)

System Operator from US 

incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14664
votes: 99


Even if they did run javascript it's likely that Google would block them.
10:14 pm on July 14, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 6, 2002
posts:1836
votes: 22


What are they doing?

They are crawling your pages, NOT indexing them.

11:13 pm on July 14, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 20, 2004
posts:703
votes: 0


I have similar problems with Yahoo crawlers indexing my sites far too frquently and to make it worse they often use different IP's. I have seen hundreds if visits in a few weeks from Yahoo and dozens of their IP addresses. That compares to only a few visits from Google in the same time span and not using multiple IP's.

My question is since Yahoo visits so excessively do my traffic stats programs count them as unique visitors and therfeore inflate my stats? With some of my lower traffic or new websites it sometimes appears the majority of its traffic comes from the Yahoo crawlers.

Why does Yahoo do that anyway? Seems like a waste of resources and bandwith.

P.S. What is the difference between a crawl and indexing?

11:41 pm on July 14, 2007 (gmt 0)

System Operator from US 

incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14664
votes: 99


to make it worse they often use different IP's

That's just standard operating procedure for ALL search engines as the crawlers are distributed across multiple servers and multiple IPs.

More IPs doesn't make it worse, it's just how it works normally.

7:10 am on July 15, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member hobbs is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 19, 2004
posts:3056
votes: 5


Bill,
Google comes from a consistent IP
Yahoo comes from many PLUS Yahoo China now too
MSN is all over the place with IPs
7:21 am on July 15, 2007 (gmt 0)

System Operator from US 

incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14664
votes: 99


Google comes from a consistent IP

Um no, Google has a LOT of IP's, it doesn't just crawl from a single IP.

Even with Yahoo and MSN crawling from China it's still a consistent block of IPs.

[edited by: incrediBILL at 7:21 am (utc) on July 15, 2007]