Mediapartners crawling a new site without AdSense?

Forum Moderators: martinibuster

Message Too Old, No Replies

Mediapartners crawling a new site without AdSense?

...it's still being developed!

icedowl

6:11 pm on Sep 10, 2004 (gmt 0)

Okay, I'm really puzzled now. I've got a brand new domain that I've just started to develop and add content. I have not submitted the domain to any search engine nor have I placed my AdSense code on any of its pages. It's simply too soon in the whole process.

Out of curiosity, I was just looking at my usage statistics for the site - not really expecting to see anything other than my own visits. I found 12 requests/6 pages made by Mediapartners-Google/2.1 in the browser summary section.

I don't remember using the preview tool on this site though it's possible that I may have used it. Could that be why? Or?

loanuniverse

6:40 pm on Sep 10, 2004 (gmt 0)

Could it be someone wandering over and using the tool?

Could it be you or someone else wandering over and using an unregistered version of Opera?

Do you have the logs from the time the site went live, can you look for ips other than your own and google's bot.

icedowl

6:57 pm on Sep 10, 2004 (gmt 0)

Could it be someone wandering over and using the tool?

Doubtful. Other than my hosting company and registrar, I am the only person that is really aware of the site.

Could it be you or someone else wandering over and using an unregistered version of Opera?

Again, doubtful. I don't even have Opera.

Do you have the logs from the time the site went live, can you look for ips other than your own and google's bot.

The online logs only reveal "Faxobot/1.0", "Mediapartners-Google/2.1", and "SurveyBot/2.3 (Whois Source)" in addition to my own visits.

I've yet to download the raw logs, but I suppose that will be my next step. The DNS resolved on September 2nd, but it really wasn't until this past Monday that I was able to start adding content to it due to my job getting in my way.

diamondgrl

7:20 pm on Sep 10, 2004 (gmt 0)

Just because it says Googlebot or Mediapartners or Yahoo Slurp or whatever, it doesn't mean it is. Some folks - for whatever their reasons - spoof the bot. Probably because they find it easier to gain acceptance from an eager webmaster who might otherwise ban pesky bots from unknown sources.

Do some research on the IP address to find more info.

icedowl

7:36 pm on Sep 10, 2004 (gmt 0)

The raw logs didn't reveal much of real interest except that it did visit my robots.txt file first before crawling further. I ran a trace with NeoTrace and the IP does belong to google.

dmorison

7:50 pm on Sep 10, 2004 (gmt 0)

Are you sure that your domain hasn't had a previous life?

icedowl

7:55 pm on Sep 10, 2004 (gmt 0)

Are you sure that your domain hasn't had a previous life?

Pretty sure. It's a domain name I've had my eye on for a few years but didn't actually get until recently.

dmorison

8:05 pm on Sep 10, 2004 (gmt 0)

>Pretty sure.

Zero backlinks then?

icedowl

8:14 pm on Sep 10, 2004 (gmt 0)

Yep. Zip point zero backlinks. No PR yet either.

Never_again

9:00 pm on Sep 10, 2004 (gmt 0)

Are you using Google's Toolbar? If so, they know about the site.

wshi88

9:03 pm on Sep 10, 2004 (gmt 0)

Do you have google toolbar installed?

When you browse your site, it's possible that Google knows about your site via toolbar.

PatrickDeese

9:14 pm on Sep 10, 2004 (gmt 0)

> "SurveyBot/2.3 (Whois Source)"

well - a lot of promoters use whois source to look for domain names - in general, querying a domain via their site that isn't indexed sends the bot out.

It could be that someone wanted to see what was on the domain, and used the preview tool to see what kind of adverts your content might pull.

icedowl

9:35 pm on Sep 10, 2004 (gmt 0)

Yes, I do have the google toolbar installed. It was 5 days after the DNS finished propagating when the bot made its visit.

As for the whois bot, I guess that is a possibility for why it visited though I don't really understand why anyone would bother to take a look.

This also makes me wonder if the AdSense preview tool will send the mediapartners bot to crawl any site it's used on, whether or not the site is using AdSense.

In the back of my mind I suspect that I might have used the preview tool, but the visits were during the same time that the preview tool was having troubles and was giving errors to everyone. I may have had the site open and used it to see if the tool's troubles had been cured. I just don't know.

AdSenseAdvisor

10:06 pm on Sep 10, 2004 (gmt 0)

This also makes me wonder if the AdSense preview tool will send the mediapartners bot to crawl any site it's used on, whether or not the site is using AdSense.

You're right- if you use the preview tool on a page that hasn't been crawled and/or doesn't contain the AdSense code, doing so will trigger the Mediapartners bot to crawl your site. That's why it says in the preview tool FAQ to try waiting an hour before using the preview tool again on your page. (see [google.com...]

ASA

allinternet

11:58 pm on Sep 10, 2004 (gmt 0)

I wonder if using the Preview toolbar (I haven't installed it yet) on my home page would "invite" a re-indexing so my ads on that page would be on target? It's driving me crazy. Yet Mediapartners visits daily.

Tom

icedowl

8:41 am on Sep 11, 2004 (gmt 0)

ASA, thank you for the explanation.

I wonder if using the Preview toolbar (I haven't installed it yet) on my home page would "invite" a re-indexing so my ads on that page would be on target?

Doesn't sound like that would happen from ASA's explanation (key words being "hasn't been crawled and/or doesn't contain the AdSense code").