Spider Site

Is there a web site that will spider my site for me?

[wildfiction]

Say I have just added 1,000 pages to a web and want to open each page once to trigger the AdSense code so that Google will crawl each page.

Is there a web site that I can submit my URL to that will crawl/spider the site and trigger the AdSense code for each page just once?

[DanA]

Did you try to use an offline browser such as httrack or wysigot or Wget...?

[wildfiction]

No I haven't tried those programs but just been looking at their webs and they will do the job if I can't find a web site to do it for me. Many thanks for the info - very helpful.

[hyperkik]

I would be cautious about using a program to load every page, given the size and scope of your site - Google might view that as a violation of its terms of service.

[jomaxx]

A regular spider won't do what you want, and as hyperkik suggests you would probably be better off staying away from the idea altogether. Especially now that they're introducing CPM pricing.

[wildfiction]

Very good point - i'd forgotten about the CPM - that would certainly be viewed as fraud. I'll drop this idea.

[Sunflux]

It wouldn't be a problem if you could disable the execution of javascript.

[sailorjwd]

I read somewhere that Google is usually able to deliver on-target ads on the very first view of a page... Something about keeping the connection open for a second and they somehow get the info they need. I've noticed this (good targeting) on pages that I have recently created.

I would definitely not run any program that would artificially cause page views - you'd be asking for trouble.

[OptiRex]

In my experience I have found that Adsense automatically comes to the page to spider current content when a request is made from an infrequently used page.

I have no proof of this however other than checking one of these pages and seeing the following:

1. No Ads
2. PSA Ads
3. Ad Links Search Box

I have also found that within seconds a refresh will generate the ads correctly.

I don't believe that Google keep a cache type copy of a site to auto-serve however for popular pages it does serve up Adsense immediately since they are in regular demand.

Assuming I am correct then your spidering idea would serve no purpose for the less used pages.

[cat5]

"so that Google will crawl each page"
take life easy my friend if you are on a hurry take a taxi .

[cat5]

by the way if your page is popular ,frequent updated and has many visitors pr day ,Googlebot will come,don't worry:)

[EricGiguere]

If you read the AdSense patent description, you'll see that AdSense takes the content of the other pages on your site into account when serving up ads. So even though AdSense may not have crawled your page in particular, it can still make a good guess at what kinds of ads to display. So you won't necessarily get PSAs.

Eric