Would it be Good to Let Us Schedule Googlebot Visits?

[kpaul]

Let me first say that I in no way mind Google (and the other SEs crawling my site...)

When two or more bots hit at the same time, though, it sometimes slows things down.

Tonight I wondered about Google letting us put something in the robots.txt to let us tell it when to crawl the site (late at night, etc.)

Stupid idea? Has some merit?

Thanks,
kpaul

[jcoronella]

Hmm... time based cloaking? ;)

[kpaul]

with that, though, wouldn't googlebot visit, not see you and not come back?
<p>

[Dpeper]

Mabey when they learn to provide good serps they can start working on more "advanced" ideas.

[trimmer80]

Mabey when they learn to provide good serps they can start working on more "advanced" ideas.

Dpeper - How about trying to keep the comments relevant.

kpaul,
I doubt google will do scheduled crawling ever.
It would make load bearing too difficult as most people in the geographical regions would want to do it at the same time. This would raise their costs.
Plus there are the costs of storing these times and calculating who to visit when.

Too many costs, too little benefit to google.

[karmov]

trimmer80 is right, if we were permitted to schedule googlebot visits, it would be very boom bust work for google. They'd have to do 90% of their crawling during 10% of the day. As it is now, they can freely cawl at their own steady pace and get far more bang for their hardware buck.

[mmarlor]

Dear Google,

My site runs 24x7 and I get quite a respectable level of traffic at all hours. However, I have noticed that 4:55pm - 5:00pm is a dead time on Mondays and Saturdays. Could you please only crawl my site during those windows - and make sure you get it all!

;-)

[digitalv]

When two or more bots hit at the same time, though, it sometimes slows things down.

Sounds like you need a better host or better server - be thankful you're getting indexed at all :)

[trillianjedi]

I agree with digitalv - if google traffic is important to you, you need to upgrade your host.

If you don't need google traffic, then block googlebot in your robots file.

TJ

[Leosghost]

I saw the title of this thread ..and thought ...

Only in the wildest dreams of the Owners/Admins/Mods of here and after much celebrations and pubcons ...

[kpaul]

yeah, i figured i would have to bite the bullet and upgrade. it *is* kinda good news, though, i guess ;)

thanks all.

-kpaul