robots.txt

can we get googlebot to slow down a bit?

[MiG25]

Hi there,

I'm running adsense on a couple of sites that are dynamically generated, with PHP and databases, etc. When the mediapartners robot visits, it makes 60-70 connections at a time, which slows the site down somewhat.

Is there a robots.txt setting that would force the mediapartners spider to request one page at a time?

[trillianjedi]

Nope, but if you email google they can turn it down a bit for you.

Before doing that though, make sure you're not serving mediabot with session ID's.

TJ

[mlemos]

No, but you can use a script execution accelerators [turck-mmcache.sourceforge.net] and content caching [phpclasses.org] to reduce the database overhead.

If you use persistent database connections and PHP runs as a Apache 1.x module, you can to reduce drastically the number of Apache processes and the number of opened simultaneous connections by making all static content (basically all images) be served in by separate HTTP server dedicated to serve static content like thttpd [acme.com]. This way Apache does not fork more processes to serve database driven content than necessary.

[MiG25]

No, no session IDs happening, fixed that already.

Thanks for the suggestions.

[trillianjedi]

Great post mlemos - I'll have to digest that one myself when I have a bit more time.

TJ