How to handle shutting off the site for a day?

We're going over our quota and have to turn off but what about the bots?

[EliteWeb]

I run plenty of highly trafficked sites that have many files on them for people to download. The only problem is if I don't shut off the site for a day I'll have to pay for additional bandwidth.

My concern is of this, since my sites get crawled almost every day by one spider or another will I penalize myself too much. I don't want to see the search results come up showing the results for my page giving people a reason why the site is offline. Too much position to lose with the engines also is a concern.

How to handle this?

[john316]

Could you just disable the downloads? That won't affect page content (what gets spidered).

Maybe put an .htaccess file in your downloads directory that redirects to a "downloads disabled today" page.

[EliteWeb]

Not able to disable the downloads because they are all throughout the site, not database driven yet either. No CGI counting downloads or regulating :)

[starec]

Turning off the site is not a good solution. Start working on some download regulation, otherwise soon you will have to be off-line more and more. (Your traffic is growing I assume.)

I had a few black-outs in the last few months and it always takes several days before the traffic comes back to its original levels if I am offline for a day or two. As if people thought what is offline for 2 days is offline permanently?

[Crazy_Fool]

turning off isnt a solution. go over quota this month and spend a bit of time either changing your site so you don't go over quota next month, or finding another host that gives you a bigger allowance for next month.

[mack]

This may not be a workable solution but why not use a mirror for sharing the load for the downloads.

[EliteWeb]

70 gigs transfer a month is my quota :) The funding for a mirrored site to do round-robin handling of bandwidth i could do but id need someone to mirror the site free of change ;)

[lawman]

If these are commercial sites, doesn't it come down to which costs more, the extra bandwidth or the loss of business?

lawman

[EliteWeb]

I'm just concerned with the search engines on these sites. Business will always come back its more of a hobby site for those type of people out htere.

[seth_wilde]

You could cloak it...

Send all spiders to the real content and send all users to a page telling them to come back in a day...

[mivox]

I vote for the cough up the dough for the extra bandwidth this time while you re-organize the site file structure.

Once you get all your download files into one directory, you can set up a simple .htaccess redirect sending download requests to an "I'm sorry" page next time you're going over bandwidth. Upload it when needed, delete it when the bandwidth "clock" is re-set.

And next time you build a download site, plan ahead... ;)

[piskie]

House all your downloads into one folder as a starter. This will open up all sorts of D/L control options.

[Key_Master]

What type of downloads are they? You might be able to use the following code in your .htaccess file. Just change the .gif extension to the extension the download files have.

<Files ~ "\.gif$">
order deny,allow
deny from all
</Files>

To allow Google to grab the files use this instead:

SetEnvIf User-Agent ^Google welcome
<Files ~ "\.gif$">
order deny,allow
deny from all
allow from env=welcome
</Files>

[EliteWeb]

heheheh i think for now since this site is non-profit i will have to shut it off for a day. My ISP was so nice enough to e-mail me with the following:

The current non-billable allocation on your server is 65 GB
The total traffic for the billing period to date is 32 GB
The calculated projection for this billing period is 128 GB
The estimated overage charge at this time is $189
Your billing date is the 23rd of each month.