A better way to run a PFI?

You decide when to spider your site.....

[born2drv]

I was just thinking how expensive it must be for Inktomi to spider their paying members' sites every 48 hrs.... when let's face it, most of us do not update our site every 48 hrs.

I bet they would cut their spidering costs down by over 90% and make webmasters much happier if there was a Big Red Index Me! Button So for example, I would press the button and all URL's in my account are refreshed within 6 hrs or something.

I'm happy because I don't get upset if the spider is late after 48 hrs, and I get my content in faster, and they save a lot of money, WIN-WIN!

[Brett_Tabke]

But if they do that, you could algo investigate easier. (easier than it already is).

Same deal with Google's PFI spider (erm, I guess we call that one FreshBot ;-).

I do agree some time frame setting would be nice.

[born2drv]

Well 48 hours is already pretty quick to try and crack their algo don't you think?

I was just thinking it would save them a ton of cash to not have to re-spider pages that never change in content, let the user decide when it's time to refresh.

For example, I probably would only press my big red button 2-3x a month as I upload pages and new content. Compared to 15x a month, thats a huge savings in bandwidth.

[Brett_Tabke]

Let's say you had just 20 pages in there. In just 10 days, that is 100 variables you could test. You could have some of the pfi systems (all but link pop and the pfi cheat factor) within a couple of weeks of hard testing.