Welcome to WebmasterWorld Guest from 54.147.10.72

Forum Moderators: open

Message Too Old, No Replies

A better way to run a PFI?

You decide when to spider your site.....

     

born2drv

9:49 am on Mar 15, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I was just thinking how expensive it must be for Inktomi to spider their paying members' sites every 48 hrs.... when let's face it, most of us do not update our site every 48 hrs.

I bet they would cut their spidering costs down by over 90% and make webmasters much happier if there was a Big Red Index Me! Button So for example, I would press the button and all URL's in my account are refreshed within 6 hrs or something.

I'm happy because I don't get upset if the spider is late after 48 hrs, and I get my content in faster, and they save a lot of money, WIN-WIN!

Brett_Tabke

10:05 am on Mar 15, 2003 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



But if they do that, you could algo investigate easier. (easier than it already is).

Same deal with Google's PFI spider (erm, I guess we call that one FreshBot ;-).

I do agree some time frame setting would be nice.

born2drv

11:16 am on Mar 15, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Well 48 hours is already pretty quick to try and crack their algo don't you think?

I was just thinking it would save them a ton of cash to not have to re-spider pages that never change in content, let the user decide when it's time to refresh.

For example, I probably would only press my big red button 2-3x a month as I upload pages and new content. Compared to 15x a month, thats a huge savings in bandwidth.

Brett_Tabke

12:38 pm on Mar 15, 2003 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Let's say you had just 20 pages in there. In just 10 days, that is 100 variables you could test. You could have some of the pfi systems (all but link pop and the pfi cheat factor) within a couple of weeks of hard testing.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month