homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
A better way to run a PFI?
You decide when to spider your site.....

 9:49 am on Mar 15, 2003 (gmt 0)

I was just thinking how expensive it must be for Inktomi to spider their paying members' sites every 48 hrs.... when let's face it, most of us do not update our site every 48 hrs.

I bet they would cut their spidering costs down by over 90% and make webmasters much happier if there was a Big Red Index Me! Button So for example, I would press the button and all URL's in my account are refreshed within 6 hrs or something.

I'm happy because I don't get upset if the spider is late after 48 hrs, and I get my content in faster, and they save a lot of money, WIN-WIN!



 10:05 am on Mar 15, 2003 (gmt 0)

But if they do that, you could algo investigate easier. (easier than it already is).

Same deal with Google's PFI spider (erm, I guess we call that one FreshBot ;-).

I do agree some time frame setting would be nice.


 11:16 am on Mar 15, 2003 (gmt 0)

Well 48 hours is already pretty quick to try and crack their algo don't you think?

I was just thinking it would save them a ton of cash to not have to re-spider pages that never change in content, let the user decide when it's time to refresh.

For example, I probably would only press my big red button 2-3x a month as I upload pages and new content. Compared to 15x a month, thats a huge savings in bandwidth.


 12:38 pm on Mar 15, 2003 (gmt 0)

Let's say you had just 20 pages in there. In just 10 days, that is 100 variables you could test. You could have some of the pfi systems (all but link pop and the pfi cheat factor) within a couple of weeks of hard testing.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved