homepage Welcome to WebmasterWorld Guest from 54.167.179.48
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
immediate dropping of ppi page if server down when spider on
no leeway whatsoever
jamie

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 867 posted 9:46 am on May 1, 2003 (gmt 0)

this has happened to us 2 times in the last 3 months.

we use the ppi for our index page.

in the last 3 months i have been making several server modifications, (e.g. setting up quotas) we have also had one very dodgy afternoon with network problems from our serverhost.

all this resulted in our server being unreachable several times, each for 2 minutes, whilst i rebooted after making modifications and once for several hours during the network connectivity problems.

it appears that the fast crawler was on our site when this happened - which meant immediate dropping from their index.

i wrote to lycos ppi service and was told that if a (ppi) page could not be found or if the spider was thrown off for some reason (e.g. reboot), it would be removed from the index forthwith.

we then had to wait a week for the spider to recrawl our ppi page and we were then reincluded.

i am not sure if anyone can help here ;-)

but i did want to get this off my chest. this is really frustrating - it has never happened with any other engine. and in the case of implementing server quotas - i HAD to reboot the server.

here's hoping someone from fast might read this and is able to consider changing the situation.

cheers ;-)

p.s. i did mention this in my last mail to the lycos ppi help service, and they said they would pass it on to fast. but the same thing just happened to us again (1 month later)

 

Rumbas

WebmasterWorld Administrator 10+ Year Member



 
Msg#: 867 posted 10:09 am on May 1, 2003 (gmt 0)

jamie, you gotta keep that server running 24/7 and you can't blame them for throwing out a page that's not there when they try to fetch it :)

I really don't think it would be a major problem as the crawler picks you right up again on the next visit - which usually happens on more frequent basis that once a week.

jamie

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 867 posted 11:16 am on May 1, 2003 (gmt 0)

rumbas

" jamie, you gotta keep that server running 24/7 and you can't blame them for throwing out a page that's not there when they try to fetch it :) "

at the end of the day i know, i know ;-)

but google / altavista /inktomi have never dropped us for that.

anyway - i hope the server mods are finished now, so we can look forward to a long period of uninterupted spidering!

cheers

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved