Welcome to WebmasterWorld Guest from 54.234.63.187

Forum Moderators: open

Message Too Old, No Replies

immediate dropping of ppi page if server down when spider on

no leeway whatsoever

     

jamie

9:46 am on May 1, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



this has happened to us 2 times in the last 3 months.

we use the ppi for our index page.

in the last 3 months i have been making several server modifications, (e.g. setting up quotas) we have also had one very dodgy afternoon with network problems from our serverhost.

all this resulted in our server being unreachable several times, each for 2 minutes, whilst i rebooted after making modifications and once for several hours during the network connectivity problems.

it appears that the fast crawler was on our site when this happened - which meant immediate dropping from their index.

i wrote to lycos ppi service and was told that if a (ppi) page could not be found or if the spider was thrown off for some reason (e.g. reboot), it would be removed from the index forthwith.

we then had to wait a week for the spider to recrawl our ppi page and we were then reincluded.

i am not sure if anyone can help here ;-)

but i did want to get this off my chest. this is really frustrating - it has never happened with any other engine. and in the case of implementing server quotas - i HAD to reboot the server.

here's hoping someone from fast might read this and is able to consider changing the situation.

cheers ;-)

p.s. i did mention this in my last mail to the lycos ppi help service, and they said they would pass it on to fast. but the same thing just happened to us again (1 month later)

Rumbas

10:09 am on May 1, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



jamie, you gotta keep that server running 24/7 and you can't blame them for throwing out a page that's not there when they try to fetch it :)

I really don't think it would be a major problem as the crawler picks you right up again on the next visit - which usually happens on more frequent basis that once a week.

jamie

11:16 am on May 1, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



rumbas

" jamie, you gotta keep that server running 24/7 and you can't blame them for throwing out a page that's not there when they try to fetch it :) "

at the end of the day i know, i know ;-)

but google / altavista /inktomi have never dropped us for that.

anyway - i hope the server mods are finished now, so we can look forward to a long period of uninterupted spidering!

cheers

 

Featured Threads

Hot Threads This Week

Hot Threads This Month