Forum Moderators: phranque

Message Too Old, No Replies

some Spiders do need more bandwidth?

Other than Gbot or Msn does others take more bandwidth when crawling?

         

bnvrc

8:39 am on Jun 13, 2005 (gmt 0)

10+ Year Member



hi,

my server is able to handle the requests from Gbot(100,00+) and MSN(20,000) pretty well.

other than these many other search engines(ask jeeves, bigclique) are spidering with big strings as urls,,, does this require more of our bandwith?

i am observing that, our revenues are down when the robot hits are at its peak..

so what shall be the solution?
Shall I increase the server capacity to accomodate all other search spiders request? (i have doubt on Ask Jeeves, does it happen that it hits with crawler/robots asking for more bandwidth)

I am very new into this search siders topic,,,
hence could any help me out?

Advanced Thanks...