Forum Moderators: goodroi
my server is able to handle the requests from Gbot(100,00+) and MSN(20,000) pretty well.
other than these many other search engines(ask jeeves, bigclique) are spidering with big strings as urls,,, does this require more of our bandwith?
i am observing that, our revenues are down when the robot hits are at its peak..
so what shall be the solution?
Shall I increase the server capacity to accomodate all other search spiders request? (i have doubt on Ask Jeeves, does it happen that it hits with crawler/robots asking for more bandwidth)
I am very new into this search spiders topic,,,
hence could any help me out?
Advanced Thanks..
The spider shouldn't really have a direct effect on your revenues besides the fact that hosting cost may be more expensive. However if you can see a correlation between the frequency of spider hits and your revenue on a given day, I guess there might be a performance issue. If the pages load too slow due to high load (heavy spidering) it might happen that people leave the site fast and are less likely to buy. But this is just a guess.
