Forum Moderators: open

Message Too Old, No Replies

google freshbot, not navigating but checking back

freshbot, deepbot

         

snook

10:15 am on Jun 2, 2003 (gmt 0)

10+ Year Member



Ok freshbot has been to our site for the last few days. But it doesn't seem to be navigating through the site, but insteading coming back looking for the pages deepbot spidered 6 weeeks ago.

Pages its going to, were ONLY there for a span of a couple days, as deepbot came while we were restructing the site. (back in Aprils deepbot)
Also pages it missed then, freshbot is missing now (which there is no way they should not be hitting them now)
Does that sound right?

And these pages, that don't really exsist, if you were try to go to the link we show freshbot went to, you would get a mysql_fetch_array error.
What can I do about making sure these pages don't get picked up, and the errors shown..

Thanks

Krapulator

10:22 am on Jun 2, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Block them with your robots.txt

trillianjedi

10:25 am on Jun 2, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member




[webmasterworld.com...]

Basic conclusion is that freshbot is crawling the April deepcrawl data index, not the current index.

TJ