Forum Moderators: open
I'm being hit very hard by google's freshbot at the moment, and going deep too. At first glance at what is currently going on with the little guys, I had to check and double check that the IP's were 64.... (they are).
It's behaviour, in terms of hard hitting and depth of crawl (it's going through the entire site) is more like the character of the old deepbot.
In fact, it's identical behaviour to deepbot the last time it crawled this site back in April.
I'm interested in hearing from others who are seeing the same.
TJ
Any ideas?
I know it's already been established that Googlebot doesn't seem to "know" about the April deepcrawl, yada yada. I'm just one more confirmation.
It is frustrating, though.
But are you seeing the pages the FB grabs show up in the index? From what I'm seeing, the FB system is only funtioning properly for those pages coming from sites that were already in the Feb/March db.
On new sites, FB is crawling regularly, but those new pages aren't showing up.