Fresh tags every day, but...

Freshie won't take the bait on my new pages

[FleaPit]

I don't understand the way freshie is working a few of my sites. I see a fresh tag near enough everyday but new pages never seem to get crawled. Freshie updates changes made on existing pages but how can I get the little critter to take the bait on my new pages? All my new pages have a lovely little href hook on the homepage so why is it not happening?

Anybody experiencing the same and any ideas on how to make it work?

[shrirch]

I've had this problem with one site in particular. 10-15 visits by Google and all it does is requests robots.txt and a particular image file.

My other sites on the servers seem to be doing ok. Weird. :)

[Hardwood Guy]

I'm waiting for those ideas too Fleapit. I've done exactly as you, but not until very recent. The site has been online for almost a year with a PR5...and has been getting hit by Freshie very sporadic lately.

[Nicke]

My index page has a fresh tag everyday but same as you, new pages gets indexed slowly.

Now I wait Google to fully index my forums, it grabbed a part of it recently but most forum pages is not indexed yet. (I upgraded to a SE friendly version of my forum software recently.)

[FleaPit]

So it would seem there is no way to encourage freshbot to crawl new pages. It either does it or it doesn't... Has anyone noticed any particular date or day during the month when new pages do appear to be addded by Freshbot? I can't understand this activity as I would assume the whole idea would be to introduce new pages on each crawl not simply update existing ones.

[martinibuster]

Gbot crawls on a multiple schedule. Some sites get it daily. Some sites get it on a 3-4 week schedule.

That's according to a Google scientist at the "Meet the Crawlers" session at last week's SES in San Jose, in response to a question about the "continuous update" in the context of refreshing the index.

Someone, (using WW jargon :) ) asked her to elaborate further, but she declined.

If that's how G is doing it, then I would "speculate" that your deeper pages may get it in on the 3-4 week schedule... perhaps?