Freshbot.. ahhh......damm you!

How do I get this guy to like me!

[askjoe]

I have a webpage whose index is hit at least once a day by freshy. I have all the pages I want indexed linked to from this page(index page). However, freshbot simply refuses to go any further past my index. Seriously, the pages I want freshbot to index are only one page deep into my site! - not too much to ask, huh?

How do I get freshy to take that extra step and index these pages? It's these pages that are optimized for google the best.

I've tried some of the advice you guys have given me before ... no success yet so I'm fish'n for more clues.

My sites been up now for about 40 days.

[hakre]

eh askjoe ;),
why not taking a look here:

Does a Freshbot Crawl Get You Listed [webmasterworld.com]

[Jesse_Smith]

Do they have session IDs in the URLs? Ask GoogleGuy what he thinks about those!

[hakre]

if sitesearch would be that cool like google, would anyone se-optimize it's thread to get topranking positions in it? and then? would it be used more and more often?

for example: if an idea is propagted (like: is your site-structure se-optimzed, ie a dynamic page) why not put a link into the answer to direct to one or two according threads?

[askjoe]

Can you guys take a look at my site and tell me what I'm doing wrong? It's in my profile. Seriously, the pages I want indexed are only 1 page deep - how can I get freshbot to go through my index page, spider these pages, then post them within google.

Like I said before, I've tried several ideas posted here but nothing has really worked. Only my home page continues to get indexed.

There's no hidden text, spam etc. that should keep freshy from proceeding through my site - I'm seriously stumpd here.

Thanks

[heini]

Sorry, Joe but we don't do sitereviews.
As far as I know there are no clear cut answers as to what triggers fresh listings anyway.

[MetropolisRobot]

Are they completely new pages?

I have seen behavior like this too. Freshbot comes and looks at a few pages and leaves. Fine. No worries.

Later it comes back with a few buddies and they work over your site. However it may be days or weeks before this happens.

I thought this maight be related to the "new page" syndrome where if a page is new, then the link is stored by Google but then it comes and spiders that link when it feels like it.

Make sure that the links are legal though (no really small text, no text the same color as the background, no session ids etc)

[caine]

GoogleBot is your friend, freshbot only when your already indexed.

[askjoe]

So what you guys are saying is that freshbot won't index my site until googlebot does? So I have to wait until the update? This is all becomming very frustrating to me!

[askjoe]

What I mean to say is freshbot will not continue to come back into my site and spider the whole thing until my site is fully indexed?

[TheComte]

It gets even more frustrating. I have two redesigned sites (late January)which have been deep crawled and of course no permanence yet since the dance has been postponed. Freshbot has been hitting these sites like crazy, doing it's own deep crawl, but only the main (index) page shows up in the index, and several old pages which no longer exist. Things have really been crazy since the last dance that started January 20. I think some big changes are in the works, or Google is suffering a severe meltdown. I'm starting to see the new pages in other search engines, but not Google. This is something I haven't noticed before.

[aspdesigner]

Try increasing your PageRank (i.e. - more good inbound links to you site).

[decaff]

AskJoe,

Freshbot represents Google's attempt to keep content fresh from sites that are already in the index. If a site is well established and has strong PR then freshbot will be more inclined to deepcrawl...instead of just the index page...however...it is advisable to keep your content fresh somehow on your index page to keep freshbot interested on a regular basis..

The deep crawl, googledance, googlebot (216.etc..)comes around periodically (and who says it has to be once a month...)...and validates all the incoming links into your site...once this is done than your site will attain its current status in the eyes of google and from there you can then make further adjustments...

There's no way to force or convince freshbot to crawl deep .... it does this in a somewhat random fashion...

decaff

[TheComte]

(and who says it has to be once a month...)...

No one, but it would probably be in their interest to try and stay as fresh as some of the other search engines.