Tracking the Google Crawl

[budbiss]

How will I know when my page has been crawled? In my access.log I see that I have been visited quite frequently in the past 24 hours by Googlebot 2.1...I am assuming this is the crawler...so how do I know when it has finished? How can I find out how my page has ranked?

[Stefan]

Yes, you've been visited by a googlebot.

so how do I know when it has finished?

Actually, it will never be finished, (unless it hates your site for some reason). Googlebots will visit regularly, probably concentrating on your index/default page at first.

How can I find out how my page has ranked?

Do a search for the title of your index page. You'll probably have the kw's in it that you hope people will find your site with. Also, try searches on different combinations of all the kw's you hope to have show in the listings and tweak accordingly.

[GoogleGuy]

And just to add to what Stefan wrote, you can also check for a page with a combo of operators like
site:mydomain.com inurl:blue-widgets uniqueword
and so on. That will tell you if the page is crawled/indexed, but not how it ranks, of course. If you see just the url come up without any snippet, then we probably saw a link to that url (e.g. from some other page such as your site map) but didn't get a chance to crawl it. Google calls that "partially indexed" because we can index the url and the anchortext pointing to that page, but we haven't actually fetched the page. In general the larger the number of important sites that point to your site, the more we'll trust that your site is important and the more we'll try to crawl deeply within your site.

[DerekH]

GoogleGuy wrote

the larger the number of important sites that point to your site, the more we'll trust that your site is important and the more we'll try to crawl deeply within your site.

I've always found that GoogleGuy is like a good crossword - the clue is satisfying in itself, the answer, if only you can find it, is moreso. Guessing the answer isn't the way to proceed - one needs to look at other clues that intersect with the one we're trying to solve...

This time, though, I've drawn a blank. Oh no, that's dominoes, not crosswords...

The words "important sites" confuse me...
I thought the magic words were "important pages"

The difference is both subtle and immense...
DerekH

[Bones]

I think you're perhaps over-analysing a straight-forward GG reply there, DerekH! :)

[GoogleGuy]

Yah, sometimes I type too fast. :) It helps to have important pages on important sites point to you, i.e. a little thing we call PageRank. :)

[Miop]

It would help if Google would actually pick up the decent links I've got. Since the you know what, the only links I've got are malls and my own site, yet I have been linked to by a steady number 1 site for months, E*bay, and a green widget authority site - none of those links show up at all.
Is there any reason why that should be?
What is the point of building quality links if G takes no notice of them?

[steveb]

"Is there any reason why that should be?"

Sure, if the links are complexed up in some way, javascripted or redirected in some way. That's just one obvious reason they wouldn't show up. Several other possible reasons like pagerank of the linked from pages, last time they were crawled...

====

Page, not site.

[edited by: steveb at 11:08 pm (utc) on Dec. 7, 2003]

[Miop]

Thanks Steve B. Would it be the PR of the page pointing at me or the index page of the site that is pointing at me that counts? (I would guess it's the PR of the page pointing at me, but I'm just making sure!)

[rustybrick]

Thanks Steve B. Would it be the PR of the page pointing at me or the index page of the site that is pointing at me that counts? (I would guess it's the PR of the page pointing at me, but I'm just making sure!)

You are right, it is the PR of the page that is pointing to your page.

[rfgdxm1]

>You are right, it is the PR of the page that is pointing to your page.

And notice GG mentioned the plural pages there. The more pages that link to you, and the higher their PR, the better.

[Miop]

The problem is that prior to the update, the pages were higher pr than mine, and now they aren't!

[anime_otaku]

what gg might have said to be more accurate (post #3) about how often a page is crawled is how many inbounds there are to it. a general rule is a site might get crawled across the whole domain maybe once a month but if several pages link to a certain page (deep link) then that specifically linked page gets crawled more often than once a month. since the general link is to the root of a domain usually the homepage gets indexed more than once for most sites, but if you have deep links then they get cached more often as well.

[wanna_learn]

In general the larger the number of important sites that point to your site, the more we'll trust that your site is important and the more we'll try to crawl deeply within your site.

>>>> Now who is teasing and seducting for lil thing called "Link Exchange"?

[zgb999]

GoogleGuy said "If you see just the url come up without any snippet, then we probably saw a link to that url (e.g. from some other page such as your site map) but didn't get a chance to crawl it. Google calls that "partially indexed" because we can index the url and the anchortext pointing to that page, but we haven't actually fetched the page. "

Does anybody know what it means if you have "partially indexed" pages that are online for one year or more (only showing a titel and no description).

We have several pages with similar content (different products but similar description) showing this pattern. The strange thing is that some of those pages are indexed completely while others (having the same backlinks...) are only "partially indexed".