Spiders

Forum Moderators: open

Message Too Old, No Replies

Spiders

What makes Googlebot come...

thehittmann

5:46 pm on Jan 4, 2004 (gmt 0)

What makes SE spiders come to your site more often?

I know that incomming links from higher PR sites has a bit to do with it. But the fast spider comes to my site everyday a few times a day. If I make a change without a doubt the very next day I can see my change on alltheweb. Googlebot comes about one a month and even though I keep building my incomming links...this doesnt seem to be increasing.

AjiNIMC

9:10 pm on Jan 4, 2004 (gmt 0)

Keep changing your contents, add the date e.tc this helps.

Aji

millie

12:06 am on Jan 5, 2004 (gmt 0)

>... add the date e.tc this helps<

Could you clarify this for me ... why does adding the date help and where - ie, do you mean Last Modified or in the page content?

Thanks, M

vrtlw

12:15 am on Jan 5, 2004 (gmt 0)

I think Aji means a date #include statement.

With SSI (.shtml) extension you could use

To take it a stage further you could use an include statement to insert some text that changes on a regular basis (like a latest info column).

dasboot

12:16 am on Jan 5, 2004 (gmt 0)

High PR = more spidering from Google. So if you are already a huge, massive, rich company with a megasite - you're o.k.

Does that help?

thehittmann

2:26 am on Jan 5, 2004 (gmt 0)

Thanks everyone. I think I'll add a date like you mentioned. My site is not for a company. It is a music website.

g1smd

11:01 pm on Jan 12, 2004 (gmt 0)

Google usually shows the latest version of a site in the cache, but every few weeks it can revert to showing an older cached copy. Having an embedded date on the page can help you to determine what Google is up to more readily.

Worth doing.

allanp73

11:47 pm on Jan 12, 2004 (gmt 0)

Has anyone noticed that Google isn't indexing as often? I have a new client with a pr3 site which I made major changes to about a week ago and still the new index page hasn't been indexed. I got desparate and even submitted it to Google. I never experienced such a slow indexing.

thehittmann

5:43 am on Jan 13, 2004 (gmt 0)

I have a medium size site (30 pages) that gets indexed about once a month, I have a tiny little blog that gets indexed roughly every 3 days. The pr of my blog is 1 but it gets spidered and indexed more often than my other which is PR 4 and climbing.

g1smd

8:30 pm on Jan 14, 2004 (gmt 0)

Google has seen that the blog is updated more often than the rest of the site, that's why it keeps coming back to it.

trimmer80

8:42 pm on Jan 14, 2004 (gmt 0)

anybody know how the bot tells when the page is updated. E.g. does it look at the file properties for last modified or compare content on the page with past content. The reason I ask this is I have sites that are dynamic and the content on all pages are updated via the database. Thus the file itself doesn't change but the content does.

MarkHutch

8:45 pm on Jan 14, 2004 (gmt 0)

I think most go by the file date and time on the server. Last mod I believe.

trimmer80

8:56 pm on Jan 14, 2004 (gmt 0)

if this is the case, i'm better off finding a small element that is in every page of my code, e.g. <HTML>, doing a find and replace on this to something unique <ABCDEFG> and then another find and replace back to <HTML>. Thus changing the last modified on all files in site.

Stefan

3:59 am on Jan 15, 2004 (gmt 0)

Crawling frequency is very much related to PR.
Making regular changes to pages, and adding internal links to new good content pages, might help as well... it sure doesn't hurt.

sidyadav

5:34 am on Jan 15, 2004 (gmt 0)

yep, you should get crawled often if:

you are a PR4 or higher.

you have a link from a PR4 or higher page, but you yourself don't have a PR4 or higher.

Google has calculated your update frequency and indexes your site very rapidly even though you don't have a high PR or have a link from one. It usually helps if you add the date at top of your page, something like the php date() [nz2.php.net] function.

Sid

trimmer80

8:40 pm on Jan 15, 2004 (gmt 0)

a php date function will not change the last modified date. Does this mean you believe that it compares new content to old and not the last modified?

thehittmann

1:06 am on Jan 16, 2004 (gmt 0)

if the content on the page has changed it will see it as being modified even if the server doesnt support the last modified header. But most do anyway.

Spiders

What makes Googlebot come...

thehittmann

AjiNIMC

millie

vrtlw

dasboot

thehittmann

g1smd

allanp73

thehittmann

g1smd

trimmer80

MarkHutch

trimmer80

Stefan

sidyadav

trimmer80

thehittmann

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week