Welcome to WebmasterWorld Guest from 54.225.47.94

Forum Moderators: open

Message Too Old, No Replies

Spiders

What makes Googlebot come...

     
5:46 pm on Jan 4, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Nov 15, 2003
posts:329
votes: 0


What makes SE spiders come to your site more often?

I know that incomming links from higher PR sites has a bit to do with it. But the fast spider comes to my site everyday a few times a day. If I make a change without a doubt the very next day I can see my change on alltheweb. Googlebot comes about one a month and even though I keep building my incomming links...this doesnt seem to be increasing.

9:10 pm on Jan 4, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 8, 2003
posts:1419
votes: 0


Keep changing your contents, add the date e.tc this helps.

Aji

12:06 am on Jan 5, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 9, 2003
posts:110
votes: 0


>... add the date e.tc this helps<

Could you clarify this for me ... why does adding the date help and where - ie, do you mean Last Modified or in the page content?

Thanks, M

12:15 am on Jan 5, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 26, 2003
posts:311
votes: 0


I think Aji means a date #include statement.

With SSI (.shtml) extension you could use <!--#echo var="DATE_LOCAL" -->

To take it a stage further you could use an include statement to insert some text that changes on a regular basis (like a latest info column).

12:16 am on Jan 5, 2004 (gmt 0)

Junior Member

joined:Dec 29, 2003
posts:47
votes: 0


High PR = more spidering from Google. So if you are already a huge, massive, rich company with a megasite - you're o.k.

Does that help?

2:26 am on Jan 5, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Nov 15, 2003
posts:329
votes: 0


Thanks everyone. I think I'll add a date like you mentioned. My site is not for a company. It is a music website.
11:01 pm on Jan 12, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Google usually shows the latest version of a site in the cache, but every few weeks it can revert to showing an older cached copy. Having an embedded date on the page can help you to determine what Google is up to more readily.

Worth doing.

11:47 pm on Jan 12, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 22, 2002
posts:646
votes: 0


Has anyone noticed that Google isn't indexing as often? I have a new client with a pr3 site which I made major changes to about a week ago and still the new index page hasn't been indexed. I got desparate and even submitted it to Google. I never experienced such a slow indexing.
5:43 am on Jan 13, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Nov 15, 2003
posts:329
votes: 0


I have a medium size site (30 pages) that gets indexed about once a month, I have a tiny little blog that gets indexed roughly every 3 days. The pr of my blog is 1 but it gets spidered and indexed more often than my other which is PR 4 and climbing.
8:30 pm on Jan 14, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Google has seen that the blog is updated more often than the rest of the site, that's why it keeps coming back to it.
8:42 pm on Jan 14, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Oct 23, 2003
posts:314
votes: 0


anybody know how the bot tells when the page is updated. E.g. does it look at the file properties for last modified or compare content on the page with past content. The reason I ask this is I have sites that are dynamic and the content on all pages are updated via the database. Thus the file itself doesn't change but the content does.
8:45 pm on Jan 14, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 6, 2002
posts:742
votes: 0


I think most go by the file date and time on the server. Last mod I believe.
8:56 pm on Jan 14, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Oct 23, 2003
posts:314
votes: 0


if this is the case, i'm better off finding a small element that is in every page of my code, e.g. <HTML>, doing a find and replace on this to something unique <ABCDEFG> and then another find and replace back to <HTML>. Thus changing the last modified on all files in site.
3:59 am on Jan 15, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 4, 2002
posts:1687
votes: 0


Crawling frequency is very much related to PR.
Making regular changes to pages, and adding internal links to new good content pages, might help as well... it sure doesn't hurt.
5:34 am on Jan 15, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 11, 2003
posts:955
votes: 0


yep, you should get crawled often if:
  • you are a PR4 or higher.
  • you have a link from a PR4 or higher page, but you yourself don't have a PR4 or higher.
  • Google has calculated your update frequency and indexes your site very rapidly even though you don't have a high PR or have a link from one. It usually helps if you add the date at top of your page, something like the php date() [nz2.php.net] function.

    Sid

  • 8:40 pm on Jan 15, 2004 (gmt 0)

    Full Member

    10+ Year Member

    joined:Oct 23, 2003
    posts:314
    votes: 0


    a php date function will not change the last modified date. Does this mean you believe that it compares new content to old and not the last modified?
    1:06 am on Jan 16, 2004 (gmt 0)

    Full Member

    10+ Year Member

    joined:Nov 15, 2003
    posts:329
    votes: 0


    if the content on the page has changed it will see it as being modified even if the server doesnt support the last modified header. But most do anyway.
     

    Join The Conversation

    Moderators and Top Contributors

    Hot Threads This Week

    Featured Threads

    Free SEO Tools

    Hire Expert Members