Welcome to WebmasterWorld Guest from 50.16.24.12

Forum Moderators: open

Spiders

What makes Googlebot come...

   
5:46 pm on Jan 4, 2004 (gmt 0)

10+ Year Member



What makes SE spiders come to your site more often?

I know that incomming links from higher PR sites has a bit to do with it. But the fast spider comes to my site everyday a few times a day. If I make a change without a doubt the very next day I can see my change on alltheweb. Googlebot comes about one a month and even though I keep building my incomming links...this doesnt seem to be increasing.

9:10 pm on Jan 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Keep changing your contents, add the date e.tc this helps.

Aji

12:06 am on Jan 5, 2004 (gmt 0)

10+ Year Member



>... add the date e.tc this helps<

Could you clarify this for me ... why does adding the date help and where - ie, do you mean Last Modified or in the page content?

Thanks, M

12:15 am on Jan 5, 2004 (gmt 0)

10+ Year Member



I think Aji means a date #include statement.

With SSI (.shtml) extension you could use <!--#echo var="DATE_LOCAL" -->

To take it a stage further you could use an include statement to insert some text that changes on a regular basis (like a latest info column).

12:16 am on Jan 5, 2004 (gmt 0)



High PR = more spidering from Google. So if you are already a huge, massive, rich company with a megasite - you're o.k.

Does that help?

2:26 am on Jan 5, 2004 (gmt 0)

10+ Year Member



Thanks everyone. I think I'll add a date like you mentioned. My site is not for a company. It is a music website.
11:01 pm on Jan 12, 2004 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Google usually shows the latest version of a site in the cache, but every few weeks it can revert to showing an older cached copy. Having an embedded date on the page can help you to determine what Google is up to more readily.

Worth doing.

11:47 pm on Jan 12, 2004 (gmt 0)

10+ Year Member



Has anyone noticed that Google isn't indexing as often? I have a new client with a pr3 site which I made major changes to about a week ago and still the new index page hasn't been indexed. I got desparate and even submitted it to Google. I never experienced such a slow indexing.
5:43 am on Jan 13, 2004 (gmt 0)

10+ Year Member



I have a medium size site (30 pages) that gets indexed about once a month, I have a tiny little blog that gets indexed roughly every 3 days. The pr of my blog is 1 but it gets spidered and indexed more often than my other which is PR 4 and climbing.
8:30 pm on Jan 14, 2004 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Google has seen that the blog is updated more often than the rest of the site, that's why it keeps coming back to it.
8:42 pm on Jan 14, 2004 (gmt 0)

10+ Year Member



anybody know how the bot tells when the page is updated. E.g. does it look at the file properties for last modified or compare content on the page with past content. The reason I ask this is I have sites that are dynamic and the content on all pages are updated via the database. Thus the file itself doesn't change but the content does.
8:45 pm on Jan 14, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I think most go by the file date and time on the server. Last mod I believe.
8:56 pm on Jan 14, 2004 (gmt 0)

10+ Year Member



if this is the case, i'm better off finding a small element that is in every page of my code, e.g. <HTML>, doing a find and replace on this to something unique <ABCDEFG> and then another find and replace back to <HTML>. Thus changing the last modified on all files in site.
3:59 am on Jan 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Crawling frequency is very much related to PR.
Making regular changes to pages, and adding internal links to new good content pages, might help as well... it sure doesn't hurt.
5:34 am on Jan 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



yep, you should get crawled often if:
  • you are a PR4 or higher.
  • you have a link from a PR4 or higher page, but you yourself don't have a PR4 or higher.
  • Google has calculated your update frequency and indexes your site very rapidly even though you don't have a high PR or have a link from one. It usually helps if you add the date at top of your page, something like the php date() [nz2.php.net] function.

    Sid

  • 8:40 pm on Jan 15, 2004 (gmt 0)

    10+ Year Member



    a php date function will not change the last modified date. Does this mean you believe that it compares new content to old and not the last modified?
    1:06 am on Jan 16, 2004 (gmt 0)

    10+ Year Member



    if the content on the page has changed it will see it as being modified even if the server doesnt support the last modified header. But most do anyway.
     

    Featured Threads

    My Threads

    Hot Threads This Week

    Hot Threads This Month

    All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
    Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
    WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
    © Webmaster World 1996-2014 all rights reserved