Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google crawls monthly, but my site is updated weekly

Need more frequent crawling

         

jcodemasters

8:53 pm on Apr 30, 2006 (gmt 0)

10+ Year Member



Hello,
I have a site which has PR 4.. I update contents on weekly basis.. but when I search it i dont see them.. google crawls my site every month.. in the sitemap i have put page importance thats i believe it is used for site crawling.. i have another site which crawls every week .. i didn't even submitted sitemap or robots on google!
Can you please tell me any hint to crawl my site every week..

Thanks

tedster

10:34 pm on Apr 30, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I am assuming that by "update contents" you mean introduce new pages. If this involves changing any existing pages, you can make sure that you give those urls a <changefreq> of weekly.

No matter what you put into your Sitemap, Google will use it more as a helpful hint than as a command or rule. It still has its own internal logic for crawl frequency, based on PR, historical change factors, how your current pages are indexed and probably a lot more.

Probably the best thing you can do is find ways to increase your site's traffic and therefore inbound links, remembering deep pages whereever appropriate.

jcodemasters

10:49 pm on Apr 30, 2006 (gmt 0)

10+ Year Member



i have set changefreq to weekly.. see my sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset
xmlns="http://www.google.com/schemas/sitemap/0.84"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84
[google.com...]

<url>
<loc>http://www.mysite.org/</loc>
<priority>10</priority>
<changefreq>weekly</changefreq>
</url>
<url>
<loc>http://www.mysite.org/services.php</loc>
<priority>10</priority>
<changefreq>weekly</changefreq>
</url>

and robots.txt
#
# This file is used to allow crawlers to index our site.
#
# List of all web robots: [robotstxt.org...]
#
# Check robots.txt at:
# [searchengineworld.com...]
#

# Details about Googlebot available at: [google.com...]
# The Google search engine can see everything
User-agent: Googlebot
Disallow:

# All other robots will be restricted from accessing the Google-specific index pages
User-agent: *
Disallow: /css/ /cgi-bin/ /images/

I am wondering if you can help me to resolve this!.. the monthly traffic on my site is 2000 to 3000 visitors

tedster

11:02 pm on Apr 30, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I suggest changing the robots.txt file so that each Disallow: has its own separate line -- however, that "should not" affect googlebot since you've given it its own line first.

Other than that, it sounds like you're doing what you need to do. Googlebot crawling has been changed significantly this past month to include all of Google's bots in the process of feeding one shared cache -- and there may well be bugs in there, still.

At some point, it's up to them. All we can do is be vigilant -- and this forum is certainly not a direct pipeline to the Google Gods. We may discuss Google, but we aren't Google. At a certain point, we are still in the dark and subject to whatever they are currently putting into their secret recipes.

jcodemasters

11:46 pm on Apr 30, 2006 (gmt 0)

10+ Year Member



Thanks tedster.. for your reply.. what I believe is that i am doing any mistake.. do you think should I pay for premier services of google? but i am wondering i have opened 3 sites recently and they all are crawling prefect.. i see the stats by logging in google site it only crawls at the end of month or in the middle!