Daily spidering

Is that a lot?

[WallaceCleaver]

Hi,

My site is getting crawled on a daily basis by Googlebot and on an almost daily basis by MSN and Yahoo. Is this normal? I have never had a site that was crawled so much by the seach engines.

[rich42]

Google seems to crawl daily starting around pagerank 5 or so.

I have a PR6 site that gets crawled about every other hour.

-I think Google's been crawling more since I started adding more content on a regular basis-

I actually modified some pages yesterday and found they were in the index today (usually it takes several days though).

[robotsdobetter]

PR has something to do with how often you get spidered, but it's really how many links to your site that matters. You can have a PR of 5 or 6 and not get spidered everyday.

[j4mes]

I have a PR4 that gets more hits from Googlebot than from people! :)

[RenFromPenn]

I don't even have a page rank and very few links in so I don't get it. I too am getting more visits from Google than people at this point because the site is so new.

[justgowithit]

I have a PR4 that gets more hits from Googlebot than from people!

I don't even have a page rank and very few links in so I don't get it. I too am getting more visits from Google than people at this point because the site is so new.

Sometimes things just don't need an explanation. If I were you, I would count my lucky stars and enjoy the attention. For your sake I hope it continues for an extended period of time. However, I would strongly suggest adding new content everyday.

[j4mes]

I would strongly suggest adding new content everyday.

Doing it already, and very good advice it is too :) keeps Googlebot very happy indeed!

[sirkei]

Hi all,

My website is also being crawled by googlebot and MSN bot a lot since the beginning of this month. My site is still 1 1/2 months old , so i did not expect a pagerank so fast. Anyways, just to add that i update contents half a day and i found that one of the famous related keyword keeps crawling up from 500+ position till 80+. Well i do use adwords to promote and it costs me a lot. I am kind of worried bout the budget. I stopped adword sometimes because i can get over 300+ clicks per day and it costs me a lot. Hope that i can get free visitors soon instead of paying to get visitors.

Cheers.

[WallaceCleaver]

I know that this is probably going to sound uneducated in the ways of SEs, but what benefit is there to having the bots visit a lot? Do you get a higher ranking in the SEs than sites that aren't visited often?

[sirkei]

well, what i think is if you got more bot visits, your index get updated more frequently.

[OlRedEye]

Hi,
Can somebody pls tell me : my index.htm and pages are in public_html folder. Should my robots.txt be in that folder or root?

[j4mes]

robots.txt should be in public_html.

The outside world does not have access to your root folder, mainly because from there they could access (and attempt to crack) /etc/passwd, which would not be good :)

J.

[justgowithit]

public_html it is the root directory. Your robots.txt files should reside in this directory.