homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Crawl Rate: How do I know when you request slow versus fast
Does a long maximum time spent crawling a page have adverse impact on targe

5+ Year Member

Msg#: 3759885 posted 6:32 pm on Oct 6, 2008 (gmt 0)

Here is the scenario: 10 year old authority site; almost 600 backlinks; very good pagerank and serps for popular words.

I am afraid that because my pages are so long that I am missing the boat regarding getting all of my content spidered.

Currently the page is getting crawled by normal rate.

When I saw the "time spent downloading a page" I became concerned.

Number of pages crawled per day
Maximum 6
Average 2

Number of kilobytes downloaded per day
Maximum 387
Average 96
Minimum 0

Time spent downloading a page (in milliseconds)
Maximum 3511
Average 375
Minimum 0

Do I have a problem? Is it a non-issue - I am thinking that all of the pages will get spidered just that it will take a montnh f or a complete crawl as opposed to 12 days. Please help me :)



WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3759885 posted 7:50 pm on Oct 6, 2008 (gmt 0)

Time downloading - the average of 375 ms looks OK.

Look at the graph, have you just got one or two days that spike high?

If so, then that's not really an issue.

If it is most days (and it obviously isn't, from the average) then there is a problem.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved