Forum Moderators: open

Message Too Old, No Replies

What's going on?

Why aren't my pages getting crawled?

         

zsalmon

1:53 am on Oct 22, 2002 (gmt 0)

10+ Year Member



Do you think my website is penalized for some reason? I have a PR of 4, I am part of DMOZ and I only use legitimate ways of optimizing my website. What's weird is that unlike other pages, I have not been crawled by Googlebot since the end of the last dance. Why have I not been crawled especially since pages ranked above and below me have? Google also has crawled pages linking to my site, but have not followed the link and crawled my site!

Should I be concerned? Will I get crawled during the dance?

Thanks,
Zach

Powdork

2:51 am on Oct 22, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Its hard to know when a site becomes eligible for fresh crawls. It seems to be around pr 5 lately but pr may have nothing to do with it.
Does your server configured to support the "if modified since" header?My own experience suggests that sites that change alot get spidered more frequently.

zsalmon

4:09 am on Oct 22, 2002 (gmt 0)

10+ Year Member



I will try adding that header. Do you think that I will will be crawled during the dance?

Thanks,
Zach

Powdork

5:32 am on Oct 22, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I think you will most definitely be crawled assuming
1. You have incoming links by sites indexed by Google.
2. Your server is up and functioning when the crawl comes.
3. Your robots.txt file exists OR your server is returning the proper 404 error.

You have been crawled before so I would assume you will be definitely be crawled again. Unless you used to be crawled by the fresh bot and now you no longer are. Even then you still most likely will be; Google wants to get to 2.5 billion and yours just might put them over the top.:)