Forum Moderators: open

Message Too Old, No Replies

Missing Pages Keep Getting Crawled

Is this a bad thing for my site?

         

Brinaf

5:12 am on Aug 16, 2002 (gmt 0)



So, my site had a pretty major face-lift not too long ago and I replaced the htm files with shtml files. I was told to remove old copies of pages from my server so I did. I also had created a doorway page with a bunch of links and was instructed to get rid of that too so as not to be seen as a spammer by google.

Well, now I look into my log files every day or so and there is the googlebot, looking for my missing htm files and that silly doorway file. And it isn't there...

What does this mean for me? Why isn't it indexing my new site? Any help would be much appreciated.

Love this forum, Brina

SmallTime

6:00 am on Aug 16, 2002 (gmt 0)

10+ Year Member



I assume there must be links to those old pages out there. If there are a significant number, you might put in permanent redirects to the new page, and/or contact the webmasters of the linking sites and give them an updated address.

mack

6:06 am on Aug 16, 2002 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



From what I have heard Google will make several attempts to crawl a page before giving up and removing it from the index. This is a good thing because it allows for technical faults like a server going down at the wrong time. You also might want to check you don’t have any links from within your own pages linking to the missing files. This could be a problem because Google will think of your site as having broken links.

deltakits

11:56 am on Aug 16, 2002 (gmt 0)



Can't you also update your robots.txt to not allow robots in to the old pages?

Brinaf

11:00 pm on Aug 16, 2002 (gmt 0)



Yeah - these pages are dead - no links to them at all. So, I guess these crawls are just based on what is already in the index now and hopefully my new stuff will get indexed at the end of this month.

Thanks!