Forum Moderators: open
I am seeking advice on the current problem I am facing. I recently redesigned (3 weeks ago) my website and changed the entire structure. Prior to this redesign effort, this website had not been touched since May 1997. There are several old pages from this website in the google index, which are now obsolete and generate 404 errors.
On my newly redesigned website, Googlebot visits it and leaves after fetching only the robots.txt and index file. It simply refuses to spider the entire website.
FYI, As suggested in the posts on this forumn, I frequently change my index page (adding content/links) and modify the robots.txt file in order to seek the interest of the googlebot. However, nothing seems to be working.
My site has a page rank of 4 and a few incomming links from PR 5-6. My site navigates well on lynx.
I'd like to abide by the rules of this forumn and not place my URL in this posting. If anyone wants to see the site then send me an email.
I want the googlebot to crawl my site and include the newly designed web pages. I also want my older files deleted fom the google index.
Also, adding any new links to the home page couldn't hurt either.
However, I would imagine that any site will eventually get reindexed if it is already in the index.
Send me a sticky with the url and I'll take a look at it.
Google doesn't care, but if dynamic, are you certain that all the headers are constructed correctly?
You said that you've changed your robots.txt file several times. Why not try using a blank robots.txt file. Also ensure that your existing one validates. (Be aware that a missing robots.txt file may result in an error pages being served instead.)
You should also validate the your index page. I have never used Lynx but my understanding is that it is a browser not a validator. There may be a badly formed tag somewhere that causes Googlebot to puke.
Kaled.
Try [searchengineworld.com...]
and see how things look..
Further to add that all the currently indexed pages from my site at google when one goes to "site:www.sitename.com site" are giving 404 errors except the home page. The older files have been deleted.
Since we have created newer content and totally changed the way we run the business, I am unable to set 301 or 302 redirects.
Thanks once again to all who responded. My fingers are crossed. I hope the site gets indexed some day.
Since we have created newer content and totally changed the way we run the business, I am unable to set 301 or 302 redirects.
One of my clients had a very similar problem a year or so ago. Two things that can help are ...
I would really think about repopulating my pages with the old URLS again, if waiting a few months is not an option.
Otherwise sit tight and wait a few months. Your new pages will eventualy be indexed and the old ones will drop. However since many of us have seen 1 or 2 year old removed pages come back into the Googl eindex after the past few updates, I still recommend somehow keeping the old file names.
One thing though: I am nto sure if setting all the 404's to the sitemap and letting it get indexed is a good option. YOu might need to make sure if you do that, theat it does nto get indexed, otherwise you would have a lot of duplicate or identical pages which is not a good thing! ;)
TheVoodoo
YOu might need to make sure if you do that, theat it does nto get indexed, otherwise you would have a lot of duplicate or identical pages which is not a good thing!
Seems a little unlikely that you would be penalised for using a standard mechanism for dealing with non-existent pages ...
AntSaint, Those are even more horrid than using a unified Sitemap. Those methods suggest depricate spamming methods.
CyberCeo, I think if you email google with your problem and wait a few weeks you wil receive a response. and then you can enlighten all of us if the sitemap method would be harmless according to google. You will be amazed at how much specific help Google will provide if you email them patiently.
I must say you can not be too google-cautious or paranoid when it comes to dealing with pages and content on your website.
Updates come and go. I just try to forget the paranoia and build web sites that are interesting and on topic. Works for me ;)
That is the best practice. However keep caution lingering in the back at all times. Because even if you are innocent and not trying to do anytyhign wrong you can get slammed by updates. That has happened to my personal site with not even a single trace of SEO in it.
Unfortunately the internet, much like the world we live in, is full of spammers and crooks and it is always imperative that good people get burnt along side the bad ones. Not all of them get burnt ofcouse, but you always want to be wearing your anti-flammatory gear. hehe
Thevoodoo
Because even if you are innocent and not trying to do anytyhign wrong you can get slammed by updates.
Quite right, thevoodoo.
So what's the point in worrying about it? Building a clean, on topic site without using any tricks means that you are creating the type of content that the perfect search engine would love.
And all of the search engines are striving to be perfect ... right? ;)
