Forum Moderators: goodroi
I recently developed a small site upon a domain which I'd previously had parked for a couple of months. White hat site, unique content, a few relevant backlinks to get it rolling. Google sitemap, robots meta tags (index, follow) and a bare-bones "allow everything to everybody" robots.txt.
Yahoo picked it up, MSN picked it up. Google? Nothing. Weeks, a couple of months, still nothing in the index (it was a low priority site or I'd have pursued a solution sooner)
A look at the Google webmaster tools just now showed the problem - Google was still showing the text of the robots.txt used by the parking service where it formerly resided - yet Google was claiming this was my site's robots.txt as of today!
I tried manually entering my robots.txt on their diagnostics page and doing a test with it - checked out OK. After I did that, a subsequent check on the domain APPEARS to have picked up my "real" robots.txt. Haven't looked at my logs yet to see if or what they've chosen to visit.
Has anyone else run into this problem? Have a pre-emptive solution? Very strange ...
Another reason could be that Google did try to visit, had a problem connecting to the server and your site got skipped till the next round of indexing. This could also explain why Google was slow on picking up on your changes.
If you work on more links and having quality, constantly updated content it will improve the chances of google indexing you faster.
robots.txt URL [http://www.mysitesname.tld/robots.txt]
last downloaded: July 2, 2007 10:20:39 AM PDT
status: 200 (success)
...
Followed by the text of the obsolete robots.txt (from the parking site).
So it looks like its trying to access, but for some reason keeps pulling up the old pages? Or am I misinterpreting something here?
If it's got the parked site cached somewhere that would explain the lack of indexing too - it was a parked page, it was not in the index.
[edited by: MamaDawg at 11:29 pm (utc) on July 2, 2007]
There's a good chance that they are framing your site, or conditionally redirecting to it when your domain is requested, and that, in fact, they continue to control the robots.txt for your site...
Look at your site's pages using 'View page source' in your browser and look for HTML frames or iFrames being used. Visit your domain while checking server response headers for 301 or 302 redirects (The "Live HTTP Headers" extension for Firefox and Mozilla browsers is a great free tool for this job).
Something is very, very fishy, and it's likely not Google...
Jim
This one really has me baffled - Is it possible Google found it once while it was parked and said "eww - parked page!", spat it out and put it in the "do not index" list? Is it reinclusion request time?
Service is a popular one which is very widely used - I have a number of domains which I plan to develop, but my own projects are always getting pushed to the back burner, hence the parking.
I just checked another domain parked in the same place - no redirects, no frames, the response is:
HTTP/1.x 200 OK
Date: Tue, 03 Jul 2007 11:03:55 GMT
Server: Apache
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Tue, 03 Jul 2007 11:03:55 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
?
[edited by: MamaDawg at 11:08 am (utc) on July 3, 2007]
