Welcome to WebmasterWorld Guest from 54.145.173.36

Message Too Old, No Replies

Google visits unlinked new URLs

   
8:50 pm on Jul 20, 2005 (gmt 0)

5+ Year Member




The url was registered in april and has been in development since. I recently noticed that the site had had a visit from googlebot and was since added to googles index. The site is not linked so google seem to have picked it up via domain registers. (.se domain)

I thought incoming links was the only way into the index.

Not sure if this is old news. Hadn't seen it before.

1:52 pm on Jul 21, 2005 (gmt 0)



I have some pages at my site that are NOT LINKED TO by ANY pages (like test pages for example), and they've shown in the G index.
3:50 pm on Jul 21, 2005 (gmt 0)

10+ Year Member



I guess the advanced features of the google toolbar could trigger google to know about your pages!

I know in the case of Alexa this is the case... an example is this: I wrote an admin area, and was running some tests online (I had not set-up any password on the pages yet). Anyway, I came back the next day and all the links in the admin area had been visited... the problem was the links where things like 'Edit' or 'Delete'. Imagine my shook when I came back the next day to find all my content deleted, LOL. A hard lesson learned. (My logs showed alexa had visted those pages!)

I know that the google toolbar does take note of pages visited if the advanced features (i.e. pagerank ect) are enabled.... perhaps this is how they found the pages! Although I can't say with certainty they record all visited pages - but I guess they need to, to try and get the PR of that page!

4:36 pm on Jul 21, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hey jmswe,

Welcome to Webmaster World!

4:46 pm on Jul 21, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I had my local, development pages spidered by Yahoo. I was glad I noticed it in time and added the robots.txt to the local server.
5:20 pm on Jul 21, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



When designing a website I always use a robots text disallowing all SEs as well as a nofollow/noindex meta tag on each page until it's ready and I don't recall an SE ever listing anything before I wanted it to.

Also when redesigning a site I usually post it on my own domain until its ready to be uploaded to the client site. I use the nofollow/noindex meta tags and robots text on it also until time to upload.

No problems encountered.

4:42 pm on Jul 22, 2005 (gmt 0)

10+ Year Member



Wow, this is all news to us. We have an entire duplicate site, a test duplicate, for a site dealing with [well...], a test site where amateur-SEO friends read new topics we don't want the world to see yet, due to the fact that we're amateurs ourselves and often under the influence. In other words, no one knows about this test site but us and close friends with too much time on their hands. Or so we thought.

Okay, we just now checked and it seems no one else knows about it, but maybe we should make preparations.

We've always left our test pages "live," never having considered the possibility that anyone could ever find them. We actually started doing this because we trust live pages over our design app's test feature and we're always making modifications.

7:54 pm on Jul 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Production site/server

Never trust a toolbar. It is a security hole.

Never have test files on a live site.

Never have test data in a live database.

Never leave any startup code.

Always block the automatic generation of directory index listings.

Remove all scripts that aren't used (that you might use them doesn't negate this rule)

Test site/server

Never trust a toolbar. It is a security hole.

On all test sites, disallow all robots.

Require a password.

Always block the automatic generation of directory index listings.

And there are more but this is a start.

3:17 am on Jul 23, 2005 (gmt 0)

10+ Year Member



Good to know; thanks. More rules than high school....
9:00 am on Jul 24, 2005 (gmt 0)

5+ Year Member



I know this whole issue with google seems a little weird, but have you considered using robots.txt? lol
10:44 am on Jul 24, 2005 (gmt 0)

10+ Year Member



I was making a new site from last 2 months...I didn't disallow my site to SE's.
In the latest update...Google crawled my site and gave home page PR - 2... Though internal pages are still 0..
My site has not compleated yet even donot linked properly..
What should I do now? I think now its difficult to improve my PR in future...will it remain 2?

need advise...

10:54 am on Jul 24, 2005 (gmt 0)

5+ Year Member



uhh.. did you try contacting google? lol
11:27 am on Jul 24, 2005 (gmt 0)

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Very funny, dude. :)

>>What should I do now? I think now its difficult to improve my PR in future...will it remain 2?

What you should do now is keep making your site better for your visitors. That's your job. The PR is Google's job. No, it won't remain PR2 - get a *few* good quality links at a time and it'll start to go up - but don't obsess about PR, just work on the site and give it time.

1:41 pm on Jul 24, 2005 (gmt 0)

10+ Year Member



Window said:

Google crawled my site and gave home page PR - 2... Though internal pages are still 0..

Totally not joking here ... really ... but IMDb has really good PR, yet all the pages that link to us are really low, even though IMDb is a directory edited by people (as opposed to some place where you go and just add your link).

I was under the impression that PR spread from page to page, and I often notice that many sites have a pretty consistent PR throughout the whole site. I assume, then, that Window's site will even out after a while, as do most, or am I just way off?

2:01 pm on Jul 24, 2005 (gmt 0)



FWIW, each of my sites have far different PR's throughout their pages. I think it depends some on how/if you link to each of your pages FROM each of your pages, like a nav menu that has ALL links in it on each page. Now I even do that for most pages and of those pages even they have different PR's. That would probably be indicative of whom and how many are linking to your internal pages.
4:36 pm on Jul 24, 2005 (gmt 0)

10+ Year Member



Clint said:

each of my sites have far different PR's throughout their pages. I think it depends some on how/if you link to each of your pages FROM each of your pages, like a nav menu that has ALL links in it on each page.

Well, besides being free of social restraints ("make your pictures smaller"; "spend more time on design and less time screen capping every blessed frame of the movie"; "do not have an opinion"...), being unpopular does have another advantage: for a very long time, each and every page of your site will have exactly the same PR! :o)

We're just now learning how to properly utilize the template feature; what you're saying about links in the nav menu will come in mighty handy, so thank you.