Welcome to WebmasterWorld Guest from 34.239.158.107

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google visits unlinked new URLs

     
8:50 pm on Jul 20, 2005 (gmt 0)

New User

10+ Year Member

joined:Nov 3, 2008
posts:
votes: 0



The url was registered in april and has been in development since. I recently noticed that the site had had a visit from googlebot and was since added to googles index. The site is not linked so google seem to have picked it up via domain registers. (.se domain)

I thought incoming links was the only way into the index.

Not sure if this is old news. Hadn't seen it before.

1:52 pm on July 21, 2005 (gmt 0)

Full Member

joined:Jan 12, 2004
posts:334
votes: 0


I have some pages at my site that are NOT LINKED TO by ANY pages (like test pages for example), and they've shown in the G index.
3:50 pm on July 21, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Feb 19, 2003
posts:227
votes: 0


I guess the advanced features of the google toolbar could trigger google to know about your pages!

I know in the case of Alexa this is the case... an example is this: I wrote an admin area, and was running some tests online (I had not set-up any password on the pages yet). Anyway, I came back the next day and all the links in the admin area had been visited... the problem was the links where things like 'Edit' or 'Delete'. Imagine my shook when I came back the next day to find all my content deleted, LOL. A hard lesson learned. (My logs showed alexa had visted those pages!)

I know that the google toolbar does take note of pages visited if the advanced features (i.e. pagerank ect) are enabled.... perhaps this is how they found the pages! Although I can't say with certainty they record all visited pages - but I guess they need to, to try and get the PR of that page!

4:36 pm on July 21, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 3, 2002
posts:894
votes: 0


Hey jmswe,

Welcome to Webmaster World!

4:46 pm on July 21, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 18, 2003
posts:1929
votes: 0


I had my local, development pages spidered by Yahoo. I was glad I noticed it in time and added the robots.txt to the local server.
5:20 pm on July 21, 2005 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts:1908
votes: 3


When designing a website I always use a robots text disallowing all SEs as well as a nofollow/noindex meta tag on each page until it's ready and I don't recall an SE ever listing anything before I wanted it to.

Also when redesigning a site I usually post it on my own domain until its ready to be uploaded to the client site. I use the nofollow/noindex meta tags and robots text on it also until time to upload.

No problems encountered.

4:42 pm on July 22, 2005 (gmt 0)

New User

10+ Year Member

joined:May 10, 2005
posts:7
votes: 0


Wow, this is all news to us. We have an entire duplicate site, a test duplicate, for a site dealing with [well...], a test site where amateur-SEO friends read new topics we don't want the world to see yet, due to the fact that we're amateurs ourselves and often under the influence. In other words, no one knows about this test site but us and close friends with too much time on their hands. Or so we thought.

Okay, we just now checked and it seems no one else knows about it, but maybe we should make preparations.

We've always left our test pages "live," never having considered the possibility that anyone could ever find them. We actually started doing this because we trust live pages over our design app's test feature and we're always making modifications.

7:54 pm on July 22, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 19, 2003
posts:804
votes: 0


Production site/server

Never trust a toolbar. It is a security hole.

Never have test files on a live site.

Never have test data in a live database.

Never leave any startup code.

Always block the automatic generation of directory index listings.

Remove all scripts that aren't used (that you might use them doesn't negate this rule)

Test site/server

Never trust a toolbar. It is a security hole.

On all test sites, disallow all robots.

Require a password.

Always block the automatic generation of directory index listings.

And there are more but this is a start.

3:17 am on July 23, 2005 (gmt 0)

New User

10+ Year Member

joined:May 10, 2005
posts:7
votes: 0


Good to know; thanks. More rules than high school....
9:00 am on July 24, 2005 (gmt 0)

New User

10+ Year Member

joined:July 24, 2005
posts:5
votes: 0


I know this whole issue with google seems a little weird, but have you considered using robots.txt? lol
10:44 am on July 24, 2005 (gmt 0)

New User

10+ Year Member

joined:Mar 25, 2005
posts:8
votes: 0


I was making a new site from last 2 months...I didn't disallow my site to SE's.
In the latest update...Google crawled my site and gave home page PR - 2... Though internal pages are still 0..
My site has not compleated yet even donot linked properly..
What should I do now? I think now its difficult to improve my PR in future...will it remain 2?

need advise...

10:54 am on July 24, 2005 (gmt 0)

New User

10+ Year Member

joined:July 24, 2005
posts:5
votes: 0


uhh.. did you try contacting google? lol
11:27 am on July 24, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 29, 2000
posts:12095
votes: 0


Very funny, dude. :)

>>What should I do now? I think now its difficult to improve my PR in future...will it remain 2?

What you should do now is keep making your site better for your visitors. That's your job. The PR is Google's job. No, it won't remain PR2 - get a *few* good quality links at a time and it'll start to go up - but don't obsess about PR, just work on the site and give it time.

1:41 pm on July 24, 2005 (gmt 0)

New User

10+ Year Member

joined:May 10, 2005
posts:7
votes: 0


Window said:

Google crawled my site and gave home page PR - 2... Though internal pages are still 0..

Totally not joking here ... really ... but IMDb has really good PR, yet all the pages that link to us are really low, even though IMDb is a directory edited by people (as opposed to some place where you go and just add your link).

I was under the impression that PR spread from page to page, and I often notice that many sites have a pretty consistent PR throughout the whole site. I assume, then, that Window's site will even out after a while, as do most, or am I just way off?

2:01 pm on July 24, 2005 (gmt 0)

Full Member

joined:Jan 12, 2004
posts:334
votes: 0


FWIW, each of my sites have far different PR's throughout their pages. I think it depends some on how/if you link to each of your pages FROM each of your pages, like a nav menu that has ALL links in it on each page. Now I even do that for most pages and of those pages even they have different PR's. That would probably be indicative of whom and how many are linking to your internal pages.
4:36 pm on July 24, 2005 (gmt 0)

New User

10+ Year Member

joined:May 10, 2005
posts:7
votes: 0


Clint said:

each of my sites have far different PR's throughout their pages. I think it depends some on how/if you link to each of your pages FROM each of your pages, like a nav menu that has ALL links in it on each page.

Well, besides being free of social restraints ("make your pictures smaller"; "spend more time on design and less time screen capping every blessed frame of the movie"; "do not have an opinion"...), being unpopular does have another advantage: for a very long time, each and every page of your site will have exactly the same PR! :o)

We're just now learning how to properly utilize the template feature; what you're saying about links in the nav menu will come in mighty handy, so thank you.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members