Forum Moderators: open

Message Too Old, No Replies

PR and robots.txt

if i block G bot for a particular page , will my PR get affected

         

AjiNIMC

12:42 pm on Jan 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi,

If I have a page xyz.html, where I need to put all the services offered by widgets, like blue widgets, red widgets....., that will help the users a lot but I donot want to take a risk from SE point of view. I will be happy if G bot or any other bot doesnt crawl it. I am adding this code to the robots.txt


# applies to all robots
User-agent: *
# disallow indexing of all pages
Disallow: /xyz.html

I want to ask whether that will affect the PR of the page (which is 5 as of now)? Does it have any other negative point? Your expertise views will be highly appreciated.

Thanks
Aji

ciml

5:40 pm on Jan 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Those pages will still be able to receive PR, as long as they're listed. They won't be able to pass any on, so they won't get any PR feedback from pages they link to (not that PR feedback has much bearing on Google rankings anyway).

AjiNIMC

6:41 pm on Jan 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks ciml

Those pages will still be able to receive PR, as long as they're listed.

Does this mean that slowly the pr will get reduced, as that page will no longer be listed in Google?

Aji

ciml

6:47 pm on Jan 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If there are good links to those URLs, then they'll probably still be listed. As they're not fetched, Google can't track whom they link to.

AjiNIMC

5:29 am on Jan 9, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks ciml,
sorry to bug you again as I want to be very careful when doing this extra work.

If x,y,z are giving a link to widgets.html, that means the PR will pass on to the page widgets.html irrespective of whether google can crawl widgets.html or not. But if a,b, c are placed on the page widgets.html then no PR will be passed on to a,b,c as google cannt crawl the page.

Am I correct of what I have understood?
Are there any more side effects to my overall site?

Thanks Ciml
Aji

antrat

6:36 am on Jan 9, 2004 (gmt 0)



...that will help the users a lot but I donot want to take a risk from SE point of view

What risk? If it helps users, then it will probably be liked by Google.

AjiNIMC

7:28 am on Jan 9, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



antrat,

There will be heavy repeation of my target keyword and keypharse "blue widgets" on the page and I have no other good alternative.

From SE's point of view that may turn to be spam today or tommorow. I know as of now it may be save but why to take a risk when I have alternatives(stopping by robots.txt).

Still I am afraid of the side effects if there are any such. If there are any side effects please let me know, then i will make the next move accordingly.

Thanks
Aji

antrat

8:00 am on Jan 9, 2004 (gmt 0)



There will be heavy repeation of my target keyword and keypharse "blue widgets" on the page and I have no other good alternative.

Key word density is not that important IMO. If your page reads ok to humams then don't worry about it. If you think the KWD a bit high simply dilute it down with small talk.

Content is king and every bit helps!

AjiNIMC

8:52 am on Jan 9, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I know content is the king, it helps when you are in the SERPs, if in case (.1% chance) I get a penalty then nothing counts. So better to play safe and sacrifice some good things for better ones.

Thanks I will keep it as it is but still will like the know the indepth of it, if it will have a negative effect on my ranking( I can adjust zero effect).

Aji

antrat

9:28 am on Jan 9, 2004 (gmt 0)



Why/what do you think Google will penalize you for? If it helps your site visitors I believe Google will have no problem.

On my site I have a 3 pages of widgets in aplha order. On other pages I have the same widgets categorized. This is really only to help my site visitors, but these pages have been picked up by Google many moons ago.

AjiNIMC

9:43 am on Jan 9, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Even my site was doing ok before Florida but after it my site disappeared. Now I am trying to bring it back by cleaning my sites one by one, like keyword density, adding related keywords e.t.c.

I still donot have any idea why my site was thrown out of the world during florida update but now it is coming back, we have made it to 270s.

So thats why I donot want to take any more risk.

Thanks
Aji

antrat

10:03 am on Jan 9, 2004 (gmt 0)



I agree that 'risks' are not worth it, but exactly what risk do you see? I can see no risk at all with what you mention.

KWD is the easiest of all to modify etc. Unless you get to the point where it is not readable for humans, there is no risk. I have *never* bothered checking KWD and simply write to describe the subject.

I honestly think you are over complicating things. Getting Google to list pages is really as simple as ABC, despite what SEO companies say. Just read through these from Google and you will have o problem
[google.com...]

ciml

2:08 pm on Jan 9, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If x,y,z are giving a link to widgets.html, that means the PR will pass on to the page widgets.html irrespective of whether google can crawl widgets.html or not. But if a,b, c are placed on the page widgets.html then no PR will be passed on to a,b,c as google cannt crawl the page.

Yes Aji, this is what should happen with widgets.html being excluded by /robots.txt

If you use a robots META tag with NOINDEX, then the page won't be listed and therefore will have no PageRank. (In truth, I can't say that it has no PR but from your point of view there's no difference.)

I completely agree with antrat though, it really doesn't matter if you have a page on your site that has a word repeated often.

AjiNIMC

4:00 pm on Jan 9, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanka a lot to all, I think I should give it another thought before taking such a step.

Thanks once again.

Aji

g1smd

7:44 pm on Jan 14, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



>> If you use a robots META tag with NOINDEX, then the page won't be listed and therefore will have no PageRank. (In truth, I can't say that it has no PR but from your point of view there's no difference.) <<

That's interesting. I have a few pages that are excluded from indexing by noindex,follow and the Toolbar still shows PR for those pages. The pages have been online for at least 6 months.

ciml

8:00 pm on Jan 14, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Is the URL listed in Google? If it is, then the noindex isn't being seen for whatever reason. Does the cache show the noindex?

If it has a URL-only listing, or an old listing, then it's still there because Google haven't fetched it since you added the noindex.

g1smd

11:21 pm on Jan 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Sometimes noindex pages can show up in allinurl or in related pages searches, but always without any title or description snippet shown at all.