Website Indexation Checking

[chriskriag]

Recently I updated robots.txt of my website as shown below:

User-agent: *
Disallow: /xyz/

Here xyz - is the site directory which I wanna block.

Before robots.txt the site has following index status in Google.com

Google - site:http://www.xyz.com/
127 results

After robots.txt the site has following index as per Google.com

Google - site:http://www.xyz.com/
137 results

then after few days

Google - site:http://www.xyz.com/
143 results

and Now Google - site:http://www.xyz.com/
132 results

Moreover when I do site:xyz.com and remove http:// it gives me

Google - site:www.xyz.com
55 results

and without "www" it gives me

Google - site:xyz.com/
65 results

and without / it gives me

Google - site:xyz.com
56 results

Tel me why I am seeing such different index pages no. also tel me which is the right way to know how many pages are indexed in Google using site: operator?

[adder]

The "site:" operator is not very precise, you shouldn't worry too much. In many cases, you will find that if you paginate to the end of Google SERPs the actual number of displayed results is different from the number it states at the beginning.

The only way to check for sure is looking at your WMT -> Google Index -> Index Status

You can also check WMT -> Google Index -> Blocked Resources and
WMT -> Crawl -> Robots Tester
to make sure your robots.txt works as intended.

Regarding the www/non-www issue, having a discrepancy (55 vs 132 results) may indicate that you're not redirecting the non-www version properly. In other words, if
www.xyz.com/page-name
and
xyz.com/page-name
both resolve to content (status OK 200), then you have a potential problem.
Make sure that each page can only be accessed via ONE url. Also, indicate the preferred domain on your WMT profile Site Settings (the cogwheel icon)

[lucy24]

wmt will give you a general notion. But the only absolutely reliable way to tell if some specific page has been indexed is to search for some piece of exact text, with the "site:" operator if necessary. (If your text is very unusual, you don't need the "site:" part. As a bonus, you may discover some scrapers.)

Psst! If you use example.com as your domain name, the way the forums keep telling you to, nobody will have the transitory confusion of an apparent "www.xyz.com/xyz/" which doesn't seem to be what you meant.