Forum Moderators: open

Message Too Old, No Replies

Subpages not crawled

Subpages not crawled by Google, MSN

         

sivas78

3:52 am on Dec 21, 2004 (gmt 0)

10+ Year Member



Hi,

All the major search engines have crawled my website by only the entry pages are scanned. I have over 20,000 pages which are not being referred to.

Any inputs on this please?
Enclosed robots.txt which disallows only /bin folder

# Robots.txt file
#

User-agent: *
Disallow: /bin
Disallow: /cgi-bin

diamondgrl

4:02 am on Dec 21, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



How long has the site been up? It can take some time for sites to show up in the indexes.

sivas78

4:24 am on Dec 21, 2004 (gmt 0)

10+ Year Member



Hi,

The site is up for 6 months, and google, yahoo, msn have been crawling on a fortnight/monthly basis, i could see the log and email analysis.

however, subpages are not indexed only root pages!
any idea?