Hi,
All the major search engines have crawled my website by only the entry pages are scanned. I have over 20,000 pages which are not being referred to.
Any inputs on this please?
Enclosed robots.txt which disallows only /bin folder
# Robots.txt file
#
User-agent: *
Disallow: /bin
Disallow: /cgi-bin