I think google can't access my pages.

[rj87uk]

Hey,

My robots.txt reads...

User-agent: *
Disallow: /productimages/
Disallow: /admin/
User-agent: googlebot
Disallow: *.asp

Will this also block www.example.co.uk/test.aspx?

What exactly does this do?

I think the old website was done in ASP & it is still live so the guy before me wrote this tobots.txt file.

Any help would be great!

[Quadrille]

Always leave a blank line between user agents.

User-agent: *
Disallow: /productimages/
Disallow: /admin/

User-agent: googlebot
Disallow: *.asp

If the pages are no longer required, the safe and sensible thing to do is delete them - the web has quite enough clutter; also, particularly if any links exist to the unwanted pages, they may still appear in the serps as URLs or supplementary results.

[rj87uk]

Thanks for the reply!

I can't delete them just yet. I need to do research and find out more information about them.

Can spiders index the new pages that have the .aspx?

Also, Why is that space needed?

Thanks!
RJ

[abates]

Also note that this robots.txt will allow googlebot to index /productimages/ and /admin/ because they're not in the Googlebot specific section.

[rj87uk]

Anyone else?

Can spiders index the new pages that have the .aspx?

Thanks...

[krt1]

Yes, if you don't want them to, use:

Disallow: *.asp*