Welcome to WebmasterWorld Guest from 54.224.234.8
Forum Moderators: goodroi
My robots.txt reads...
User-agent: *
Disallow: /productimages/
Disallow: /admin/
User-agent: googlebot
Disallow: *.asp
Will this also block www.example.co.uk/test.aspx?
What exactly does this do?
I think the old website was done in ASP & it is still live so the guy before me wrote this tobots.txt file.
Any help would be great!
User-agent: *
Disallow: /productimages/
Disallow: /admin/
User-agent: googlebot
Disallow: *.asp
If the pages are no longer required, the safe and sensible thing to do is delete them - the web has quite enough clutter; also, particularly if any links exist to the unwanted pages, they may still appear in the serps as URLs or supplementary results.