Welcome to WebmasterWorld Guest from 54.162.155.183

Forum Moderators: goodroi

Message Too Old, No Replies

I think google can't access my pages.

     
9:53 am on Aug 24, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hey,

My robots.txt reads...

User-agent: *
Disallow: /productimages/
Disallow: /admin/
User-agent: googlebot
Disallow: *.asp

Will this also block www.example.co.uk/test.aspx?

What exactly does this do?

I think the old website was done in ASP & it is still live so the guy before me wrote this tobots.txt file.

Any help would be great!

2:43 pm on Aug 24, 2006 (gmt 0)

WebmasterWorld Senior Member quadrille is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Always leave a blank line between user agents.

User-agent: *
Disallow: /productimages/
Disallow: /admin/

User-agent: googlebot
Disallow: *.asp

If the pages are no longer required, the safe and sensible thing to do is delete them - the web has quite enough clutter; also, particularly if any links exist to the unwanted pages, they may still appear in the serps as URLs or supplementary results.

3:14 pm on Aug 24, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks for the reply!

I can't delete them just yet. I need to do research and find out more information about them.

Can spiders index the new pages that have the .aspx?

Also, Why is that space needed?

Thanks!
RJ

10:06 pm on Aug 24, 2006 (gmt 0)

10+ Year Member



Also note that this robots.txt will allow googlebot to index /productimages/ and /admin/ because they're not in the Googlebot specific section.
4:27 pm on Aug 29, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Anyone else?

Can spiders index the new pages that have the .aspx?

Thanks...

10:16 am on Sep 4, 2006 (gmt 0)

10+ Year Member



Yes, if you don't want them to, use:

Disallow: *.asp*

 

Featured Threads

Hot Threads This Week

Hot Threads This Month