Welcome to WebmasterWorld Guest from 23.22.182.29

Forum Moderators: goodroi

Message Too Old, No Replies

I think google can't access my pages.

     
9:53 am on Aug 24, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 30, 2004
posts:1148
votes: 0


Hey,

My robots.txt reads...

User-agent: *
Disallow: /productimages/
Disallow: /admin/
User-agent: googlebot
Disallow: *.asp

Will this also block www.example.co.uk/test.aspx?

What exactly does this do?

I think the old website was done in ASP & it is still live so the guy before me wrote this tobots.txt file.

Any help would be great!

2:43 pm on Aug 24, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member quadrille is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Feb 22, 2002
posts:3455
votes: 0


Always leave a blank line between user agents.

User-agent: *
Disallow: /productimages/
Disallow: /admin/

User-agent: googlebot
Disallow: *.asp

If the pages are no longer required, the safe and sensible thing to do is delete them - the web has quite enough clutter; also, particularly if any links exist to the unwanted pages, they may still appear in the serps as URLs or supplementary results.

3:14 pm on Aug 24, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 30, 2004
posts:1148
votes: 0


Thanks for the reply!

I can't delete them just yet. I need to do research and find out more information about them.

Can spiders index the new pages that have the .aspx?

Also, Why is that space needed?

Thanks!
RJ

10:06 pm on Aug 24, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Aug 22, 2003
posts:333
votes: 0


Also note that this robots.txt will allow googlebot to index /productimages/ and /admin/ because they're not in the Googlebot specific section.
4:27 pm on Aug 29, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 30, 2004
posts:1148
votes: 0


Anyone else?

Can spiders index the new pages that have the .aspx?

Thanks...

10:16 am on Sept 4, 2006 (gmt 0)

New User

10+ Year Member

joined:May 15, 2005
posts:28
votes: 0


Yes, if you don't want them to, use:

Disallow: *.asp*