homepage Welcome to WebmasterWorld Guest from 54.204.182.118
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
I think google can't access my pages.
rj87uk

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3058459 posted 9:53 am on Aug 24, 2006 (gmt 0)

Hey,

My robots.txt reads...

User-agent: *
Disallow: /productimages/
Disallow: /admin/
User-agent: googlebot
Disallow: *.asp

Will this also block www.example.co.uk/test.aspx?

What exactly does this do?

I think the old website was done in ASP & it is still live so the guy before me wrote this tobots.txt file.

Any help would be great!

 

Quadrille

WebmasterWorld Senior Member quadrille us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3058459 posted 2:43 pm on Aug 24, 2006 (gmt 0)

Always leave a blank line between user agents.

User-agent: *
Disallow: /productimages/
Disallow: /admin/

User-agent: googlebot
Disallow: *.asp

If the pages are no longer required, the safe and sensible thing to do is delete them - the web has quite enough clutter; also, particularly if any links exist to the unwanted pages, they may still appear in the serps as URLs or supplementary results.

rj87uk

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3058459 posted 3:14 pm on Aug 24, 2006 (gmt 0)

Thanks for the reply!

I can't delete them just yet. I need to do research and find out more information about them.

Can spiders index the new pages that have the .aspx?

Also, Why is that space needed?

Thanks!
RJ

abates

10+ Year Member



 
Msg#: 3058459 posted 10:06 pm on Aug 24, 2006 (gmt 0)

Also note that this robots.txt will allow googlebot to index /productimages/ and /admin/ because they're not in the Googlebot specific section.

rj87uk

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3058459 posted 4:27 pm on Aug 29, 2006 (gmt 0)

Anyone else?

Can spiders index the new pages that have the .aspx?

Thanks...

krt1

5+ Year Member



 
Msg#: 3058459 posted 10:16 am on Sep 4, 2006 (gmt 0)

Yes, if you don't want them to, use:

Disallow: *.asp*

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved