homepage Welcome to WebmasterWorld Guest from 54.167.41.199
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
I think google can't access my pages.
rj87uk




msg:3058461
 9:53 am on Aug 24, 2006 (gmt 0)

Hey,

My robots.txt reads...

User-agent: *
Disallow: /productimages/
Disallow: /admin/
User-agent: googlebot
Disallow: *.asp

Will this also block www.example.co.uk/test.aspx?

What exactly does this do?

I think the old website was done in ASP & it is still live so the guy before me wrote this tobots.txt file.

Any help would be great!

 

Quadrille




msg:3058748
 2:43 pm on Aug 24, 2006 (gmt 0)

Always leave a blank line between user agents.

User-agent: *
Disallow: /productimages/
Disallow: /admin/

User-agent: googlebot
Disallow: *.asp

If the pages are no longer required, the safe and sensible thing to do is delete them - the web has quite enough clutter; also, particularly if any links exist to the unwanted pages, they may still appear in the serps as URLs or supplementary results.

rj87uk




msg:3058792
 3:14 pm on Aug 24, 2006 (gmt 0)

Thanks for the reply!

I can't delete them just yet. I need to do research and find out more information about them.

Can spiders index the new pages that have the .aspx?

Also, Why is that space needed?

Thanks!
RJ

abates




msg:3059432
 10:06 pm on Aug 24, 2006 (gmt 0)

Also note that this robots.txt will allow googlebot to index /productimages/ and /admin/ because they're not in the Googlebot specific section.

rj87uk




msg:3064522
 4:27 pm on Aug 29, 2006 (gmt 0)

Anyone else?

Can spiders index the new pages that have the .aspx?

Thanks...

krt1




msg:3071148
 10:16 am on Sep 4, 2006 (gmt 0)

Yes, if you don't want them to, use:

Disallow: *.asp*

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved