homepage Welcome to WebmasterWorld Guest from 54.204.182.118
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Why is Google following my exclusions?
Robots.txt file says dont go and Google does!
anxvariety

10+ Year Member



 
Msg#: 421 posted 4:30 am on Jul 14, 2004 (gmt 0)

My robots.txt file at www.mysite.com/robots.txt contains the following:

User-agent: *
Disallow: alert.asp

But google follows every single one of them! I have a alert.asp?issue=1-10000.. It follows every single page.

What am I doing wrong?

 

digitalv

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 421 posted 4:41 am on Jul 14, 2004 (gmt 0)

How new is your robots.txt? Google often caches robots.txt for a while before picking up a fresh copy. If there is nothing wrong with your robots.txt file then just give it some time.

ogletree

WebmasterWorld Senior Member ogletree us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 421 posted 4:54 am on Jul 14, 2004 (gmt 0)

If that is true why do they read the thing every day. I have never seen a site that did not get at least 2 hits a day on gobot one to index and one to the robots.txt. I'm sure some smaller pages don't get that but any site that has pages indexed in G and has a PR4 or better should get that.

digitalv

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 421 posted 4:56 am on Jul 14, 2004 (gmt 0)

You know one other way you could do it would be to do useragent detection via ASP:

Put this at the top of your ASP page:

If instr(request.servervariables("HTTP_USER_AGENT"),"googlebot") Then
response.write "Sorry google, no access baby"
Else
... the rest of your page

Then throw an "end if" at the bottom. That would go into effect immediately.

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 421 posted 5:49 am on Jul 14, 2004 (gmt 0)

Simple fix:

User-agent: *
Disallow: [b]/a[/b]lert.asp

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved