homepage Welcome to WebmasterWorld Guest from 54.145.183.169
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Incorrect Robots.txt URL in server Logs
triggerfinger

5+ Year Member



 
Msg#: 4496320 posted 4:59 pm on Sep 17, 2012 (gmt 0)

Hey Guys,

I'm seeing some seriously strange stuff in our log files.

After receiving warning in GMT about robots.txt inaccessible, we checked the server logs and are seeing the following:

66.249.73.200 www.example.com - [16/Sep/2012:12:21:54 -0400] "GET /exampleproducts/product-2012.htmlrobots.txt HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"

Any idea why Google would request incorrect URLs like this? Anyone seeing anything similar?

Thanks,

-t

 

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4496320 posted 5:06 pm on Sep 17, 2012 (gmt 0)

That request is being redirected.

You should check where to. That could be an even bigger problem.

triggerfinger

5+ Year Member



 
Msg#: 4496320 posted 5:21 pm on Sep 17, 2012 (gmt 0)

It gets redirected to the products page. Hence why we get errors, but why would google request a bogus URL like this?

triggerfinger

5+ Year Member



 
Msg#: 4496320 posted 6:42 pm on Sep 17, 2012 (gmt 0)

Another clue: All the URLs seem to have vanity tld URLs redirecting to them. Is G trying to access the robots.txt of these URLs and instead requesting it from the deep page? Seems like a rather dumb idea for such a smart algorithm.

examplevanityurl.com -> 301 -> example.com/deepURL.html
examplevanityurl.com/robots.txt -> example.com/deepURL.html/robots.txt

Seems silly, no?

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4496320 posted 7:57 pm on Sep 17, 2012 (gmt 0)

if examplevanityurl.com is yours i would look for why that server is doing essentially a sitewide redirect to a subdirectory of example.com and fix it so it redirects to the root specifically for robots.txt request.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved