homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum


WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

Msg#: 4551108 posted 11:41 pm on Mar 4, 2013 (gmt 0)

Was all set to post this in SSID but happily this subforum jumped up and hit me in the face before I could get there.

Met this one in today's 404s: - - [04/Mar/2013:05:23:01 -0800] "GET /trafficbasedsspsitemap.xml HTTP/1.1" 404 930 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe"

Some cursory googling hahaha leads to this page [bing.com].

Apparently I got lucky with just the one request. Some folks have been getting so many that they've resorted to locking out bing. (Seems a bit pointless to me: unless you've got a firewall setup, a 404 probably uses fewer resources than a 403.)

Why they expect to find this server-plugin-generated file on a shared-hosting site-- especially one whose robots.txt includes an explicit sitemap reference-- must remain a mystery.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved