homepage Welcome to WebmasterWorld Guest from 54.226.43.155
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Entries to robots.txt added automatically? Help
Potsie88




msg:3975065
 4:35 pm on Aug 19, 2009 (gmt 0)

Hello, and apologies if there is a similar question posted, I was unable to find the answer otherwise.

I attempted to add a sitemap for my new url. When I submitted it, I received a message stating that the file was blocked by robots.txt I found this odd as I didn't have a robots.txt that I was aware of. So I generated one allowing all robots. I uploaded it to the root directory and resubmitted. Again, I received the same exact error message. I went to the 'crawler access' tab and I was surprised to see that my robots.txt file as displayed on the Google webmaster tools...tool had a line added -
"User-agent: *
Disallow: /

Sitemap: http://example.net/sitemap.xml.gz"

I did not add "Sitemap:....." to the .txt file, it just appears on it's own. Where do you suppose this is being generated from? I have other domains hosted through my webhost, and this is the only one that is having this problem. I have removed the sitemap.xml.gz and only the sitemap.xml remains. (Don't even know what the .gz is) I am using WordPress and the google xml sitemaps generator plugin, which I've used with other domains with no problems. Thank you again for your much appreciated help!

[edited by: tedster at 5:40 pm (utc) on Aug. 19, 2009]
[edit reason] switch to example.net [/edit]

 

Potsie88




msg:3975141
 6:26 pm on Aug 19, 2009 (gmt 0)

When I download the robots.txt file that is on the server, it doesn't have this added line, maybe only google reads it that way?

Potsie88




msg:3975164
 6:47 pm on Aug 19, 2009 (gmt 0)

This is what the robots.txt file text is that I uploaded...

User-agent: *
Allow: /

This is what Google sees it as

User-agent: *
Disallow: /

Sitemap: [#*$!-#*$!x.net...]

Is there anyone who can help solve this mystery?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved