homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Site Map Verses Robot Text File
Nouf Taraman, Text,

Msg#: 4505686 posted 1:35 am on Oct 9, 2012 (gmt 0)


My name is Nouf Taraman I am very new to internet marketing and I was just curuios to ask, If I have a site map for my wordpress blog and I created a robot text file to block the search engines, will my site still show up with the site map embedded?

Nouf Taraman



WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 4505686 posted 2:18 am on Oct 9, 2012 (gmt 0)

welcome to WebmasterWorld, Nouf Taraman !

it depends on what is in your robots.txt file.

if a site map is discovered by the search engines, the urls listed in that site map will likely be crawled.
if the search engine's crawler complies with the robots exclusion protocol it will look for excluded patterns in robots.txt before requesting any url.
the content on pages on your site that are not excluded and get crawled have a good chance of showing up.
some of your excluded content may also show up with an abbreviated snippet if those urls are linked from other crawled content.
by the way if your site map url is excluded by robots.txt the search engines may never discover your site map urls.
it's not a good idea to list excluded urls in your xml site map.

i'm not sure what "with the site map embedded" means.
are you referring to an html site map or an xml site map?

you should get accounts on the search engine webmaster consoles.
this will help you figure out some of these things.
(like the "fetch as googlebot" tool on GWT)

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved