Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Confused by how "st:robots" meta tag would affect Google indexing

         

maha

5:13 pm on Oct 15, 2016 (gmt 0)

10+ Year Member



I have a page with this meta tag:

<meta name="st:robots" content="nofollow, noindex">

Does it mean all the links in this page are "nofollow"? I've never seen the "st:robots" before, it's usually "robots", not sure what that means or if it makes any difference

Robert Charlton

1:18 am on Oct 16, 2016 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I have a page with this meta tag:

<meta name="st:robots" content="nofollow, noindex">
maha, I've never seen this before, but I did a search on a partial text string from the tag that I thought might be mentioned in html text somewhere in a discussion about this tag... and therefore findable via search....

[name="st:robots"]

It turns out that the "st" stands for Swiftype, which is site search software, apparently with its own crawler. The first result led me to this page [swiftype.com...]

The page explains how to use both the standard robots exclusion meta tag and Swiftype's proprietary tag, which uses the "st:", to control indexing of your site's pages by the Swiftype crawler in combination with other standard bots.
Robots Meta Tag Support

The Swiftype web crawler supports the robots meta tag standard. This allows you to control how Swiftype indexes pages on your site.

So, depending on how you set it up, Swiftype results might show some pages that you've excluded, say, from Google, or vice versa.

PS:
Does it mean all the links in this page are "nofollow"?
Yes, but note that the st tag applies to Swiftype only. The "noindex" attribute should keep anything on the page from appearing in the Swiftype search results, and the "nofollow" attribute should keep the st bot from crawling any of the links on that page. A "nofollow" robots meta in Google also keeps PageRank from circulating through that page. Not clear that Swiftype's algo has anything analogous to those kinds of considerations.

From the wording of your question, I can't tell how aware you are that "noindex" and "nofollow" do different things. I'm thinking that reading and studying this discussion might be helpful. It discusses a great many misconceptions about the robots meta tag and also robots.txt, with regard to Google....

Pages are indexed even after blocking in robots.txt
https://www.webmasterworld.com/google/4490125.htm [webmasterworld.com]

Again, though, with the st bot, you'd be dealing with a different "back end" than you would be with Googlebot... assuming ie that you're asking about a site that you're working on, and that it is using Swiftype as site search.

maha

4:57 am on Oct 16, 2016 (gmt 0)

10+ Year Member



I'm asking because I have a link from a page with this meta tag:
<meta name="st:robots" content="nofollow, noindex">

I cannot figure out if this means all bots (including Googlebot) are block? Sounds like it doesn't.

Robert Charlton

8:08 am on Oct 16, 2016 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I cannot figure out if this means all bots (including Googlebot) are block? Sounds like it doesn't.

You're right... "st:robots" doesn't affect other bots at all... and if this isn't your site, this setup shouldn't affect you at all.

If anything, it makes it less likely that the st:robot is going to start spidering your site.

maha

2:02 pm on Oct 16, 2016 (gmt 0)

10+ Year Member



thanks Robert