Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Hope this is not cloaking

disallowing robots 50% pages using robots.txt

3:21 pm on Jan 26, 2009 (gmt 0)

Junior Member

10+ Year Member

joined:July 28, 2008
posts: 46
votes: 0

I have a rich media site that is not being indexed well by Google. Would the following methodology be counted as cloaking?

[A] We create a new folder in the site root (say, it's /html/ containing pure html pages. These pure html pages serve rich media content in simplest possible html coding. They may not be visually appealing, but are made friendly to search engines, strictly using all white hat practices.

[B] Using robots.txt, robots are disallowed from indexing rich media pages. Instead, robots are directed to corresponding /html/ pages for all rich media pages.

[C] Besides on-page optimization, we do all the off page optimization work for pages in /html/ folder only.

[D] Coming in from home page, site visitors normally see rich media pages, but pages contained in /html/ folder remain landing pages for customers when they come in from search engines.

Would this be termed as cloaking? I hope not.

Thanks in advance for your involvement and caring to help out.



4:54 pm on Jan 26, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0

It is cloaking, and if the separate pages for search engines can be seen as "differently-focused" as well as "better for search engines," then it could be interpreted as "cloaking with intent to deceive." The big problem here is that this would be a judgment call by the search engine employee(s) who reviewed the site -- Likely because of a complaint from a competitor.

Frankly, this sounds like the wrong solution because in addition to leaving your fate in the hands of a search engine employee tasked to review your site, it requires you to create and maintain a whole new set of pages. It also delivers a non-optimum on-site experience to visitors coming from search engines, and involves a confusing URL-structure.

I would recommend that you determine the actual reasons for your ranking difficulties before you do something that is risky and may make things worse. For example, re-assess the need to have such "rich" pages, and try to make search engines happier with your "real" pages before embarking on this labor-intensive and difficult-to-maintain project.

Very often, improving the accessibility of your site to visually- and hearing-impaired visitors will greatly improve its indexability as well. Use plain text where plain text is to be displayed (rather than embedding it in images), limit the dependency of the site on video or audio, and use client-side scripting (e.g. JavaScript and AJAX) only where actually required to support important functions which cannot be done any other way. Make proper use of the <alt> tag for images, and use the <title> attribute on objects which can be titled. Beware especially the 'expert' in a single technology. For example, if your coder only knows JavaScript, then functions which could easily be implemented in a far more usable way using PHP on the server will be implemented in JavaScript: "If your only tool is a hammer, then everything looks like a nail."

Full disclosure: I'm one of those who has little use for fancy, flashy sites with complex scripting used for page rendering; I judge sites by whether they deliver what I came for using the fewest clicks possible, and in a straightforward, simple way. Your mileage may vary... :)

I would also recommend posting this question in one of the forums where SEO is discussed here at WebmasterWorld; Your subject is far wider than the more-or-less "mechanical" subject of robots.txt function and implementation. I recommend thoroughly studying your plan and its implications before rushing to implement it in any particular way.

Also, do visit the major search engines' "Webmaster Help" sections, and read their descriptions of cloaking -- And remember *they* get to interpret what they write there, so don't read it to mean what you hope it means, read it from a "worst-case" viewpoint.



Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members