Forum Moderators: open

Message Too Old, No Replies

robots.txt to prevent google indexing cloaked URLs

Site recently banned from google due to cloaked urls

         

wellnesscafe

11:28 pm on Jan 17, 2003 (gmt 0)

10+ Year Member



To prevent google to index my cloaked files, I'm using the robots.txt file with the following command: User-agent: * Disallow: /a/
(/a/ is the directory where I have all the cloaked files). Is this the correct command? Should I add an * at the end (/a/*)

Will robots.txt file satisfy google's guideline? My site was recently banned from its search engine because of these cloaked urls I think. Any suggestions?

Is there something else I should be doing? Should I re-submit my site to google or wait until google re-indexes next month? What are my chances to have my site re-admitted back into google?

Brett_Tabke

7:47 am on Jan 18, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Just the /a/ is the proper form.

However, there is a bit of disagreement on how strict the standard is on robots. The question is if the exclusion is from crawling or just indexing.

What some have found is google crawling even pages that are blocked by robots.txt, but not indexing them or adding them to the engine.

In other words, just because you list a directory in a robots.txt doesn't mean google won't crawl the directory. eg: it will find the files and do what ever it will with them, but not index them.

wellnesscafe

1:48 pm on Jan 18, 2003 (gmt 0)

10+ Year Member



Thank you Brett. The only important thing for me is to please google so that they put site be back in their search engine. Personally it matters not to me if google crawls the directory. The question is, will google be satisfied when I disallow its robot from indexing the files in directory /a/?

With regard to cloaking, JD Morgan (Jim) replied to one of my post last night as follow "Cloaking is feeding a search engine spider something materially different from what a visitor gets in order to fool it into ranking your page higher than it would otherwise be ranked. Redirecting off-site links in order to count them or for other reasons is not necessarily cloaking." [webmasterworld.com...]

I'm cloaking URLs to redirect off-site links. Google may not be upset by that after all. With regard to being banned from google, I've now come to realize that I had done some infractions (hidden dots to position image and text, keywords at the bottom of files, etc) which I have now cleaned up. Initially I thought the cloaking of urls were also an infraction to google's policy.

I have written to google to tell them that I have cleaned up and I apologize for my unintentional mistakes. I hope that google will re-index my site in their search engines and give me the same ranking I had before I was banned. Do you have any information about this? What are my chances?

Thank you for you help.