homepage Welcome to WebmasterWorld Guest from 54.242.126.126
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Google spidering past secure server
Robots.txt file to boot the bot?
xbase234

10+ Year Member



 
Msg#: 259 posted 9:53 pm on Jan 29, 2004 (gmt 0)

The Googlebot has been spidering my client's site, which is hosted on a secure server (string begins with HTTPS), and the legal team wants it out of there.

Anything besides a robots.txt to get rid of Googlebot?

 

ScottM

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 259 posted 12:52 am on Jan 31, 2004 (gmt 0)

Sure,

According to some just put in some hidden text and your gone!

Just kidding, it almost sounds to me like you need to contact them via this url, and explain the reason in the form box.

[google.com...]

There is some great information on that page.

Not sure if that helps or not.

You could secure it with a password?...but that's beyond my post.

My second thought about this is: If it's really secure and googlebot can spider it...is it really secure?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved