Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Google spidering past secure server

Robots.txt file to boot the bot?

9:53 pm on Jan 29, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 26, 2002
votes: 0

The Googlebot has been spidering my client's site, which is hosted on a secure server (string begins with HTTPS), and the legal team wants it out of there.

Anything besides a robots.txt to get rid of Googlebot?

12:52 am on Jan 31, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 1, 2002
votes: 2


According to some just put in some hidden text and your gone!

Just kidding, it almost sounds to me like you need to contact them via this url, and explain the reason in the form box.


There is some great information on that page.

Not sure if that helps or not.

You could secure it with a password?...but that's beyond my post.

My second thought about this is: If it's really secure and googlebot can spider it...is it really secure?


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members