Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt not functioning!

Added the file months ago but results still coming up

         

nwhorton

4:47 pm on Nov 28, 2005 (gmt 0)

10+ Year Member



At work we have a development server for our website alongside our actual site's server. A while ago we decided it would be a good idea if we made it so the development server could not be searched. I added a robots.txt file to the root of that server that looks like this


User-agent: *
Disallow: /

It's been a few months now, and if I go to our university's google search and search for the development server's domain results are still coming up. Does anyone have an explanation for this?

I'm sure this issue comes up often, so I apologize if I'm being redundant, but from everything I've read, this SHOULD be working by now. Thanks.

DanA

6:00 pm on Nov 28, 2005 (gmt 0)

10+ Year Member



Do robots still crawl the site?
They shouldn't (http://www.webmasterworld.com/robots.txt)
Does Google show recent modifications or still show in its cache (have a look at the date there) the pages it indexed before reading robots.txt?

Lord Majestic

6:14 pm on Nov 28, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



You may want to try to use Google's url removal facility to remove those pages that they have already indexed - robots.txt standard does not require search engines to remove already cralwed pages from their index if such pages became disallowed at a later date.

nwhorton

6:55 pm on Nov 28, 2005 (gmt 0)

10+ Year Member



Ah yes, it's looking like the pages are all cached well before the robots.txt file was added. I'll look into having these pages removed...does anyone have any tips on the topic of removal?