Welcome to WebmasterWorld Guest from 100.24.122.228

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Stopping url being indexed?

Not with robots.txt

     
11:32 am on Nov 29, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:May 9, 2005
posts:172
votes: 0


Is there any other way to stop a URL being indexed other than use robots.txt? We have a statistics script that loads at the end of the html but robots have been following it and getting caught in an endless loop.

I have posted an example of the code before,

<img src="http://www.mysite.fr/stats/getStats.cfm?passed_sessionkey=1132945495-14913556&passed_scriptname=&passed_jobref=&passed_http_request_method=GET&passed_path_translated=D:\eFC\index.cfm&passed_script_name=/index.cfm&passed_query_string=&passed_HTTP_referer=&partnerId=003&passed_uniqueUser=C2430036-1321-B04D-82CCC92A61FA1FAC" width="0" height="0" style="display:none">

Anyone know a way in html that could be used to stop spiders from indexing this link - notice the link is dynamic and will change each visit.

Any help apprecaited.

7:07 pm on Dec 1, 2005 (gmt 0)

New User

10+ Year Member

joined:Nov 18, 2005
posts:27
votes: 0


Ya, just add

<META NAME="ROBOTS" CONTENT="NOINDEX">

and if your problem is also with google, try

<META NAME="GOOGLEBOT" CONTENT="NOARCHIVE">

in your <HEAD></HEAD>.

That should fix you up.

Regards,
- Shaun

6:52 pm on Dec 2, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 11, 2004
posts:62
votes: 0


I'm just thinking out loud here so somebody please correct me if i'm wrong, but with your link

getStats.cfm?passed_sessionkey=1132945495..etc..etc

what if because the URL is different you're not properly definining the file name in robots.txt. Perhaps in robotos.txt deny all for the directory so that everything within it is banned. Also adding the no index meta tag to the page too sounds good.

Also still, if it's a bad bot.. try .htaccess in that directory and ban by IP.

I hope this helps.