Anyone know if this robot is good or bad?

[candlelady]

RedCarpet/1.0(http://www.redcarpet-inc.com/robots.html had been
searching my site several times now. Today when I clicked on the referrer link, there was a warning header that came up on my shopping cart "sign in" (forgot password)stating (Warning: Cannot modify header information - headers already sent by}. Is it trying to do something malicious?

This is almost becoming a full-time endeavor to weed out the bad guys.

Thanks for any help.

[volatilegx]

Hi candlelady and welcome to WebmasterWorld :)

This is the first I've heard of this bot. Tough to know if it is "good" or "bad". Their site isn't really live yet. Does the bot obey robots.txt rules? The fact that it sends a referrer header is very unusual.

[candlelady]

How would I know if it "obeyed" the robot.txt message or not? I know just enough "webese" to be dangerous!

[volatilegx]

A robots.txt file is a way of telling spiders which files they may access. If the spider requests a file that has been forbidden by the robots.txt file, then it is not obeying robots.txt. You can tell this by reviewing your website's log files.

Some of us will create a "bot trap" web page that is linked to from our index page (with an obscure link that humans wouldn't click), but disallowed in our robots.txt. This bot trap page will be a PHP (or other CGI) script that will log the access by the "bad" bot and possibly ban the bot from the server.