Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: Ocean10000
Didn't check robots.txt. The reference page says:
MrCarlito-0.1 is an experimental spider that collects header & link information from web pages. The spider is written in PERL (Practical Extraction and Report Language), and uses the LWP::UserAgent Class. Currently this spider does not delve into websites, it simply obtains the headers & hostnames contained in your web page index.
IMHO - it would be more polite if Mr.Carlito bothered to check with robots.txt to see if he's welcome. I guess that's not Carlito's Way.
[edited by: incrediBILL at 8:12 pm (utc) on July 6, 2008]
[edit reason] fixed formatting and link [/edit]