Perl using Curl Banned

can not crawl

[aff_dan]

Hi Sirs,

I have a perl script that copy news from different sites. Some sites now changed something, and I'm get this error when I make command perl News_Site.cgi:

curl (7) socket error: 111
failed

Ths site when I want to extract the news do not respond to PINGs, but is online. I tryed a search engine simulator and that site is live. I think that site block somehow PINGS and CURL. Am I wrong?

My script starts:

##########################################################
my $newDumpFlag=int($ARGV[2]); ## if zero then start the new dump . If 1 then continue/overwrite the old dump
###########################################################
$startPgNo=int($ARGV[0]);
if(int($startPgNo) ==0)
{
$startPgNo=1;
}

Did you resolve this somehow this problem? Any help please?

Many thanks,
Dan

[perl_diver]

sites can block pinging, not sure about curl.

[jatar_k]

I don't know about blocking curl, it should just be a regular request

they could block a suspicious request. Maybe something you are doing makes it look like a scraper.

user agent
requesting cgi script with out some param
etc.