Welcome to WebmasterWorld Guest from 54.145.246.183

Forum Moderators: coopster & jatar k & phranque

Perl using Curl Banned

can not crawl

   
1:02 pm on Jan 3, 2006 (gmt 0)

10+ Year Member



Hi Sirs,

I have a perl script that copy news from different sites. Some sites now changed something, and I'm get this error when I make command perl News_Site.cgi:

curl (7) socket error: 111
failed

Ths site when I want to extract the news do not respond to PINGs, but is online. I tryed a search engine simulator and that site is live. I think that site block somehow PINGS and CURL. Am I wrong?

My script starts:

##########################################################
my $newDumpFlag=int($ARGV[2]); ## if zero then start the new dump . If 1 then continue/overwrite the old dump
###########################################################
$startPgNo=int($ARGV[0]);
if(int($startPgNo) ==0)
{
$startPgNo=1;
}

Did you resolve this somehow this problem? Any help please?

Many thanks,
Dan

11:45 pm on Jan 5, 2006 (gmt 0)

5+ Year Member



sites can block pinging, not sure about curl.
11:49 pm on Jan 5, 2006 (gmt 0)

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I don't know about blocking curl, it should just be a regular request

they could block a suspicious request. Maybe something you are doing makes it look like a scraper.

user agent
requesting cgi script with out some param
etc.

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month