homepage Welcome to WebmasterWorld Guest from 54.197.65.82
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Code, Content, and Presentation / Perl Server Side CGI Scripting
Forum Library, Charter, Moderators: coopster & jatar k & phranque

Perl Server Side CGI Scripting Forum

    
Perl using Curl Banned
can not crawl
aff_dan

10+ Year Member



 
Msg#: 4330 posted 1:02 pm on Jan 3, 2006 (gmt 0)

Hi Sirs,

I have a perl script that copy news from different sites. Some sites now changed something, and I'm get this error when I make command perl News_Site.cgi:

curl (7) socket error: 111
failed

Ths site when I want to extract the news do not respond to PINGs, but is online. I tryed a search engine simulator and that site is live. I think that site block somehow PINGS and CURL. Am I wrong?

My script starts:

##########################################################
my $newDumpFlag=int($ARGV[2]); ## if zero then start the new dump . If 1 then continue/overwrite the old dump
###########################################################
$startPgNo=int($ARGV[0]);
if(int($startPgNo) ==0)
{
$startPgNo=1;
}

Did you resolve this somehow this problem? Any help please?

Many thanks,
Dan

 

perl_diver

5+ Year Member



 
Msg#: 4330 posted 11:45 pm on Jan 5, 2006 (gmt 0)

sites can block pinging, not sure about curl.

jatar_k

WebmasterWorld Administrator jatar_k us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4330 posted 11:49 pm on Jan 5, 2006 (gmt 0)

I don't know about blocking curl, it should just be a regular request

they could block a suspicious request. Maybe something you are doing makes it look like a scraper.

user agent
requesting cgi script with out some param
etc.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / Perl Server Side CGI Scripting
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved