Is yahoo blocking automated script?

[caragh]

Hi All,

I have an automated script that returns a number of results from Yahoo.
It uses php curl to return the results. However, sometimes it will not return anything - out of the blue.

Does Yahoo identify the use of an automated script on it? and do they then block it? If so - what is the allowance before they block the script?

Any help would be appreciated on this.
cheers
CAD

[eelixduppy]

You should be getting an appropriate XML error response from Yahoo if there was something they were doing. For instance, if you exceed the daily limit you would receive a response similar to the following from the Search API:

 <Error xmlns="urn:yahoo:api"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:noNamespaceSchemaLocation="http://api.yahoo.com/Api/V1/error.xsd">
   The following errors were detected:
  <Message>limit exceeded</Message>
 </Error>

My guess is that your script is hanging up somewhere, so check your error logs; there is also something that is overlooked with these things.

Also, you might find Yahoo's API FAQ [developer.yahoo.com] useful.

[caragh]

sorry - i wasn't clear - I'm not using the API.
The script doesn't hang - as it makes it to the end. (echos 'end of script')

[eelixduppy]

Not sure what the issue is, then. Check your error logs to see if anything comes up. There is obviously something that isn't processing correctly. I do have a question, though: Why aren't you using the API? It will make things easier for you. It is certainly much more dynamic, too, as far as working with the data returned.

[caragh]

not using the api as the scraping method is working with all other engines I look at - google etc.
And moving just one to use an api means maintaining that api install etc.

It looks like I'll have to move this one to using the api though. However, I do know that using the api will monitor how much use I am making. And it's not clear what the limits are. (http://developer.yahoo.com/search/rate.html) I'd like to know will I be hitting that limit immediately -- if so I should avoid the api and try and figure out my problem above.

Any ideas? cheers for input so far

[eelixduppy]

Not sure about their limit as they aren't specific anywhere I look, however, you might want to take a look at their Terms of Use [info.yahoo.com].

As far as switching over, if you have a lot of time invested in this particular script it might be worth just looking for a simple bug (what might be simple) and fixing it. I would have started with the API if I had to do a project like this, but that's just me. If you cannot fix the problem then I'd probably switch over to the API if it isn't going to be too much of a hassle for you.