homepage Welcome to WebmasterWorld Guest from 54.166.65.9
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / UK Search and Internet Marketing News
Forum Library, Charter, Moderators: IanTurner & engine

UK Search and Internet Marketing News Forum

    
Getting more than a 1,000 results
findtheneedle




msg:257735
 11:29 am on Nov 22, 2005 (gmt 0)

Is it possible to get more than 1,000 results in google with a program of some sort?

 

DaveN




msg:257736
 1:20 pm on Nov 22, 2005 (gmt 0)

yes .. but i think you will need a good inhouse programmer

findtheneedle




msg:257737
 3:01 pm on Nov 22, 2005 (gmt 0)

Ok is there a program that you could recommend?

DaveN




msg:257738
 3:08 pm on Nov 22, 2005 (gmt 0)

you need to scape and rape the serps, you will need a program written for you.

DaveN

inbound




msg:257739
 3:15 pm on Nov 22, 2005 (gmt 0)

Getting more than 1000 results on Google.co.uk:

Do searches for UK sites but limit results to .co.uk

e.g.

widget site:co.uk

Then repeat for .com .org.uk .net etc.

If you need more than that then try also using a date operator to restrict results to a set period and then get rid of the duplicates.

If you need more than that you could try doing something similar with Yahoo and MSN and combining the results OR use common terms with positive and negative operators to split results e.g

widget site:co.uk -commomterm
widget site:co.uk commomterm

gives 2000 results, be careful to use terms that are likely to appear around an even amount of times otherwise you will skew your result set.

Other ways include:

restricting the file format to return Word or PDF etc

Pay Gigablast for a commercial feed and get 10,000 at a time from their data :)

Use common first or surnames as positive qualifiers - e.g. widget john site:co.uk

Use large town names as positive qualifiers (this will also allow you to filter out directory sites as they will have many appearances in the goegraphical lists.

IF you still need more then I think that buying Google would be the next step, certainly easier than combining all of that ;)

inbound




msg:257740
 3:17 pm on Nov 22, 2005 (gmt 0)

[google.com...] is the home of the legitimate way to query Google en masse, however it's severely limited by the number of daily queries so you may need to scrape as suggested.

DaveN




msg:257741
 3:21 pm on Nov 22, 2005 (gmt 0)

inbound did you find that maxresult will only return 10 even if set to 100?/

DaveN

inbound




msg:257742
 6:35 pm on Nov 22, 2005 (gmt 0)

Yes, the Google Search API is a pain with it's throttling, I just thought I had to include it to give the legitimate route to get the results. It's not so much of an issue if you have a program set up to churn away but it severley restricts the applications you can build for realtime queries, which is kind of daft.

engine




msg:257743
 7:22 pm on Nov 22, 2005 (gmt 0)

which is kind of daft.

hmmm, I think you're looking at it from the wrong angle. With an API key it allows people to run their apps without overloading Google's system. Google could remove the facility and then where'd we be.

DaveN




msg:257744
 7:36 pm on Nov 22, 2005 (gmt 0)

nah .. what we mean Engine is that you have a maxresults setting like NUM=100 in normal google .. but maxresults is 10 or 9,8,7,6,5,4,3,2,1

but no more than 10.. so it's poor .. even multi threading you use up 10 requests to get 100 results back .. when you only have 1000 request that soon adds up to .. api key query depletion

DaveN

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / UK Search and Internet Marketing News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved