Welcome to WebmasterWorld Guest from 54.227.110.209

Forum Moderators: IanTurner & engine

Message Too Old, No Replies

Getting more than a 1,000 results

     

findtheneedle

11:29 am on Nov 22, 2005 (gmt 0)

10+ Year Member



Is it possible to get more than 1,000 results in google with a program of some sort?

DaveN

1:20 pm on Nov 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



yes .. but i think you will need a good inhouse programmer

findtheneedle

3:01 pm on Nov 22, 2005 (gmt 0)

10+ Year Member



Ok is there a program that you could recommend?

DaveN

3:08 pm on Nov 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



you need to scape and rape the serps, you will need a program written for you.

DaveN

inbound

3:15 pm on Nov 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Getting more than 1000 results on Google.co.uk:

Do searches for UK sites but limit results to .co.uk

e.g.

widget site:co.uk

Then repeat for .com .org.uk .net etc.

If you need more than that then try also using a date operator to restrict results to a set period and then get rid of the duplicates.

If you need more than that you could try doing something similar with Yahoo and MSN and combining the results OR use common terms with positive and negative operators to split results e.g

widget site:co.uk -commomterm
widget site:co.uk commomterm

gives 2000 results, be careful to use terms that are likely to appear around an even amount of times otherwise you will skew your result set.

Other ways include:

restricting the file format to return Word or PDF etc

Pay Gigablast for a commercial feed and get 10,000 at a time from their data :)

Use common first or surnames as positive qualifiers - e.g. widget john site:co.uk

Use large town names as positive qualifiers (this will also allow you to filter out directory sites as they will have many appearances in the goegraphical lists.

IF you still need more then I think that buying Google would be the next step, certainly easier than combining all of that ;)

inbound

3:17 pm on Nov 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



[google.com...] is the home of the legitimate way to query Google en masse, however it's severely limited by the number of daily queries so you may need to scrape as suggested.

DaveN

3:21 pm on Nov 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



inbound did you find that maxresult will only return 10 even if set to 100?/

DaveN

inbound

6:35 pm on Nov 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes, the Google Search API is a pain with it's throttling, I just thought I had to include it to give the legitimate route to get the results. It's not so much of an issue if you have a program set up to churn away but it severley restricts the applications you can build for realtime queries, which is kind of daft.

engine

7:22 pm on Nov 22, 2005 (gmt 0)

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



which is kind of daft.

hmmm, I think you're looking at it from the wrong angle. With an API key it allows people to run their apps without overloading Google's system. Google could remove the facility and then where'd we be.

DaveN

7:36 pm on Nov 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



nah .. what we mean Engine is that you have a maxresults setting like NUM=100 in normal google .. but maxresults is 10 or 9,8,7,6,5,4,3,2,1

but no more than 10.. so it's poor .. even multi threading you use up 10 requests to get 100 results back .. when you only have 1000 request that soon adds up to .. api key query depletion

DaveN

 

Featured Threads

Hot Threads This Week

Hot Threads This Month