Forum Moderators: open
Curiousity turned into "checking a few things out", which turned into "examining a few SOAP implementations", which turned into "running a few experiments", which turned into "writing a program or two", which turned into... "an all-nighter"! (The programmers among you should be failiar with this phenomena! ;) ) The end result was a collection of nifty Google Tools written in Visual Basic, one of which is detailed below.
The Visual Basic Google Rank Check program has the following features:
> Supports many versions of Windows (all the way back to Win95 OSR2!)
> Does not require Visual Studios .NET (even works with old VB5!)
> Does not require you to install a bunch of XML drivers and Toolkits
> Does not require a PERL or JAVA interpreter to run
> Not limited to only first 10 results, does high speed "auto-scan" to any desired search depth, up to the API maximum of 1,000 results!
> Auto-halts deep scans as soon as your page is found (avoids wasting precious API daily search limits!)
> Displays interim results and status during deep scans.
> Traps and reports SOAP-based Google API Server Errors.
> Easy to Use
For the VB programmers among you, following are full instructions and source code! To use this you need to be 1) familiar with Visual Basic programming, and 2) have signed-up for the Google API and gotten your own "key".
SOAPing-Up Your PC:
As the Google API uses SOAP/XML to communicate with their server, the first thing you need is some way of accessing SOAP from Visual Basic without needing .NET! I looked at several different packages, some were designed primarily for non-VB environments (like SOAPLite); and some were just too convoluted, required other software to run, or buggy (like Microsoft). I finally settled on PocketSOAP.
I choose it because a) it was originally written for the PocketPC, so it is compact and efficient, b) it is fully self-contained, no extra XML drivers required, c) is supports various versions of Windows, including older ones, d) it works well with Visual Basic, and e) it's free!
Installing PocketSOAP was easy. If you are running Windows ME, 2000 or XP, just download [pocketsoap.com] (small, < 1mb) and install!
(If you are running an old version of Windows (95/98/NT), you first need to upgrade to a new version of the Windows Installer program, which you can get for either Win 95/98 [pocketsoap.com] or for NT [pocketsoap.com], then you can download and install PocketSOAP)
That's it! Your PC now has SOAP/XML support!
NEXT STEP - "THE FORM"
OK, so your PC is all SOAPed up. Now what?
Fire-up Visual Basic, and create a New (Standard EXE) project. Save and name the Project and Form.
First thing we need to do is tell Visual Basic that we want to use SOAP in this program, so go to Project/References, find the line "PocketSOAP 1.3.0 Type Library", and check the little box beside it, then click OK. (Programming Hint - if you ever get an compiler error "User defined type not defined" on the first line of the program, go back and make sure that this box is still checked!)
Now, make your new form large enough to hold everything (mine is 8000 x 8000). Put a nice title at the top, add some instructional text below if desired. The main thing is to make sure that you add the following required items to your form, otherwise it won't work! -
txtQuery - a single-line text box, long enough to hold your search phrase. Make sure that you name it txtQuery.
txtURL - a single-line text box, long enough to hold the URL that you are scanning for. Make sure that you name it txtURL.
txtMax - a single-line text box, to hold the desired search depth. A width of 1200 to 1500 is fine. Make sure that you name it txtMax.
lblCurrent - a label, which will be used to display the current progress during deep scans, and the actual scan depth when a search is completed. A width of about 1200 to 1500 is fine. Set the caption to empty. Make sure that you name it lblCurrent. You may also want to add another label to the left of it, with the text "Scanning: "
cmdDoSearch - a command button, which will be used to perform the search. You can set the caption to "Do Search!" Make sure that you name it cmdDoSearch.
txtResults - a multi-line text box, to hold the scan results. Make sure to set the MultiLine property to True, and the ScrollBars property to 3-Both. Adjust the size so that it is large enough to hold your results, I used HEIGHT 2500 and WIDTH 6900. Make sure you name it txtResults.
That's it! You may want to add some labels to identify what these boxes are for, and make it look pretty. As long as you have the required items listed above, then your form's ready to rock and roll!
Save your form and project now, so you don't loose anything.
NEXT STEP - THE PROGRAMMING
OK, you have a pretty form that does nothing. Now we need to add some code to make it fly!
First, go to the Declarations section at the top of the code view of your new form, and add the following declarations -
Option Explicit
Dim SOAPRequest As CoEnvelope
Dim Transport As HTTPTransport
' Google API SOAP Params
Dim GoogleSearchMethodName As String
Dim GoogleNameSpaceURI As String
Dim GoogleSOAPAction As String
Dim GoogleEndPoint As String
Dim MyKey As String
Dim DupFilter As Boolean
Dim FamilyFilter As Boolean
You'll notice that I'm using variables, rather than hard-coding some of the SOAP-related information. This makes it easier to understand what the code is doing, or to change it later if needed.
Next, add the following initialization code to Form_Load. Make sure that you replace the "put your key here" with the actual value of the API key code you got from Google, or else it won't work! -
Private Sub Form_Load()
' Init Vars
GoogleSearchMethodName = "doGoogleSearch"
GoogleNameSpaceURI = "urn:GoogleSearch"
GoogleSOAPAction = "urn:GoogleSearchAction"
GoogleEndPoint = "http://api.google.com/search/beta2"
MyKey = "put your key here"
DupFilter = True
FamilyFilter = False
' Init Form vars
txtQuery = ""
txtURL = ""
txtMax = 100
lblCurrent = 0
txtResults = ""
End Sub
Notice the variables DupFilter and FamilyFilter. These control how the search is done. DupFilter eliminates "very similar" duplicate listings in the results, and limits each domain to 2 listings max, this is the normal setting for Google searches unless you change it. FamilyFilter controls the "Family Safe" filtering of search results. You can change these settings if you wish.
(On the subject of settings, you will see below that the API restrict, lr, ie, and oe options are presently set to default, you only need these if you wish to limit searches to a specific country or language, or search with an non-standard character set (like Chinese), see the API doc's if you need to do this.)
Next, add the Private Function ValidateForm, and add the following code. This function does simple error-checking on the user's input, you can add to this if you wish -
Private Function ValidateForm() as Boolean
Dim OKFlag as Boolean
OKFlag = True
If txtMax = "" Then txtMax = 100
If txtMax > 1000 Then
MsgBox "Error - API Cannot scan past 1,000 listings!", _
vbExclamation, "Error - # of Listings too large!"
OKFlag = False
End If
If Trim(txtQuery) = "" Then
MsgBox "Error - No Search Query Specified!", _
vbExclamation, "No Search Query Specified!"
OKFlag = False
End If
If Trim(txtURL) = "" Then
MsgBox "Error - No URL to Search for!", _
vbExclamation, "No URL to Search for!"
OKFlag = False
End If
ValidateForm = OKFlag
End Function
Finally, add the following code to your command button's cmdDoSearch_click() event. This is the meat and potatoes of the program, and actually performs the scan and generates the results -
Private Sub cmdDoSearch_Click()
Dim CurrentOffset As Long
Dim RequestSize As Long
Dim Listings As Variant
Dim CurrentURL As String
Dim MyCounter As Long
Dim strResults As String
Dim MatchFound As Boolean
MatchFound = False
On Error goto SOAPErr
If ValidateForm() Then
strResults = ""
Set Transport = New HTTPTransport
CurrentOffset = 0
RequestSize = 10
If (txtMax < 10) And (txtMax > 0) Then
RequestSize = txtMax
End If
lblCurrent.Caption = "0"
APILoop:
Set SOAPRequest = New CoEnvelope
SOAPRequest.SetMethod GoogleSearchMethodName, _
GoogleNameSpaceURI
SOAPRequest.Parameters.Create "key", MyKey
SOAPRequest.Parameters.Create "q", Trim(txtQuery)
SOAPRequest.Parameters.Create "start", CurrentOffset
SOAPRequest.Parameters.Create "maxResults", RequestSize
SOAPRequest.Parameters.Create "filter", DupFilter
SOAPRequest.Parameters.Create "restrict", ""
SOAPRequest.Parameters.Create "safeSearch", FamilyFilter
SOAPRequest.Parameters.Create "lr", ""
SOAPRequest.Parameters.Create "ie", ""
SOAPRequest.Parameters.Create "oe", ""
Transport.SOAPAction = GoogleSOAPAction
Transport.Send GoogleEndPoint, SOAPRequest.Serialize
SOAPRequest.Parse Transport
Listings = SOAPRequest.Parameters.Item(0).Nodes.ItemByName("resultElements").Value
If (CurrentOffset = 0) Then ' First Time thru, get total "hits"
strResults = strResults & "Out of "
If (True = SOAPRequest.Parameters.Item(0).Nodes.ItemByName("estimateIsExact").Value) Then
strResults = strResults & "Exactly "
Else
strResults = strResults & "Approxiately "
End If
strResults = strResults & _
SOAPRequest.Parameters.Item(0).Nodes.ItemByName("estimatedTotalResultsCount").Value & _
" hits" & vbCrLf & vbCrLf
txtResults = strResults
txtResults.Refresh
End If
For MyCounter = 0 To UBound(Listings)
CurrentURL = Listings(MyCounter).Nodes.ItemByName("URL").Value
If (InStr(1, CurrentURL, txtURL, vbTextCompare) > 0) Then
' Match found
strResults = strResults & "#" & CStr(CurrentOffset + MyCounter + 1) & " "
strResults = strResults & Listings(MyCounter).Nodes.ItemByName("title").Value & vbCrLf
strResults = strResults & Listings(MyCounter).Nodes.ItemByName("URL").Value & vbCrLf & vbCrLf
txtResults = strResults
txtResults.Refresh
MatchFound = True
End If
Next
lblCurrent.Caption = CStr(CurrentOffset + UBound(Listings) + 1)
lblCurrent.Refresh
CurrentOffset = CurrentOffset + 10
If ((txtMax - CurrentOffset) < 10) Then
RequestSize = txtMax - CurrentOffset
End If
If (CurrentOffset < txtMax) And (UBound(Listings) >= 8) _
And Not (MatchFound) Then
GoTo APILoop
End If
strResults = strResults & "Done." & vbCrLf
txtResults = strResults
Set SOAPRequest = Nothing
Set Transport = Nothing
End If
Exit Sub
SOAPErr:
MsgBox "Err # " & Err.Number & vbCrLf & vbCrLf & _
Err.Description, vbExclamation, _
"Google API SOAP/XML Error!"
Exit Sub
End Sub
That's it! Look over it for any typos. Now be sure to save the form again so you don't loose anything!
NEXT - HOW TO USE
IGNORETHISJUSTABUNCHOFTEXTSOTHATTHECODELISTINGIN THISFORMABOVEDOESNTWRAP AROUNDWHEREITSHOULDNTBEWRAPPING
(edited by: aspdesigner at 6:40 am (utc) on May 31, 2002)
To use this, make sure that you are connected to the Internet, then run the program. Enter the desired search phrase into the txtQuery box, the URL to search for in the txtURL box, change the search depth in the txtMax box if desired, and then click the "Do Search!" button.
The URL can be either an exact page (www.mysite.com/mypage.html), or just a domain name (http://www.mydomain.com), or even a partial domain name! (intel) You can either include the "http://" or not, both ways will work. The program will scan until it finds a listing who's URL contains what you typed.
The program will do an automatic looping high-speed scan to whatever search depth you specify, up to the maximum API limit of 1,000 listings, via the txtMax box. It is recommended that you leave this set to the default unless you need to do a deep scan. Very deep scans take longer, use resources, and do you really need to know if your page comes up #935 on some obscure search term?
More important, it is very easy to use up you daily maximum API search allotment using this program! At the maximum API search depth, doing only 10 maximum-depth searches could use up you quota for the entire day! So be considerate!
In order to increase efficiency and reduce search load, this program is also designed to automatically stop any search the instant that results are found. So if you are set to scan 100 results, and your page is found on #10, the program terminates the scanning at that point.
During a scan, lblCurrent will display the # of listings scanned so far. After the first block is read, the txtResults will display the total # of hits for that search phrase, and whether Google indicated that # was exact, or just an estimate (approximate). If your page is found, it will display the ranking, the page title, and the URL. If more than one result is found in the same block of 10 results (such as when you have back-to-back listings like #1 & #2), then both results will be displayed. The scan will terminate when you page is found, or the specified scan depth is reached.
A more enhanced version of this program is in process, will keep you informed.
If your a Visual Basic programmer, I discovered SOAP/XML can be a lot of fun! You can use the tools I have discussed to "get your feet wet", and maybe invent the next great application for the Google API in the process! Good luck!
NutsAndBolts, LOL ;).
I respectably disagree with your initial impression as to the proper forum, and request that you reconsider.
I gave some thought to the best forum to place this in. I did not add it to the forum in which your excellent program which inspired me into writing this was placed, as this is not a PERL or PHP program.
For the same reason, it would not be appropriate to place it in a .net/asp forum.
The whole design philosophy was to find a way to accomplish this in Windows WITHOUT having to resort to Microsoft-centric software or .NET!
I wanted to make something that would not require the latest flavor of Windows to work, and that could be experimented with by the average Joe who doesn't have $1,799 or more to spend on Visual Studio .Net!
I used NON-MICROSOFT communications software, and a decidedly NON-.NET design. Had I wanted to do this as a .NET program, it would have been considerably easier, as a .NET example program is included in the Google API ! The approach I took was a LOT more work!
This program was designed for VB5/6. The communications code is COMPLETELY DIFFERENT from how you would do it in .NET, indeed, I don't think that this program would even compile clean with VB.Net, due to the incompatibilities between the two languages!
If you had a VB5/6 forum, or a general Windows applications forum, that might be more understandable. But moving this to a .Net/asp forum makes about as much sense as moving a Unix post to a Windows forum!
Even if you had a VB5/6 forum, the second issue is the audience to which this topic best applies. This topic is of GREAT potential interest to people involved in Google SEO. It would be of LITTLE or NO interest to a programmer who is NOT involved with Google. The fact that a certain expertise beyond running WPG may be useful to gain the greatest possible benefit from this post does not detract from the target audience that this was written for.
For the rest of you, you can read all about Google's Beta-Test Web API Service here [google.com].
Seriously though, aspdesigner, that is absolutely fantastic. From finding the thread to running a query and getting some results took only a few minutes, Im very thankful.
And thanks to Google too for allowing us to use such tools.
I guess Id better keep practising, I wanna write great programmes that everyone wants!
All microsoft related 'code' internet stuff can go in this forum. Here, php/perl, and the html/css forum, are the only places we'd like to see code.
But seriously, More VB coders will find your code here then in the Google forum. Your code will be buried in the Google forum with in a week. Never to be seen again.
I've fixed the code. For some reason .net failed to pick up the reference properly. Instead of 'CoEnvelope' working I had to use 'PocketSOAP.CoEnvelope'. Also, I had to alter some code to make it .net complient.
Chris
if you're a student I believe there is a cheaper version of VB out there. $100
I guess you are talking about the Learning Edition. The problem with it is that it does not allow you to complile the code into executables.
i don't even know if that's possibe, but i'll hear i guess, executable would be great also, before i start cracking my head over vb programming language.
thanks
Is the code portable to this Brett?
Reading the thread got me thinking about discussion in another thread [webmasterworld.com ] where I asked Googleguy why automated position checkers like WPG haven't struck a deal with Google to use the API license keys on a fee basis. My thoughts were that if API keys were used by WPG, Google could charge WPG for the resources searches tie up and also discount WPG searches from their search calculations.
Googleguy's response was that assumes a world in which WPG is willing to pay for the thousands of queries a day that are issued. :)
Fair enough reply. Googleguy - this thread got me thinking if WPG won't play ball how about dealing with the WPG end-users? Why doesn't Google offer owners of automted ranking software like WPG the option of buying the equivalent of an API key. $X would allow 1,000 queries per day.
I'd certainly be happy to pay for this facility.
[edited by: biggles at 4:38 am (utc) on Oct. 1, 2002]
gopi, I believe it's much safer to use a program like this. It allows us to throttle back usage or delay results a little if the hardware is crunching. Compared to rank checking programs that search us without paying, I like the API route.
