| This 35 message thread spans 2 pages: 35 (  2 ) > > || |
|Visual Basic Google API Rank Checker Program |
Deep scanning, Win Pgm with source code!
| 6:21 am on May 31, 2002 (gmt 0)|
I was quite pleased to find Brett's excellent program [webmasterworld.com]. Unfortunately, it requires PERL, which I don't have on my Windows-based PC. :( I wanted to know if there was a way to do something like this in Windows from Visual Basic, without requiring Visual Studios .NET (expensive!)
Curiousity turned into "checking a few things out", which turned into "examining a few SOAP implementations", which turned into "running a few experiments", which turned into "writing a program or two", which turned into... "an all-nighter"! (The programmers among you should be failiar with this phenomena! ;) ) The end result was a collection of nifty Google Tools written in Visual Basic, one of which is detailed below.
The Visual Basic Google Rank Check program has the following features:
> Supports many versions of Windows (all the way back to Win95 OSR2!)
> Does not require Visual Studios .NET (even works with old VB5!)
> Does not require you to install a bunch of XML drivers and Toolkits
> Does not require a PERL or JAVA interpreter to run
> Not limited to only first 10 results, does high speed "auto-scan" to any desired search depth, up to the API maximum of 1,000 results!
> Auto-halts deep scans as soon as your page is found (avoids wasting precious API daily search limits!)
> Displays interim results and status during deep scans.
> Traps and reports SOAP-based Google API Server Errors.
> Easy to Use
For the VB programmers among you, following are full instructions and source code! To use this you need to be 1) familiar with Visual Basic programming, and 2) have signed-up for the Google API and gotten your own "key".
SOAPing-Up Your PC:
As the Google API uses SOAP/XML to communicate with their server, the first thing you need is some way of accessing SOAP from Visual Basic without needing .NET! I looked at several different packages, some were designed primarily for non-VB environments (like SOAPLite); and some were just too convoluted, required other software to run, or buggy (like Microsoft). I finally settled on PocketSOAP.
I choose it because a) it was originally written for the PocketPC, so it is compact and efficient, b) it is fully self-contained, no extra XML drivers required, c) is supports various versions of Windows, including older ones, d) it works well with Visual Basic, and e) it's free!
Installing PocketSOAP was easy. If you are running Windows ME, 2000 or XP, just download [pocketsoap.com] (small, < 1mb) and install!
(If you are running an old version of Windows (95/98/NT), you first need to upgrade to a new version of the Windows Installer program, which you can get for either Win 95/98 [pocketsoap.com] or for NT [pocketsoap.com], then you can download and install PocketSOAP)
That's it! Your PC now has SOAP/XML support!
NEXT STEP - "THE FORM"
| 6:22 am on May 31, 2002 (gmt 0)|
OK, so your PC is all SOAPed up. Now what?
Fire-up Visual Basic, and create a New (Standard EXE) project. Save and name the Project and Form.
First thing we need to do is tell Visual Basic that we want to use SOAP in this program, so go to Project/References, find the line "PocketSOAP 1.3.0 Type Library", and check the little box beside it, then click OK. (Programming Hint - if you ever get an compiler error "User defined type not defined" on the first line of the program, go back and make sure that this box is still checked!)
Now, make your new form large enough to hold everything (mine is 8000 x 8000). Put a nice title at the top, add some instructional text below if desired. The main thing is to make sure that you add the following required items to your form, otherwise it won't work! -
txtQuery - a single-line text box, long enough to hold your search phrase. Make sure that you name it txtQuery.
txtURL - a single-line text box, long enough to hold the URL that you are scanning for. Make sure that you name it txtURL.
txtMax - a single-line text box, to hold the desired search depth. A width of 1200 to 1500 is fine. Make sure that you name it txtMax.
lblCurrent - a label, which will be used to display the current progress during deep scans, and the actual scan depth when a search is completed. A width of about 1200 to 1500 is fine. Set the caption to empty. Make sure that you name it lblCurrent. You may also want to add another label to the left of it, with the text "Scanning: "
cmdDoSearch - a command button, which will be used to perform the search. You can set the caption to "Do Search!" Make sure that you name it cmdDoSearch.
txtResults - a multi-line text box, to hold the scan results. Make sure to set the MultiLine property to True, and the ScrollBars property to 3-Both. Adjust the size so that it is large enough to hold your results, I used HEIGHT 2500 and WIDTH 6900. Make sure you name it txtResults.
That's it! You may want to add some labels to identify what these boxes are for, and make it look pretty. As long as you have the required items listed above, then your form's ready to rock and roll!
Save your form and project now, so you don't loose anything.
NEXT STEP - THE PROGRAMMING
| 6:23 am on May 31, 2002 (gmt 0)|
OK, you have a pretty form that does nothing. Now we need to add some code to make it fly!
First, go to the Declarations section at the top of the code view of your new form, and add the following declarations -
Dim SOAPRequest As CoEnvelope
Dim Transport As HTTPTransport
' Google API SOAP Params
Dim GoogleSearchMethodName As String
Dim GoogleNameSpaceURI As String
Dim GoogleSOAPAction As String
Dim GoogleEndPoint As String
Dim MyKey As String
Dim DupFilter As Boolean
Dim FamilyFilter As Boolean
You'll notice that I'm using variables, rather than hard-coding some of the SOAP-related information. This makes it easier to understand what the code is doing, or to change it later if needed.
Next, add the following initialization code to Form_Load. Make sure that you replace the "put your key here" with the actual value of the API key code you got from Google, or else it won't work! -
Private Sub Form_Load()
' Init Vars
GoogleSearchMethodName = "doGoogleSearch"
GoogleNameSpaceURI = "urn:GoogleSearch"
GoogleSOAPAction = "urn:GoogleSearchAction"
GoogleEndPoint = "http://api.google.com/search/beta2"
MyKey = "put your key here"
DupFilter = True
FamilyFilter = False
' Init Form vars
txtQuery = ""
txtURL = ""
txtMax = 100
lblCurrent = 0
txtResults = ""
Notice the variables DupFilter and FamilyFilter. These control how the search is done. DupFilter eliminates "very similar" duplicate listings in the results, and limits each domain to 2 listings max, this is the normal setting for Google searches unless you change it. FamilyFilter controls the "Family Safe" filtering of search results. You can change these settings if you wish.
(On the subject of settings, you will see below that the API restrict, lr, ie, and oe options are presently set to default, you only need these if you wish to limit searches to a specific country or language, or search with an non-standard character set (like Chinese), see the API doc's if you need to do this.)
Next, add the Private Function ValidateForm, and add the following code. This function does simple error-checking on the user's input, you can add to this if you wish -
Private Function ValidateForm() as Boolean
Dim OKFlag as Boolean
OKFlag = True
If txtMax = "" Then txtMax = 100
If txtMax > 1000 Then
MsgBox "Error - API Cannot scan past 1,000 listings!", _
vbExclamation, "Error - # of Listings too large!"
OKFlag = False
If Trim(txtQuery) = "" Then
MsgBox "Error - No Search Query Specified!", _
vbExclamation, "No Search Query Specified!"
OKFlag = False
If Trim(txtURL) = "" Then
MsgBox "Error - No URL to Search for!", _
vbExclamation, "No URL to Search for!"
OKFlag = False
ValidateForm = OKFlag
Finally, add the following code to your command button's cmdDoSearch_click() event. This is the meat and potatoes of the program, and actually performs the scan and generates the results -
Private Sub cmdDoSearch_Click()
Dim CurrentOffset As Long
Dim RequestSize As Long
Dim Listings As Variant
Dim CurrentURL As String
Dim MyCounter As Long
Dim strResults As String
Dim MatchFound As Boolean
MatchFound = False
On Error goto SOAPErr
If ValidateForm() Then
strResults = ""
Set Transport = New HTTPTransport
CurrentOffset = 0
RequestSize = 10
If (txtMax < 10) And (txtMax > 0) Then
RequestSize = txtMax
lblCurrent.Caption = "0"
Set SOAPRequest = New CoEnvelope
SOAPRequest.SetMethod GoogleSearchMethodName, _
SOAPRequest.Parameters.Create "key", MyKey
SOAPRequest.Parameters.Create "q", Trim(txtQuery)
SOAPRequest.Parameters.Create "start", CurrentOffset
SOAPRequest.Parameters.Create "maxResults", RequestSize
SOAPRequest.Parameters.Create "filter", DupFilter
SOAPRequest.Parameters.Create "restrict", ""
SOAPRequest.Parameters.Create "safeSearch", FamilyFilter
SOAPRequest.Parameters.Create "lr", ""
SOAPRequest.Parameters.Create "ie", ""
SOAPRequest.Parameters.Create "oe", ""
Transport.SOAPAction = GoogleSOAPAction
Transport.Send GoogleEndPoint, SOAPRequest.Serialize
Listings = SOAPRequest.Parameters.Item(0).Nodes.ItemByName("resultElements").Value
If (CurrentOffset = 0) Then ' First Time thru, get total "hits"
strResults = strResults & "Out of "
If (True = SOAPRequest.Parameters.Item(0).Nodes.ItemByName("estimateIsExact").Value) Then
strResults = strResults & "Exactly "
strResults = strResults & "Approxiately "
strResults = strResults & _
SOAPRequest.Parameters.Item(0).Nodes.ItemByName("estimatedTotalResultsCount").Value & _
" hits" & vbCrLf & vbCrLf
txtResults = strResults
For MyCounter = 0 To UBound(Listings)
CurrentURL = Listings(MyCounter).Nodes.ItemByName("URL").Value
If (InStr(1, CurrentURL, txtURL, vbTextCompare) > 0) Then
' Match found
strResults = strResults & "#" & CStr(CurrentOffset + MyCounter + 1) & " "
strResults = strResults & Listings(MyCounter).Nodes.ItemByName("title").Value & vbCrLf
strResults = strResults & Listings(MyCounter).Nodes.ItemByName("URL").Value & vbCrLf & vbCrLf
txtResults = strResults
MatchFound = True
lblCurrent.Caption = CStr(CurrentOffset + UBound(Listings) + 1)
CurrentOffset = CurrentOffset + 10
If ((txtMax - CurrentOffset) < 10) Then
RequestSize = txtMax - CurrentOffset
If (CurrentOffset < txtMax) And (UBound(Listings) >= 8) _
And Not (MatchFound) Then
strResults = strResults & "Done." & vbCrLf
txtResults = strResults
Set SOAPRequest = Nothing
Set Transport = Nothing
MsgBox "Err # " & Err.Number & vbCrLf & vbCrLf & _
Err.Description, vbExclamation, _
"Google API SOAP/XML Error!"
That's it! Look over it for any typos. Now be sure to save the form again so you don't loose anything!
NEXT - HOW TO USE
IGNORETHISJUSTABUNCHOFTEXTSOTHATTHECODELISTINGIN THISFORMABOVEDOESNTWRAP AROUNDWHEREITSHOULDNTBEWRAPPING
(edited by: aspdesigner at 6:40 am (utc) on May 31, 2002)
| 6:24 am on May 31, 2002 (gmt 0)|
HOW TO USE:
To use this, make sure that you are connected to the Internet, then run the program. Enter the desired search phrase into the txtQuery box, the URL to search for in the txtURL box, change the search depth in the txtMax box if desired, and then click the "Do Search!" button.
The URL can be either an exact page (www.mysite.com/mypage.html), or just a domain name (http://www.mydomain.com), or even a partial domain name! (intel) You can either include the "http://" or not, both ways will work. The program will scan until it finds a listing who's URL contains what you typed.
The program will do an automatic looping high-speed scan to whatever search depth you specify, up to the maximum API limit of 1,000 listings, via the txtMax box. It is recommended that you leave this set to the default unless you need to do a deep scan. Very deep scans take longer, use resources, and do you really need to know if your page comes up #935 on some obscure search term?
More important, it is very easy to use up you daily maximum API search allotment using this program! At the maximum API search depth, doing only 10 maximum-depth searches could use up you quota for the entire day! So be considerate!
In order to increase efficiency and reduce search load, this program is also designed to automatically stop any search the instant that results are found. So if you are set to scan 100 results, and your page is found on #10, the program terminates the scanning at that point.
During a scan, lblCurrent will display the # of listings scanned so far. After the first block is read, the txtResults will display the total # of hits for that search phrase, and whether Google indicated that # was exact, or just an estimate (approximate). If your page is found, it will display the ranking, the page title, and the URL. If more than one result is found in the same block of 10 results (such as when you have back-to-back listings like #1 & #2), then both results will be displayed. The scan will terminate when you page is found, or the specified scan depth is reached.
A more enhanced version of this program is in process, will keep you informed.
If your a Visual Basic programmer, I discovered SOAP/XML can be a lot of fun! You can use the tools I have discussed to "get your feet wet", and maybe invent the next great application for the Google API in the process! Good luck!
| 6:39 am on May 31, 2002 (gmt 0)|
You clever bugger! Now do it in PHP ;)
| 7:08 am on May 31, 2002 (gmt 0)|
ASPDesign, You are clever indead. Now I'm off to get your program works (and make a few changes, if you don't mind). I'm going to build the ultimate GG tool and then get permission from GG to distribute it.
NutsAndBolts, LOL ;).
| 7:35 am on May 31, 2002 (gmt 0)|
This really needs to be moved to the .net/asp forum. Will do some time tomorrow after all our contestants have seen this...
| 9:22 am on May 31, 2002 (gmt 0)|
Ok. Next question. I've hunted high and low. Where do I get a GG API key. I need one to operate this code.
| 9:25 am on May 31, 2002 (gmt 0)|
Ignore me. I've found it.
| 9:26 am on May 31, 2002 (gmt 0)|
I respectably disagree with your initial impression as to the proper forum, and request that you reconsider.
I gave some thought to the best forum to place this in. I did not add it to the forum in which your excellent program which inspired me into writing this was placed, as this is not a PERL or PHP program.
For the same reason, it would not be appropriate to place it in a .net/asp forum.
The whole design philosophy was to find a way to accomplish this in Windows WITHOUT having to resort to Microsoft-centric software or .NET!
I wanted to make something that would not require the latest flavor of Windows to work, and that could be experimented with by the average Joe who doesn't have $1,799 or more to spend on Visual Studio .Net!
I used NON-MICROSOFT communications software, and a decidedly NON-.NET design. Had I wanted to do this as a .NET program, it would have been considerably easier, as a .NET example program is included in the Google API ! The approach I took was a LOT more work!
This program was designed for VB5/6. The communications code is COMPLETELY DIFFERENT from how you would do it in .NET, indeed, I don't think that this program would even compile clean with VB.Net, due to the incompatibilities between the two languages!
If you had a VB5/6 forum, or a general Windows applications forum, that might be more understandable. But moving this to a .Net/asp forum makes about as much sense as moving a Unix post to a Windows forum!
Even if you had a VB5/6 forum, the second issue is the audience to which this topic best applies. This topic is of GREAT potential interest to people involved in Google SEO. It would be of LITTLE or NO interest to a programmer who is NOT involved with Google. The fact that a certain expertise beyond running WPG may be useful to gain the greatest possible benefit from this post does not detract from the target audience that this was written for.
| 9:32 am on May 31, 2002 (gmt 0)|
OK, chris_f - consider yourself ignored LOL ;)
For the rest of you, you can read all about Google's Beta-Test Web API Service here [google.com].
| 1:37 pm on May 31, 2002 (gmt 0)|
Thank you aspdesigner, I can see that you've put a lot of effort into this.
| 4:26 pm on May 31, 2002 (gmt 0)|
Where's the exit button? I always have an exit button! But thats because I cant do much else!!
Seriously though, aspdesigner, that is absolutely fantastic. From finding the thread to running a query and getting some results took only a few minutes, Im very thankful.
And thanks to Google too for allowing us to use such tools.
I guess Id better keep practising, I wanna write great programmes that everyone wants!
| 2:27 am on Jun 1, 2002 (gmt 0)|
Is visual basic free? I can't seem to find a free download at microsoft. Looking forward to building my first app...
| 2:59 am on Jun 1, 2002 (gmt 0)|
Woops - I think my post caused it to be moved.
| 3:33 am on Jun 1, 2002 (gmt 0)|
No visual basic is not free. Pro version retails for $479. You can find it for under $200 in the standard version, or go Ebay for about that on the pro version.
All microsoft related 'code' internet stuff can go in this forum. Here, php/perl, and the html/css forum, are the only places we'd like to see code.
| 4:53 am on Jun 1, 2002 (gmt 0)|
I agree that the Microsoft code forum is where the code should be (biased because I am a moderator in that forum).
But seriously, More VB coders will find your code here then in the Google forum. Your code will be buried in the Google forum with in a week. Never to be seen again.
| 5:26 am on Jun 1, 2002 (gmt 0)|
Would it be poor ettiquette to post a link in the Google forum for this thread? At least that way he might reach all of his originally intended audience. :-)
| 12:11 pm on Jun 1, 2002 (gmt 0)|
I've fixed the code. For some reason .net failed to pick up the reference properly. Instead of 'CoEnvelope' working I had to use 'PocketSOAP.CoEnvelope'. Also, I had to alter some code to make it .net complient.
| 12:31 pm on Jun 1, 2002 (gmt 0)|
Ok. The code produces an error. Can anyone help? See here [webmasterworld.com] for details.
| 4:12 pm on Jun 26, 2002 (gmt 0)|
dantheman if you're a student I believe there is a cheaper version of VB out there. $100
| 2:53 am on Sep 30, 2002 (gmt 0)|
|if you're a student I believe there is a cheaper version of VB out there. $100 |
I guess you are talking about the Learning Edition. The problem with it is that it does not allow you to complile the code into executables.
| 5:29 am on Sep 30, 2002 (gmt 0)|
Well done, aspdesigner!
| 9:06 am on Sep 30, 2002 (gmt 0)|
what if you made a working asp file, and post the url so we can all check this out without the microsoft part?
i don't even know if that's possibe, but i'll hear i guess, executable would be great also, before i start cracking my head over vb programming language.
| 10:39 am on Sep 30, 2002 (gmt 0)|
Why not use Brett's code and install ActiveState [activestate.com] on your windows machine?
Is the code portable to this Brett?
| 3:36 am on Oct 1, 2002 (gmt 0)|
With google punishing SEO'd sites nowdays , how safe will it be to use this program?
| 4:34 am on Oct 1, 2002 (gmt 0)|
The work aspdesigner's put into this amazes a pleb non-programmer like myself. Unfortunately programming like this is way out of my league and yet I'd really like to have the ability to safely check rankings at Google. Imagine others feel the same way.
Reading the thread got me thinking about discussion in another thread [webmasterworld.com ] where I asked Googleguy why automated position checkers like WPG haven't struck a deal with Google to use the API license keys on a fee basis. My thoughts were that if API keys were used by WPG, Google could charge WPG for the resources searches tie up and also discount WPG searches from their search calculations.
Googleguy's response was that assumes a world in which WPG is willing to pay for the thousands of queries a day that are issued. :)
Fair enough reply. Googleguy - this thread got me thinking if WPG won't play ball how about dealing with the WPG end-users? Why doesn't Google offer owners of automted ranking software like WPG the option of buying the equivalent of an API key. $X would allow 1,000 queries per day.
I'd certainly be happy to pay for this facility.
[edited by: biggles at 4:38 am (utc) on Oct. 1, 2002]
| 5:36 am on Oct 1, 2002 (gmt 0)|
biggles, if someone is willing to pay a little money to search us, we much more amenable to talking. :)
gopi, I believe it's much safer to use a program like this. It allows us to throttle back usage or delay results a little if the hardware is crunching. Compared to rank checking programs that search us without paying, I like the API route.
| 6:35 am on Oct 1, 2002 (gmt 0)|
Nice comments GG. I hope somebody copies WPG & the other players on this and you guys can have a productive chat.
| 6:40 am on Oct 1, 2002 (gmt 0)|
GG, I've never used WPG but I was able to demo a program called Optilink recently. It doesn't generate many queries, but instead looks at backlinks. Have you heard of it and is it classified similarly to WPG? I've stopped using it as I'm not sure how Google will treat it's usage. Any comments would help.
| This 35 message thread spans 2 pages: 35 (  2 ) > > |