homepage Welcome to WebmasterWorld Guest from 54.204.73.126
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Microsoft / Microsoft IIS Web Server and ASP.NET
Forum Library, Charter, Moderators: ocean10000

Microsoft IIS Web Server and ASP.NET Forum

This 35 message thread spans 2 pages: 35 ( [1] 2 > >     
Visual Basic Google API Rank Checker Program
Deep scanning, Win Pgm with source code!
aspdesigner

10+ Year Member



 
Msg#: 97 posted 6:21 am on May 31, 2002 (gmt 0)

I was quite pleased to find Brett's excellent program [webmasterworld.com]. Unfortunately, it requires PERL, which I don't have on my Windows-based PC. :( I wanted to know if there was a way to do something like this in Windows from Visual Basic, without requiring Visual Studios .NET (expensive!)

Curiousity turned into "checking a few things out", which turned into "examining a few SOAP implementations", which turned into "running a few experiments", which turned into "writing a program or two", which turned into... "an all-nighter"! (The programmers among you should be failiar with this phenomena! ;) ) The end result was a collection of nifty Google Tools written in Visual Basic, one of which is detailed below.

The Visual Basic Google Rank Check program has the following features:

> Supports many versions of Windows (all the way back to Win95 OSR2!)
> Does not require Visual Studios .NET (even works with old VB5!)
> Does not require you to install a bunch of XML drivers and Toolkits
> Does not require a PERL or JAVA interpreter to run
> Not limited to only first 10 results, does high speed "auto-scan" to any desired search depth, up to the API maximum of 1,000 results!
> Auto-halts deep scans as soon as your page is found (avoids wasting precious API daily search limits!)
> Displays interim results and status during deep scans.
> Traps and reports SOAP-based Google API Server Errors.
> Easy to Use

For the VB programmers among you, following are full instructions and source code! To use this you need to be 1) familiar with Visual Basic programming, and 2) have signed-up for the Google API and gotten your own "key".

SOAPing-Up Your PC:

As the Google API uses SOAP/XML to communicate with their server, the first thing you need is some way of accessing SOAP from Visual Basic without needing .NET! I looked at several different packages, some were designed primarily for non-VB environments (like SOAPLite); and some were just too convoluted, required other software to run, or buggy (like Microsoft). I finally settled on PocketSOAP.

I choose it because a) it was originally written for the PocketPC, so it is compact and efficient, b) it is fully self-contained, no extra XML drivers required, c) is supports various versions of Windows, including older ones, d) it works well with Visual Basic, and e) it's free!

Installing PocketSOAP was easy. If you are running Windows ME, 2000 or XP, just download [pocketsoap.com] (small, < 1mb) and install!

(If you are running an old version of Windows (95/98/NT), you first need to upgrade to a new version of the Windows Installer program, which you can get for either Win 95/98 [pocketsoap.com] or for NT [pocketsoap.com], then you can download and install PocketSOAP)

That's it! Your PC now has SOAP/XML support!

NEXT STEP - "THE FORM"

 

aspdesigner

10+ Year Member



 
Msg#: 97 posted 6:22 am on May 31, 2002 (gmt 0)

THE FORM:

OK, so your PC is all SOAPed up. Now what?

Fire-up Visual Basic, and create a New (Standard EXE) project. Save and name the Project and Form.

First thing we need to do is tell Visual Basic that we want to use SOAP in this program, so go to Project/References, find the line "PocketSOAP 1.3.0 Type Library", and check the little box beside it, then click OK. (Programming Hint - if you ever get an compiler error "User defined type not defined" on the first line of the program, go back and make sure that this box is still checked!)

Now, make your new form large enough to hold everything (mine is 8000 x 8000). Put a nice title at the top, add some instructional text below if desired. The main thing is to make sure that you add the following required items to your form, otherwise it won't work! -

txtQuery - a single-line text box, long enough to hold your search phrase. Make sure that you name it txtQuery.

txtURL - a single-line text box, long enough to hold the URL that you are scanning for. Make sure that you name it txtURL.

txtMax - a single-line text box, to hold the desired search depth. A width of 1200 to 1500 is fine. Make sure that you name it txtMax.

lblCurrent - a label, which will be used to display the current progress during deep scans, and the actual scan depth when a search is completed. A width of about 1200 to 1500 is fine. Set the caption to empty. Make sure that you name it lblCurrent. You may also want to add another label to the left of it, with the text "Scanning: "

cmdDoSearch - a command button, which will be used to perform the search. You can set the caption to "Do Search!" Make sure that you name it cmdDoSearch.

txtResults - a multi-line text box, to hold the scan results. Make sure to set the MultiLine property to True, and the ScrollBars property to 3-Both. Adjust the size so that it is large enough to hold your results, I used HEIGHT 2500 and WIDTH 6900. Make sure you name it txtResults.

That's it! You may want to add some labels to identify what these boxes are for, and make it look pretty. As long as you have the required items listed above, then your form's ready to rock and roll!

Save your form and project now, so you don't loose anything.

NEXT STEP - THE PROGRAMMING

aspdesigner

10+ Year Member



 
Msg#: 97 posted 6:23 am on May 31, 2002 (gmt 0)

THE PROGRAMMING:

OK, you have a pretty form that does nothing. Now we need to add some code to make it fly!

First, go to the Declarations section at the top of the code view of your new form, and add the following declarations -

Option Explicit

Dim SOAPRequest As CoEnvelope
Dim Transport As HTTPTransport

' Google API SOAP Params
Dim GoogleSearchMethodName As String
Dim GoogleNameSpaceURI As String
Dim GoogleSOAPAction As String
Dim GoogleEndPoint As String
Dim MyKey As String

Dim DupFilter As Boolean
Dim FamilyFilter As Boolean

You'll notice that I'm using variables, rather than hard-coding some of the SOAP-related information. This makes it easier to understand what the code is doing, or to change it later if needed.

Next, add the following initialization code to Form_Load. Make sure that you replace the "put your key here" with the actual value of the API key code you got from Google, or else it won't work! -

Private Sub Form_Load()

' Init Vars
GoogleSearchMethodName = "doGoogleSearch"
GoogleNameSpaceURI = "urn:GoogleSearch"
GoogleSOAPAction = "urn:GoogleSearchAction"
GoogleEndPoint = "http://api.google.com/search/beta2"

MyKey = "put your key here"

DupFilter = True
FamilyFilter = False

' Init Form vars
txtQuery = ""
txtURL = ""
txtMax = 100
lblCurrent = 0
txtResults = ""

End Sub

Notice the variables DupFilter and FamilyFilter. These control how the search is done. DupFilter eliminates "very similar" duplicate listings in the results, and limits each domain to 2 listings max, this is the normal setting for Google searches unless you change it. FamilyFilter controls the "Family Safe" filtering of search results. You can change these settings if you wish.

(On the subject of settings, you will see below that the API restrict, lr, ie, and oe options are presently set to default, you only need these if you wish to limit searches to a specific country or language, or search with an non-standard character set (like Chinese), see the API doc's if you need to do this.)

Next, add the Private Function ValidateForm, and add the following code. This function does simple error-checking on the user's input, you can add to this if you wish -

Private Function ValidateForm() as Boolean

Dim OKFlag as Boolean

OKFlag = True

If txtMax = "" Then txtMax = 100

If txtMax > 1000 Then
MsgBox "Error - API Cannot scan past 1,000 listings!", _
vbExclamation, "Error - # of Listings too large!"
OKFlag = False
End If

If Trim(txtQuery) = "" Then
MsgBox "Error - No Search Query Specified!", _
vbExclamation, "No Search Query Specified!"
OKFlag = False
End If

If Trim(txtURL) = "" Then
MsgBox "Error - No URL to Search for!", _
vbExclamation, "No URL to Search for!"
OKFlag = False
End If

ValidateForm = OKFlag

End Function

Finally, add the following code to your command button's cmdDoSearch_click() event. This is the meat and potatoes of the program, and actually performs the scan and generates the results -

Private Sub cmdDoSearch_Click()

Dim CurrentOffset As Long
Dim RequestSize As Long
Dim Listings As Variant
Dim CurrentURL As String
Dim MyCounter As Long
Dim strResults As String
Dim MatchFound As Boolean

MatchFound = False

On Error goto SOAPErr

If ValidateForm() Then

strResults = ""

Set Transport = New HTTPTransport

CurrentOffset = 0
RequestSize = 10
If (txtMax < 10) And (txtMax > 0) Then
RequestSize = txtMax
End If

lblCurrent.Caption = "0"

APILoop:

Set SOAPRequest = New CoEnvelope
SOAPRequest.SetMethod GoogleSearchMethodName, _
GoogleNameSpaceURI

SOAPRequest.Parameters.Create "key", MyKey
SOAPRequest.Parameters.Create "q", Trim(txtQuery)
SOAPRequest.Parameters.Create "start", CurrentOffset
SOAPRequest.Parameters.Create "maxResults", RequestSize
SOAPRequest.Parameters.Create "filter", DupFilter
SOAPRequest.Parameters.Create "restrict", ""
SOAPRequest.Parameters.Create "safeSearch", FamilyFilter
SOAPRequest.Parameters.Create "lr", ""
SOAPRequest.Parameters.Create "ie", ""
SOAPRequest.Parameters.Create "oe", ""

Transport.SOAPAction = GoogleSOAPAction

Transport.Send GoogleEndPoint, SOAPRequest.Serialize

SOAPRequest.Parse Transport

Listings = SOAPRequest.Parameters.Item(0).Nodes.ItemByName("resultElements").Value

If (CurrentOffset = 0) Then ' First Time thru, get total "hits"
strResults = strResults & "Out of "
If (True = SOAPRequest.Parameters.Item(0).Nodes.ItemByName("estimateIsExact").Value) Then
strResults = strResults & "Exactly "
Else
strResults = strResults & "Approxiately "
End If
strResults = strResults & _
SOAPRequest.Parameters.Item(0).Nodes.ItemByName("estimatedTotalResultsCount").Value & _
" hits" & vbCrLf & vbCrLf
txtResults = strResults
txtResults.Refresh
End If

For MyCounter = 0 To UBound(Listings)

CurrentURL = Listings(MyCounter).Nodes.ItemByName("URL").Value
If (InStr(1, CurrentURL, txtURL, vbTextCompare) > 0) Then
' Match found
strResults = strResults & "#" & CStr(CurrentOffset + MyCounter + 1) & " "
strResults = strResults & Listings(MyCounter).Nodes.ItemByName("title").Value & vbCrLf
strResults = strResults & Listings(MyCounter).Nodes.ItemByName("URL").Value & vbCrLf & vbCrLf
txtResults = strResults
txtResults.Refresh
MatchFound = True
End If
Next

lblCurrent.Caption = CStr(CurrentOffset + UBound(Listings) + 1)
lblCurrent.Refresh

CurrentOffset = CurrentOffset + 10

If ((txtMax - CurrentOffset) < 10) Then
RequestSize = txtMax - CurrentOffset
End If

If (CurrentOffset < txtMax) And (UBound(Listings) >= 8) _
And Not (MatchFound) Then
GoTo APILoop
End If

strResults = strResults & "Done." & vbCrLf
txtResults = strResults

Set SOAPRequest = Nothing
Set Transport = Nothing

End If

Exit Sub

SOAPErr:

MsgBox "Err # " & Err.Number & vbCrLf & vbCrLf & _
Err.Description, vbExclamation, _
"Google API SOAP/XML Error!"
Exit Sub

End Sub

That's it! Look over it for any typos. Now be sure to save the form again so you don't loose anything!

NEXT - HOW TO USE


IGNORETHISJUSTABUNCHOFTEXTSOTHATTHECODELISTINGIN THISFORMABOVEDOESNTWRAP AROUNDWHEREITSHOULDNTBEWRAPPING

(edited by: aspdesigner at 6:40 am (utc) on May 31, 2002)

aspdesigner

10+ Year Member



 
Msg#: 97 posted 6:24 am on May 31, 2002 (gmt 0)

HOW TO USE:

To use this, make sure that you are connected to the Internet, then run the program. Enter the desired search phrase into the txtQuery box, the URL to search for in the txtURL box, change the search depth in the txtMax box if desired, and then click the "Do Search!" button.

The URL can be either an exact page (www.mysite.com/mypage.html), or just a domain name (http://www.mydomain.com), or even a partial domain name! (intel) You can either include the "http://" or not, both ways will work. The program will scan until it finds a listing who's URL contains what you typed.

The program will do an automatic looping high-speed scan to whatever search depth you specify, up to the maximum API limit of 1,000 listings, via the txtMax box. It is recommended that you leave this set to the default unless you need to do a deep scan. Very deep scans take longer, use resources, and do you really need to know if your page comes up #935 on some obscure search term?

More important, it is very easy to use up you daily maximum API search allotment using this program! At the maximum API search depth, doing only 10 maximum-depth searches could use up you quota for the entire day! So be considerate!

In order to increase efficiency and reduce search load, this program is also designed to automatically stop any search the instant that results are found. So if you are set to scan 100 results, and your page is found on #10, the program terminates the scanning at that point.

During a scan, lblCurrent will display the # of listings scanned so far. After the first block is read, the txtResults will display the total # of hits for that search phrase, and whether Google indicated that # was exact, or just an estimate (approximate). If your page is found, it will display the ranking, the page title, and the URL. If more than one result is found in the same block of 10 results (such as when you have back-to-back listings like #1 & #2), then both results will be displayed. The scan will terminate when you page is found, or the specified scan depth is reached.

A more enhanced version of this program is in process, will keep you informed.

If your a Visual Basic programmer, I discovered SOAP/XML can be a lot of fun! You can use the tools I have discussed to "get your feet wet", and maybe invent the next great application for the Google API in the process! Good luck!

nutsandbolts

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 97 posted 6:39 am on May 31, 2002 (gmt 0)

You clever bugger! Now do it in PHP ;)

chris_f

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 97 posted 7:08 am on May 31, 2002 (gmt 0)

ASPDesign, You are clever indead. Now I'm off to get your program works (and make a few changes, if you don't mind). I'm going to build the ultimate GG tool and then get permission from GG to distribute it.

NutsAndBolts, LOL ;).

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 97 posted 7:35 am on May 31, 2002 (gmt 0)

This really needs to be moved to the .net/asp forum. Will do some time tomorrow after all our contestants have seen this...

chris_f

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 97 posted 9:22 am on May 31, 2002 (gmt 0)

Ok. Next question. I've hunted high and low. Where do I get a GG API key. I need one to operate this code.

chris_f

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 97 posted 9:25 am on May 31, 2002 (gmt 0)

Ignore me. I've found it.

aspdesigner

10+ Year Member



 
Msg#: 97 posted 9:26 am on May 31, 2002 (gmt 0)

Brett

I respectably disagree with your initial impression as to the proper forum, and request that you reconsider.

I gave some thought to the best forum to place this in. I did not add it to the forum in which your excellent program which inspired me into writing this was placed, as this is not a PERL or PHP program.

For the same reason, it would not be appropriate to place it in a .net/asp forum.

The whole design philosophy was to find a way to accomplish this in Windows WITHOUT having to resort to Microsoft-centric software or .NET!

I wanted to make something that would not require the latest flavor of Windows to work, and that could be experimented with by the average Joe who doesn't have $1,799 or more to spend on Visual Studio .Net!

I used NON-MICROSOFT communications software, and a decidedly NON-.NET design. Had I wanted to do this as a .NET program, it would have been considerably easier, as a .NET example program is included in the Google API ! The approach I took was a LOT more work!

This program was designed for VB5/6. The communications code is COMPLETELY DIFFERENT from how you would do it in .NET, indeed, I don't think that this program would even compile clean with VB.Net, due to the incompatibilities between the two languages!

If you had a VB5/6 forum, or a general Windows applications forum, that might be more understandable. But moving this to a .Net/asp forum makes about as much sense as moving a Unix post to a Windows forum!

Even if you had a VB5/6 forum, the second issue is the audience to which this topic best applies. This topic is of GREAT potential interest to people involved in Google SEO. It would be of LITTLE or NO interest to a programmer who is NOT involved with Google. The fact that a certain expertise beyond running WPG may be useful to gain the greatest possible benefit from this post does not detract from the target audience that this was written for.

aspdesigner

10+ Year Member



 
Msg#: 97 posted 9:32 am on May 31, 2002 (gmt 0)

OK, chris_f - consider yourself ignored LOL ;)

For the rest of you, you can read all about Google's Beta-Test Web API Service here [google.com].

ciml

WebmasterWorld Senior Member ciml us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 97 posted 1:37 pm on May 31, 2002 (gmt 0)

Thank you aspdesigner, I can see that you've put a lot of effort into this.

Robber

10+ Year Member



 
Msg#: 97 posted 4:26 pm on May 31, 2002 (gmt 0)

Where's the exit button? I always have an exit button! But thats because I cant do much else!!

Seriously though, aspdesigner, that is absolutely fantastic. From finding the thread to running a query and getting some results took only a few minutes, Im very thankful.

And thanks to Google too for allowing us to use such tools.

I guess Id better keep practising, I wanna write great programmes that everyone wants!

dantheman

10+ Year Member



 
Msg#: 97 posted 2:27 am on Jun 1, 2002 (gmt 0)

Is visual basic free? I can't seem to find a free download at microsoft. Looking forward to building my first app...

dantheman

10+ Year Member



 
Msg#: 97 posted 2:59 am on Jun 1, 2002 (gmt 0)

Woops - I think my post caused it to be moved.

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 97 posted 3:33 am on Jun 1, 2002 (gmt 0)

No visual basic is not free. Pro version retails for $479. You can find it for under $200 in the standard version, or go Ebay for about that on the pro version.

All microsoft related 'code' internet stuff can go in this forum. Here, php/perl, and the html/css forum, are the only places we'd like to see code.

Lisa

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 97 posted 4:53 am on Jun 1, 2002 (gmt 0)

I agree that the Microsoft code forum is where the code should be (biased because I am a moderator in that forum).

But seriously, More VB coders will find your code here then in the Google forum. Your code will be buried in the Google forum with in a week. Never to be seen again.

Pushycat

10+ Year Member



 
Msg#: 97 posted 5:26 am on Jun 1, 2002 (gmt 0)

Would it be poor ettiquette to post a link in the Google forum for this thread? At least that way he might reach all of his originally intended audience. :-)

chris_f

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 97 posted 12:11 pm on Jun 1, 2002 (gmt 0)

ASPDesigner,

I've fixed the code. For some reason .net failed to pick up the reference properly. Instead of 'CoEnvelope' working I had to use 'PocketSOAP.CoEnvelope'. Also, I had to alter some code to make it .net complient.

Chris

chris_f

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 97 posted 12:31 pm on Jun 1, 2002 (gmt 0)

Ok. The code produces an error. Can anyone help? See here [webmasterworld.com] for details.

satanclaus



 
Msg#: 97 posted 4:12 pm on Jun 26, 2002 (gmt 0)

dantheman if you're a student I believe there is a cheaper version of VB out there. $100

saurabh

10+ Year Member



 
Msg#: 97 posted 2:53 am on Sep 30, 2002 (gmt 0)

if you're a student I believe there is a cheaper version of VB out there. $100

I guess you are talking about the Learning Edition. The problem with it is that it does not allow you to complile the code into executables.

GoogleGuy

WebmasterWorld Senior Member googleguy us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 97 posted 5:29 am on Sep 30, 2002 (gmt 0)

Well done, aspdesigner!

ikbenhet1

10+ Year Member



 
Msg#: 97 posted 9:06 am on Sep 30, 2002 (gmt 0)


what if you made a working asp file, and post the url so we can all check this out without the microsoft part?

i don't even know if that's possibe, but i'll hear i guess, executable would be great also, before i start cracking my head over vb programming language.

thanks

netcommr

10+ Year Member



 
Msg#: 97 posted 10:39 am on Sep 30, 2002 (gmt 0)


Why not use Brett's code and install ActiveState [activestate.com] on your windows machine?

Is the code portable to this Brett?

gopi

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 97 posted 3:36 am on Oct 1, 2002 (gmt 0)

With google punishing SEO'd sites nowdays , how safe will it be to use this program?

biggles

10+ Year Member



 
Msg#: 97 posted 4:34 am on Oct 1, 2002 (gmt 0)

The work aspdesigner's put into this amazes a pleb non-programmer like myself. Unfortunately programming like this is way out of my league and yet I'd really like to have the ability to safely check rankings at Google. Imagine others feel the same way.

Reading the thread got me thinking about discussion in another thread [webmasterworld.com ] where I asked Googleguy why automated position checkers like WPG haven't struck a deal with Google to use the API license keys on a fee basis. My thoughts were that if API keys were used by WPG, Google could charge WPG for the resources searches tie up and also discount WPG searches from their search calculations.

Googleguy's response was that assumes a world in which WPG is willing to pay for the thousands of queries a day that are issued. :)

Fair enough reply. Googleguy - this thread got me thinking if WPG won't play ball how about dealing with the WPG end-users? Why doesn't Google offer owners of automted ranking software like WPG the option of buying the equivalent of an API key. $X would allow 1,000 queries per day.

I'd certainly be happy to pay for this facility.

[edited by: biggles at 4:38 am (utc) on Oct. 1, 2002]

GoogleGuy

WebmasterWorld Senior Member googleguy us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 97 posted 5:36 am on Oct 1, 2002 (gmt 0)

biggles, if someone is willing to pay a little money to search us, we much more amenable to talking. :)

gopi, I believe it's much safer to use a program like this. It allows us to throttle back usage or delay results a little if the hardware is crunching. Compared to rank checking programs that search us without paying, I like the API route.

rmjvol

10+ Year Member



 
Msg#: 97 posted 6:35 am on Oct 1, 2002 (gmt 0)

Nice comments GG. I hope somebody copies WPG & the other players on this and you guys can have a productive chat.

dantheman

10+ Year Member



 
Msg#: 97 posted 6:40 am on Oct 1, 2002 (gmt 0)

GG, I've never used WPG but I was able to demo a program called Optilink recently. It doesn't generate many queries, but instead looks at backlinks. Have you heard of it and is it classified similarly to WPG? I've stopped using it as I'm not sure how Google will treat it's usage. Any comments would help.

This 35 message thread spans 2 pages: 35 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Microsoft IIS Web Server and ASP.NET
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved