homepage Welcome to WebmasterWorld Guest from 54.204.249.184
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

    
Screen scraping or faking packets?
stan




msg:659546
 10:45 am on Jan 17, 2005 (gmt 0)

Hi everyone

Iím about to commission a bit of software and Iím getting conflicting advice on the best method to use for one of the functions.
The software is designed to interact in place of or alongside a desktop client that normally interacts with its own server. Basically reading what is on the desktop client and automatically responding /interacting in place of a human.
We really want the use of this software to be hidden from the server it is receiving and sending data to.

So I have been told that screen scraping is more complicated but works the best and is ďun-spotableĒ by one developer.
While another recommends not using the desktop client at all, but instead mimicking all its functions by sniffing and copying the traffic that usually takes place with the server.

After posting on another forum i should say this application is not some sort of spy or scamware, nor a banner clicker (as they assumed it was). It's to carry out ligitemate transactions for me (saving time).

Can anyone who knows the advantages and drawbacks of both methods please give me their opinion. This would be much appreciated.

Thanks in advance

 

txbakers




msg:659547
 11:45 pm on Jan 17, 2005 (gmt 0)

Hi Stan and welcome to the webmaster world forums!

Your project sounds intriguing, but there isn't enough information there for me to determine what it is you are trying to do.

For instance, is it a screen-scraper type of application? There is a product called "J-Walk" from Seagull Software that mimics web functions, but operates as a client on a local machine, sending the information back to the server (AS/400).

The question is - what are you ultimately trying to do and could a regular web interface do it for you?

raywood




msg:659548
 1:48 pm on Jan 18, 2005 (gmt 0)

Stan, tell me if I get this right.
Present: server sends data to desktop, a human makes inputs on screen and sends back to server.

New: server sends data to desktop, an app on the desktop machine interprets the data, makes inputs and sends back to server without human intervention.

Since this is a webmaster forum, I assume we are talking about a web server and a browser client.

There are plenty of programming languages that support http request objects. You can pull in a web page, do anything you want to with it, and send data back to a server. Once you get the html file into your machine, you have free reign. Store it, manipulate it, log transactions, etc. I don't see how a server could know anything about what is happening in your desktop. It's a connectionless system. the server is just waiting for an answer. You can make it appear just like a human to the server.

stan




msg:659549
 3:56 pm on Jan 18, 2005 (gmt 0)

Thanks for replies

Raywood your description is pretty spot on.
My options are either using something to read what is actually on my screen and click the buttons for me, or do away with the desktop part of the desktop app - server arrangement and put in something that takes it's place and sends back the info. as if it had clicked the buttons. (I think Iíve made it sound even more complicated).

I believe the server and desktop app use HTTP protocols but it is encrypted.

The full program is for a type of financial trading I do online, it will allow me to leave my pc or go to the toilet without missing opportunities.

Cheers

txbakers




msg:659550
 4:30 pm on Jan 18, 2005 (gmt 0)

it will allow me to leave my pc or go to the toilet without missing opportunities.

That clears it up for me.

You basically need some type of dynamic web page that refreshes itself every few seconds. Each refresh will "fetch" the current data from the server, and based on that data, will process something.

It shouldn't be to complicated.

raywood




msg:659551
 3:04 pm on Jan 19, 2005 (gmt 0)

Hey, Stan. Sounds like some pretty fast and furious trading. Why don't you get a laptop and take it to the toilet with you? It would be cheaper and work better than a custom program.

mhhfive




msg:659552
 5:28 am on Feb 12, 2005 (gmt 0)

I'm looking for something similar -- but also different. I want to track a website for changes (for when financial data changes), and the changes occur about once a month, but not regularly. (So I can't just check on the 18th of every month.)

Is there any tool out there that can notify me when a website updates?

I've tried a couple things that work... but one of the sites I want to track requires different cookies to get information for different US states. So that's a little trickier, and I haven't found anything yet that can track website info using different cookies.

If anyone has a suggestion, I'd appreciate it. Thx in advance.

nalin




msg:659553
 10:52 pm on Feb 19, 2005 (gmt 0)

I've tried a couple things that work... but one of the sites I want to track requires different cookies to get information for different US states. So that's a little trickier, and I haven't found anything yet that can track website info using different cookies.

If anyone has a suggestion, I'd appreciate it. Thx in advance.

Wget will allow for retreiving webpages including ones that require cookies (though you have to do one request to set the cookie and a second to use it). Diff (or any number of other tools) could be used to track changes in different "versions" of files. Both are available as part of cygwin.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved