OCR and Image Recognition?

[Peter]

Is there anyone who has experience from this, and is there anyone who has done any experiements using existing (or own) libraries in EB?

I was searching for a simple way to automate softwaretesting etc over VNC/Remote Desktop (and mainly VMWare) and eventually found a software called "Automation Anywhere". This software however required Office 2003 installed, which is a big no-no for the testing I need to do since the tests I do are to be done on "clean installations" of XP/Vista. The software is also quite (read; very) pricy, even though it seems there is alot of work put down in that software. The main reason I can't use it is currently because of the dependency regarding Office, but the cost is also of great significance. It could either be a recommendation for any developers here to create a software that actually generates income, or just to create a useful software to spread across the globe. If anyone has any experience from it, recommendations of simple libraries or anything else for that matter, please do tell me.

For anyone wondering why I simply not use API:s to find the controls on the screen there are two simple answers. First, I don't know how to even do it locally when it comes to tracking and obtaining information from other processes. Secondly, I can't use the controls since when using VNC (or Remote Desktop) I can't communicate with the processes and the software will only look at the remote screen as a picture, not an actual program/window.

I would of course do this myself if I had a single clue how, but my lack of knowledge is still holding me back when it comes to certain tasks.

Regards, Peter.

[Ficko]

Hi Peter!

Your "subject" is a little bit off I can't see what your Q has to do with "OCR & Img rec" ?!

But if I understand you correctly you are looking for a way to recording macros.

This http://freelabs.info/MacroRecorder.aspx may help you.

[Peter]

Thanks Ficko, but I think you misunderstood me.

The reason that I can't use "conventional macrorecorders" is because I'm going to automate a window containing the view of another computer. Therefor I need to use Optical Character Recognition as well as Image Recognition to locate where on the screen the mouseclicks and keypresses are to be sent. When using VNC/Remote Desktop/VMWare there is no way to know what controls are active etc using API's. Instead I will have to use images and OCR to figure out what is what. Thanks for the link though, but "normal" macrorecorders are quite easy to come by, and there are plenty of advanced free ones. I would seriously consider buying a software that had the functionality of "Automation Anywhere" if the price was in an acceptable range (divided by 10/100 ) and the OCR/ImageRecognition was builtin, needing no external files. Haven't found one yet though.

What I'm thinking is something along what sapero started on for Aurora, but I would never be able to port it. I need something that is pretty advanced as well since I know that some text, that I probably (not 100% sure yet) have to read, will be placed on a non-singlecolored background. There are plenty of libraries for OCR, but I cannot port them to EB, and it would of course have been easier if there was some kind of macroapplication already containing all these functions but that seems not to be my luck. 

Regards, Peter.

[psevet]

Hello Peter,

As initially I didn't understand what you were exactly searching for, I sent you a PM today.

Now that I have a less confused view of what you are looking for, I would suggest you to consider creating a server-side based app;  this should be a less difficult task to perform to achieve your goal. All depends on the information you need once you detect a mouse click on the remote server (or computer), and how you want your program to deal with it (send information by mail or to a network directory, perform some tasks on the computer itself, and so on).

My general philosophy in computing is : any obstacle has to be bypassed ....

Regards,

Pierre

[Ficko]

OK Peter,

I think I am starting to understand what you are trying to accomplish but I think you are over-complicating the task.

You can record the macro on the remote computer and start it remotely intstead to going into OCR-aizing the remote screen!

I don't know "Anymation Anywhere" but I dont think it does OCR either it wouldn't be a reliable way to determaining target execution since the name of the icons may change.


Regards,
Ficko

[Peter]

To both of you. Since the environment which needs to be controlled is a "secure environment", there will be no direct communication with the "host" (the one running the software which controls the target). This way I can't use communication in the conventional way (I was thinking the serialport emulation built into VMWare but I'm apparently not allowed to have any communication what so ever).

As a reply to what Ficko said, that it would be hard to know the name of the icons etc, that won't be a problem. I can start the software in my emulated machine/remote desktop/VNC manually, but it's the control of the software that needs to be done remotely this way. I can start the software and locate the window with Image Recognition by triggering on the titlebar/top left corner and by using that corner together with the bottom right one emulate my own offsets to know where to look for text etc.

Automation Anywhere does indeed support OCR (if you buy the version that costs thousands of dollars) but it depends on Microsoft Office being installed on all machines using that feature, which I am not allowed to install in this environment. Due to this, I will probably end up having to do all this in .NET or similar that has support for nearly anything (thank god that the .NET Framework is allowed). I just prefer doing stuff in EB whenever I can. Thanks for all your suggestions though. I appreciate them alot. Often are the times you look at a problem and try advanced solutions when it could easily be circumvented if you just think outside the box. The box in this case is a computer running in a virtual environment though, and that box is pretty hard to avoid. :/