Developing a robot with computer vision for automating business processes of the industrial complex

Abstract

Abstract Robotic Process Automation technology is one of the solutions implemented in mechanical engineering and industrial production. A business process is automated by observing how a person is doing a task in a graphical user interface. Then the cyberphysical device repeats the same task. A significant problem of the implementation of such automated systems is that if the work is to be carried out through the interface of a virtual desktop, then it is impossible to obtain data on the elements of the graphical interface without computer vision. The object of the research is the automation of business processes of the industrial complex using computer vision. The purpose of the study is to apply computer vision methods to automate business processes and develop an appropriate software robot. The paper analyzes computer vision methods for recognizing user interface elements and text within these elements. Moreover, the authors have developed and tested a cross-platform application that reproduces a business process using computer vision. The test results have shown that selected computer vision methods will help automate industrial production business processes on any platform with a virtual desktop interface.