Build and Automate Bots Faster on a Virtual Machine with AISense

One of the key characteristics that makes Robotic Process Automation (RPA) a highly desirable automation technology is that it is application agnostic and can work across business systems just like humans do.

RPA can automate processes that involve both modern web-based applications and those on legacy technology. Humans interact with business applications visually through a screen and physically with their computer keyboard. RPA bots can mimic the human-machine interactions by directly accessing applications and breaking down the elements of the application’s user interface (UI) into objects — identifying where those objects are located on the screen and then associating rules and actions with those objects to perform a task. But what happens if the application is only available through a virtual machine (VM) without direct access to direct interaction, and what if the UI is constantly changing or an application is?

A practical yet powerful use of RPA and artificial intelligence (AI) can handle each of these challenges. Here is where Automation Anywhere AISense comes in. It’s a machine learning (ML)-based, secure screen recorder that automates seamlessly across all scenarios. AISense is designed to not only mimic how humans perceive and interact with UIs but also learn from every single interaction between a user and a UI to be able to adapt to complexities or changes that may occur.

Modern AI and ML techniques to automate in virtual environments

Enterprises often use virtual machines or environments to provide their with employees access to business applications. A common way to interact with a virtual UI is by employing Citrix Systems, a cloud computing service that enables mobile work styles. With the rise of the remote workforce, Citrix plays a vital role in many companies where applications can only be accessed remotely due to security reasons. In such a scenario, the Citrix server continuously sends screenshots or images of the live application to a client that is installed or accessed on the user’s device.

This makes object-based automation not feasible, and application-specific approaches not scalable because the object that is a constant stream of images is not static and creates a complex UI. AISense will help whenever object-based automation fails, is not reliable, or when a user is interacting with a complex UI.

Because AISense ML models are trained on thousands of images and millions of UI control examples, the recorder can efficiently identify all unique controls and objects in those images with precision and accuracy. This allows customers to automate processes and workflows, reducing the amount of time spent building and automating in virtual machines.

Keeping up with changing user interfaces

As application UIs become more advanced and tend to change faster than before to accommodate quicker release cycles, automation recorders must quickly and automatically evolve alongside. In cases where an application UI may constantly change and have objects on the screen rearranged, AISense can recognize items in your application regardless of its arrangement, allowing you to identify all objects in your respective application properly. This ensures that any existing bots do not require any modifications, even when business applications change.

How it works

AISense is built using modern ML models and architecture (deep neural networks) along with our own proprietary techniques that exploit this paradigm to model the unique nature of complex application interfaces. AISense ML models are highly accurate yet very efficient computationally and ready to run on utility CPUs available in personal computers — powerful and expensive GPUs are not required. AISense is built as a self-learning architecture and improves with every user interaction.

To continually improve our ML models, our users have the option to limit the feedback mechanism to give back metadata only, removing all sensitive information from the image protecting user privacy. This process allows us to internally recreate images that mimic the original application UI image without violating your privacy.

What separates AISense from all other recorders?

The answer is flexibility. With legacy recorders, you may be limited to automating a small number of controls or restricted to a specific browser or resolution. With AISense, you can simply record on any browser and at any resolution. Whether you are in a business process outsourcing (BPO) office or call center and frequently need to comply with systems in different resolutions, AISense makes the process seamless. In addition to supporting more resolutions and being browser agnostic, AISense introduces additional support for added controls such as image buttons, reading text from descriptions or labels, scrolling, and much more. This enables AISense to solve a typical automation problem where bots built on one user’s machine do not reliably work on the other machines due to changes in machine configuration around resolution, browser, or operating system.

Improving the experience

As application UIs become more complex with different variations of controls, users can avoid a frustrating bot execution experience by using AISense.

For example, if you are building a bot to create a lead in Salesforce Lightning UI, you can deploy that bot across different browsers such as Google Chrome, Internet Explorer, Firefox, or Microsoft Edge on an advanced UI in a Citrix environment without needing to make any script changes. Additionally, that same bot will be able to successfully run in Salesforce Classic UI where UI elements are rendered in a different layout, again without any script changes. AISense provides the resiliency our customers need to properly identify all objects in their respective applications.

Automating dynamic applications

Another AISense differentiator is its ability to automate dynamic applications where UI elements not only change their locations on the screens but also where new UI elements are introduced on the screen depending on the data being processed in the business process. This feature is now available.

Whether you are using a legacy application, an application exposed over Citrix, or accessing an application over remote desktop protocol (RDP), AISense empowers citizen developers to build more bots by reducing the technical complexity required to create automations.

AISense is the ideal solution for users who struggle with a recorder that lacks the ability to accurately identify objects, which frequently results in object-based automation failures. With AISense’s AI capabilities, users can benefit from its support for resilience to resolution, scale, and UI alterations, as well as its user-friendly and web-based interface with closed-loop feedback to reduce the amount of time spent on building and automating on virtual machines.