Perforce has previewed AI-based testing in its Perfecto user interface (UI) testing platform. Testers use plain language prompts to define tests, and the company states that its “AI validation” will recognize the content and context of what is displayed automatically – though the documentation is currently more cautious.

According to director of product marketing Clinton Sprauve, Perfecto AI validation is a step towards autonomous testing, which Sprauve described as “visual and contextual validation without human intervention.”

UI testing is complex in part because of the number of different mobile devices and/or web browsers in use, and in part because of the challenges of scripting UI interaction. We note that Perfecto is not a complete test automation platform in itself, but integrates with popular automation frameworks including Selenium, Appium, Google’s Espresso, Quantum, and more.

The documentation for Perfecto explains that AI commands are currently available for scriptless mobile testing, and for Appium tests via Perfect extensions, though it adds that “more AI-based functionality is planned and will be coming soon.”

The term “validations” is significant. Prompts to the AI must be pass or fail validation questions and not be open ended. From the documentation, “is the account balance equal to $22.00?” is a validation question but, “what’s the account balance?” is not.

Further, there are limitations, and the docs state that the AI validation models make mistakes and have gaps in knowledge, limited to what is on the screen or in the public domain. 

Acknowledging that framing unambiguous and robust questions is difficult, Perforce has added an AI validation assistant that will help to check the question. Judging by the documentation, the AI validation may fail to get the context right when passing or failing a validation question, and the assistant helps to check this – in other words, using AI to get the input right, as well to generate the output.

Although there are risks therefore in using AI for validation, there are also benefits. In particular, UI test scripts can be brittle, breaking when a new design introduces a layout change or adds new elements to the screen. Further, a UI test script typically needs to be redone for different versions of an application, such as a mobile app versus a web application. AI validation is more human-like in its approach, looking at the meaning of what is on screen, so it could answer a question like “is there a pizza on the menu that costs $30” irrespective of the exact layout or platform.

In cases where Perfecto users are struggling to get a complex AI validation to work, they are encouraged to contact support as “we love a challenge.”

A common question with AI features is whether private data is at risk. The docs state that Perfecto AI is off by default, requiring active opt-in, and that customer data is not used for training. According to the Perforce generative AI policy, the AI uses OpenAI services and Microsoft Cognitive Services.