segmentation and identification of a graphical userinterface consist of detecting the location, dimensions, and arrangement of elements of the userinterface, such as controls, labels, images, and icons, and recogniz...
详细信息
ISBN:
(纸本)9798400702068
segmentation and identification of a graphical userinterface consist of detecting the location, dimensions, and arrangement of elements of the userinterface, such as controls, labels, images, and icons, and recognizing them, respectively. While these problems have been already addressed for a graphical userinterface stored in a file and processed offline, it has received less attention for online processing when the interface evolves and is expressed in different formats, such as a whiteboard drawing or a paper sketch. To overcome these limitations, we present VisionAPI, an application programming interface trained for segmenting and identifying elements of a hand-sketched graphical userinterface both offline and online using computer vision. For this purpose, we rely on a software architecture based on Resnet101 to extract features and Faster R-CNN to build boundary boxes to obtain an 85% recognition rate for 21 classes of elements found in graphical userinterfaces: paragraph, dropdown list, checkbox, radio button, rating, toggle button, text area, date picker, stepper input, slider, video, label, table, list, header, button, image, linebreak, container, link, and text input.
暂无评论