Millions of people worldwide rely on alternative and augmentative communication devices to communicate. Visual scene displays (VSDs) can enhance communication for these individuals by embedding communication options w...
详细信息
ISBN:
(纸本)9798400706776
Millions of people worldwide rely on alternative and augmentative communication devices to communicate. Visual scene displays (VSDs) can enhance communication for these individuals by embedding communication options within contextualized images. However, existing VSDs often present default images that may lack relevance or require manual configuration, placing a significant burden on communication partners. In this study, we assess the feasibility of leveraging large multimodal models (LMM), such as GPT-4V, to automatically create communication options for VSDs. Communication options were sourced from a LMM and speech-language pathologists (SLPs) and AAC researchers (N=13) for evaluation through an expert assessment conducted by the SLPs and AAC researchers. We present the study's findings, supplemented by insights from semi-structured interviews (N=5) about SLP's and AAC researchers' opinions on the use of generative AI in augmentative and alternative communication devices. Our results indicate that the communication options generated by the LMM were contextually relevant and often resembled those created by humans. However, vital questions remain that must be addressed before LMMs can be confidently implemented in AAC devices.
This paper introduces a computing framework that combines Flow-Based programming (FBP) and Large Language Models (LLMs) to enable just-in-time programming (JITP). JITP empowers users, regardless of their programming e...
详细信息
ISBN:
(纸本)9798350308600
This paper introduces a computing framework that combines Flow-Based programming (FBP) and Large Language Models (LLMs) to enable just-in-time programming (JITP). JITP empowers users, regardless of their programming expertise, to actively participate in the development and automation process by leveraging their task-time algorithmic insights. By seamlessly integrating LLMs into the FBP workflow, the framework allows users to request and generate code in real-time, enabling dynamic code execution within a flow-based program. The paper explores the motivations, principles, and benefits of JITP, showcasing its potential in automating tasks, orchestrating data workflows, and accelerating software development. Through a fully implemented JITP framework using the Composable platform, we explore several examples and use cases to illustrate the benefits of the framework in data engineering, data science and software development. The results demonstrate how the fusion of FBP and LLMs creates a powerful and user-centric computing paradigm.
Grid displays are the most common form of augmentative and alternative communication device recommended by speech-language pathologists for children. Grid displays present a large variety of vocabulary which can be be...
详细信息
ISBN:
(纸本)9798400706776
Grid displays are the most common form of augmentative and alternative communication device recommended by speech-language pathologists for children. Grid displays present a large variety of vocabulary which can be beneficial for a users' language development. However, the extensive navigation and cognitive overhead required of users of grid displays can negatively impact users' ability to actively participate in social interactions, which is an important factor of their language development. We present a novel interaction technique for grid displays, Predictive Anchoring, based on user interaction theory and language development theory. Our design is informed by existing literature in AAC research, presented in the form of a set of design goals and a preliminary design sketch. Future work in user studies and interaction design are also discussed.
Purpose In contrast to the traditional grid-based display, visual scene displays (VSDs) offer a new paradigm for aided communication. For individuals who cannot select items from an AAC display by direct selection due...
详细信息
Purpose In contrast to the traditional grid-based display, visual scene displays (VSDs) offer a new paradigm for aided communication. For individuals who cannot select items from an AAC display by direct selection due to physical impairments, AAC access can be supported via methods such as item scanning. Item scanning sequentially highlights items on a display until the individual signals for selection. How items are highlighted or scanned for AAC access can impact performance outcomes. Further, the effectiveness of a VSD interface may be enhanced through consultation with experts in visual communication. Therefore, to support AAC access for those with physical impairments, the aim of this study was to evaluate the perspectives of experts in visual communication regarding effective methods for highlighting VSD elements. Methods Thirteen participants with expertise related to visual communication (e.g., photographers, artists) completed semi-structured interviews regarding techniques for item highlighting. Results Study findings identified four main themes to inform how AAC items may be highlighted or scanned, including (1) use of contrast related to light and dark, (2) use of contrast as it relates to colour, (3) outline highlighting, and (4) use of scale and motion. Conclusion By identifying how compositional techniques can be utilized to highlight VSD elements, study findings may inform current practice for scanning-based AAC access, along with other selection techniques where feedback or highlighting is used (e.g., eye-gaze, brain-computer interface). Further, avenues for just-in-time programming are discussed to support effective implementation for those with physical impairments.
This paper presents jacamo-web, an interactive programming IDE for developing Multi-Agent Systems (MAS). The standard programming method usually follows the sequence of compile, link and execute the application. Alter...
详细信息
ISBN:
(纸本)9783030514174;9783030514167
This paper presents jacamo-web, an interactive programming IDE for developing Multi-Agent Systems (MAS). The standard programming method usually follows the sequence of compile, link and execute the application. Alternatively, the so-called interactive programming provides a way to modify a system while it is running. Besides saving developing time, it maintains the system's availability since the application is kept running while it is being changed. To illustrate jacamo-web interactive functions, we have developed a MAS for the financial market. It checks data of companies and applies known formulae to suggest whether to buy assets or not.
暂无评论