Recent studies have adopted pre-trained language models, such as CodeT5 and CodeGPT, for automated program generation tasks like code generation, repair, and translation. Numerous language model based approaches have ...
详细信息
Recent studies have adopted pre-trained language models, such as CodeT5 and CodeGPT, for automated program generation tasks like code generation, repair, and translation. Numerous language model based approaches have been proposed and evaluated on various benchmark datasets, demonstrating promising performance. However, there is still uncertainty about the reliability of these models, particularly their realistic ability to consistently transform code sequences. This raises a question: are these techniques sufficiently trustworthy for automated program generation? Consequently, further research is needed to understand model logic and assess reliability and explainability. To bridge these research gaps, we conduct a thorough empirical study of eight popular language models on five representative datasets to determine the capabilities and limitations of automated program generation approaches. We further employ advanced explainable AI approaches to highlight the tokens that significantly contribute to the code transformation. We discover that state-of-the-art approaches suffer from inappropriate performance evaluation stemming from severe data duplication, causing overoptimistic results. Our explainability analysis reveals that, in various experimental scenarios, language models can recognize code grammar and structural information, but they exhibit limited robustness to changes in input sequences. Overall, more rigorous evaluation approaches and benchmarks are critical to enhance the reliability and explainability of automated program generation moving forward. Our findings provide important guidelines for this goal.
automated program generation (APG) is a concept of automatically making a computer program. Toward this goal, transferring automatedprogram repair (APR) to APG can be considered. APR modifies the buggy input source c...
详细信息
ISBN:
(纸本)9781665437844
automated program generation (APG) is a concept of automatically making a computer program. Toward this goal, transferring automatedprogram repair (APR) to APG can be considered. APR modifies the buggy input source code to pass all test cases. APG regards empty source code as initially failing all test cases, i.e., containing multiple bugs. Search-based APR repeatedly generates program variants and evaluates them. Many traditional APR systems evaluate the fitness of variants based on the number of passing test cases. However, when source code contains multiple bugs, this fitness function lacks the expressive power of variants. In this paper, we propose the application of a multi-objective genetic algorithm to APR in order to improve efficiency. We also propose a new crossover method that combines two variants with complementary test results, taking advantage of the high expressive power of multi-objective genetic algorithms for evaluation. We tested the effectiveness of the proposed method on competitive programming tasks. The obtained results showed significant differences in the number of successful trials and the required generation time.
This article investigates the basic design principles for a new Wireless Network Operating System (WNOS), a radically different approach to software-defined networking (SDN) for infrastructure-less wireless networks. ...
详细信息
ISBN:
(纸本)9781450357708
This article investigates the basic design principles for a new Wireless Network Operating System (WNOS), a radically different approach to software-defined networking (SDN) for infrastructure-less wireless networks. Departing from well-understood approaches inspired by OpenFlow, WNOS provides the network designer with an abstraction hiding (i) the lower-level details of the wireless protocol stack and (ii) the distributed nature of the network operations. Based on this abstract representation, the WNOS takes network control programs written on a centralized, high-level view of the network and automatically generates distributed cross-layer control programs based on distributed optimization theory that are executed by each individual node on an abstract representation of the radio hardware. We first discuss the main architectural principles of WNOS. Then, we discuss a new approach to automatically generate solution algorithms for each of the resulting subproblems in an automated fashion. Finally, we illustrate a prototype implementation of WNOS on software-defined radio devices and test its effectiveness by considering specific cross-layer control problems. Experimental results indicate that, based on the automatically generated distributed control programs, WNOS achieves 18%, 56% and 80.4% utility gain in networks with low, medium and high levels of interference;maybe more importantly, we illustrate how the global network behavior can be controlled by modifying a few lines of code on a centralized abstraction.
暂无评论