检索结果-内蒙古大学图书馆

Grasp as You Say: Language-guided Dexterous Grasp Generation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wei, Yi-Lin Jiang, Jian-Jian Xing, Chengyi Tan, Xian-Tuo Wu, Xiao-Ming Li, Hao Cutkosky, Mark Zheng, Wei-Shi School of Computer Science and Engineering Sun Yat-sen University China Stanford University United States Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China

This paper explores a novel task "Dexterous Grasp as You Say" (DexGYS), enabling robots to perform dexterous grasping based on human commands expressed in natural language. However, the development of this field is hindered by the lack of datasets with natural human guidance;thus, we propose a language-guided dexterous grasp dataset, named DexGYSNet, offering high-quality dexterous grasp annotations along with flexible and fine-grained human language guidance. Our dataset construction is cost-efficient, with the carefully-design hand-object interaction retargeting strategy, and the LLM-assisted language guidance annotation system. Equipped with this dataset, we introduce the DexGYSGrasp framework for generating dexterous grasps based on human language instructions, with the capability of producing grasps that are intent-aligned, high quality and diversity. To achieve this capability, our framework decomposes the complex learning process into two manageable progressive objectives and introduce two components to realize them. The first component learns the grasp distribution focusing on intention alignment and generation diversity. And the second component refines the grasp quality while maintaining intention consistency. Extensive experiments are conducted on DexGYSNet and real world environments for validation. Copyright © 2024, The Authors. All rights reserved.

关键词： Natural language processing systems

Loc4Plan: Locating Before Planning for Outdoor Vision and Language Navigation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Tian, Huilin Meng, Jingke Zheng, Wei-Shi Li, Yuan-Ming Yan, Junkai Zhang, Yunong School of Computer Science and Engineering Sun Yat-sen University Guangzhou China School of Computer Science and Engineering Sun Yat-sen University Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education Guangdong Provincial Key Laboratory of Information Security Technology Guangzhou China

Vision and Language Navigation (VLN) is a challenging task that requires agents to understand instructions and navigate to the destination in a visual environment. One of the key challenges in outdoor VLN is keeping track of which part of the instruction was completed. To alleviate this problem, previous works mainly focus on grounding the natural language to the visual input, but neglecting the crucial role of the agent’s spatial position information in the grounding process. In this work, we first explore the substantial effect of spatial position locating on the grounding of outdoor VLN, drawing inspiration from human navigation. In real-world navigation scenarios, before planning a path to the destination, humans typically need to figure out their current location. This observation underscores the pivotal role of spatial localization in the navigation process. In this work, we introduce a novel framework, Locating before Planning (Loc4Plan), designed to incorporate spatial perception for action planning in outdoor VLN tasks. The main idea behind Loc4Plan is to perform the spatial localization before planning a decision action based on corresponding guidance, which comprises a block-aware spatial locating (BAL) module and a spatial-aware action planning (SAP) module. Specifically, to help the agent perceive its spatial location in the environment, we propose to learn a position predictor that measures how far the agent is from the next intersection for reflecting its position, which is achieved by the BAL module. After the locating process, we propose the SAP module to incorporate spatial information to ground the corresponding guidance and enhance the precision of action planning. Extensive Copyright © 2024, The Authors. All rights reserved.

关键词： Visual languages

Benchmarking Deep Models on Salient Object Detection

学校读者我要写书评

暂无评论

SSRN

SSRN 2023年

作者： Zhou, Huajun Lin, Yang Yang, Lingxiao Lai, Jianhuang Xie, Xiaohua School of Computer Science and Engineering Sun Yat-sen University Guangzhou China Guangdong Province Key Laboratory of Information Security Technology Guangzhou China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education Guangzhou China

Deep network-based methods have continuously refreshed state-of-the-art performance on the Salient Object Detection (SOD) task. However, the performance discrepancy caused by different implementation details may conceal the real progress. In this paper, we construct a SALient Object Detection (SALOD) benchmark to ensure a fair and comprehensive evaluation by unifying the implementation settings of SOD methods. We aim to reveal the reasons behind recent progress by conducting a comprehensive examination of the impact of the network structure, training strategy, and loss function on accuracy. Based on the experimental results, first, we find that U-shaped networks, both older and more recent, still achieve better performance than other recently proposed structures. Second, the ablation study on training strategy proves its significant impact on SOD performance. Third, we provide a comprehensive analysis on existing loss functions and propose a new Edge-Aware (EA) loss function in response. At last, an objectness shifting experiment stands on a new perspective to validate the generalizability of SOD methods. Code is available at https://***/moothes/SALOD. © 2023, The Authors. All rights reserved.

关键词： Deep neural networks

Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wu, Xiao-Ming Zheng, Dian Liu, Zuhao Zheng, Wei-Shi School of Computer Science and Engineering Sun Yat-sen University China Pengcheng Lab China Guangdong Province Key Laboratory of Information Security Technology China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China

Binarization of neural networks is a dominant paradigm in neural networks compression. The pioneering work BinaryConnect uses Straight Through Estimator (STE) to mimic the gradients of the sign function, but it also causes the crucial inconsistency problem. Most of the previous methods design different estimators instead of STE to mitigate it. However, they ignore the fact that when reducing the estimating error, the gradient stability will decrease concomitantly. These highly divergent gradients will harm the model training and increase the risk of gradient vanishing and gradient exploding. To fully take the gradient stability into consideration, we present a new perspective to the BNNs training, regarding it as the equilibrium between the estimating error and the gradient stability. In this view, we firstly design two indicators to quantitatively demonstrate the equilibrium phenomenon. In addition, in order to balance the estimating error and the gradient stability well, we revise the original straight through estimator and propose a power function based estimator, Rectified Straight Through Estimator (ReSTE for short). Comparing to other estimators, ReSTE is rational and capable of flexibly balancing the estimating error with the gradient stability. Extensive experiments on CIFAR-10 and ImageNet datasets show that ReSTE has excellent performance and surpasses the state-of-the-art methods without any auxiliary modules or losses. Copyright © 2023, The Authors. All rights reserved.

关键词： Stability

Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training

学校读者我要写书评

暂无评论

Estimator Meets Equilibrium Perspective: A Rectified Straigh...

International Conference on Computer Vision (ICCV)

作者： Xiao-Ming Wu Dian Zheng Zuhao Liu Wei-Shi Zheng School of Computer Science and Engineering Sun Yat-sen University China Pengcheng Lab China Guangdong Province Key Laboratory of Information Security Technology China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China

关键词：

MLNet: Mutual Learning Network with Neighborhood Invariance for Universal Domain Adaptation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Lu, Yanzuo Shen, Meng Ma, Andy J. Xie, Xiaohua Lai, Jian-Huang School of Computer Science and Engineering Sun Yat-sen University Guangzhou China Guangdong Province Key Laboratory of Information Security Technology China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China Guangzhou China

Universal domain adaptation (UniDA) is a practical but challenging problem, in which information about the relation between the source and the target domains is not given for knowledge transfer. Existing UniDA methods may suffer from the problems of overlooking intra-domain variations in the target domain and difficulty in separating between the similar known and unknown class. To address these issues, we propose a novel Mutual Learning Network (MLNet) with neighborhood invariance for UniDA. In our method, confidence-guided invariant feature learning with self-adaptive neighbor selection is designed to reduce the intra-domain variations for more generalizable feature representation. By using the cross-domain mixup scheme for better unknown-class identification, the proposed method compensates for the misidentified known-class errors by mutual learning between the closed-set and open-set classifiers. Extensive experiments on three publicly available benchmarks demonstrate that our method achieves the best results compared to the state-of-the-arts in most cases and significantly outperforms the baseline across all the four settings in UniDA. Code is available at https://***/YanzuoLu/MLNet. Copyright © 2023, The Authors. All rights reserved.

关键词： Knowledge management

NECA: Neural Customizable Human Avatar

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Xiao, Junjin Zhang, Qing Xu, Zhan Zheng, Wei-Shi School of Computer Science and Engineering Sun Yat-sen University China Adobe Research United States Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China

Human avatar has become a novel type of 3D asset with various applications. Ideally, a human avatar should be fully customizable to accommodate different settings and environments. In this work, we introduce NECA, an approach capable of learning versatile human representation from monocular or sparse-view videos, enabling granular customization across aspects such as pose, shadow, shape, lighting and texture. The core of our approach is to represent humans in complementary dual spaces and predict disentangled neural fields of geometry, albedo, shadow, as well as an external lighting, from which we are able to derive realistic rendering with high-frequency details via volumetric rendering. Extensive experiments demonstrate the advantage of our method over the state-of-the-art methods in photorealistic rendering, as well as various editing tasks such as novel pose synthesis and relighting. The code is available at https://***/iSEE-laboratory/NECA. Copyright © 2024, The Authors. All rights reserved.

关键词： Lighting

ASAG: Building Strong One-Decoder-Layer Sparse Detectors via Adaptive Sparse Anchor Generation

学校读者我要写书评

暂无评论

ASAG: Building Strong One-Decoder-Layer Sparse Detectors via...

International Conference on Computer Vision (ICCV)

作者： Shenghao Fu Junkai Yan Yipeng Gao Xiaohua Xie Wei-Shi Zheng School of Computer Science and Engineering Sun Yat-sen University China Guangdong Province Key Laboratory of Information Security Technology China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China Pengcheng Lab China

Recent sparse detectors with multiple, e.g. six, decoder layers achieve promising performance but much inference time due to complex heads. Previous works have explored using dense priors as initialization and built one-decoder-layer detectors. Although they gain remarkable acceleration, their performance still lags behind their six-decoder-layer counterparts by a large margin. In this work, we aim to bridge this performance gap while retaining fast speed. We find that the architecture discrepancy between dense and sparse detectors leads to feature conflict, hampering the performance of one-decoder-layer detectors. Thus we propose Adaptive Sparse Anchor Generator (ASAG) which predicts dynamic anchors on patches rather than grids in a sparse way so that it alleviates the feature conflict problem. For each image, ASAG dynamically selects which feature maps and which locations to predict, forming a fully adaptive way to generate image-specific anchors. Further, a simple and effective Query Weighting method eases the training instability from adaptiveness. Extensive experiments show that our method outperforms dense-initialized ones and achieves a better speed-accuracy trade-off. The code is available at https://***/iSEE-laboratory/ASAG.

关键词：

Event-Guided Procedure Planning from Instructional Videos with Text Supervision

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Wang, An-Lan Lin, Kun-Yu Du, Jia-Run Meng, Jingke Zheng, Wei-Shi School of Computer Science and Engineering Sun Yat-sen University China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China

In this work, we focus on the task of procedure planning from instructional videos with text supervision, where a model aims to predict an action sequence to transform the initial visual state into the goal visual state. A critical challenge of this task is the large semantic gap between observed visual states and unobserved intermediate actions, which is ignored by previous works. Specifically, this semantic gap refers to that the contents in the observed visual states are semantically different from the elements of some action text labels in a procedure. To bridge this semantic gap, we propose a novel event-guided paradigm, which first infers events from the observed states and then plans out actions based on both the states and predicted events. Our inspiration comes from that planning a procedure from an instructional video is to complete a specific event and a specific event usually involves specific actions. Based on the proposed paradigm, we contribute an Event-guided Prompting-based Procedure Planning (E3P) model, which encodes event information into the sequential modeling process to support procedure planning. To further consider the strong action associations within each event, our E3P adopts a mask-and-predict approach for relation mining, incorporating a probabilistic masking scheme for regularization. Extensive experiments on three datasets demonstrate the effectiveness of our proposed model. Copyright © 2023, The Authors. All rights reserved.

关键词： Semantics