检索结果-内蒙古大学图书馆

Florenz: Scaling Laws for Systematic Generalization in Vision-Language Models

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Spravil, Julian Houben, Sebastian Behnke, Sven Fraunhofer IAIS Germany University of Applied Sciences Bonn-Rhein-Sieg Bonn Germany University of Bonn Computer Science Institute VI Center for Robotics Germany Lamarr Institute for Machine Learning and Artificial Intelligence Germany

Cross-lingual transfer enables vision-language models (VLMs) to perform vision tasks in various languages with training data only in one language. Current approaches rely on large pre-trained multilingual language models. However, they face the curse of multilinguality, sacrificing downstream task performance for multilingual capabilities, struggling with lexical ambiguities, and falling behind recent advances. In this work, we study the scaling laws of systematic generalization with monolingual VLMs for multilingual tasks, focusing on the impact of model size and seen training samples. We propose Florenz, a monolingual encoder-decoder VLM with 0.4B to 11.2B parameters combining the pre-trained VLM Florence-2 and the large language model Gemma-2. Florenz is trained with varying compute budgets on a synthetic dataset that features intentionally incomplete language coverage for image captioning, thus, testing generalization from the fully covered translation task. We show that not only does indirectly learning unseen task-language pairs adhere to a scaling law, but also that with our data generation pipeline and the proposed Florenz model family, image captioning abilities can emerge in a specific language even when only data for the translation task is available. Fine-tuning on a mix of downstream datasets yields competitive performance and demonstrates promising scaling trends in multimodal machine translation (Multi30K, CoMMuTE), lexical disambiguation (CoMMuTE), and image captioning (Multi30K, XM3600, COCO Karpathy). Copyright © 2025, The Authors. All rights reserved.

关键词： Budget control

learning a Shape-Conditioned Agent for Purely Tactile In-Hand Manipulation of Various Objects

学校读者我要写书评

暂无评论

Learning a Shape-Conditioned Agent for Purely Tactile In-Han...

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

作者： Johannes Pitz Lennart Röstel Leon Sievers Darius Burschka Berthold Bäuml Learning AI for Dextrous Robots Lab (***) Technical University of Munich Germany DLR Institute of Robotics & Mechatronics (German Aerospace Center) Machine Vision and Perception Group Technical University of Munich

ISBN: (数字)9798350377705

ISBN: (纸本)9798350377712

Reorienting diverse objects with a multi-fingered hand is a challenging task. Current methods in robotic in-hand manipulation are either object-specific or require permanent supervision of the object state from visual sensors. This is far from human capabilities and from what is needed in real-world applications. In this work, we address this gap by training shape-conditioned agents to reorient diverse objects in hand, relying purely on tactile feedback (via torque and position measurements of the fingers’ joints). To achieve this, we propose a learning framework that exploits shape information in a reinforcement learning policy and a learned state estimator. We find that representing 3D shapes by vectors from a fixed set of basis points to the shape’s surface, transformed by its predicted 3D pose, is especially helpful for learning dexterous in-hand manipulation. In simulation and real-world experiments, we show the reorientation of many objects with high success rates, on par with state-of-the-art results obtained with specialized single-object agents. Moreover, we show generalization to novel objects, achieving success rates of ~90% even for non-convex ***: https://***/manipulation/iros24

关键词： Training Visualization Three-dimensional displays Torque Shape Tactile sensors Reinforcement learning Position measurement Vectors Sensors

Weisfeiler and leman go loopy: a new hierarchy for graph representational learning 24

学校读者我要写书评

暂无评论

Weisfeiler and leman go loopy: a new hierarchy for graph rep...

Proceedings of the 38th International Conference on Neural Information Processing Systems

作者： Raffaele Paolino Sohir Maskey Pascal Welke Gitta Kutyniok Department of Mathematics LMU Munich and Munich Center for Machine Learning (MCML) Department of Mathematics LMU Munich Faculty of Computer Science TU Wien Department of Mathematics LMU Munich and Munich Center for Machine Learning (MCML) and Institute for Robotics and Mechatronics DLR-German Aerospace Center and Department of Physics and Technology University of Tromsø

ISBN: (纸本)9798331314385

We introduce r-loopy Weisfeiler-Leman (r-ℓWL), a novel hierarchy of graph isomorphism tests and a corresponding GNN framework, r-ℓMPNN, that can count cycles up to length r + 2. Most notably, we show that r-ℓWL can count homomorphisms of cactus graphs. This extends 1-WL, which can only count homomorphisms of trees and, in fact, we prove that r-ℓWL is incomparable to k-WL for any fixed k. We empirically validate the expressive and counting power of r-ℓMPNN on several synthetic datasets and demonstrate the scalability and strong performance on various real-world datasets, particularly on sparse graphs. Our code is available on https://***/RPaolino/loopy.

关键词：

Towards Map-Agnostic Policies for Adaptive Informative Path Planning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Rückin, Julius Morilla-Cabello, David Stachniss, Cyrill Montijano, Eduardo Popović, Marija Center for Robotics University of Bonn Germany DIISI3A Universidad de Zaragoza Spain MAVLab TU Delft Netherlands Lamarr Institute for Machine Learning and Artificial Intelligence Germany

Robots are frequently tasked to gather relevant sensor data in unknown terrains. A key challenge for classical path planning algorithms used for autonomous information gathering is adaptively replanning paths online as the terrain is explored given limited onboard compute resources. Recently, learning-based approaches emerged that train planning policies offline and enable computationally efficient online replanning performing policy inference. These approaches are designed and trained for terrain monitoring missions assuming a single specific map representation, which limits their applicability to different terrains. To address this limitation, we propose a novel formulation of the adaptive informative path planning problem unified across different map representations, enabling training and deploying planning policies in a larger variety of monitoring missions. Experimental results validate that our novel formulation easily integrates with classical non-learning-based planning approaches while maintaining their performance. Our trained planning policy performs similarly to state-of-the-art map-specifically trained policies. We validate our learned policy on unseen real-world terrain datasets. © 2024, CC BY.

关键词： Reinforcement learning

HyenaPixel: Global Image Context with Convolutions

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Spravil, Julian Houben, Sebastian Behnke, Sven Fraunhofer IAIS Germany University of Applied Sciences Bonn-Rhein-Sieg Germany University of Bonn Computer Science Institute VI Center for Robotics Germany Lamarr Institute for Machine Learning and Artificial Intelligence Germany

In computer vision, a larger effective receptive field (ERF) is associated with better performance. While attention natively supports global context, its quadratic complexity limits its applicability to tasks that benefit from high-resolution input. In this work, we extend Hyena, a convolution-based attention replacement, from causal sequences to bidirectional data and two-dimensional image space. We scale Hyena’s convolution kernels beyond the feature map size, up to 191×191, to maximize ERF while maintaining sub-quadratic complexity in the number of pixels. We integrate our two-dimensional Hyena, HyenaPixel, and bidirectional Hyena into the MetaFormer framework. For image categorization, HyenaPixel and bidirectional Hyena achieve a competitive ImageNet-1k top-1 accuracy of 84.9% and 85.2%, respectively, with no additional training data, while outperforming other convolutional and large-kernel networks. Combining HyenaPixel with attention further improves accuracy. We attribute the success of bidirectional Hyena to learning the data-dependent geometric arrangement of pixels without a fixed neighborhood definition. Experimental results on downstream tasks suggest that HyenaPixel with large filters and a fixed neighborhood leads to better localization performance. Copyright © 2024, The Authors. All rights reserved.

关键词： Convolution

Safe Leaf Manipulation for Accurate Shape and Pose Estimation of Occluded Fruits

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yao, Shaoxiong Pan, Sicong Bennewitz, Maren Hauser, Kris University of Illinois at Urbana-Champaign IL United States Humanoid Robots Lab University of Bonn Germany Lamarr Institute for Machine Learning and Artificial Intelligence The Center for Robotics Bonn Germany

Fruit monitoring plays an important role in crop management, and rising global fruit consumption combined with labor shortages necessitates automated monitoring with robots. However, occlusions from plant foliage often hinder accurate shape and pose estimation. Therefore, we propose an active fruit shape and pose estimation method that physically manipulates occluding leaves to reveal hidden fruits. This paper introduces a framework that plans robot actions to maximize visibility and minimize leaf damage. We developed a novel scene-consistent shape completion technique to improve fruit estimation under heavy occlusion and utilize a perception-driven deformation graph model to predict leaf deformation during planning. Experiments on artificial and real sweet pepper plants demonstrate that our method enables robots to safely move leaves aside, exposing fruits for accurate shape and pose estimation, outperforming baseline methods. Project page: https://***/lmap-ssc/. Copyright © 2024, The Authors. All rights reserved.

关键词： Fruits

WEDGE: A multi-weather autonomous driving dataset built from generative vision-language models

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Marathe, Aboli Ramanan, Deva Walambe, Rahee Kotecha, Ketan Machine Learning Department Carnegie Mellon University PA United States Robotics Institute Carnegie Mellon University PA United States India India

The open road poses many challenges to autonomous perception, including poor visibility from extreme weather conditions. Models trained on good-weather datasets frequently fail at detection in these out-of-distribution settings. To aid adversarial robustness in perception, we introduce WEDGE (WEather images by DALL-E GEneration): a synthetic dataset generated with a vision-language generative model via prompting. WEDGE consists of 3360 images in 16 extreme weather conditions manually annotated with 16513 bounding boxes, supporting research in the tasks of weather classification and 2D object detection. We have analyzed WEDGE from research standpoints, verifying its effectiveness for extreme-weather autonomous perception. We establish baseline performance for classification and detection with 53.87% test accuracy and 45.41 mAP. Most importantly, WEDGE can be used to fine-tune state-of-the-art detectors, improving SOTA performance on real-world weather benchmarks (such as DAWN) by 4.48 AP for well-generated classes like trucks. WEDGE has been collected under OpenAI’s terms1 of use and is released for public use under the CC BY-NC-SA 4.0 license. The repository for this work and dataset is available at https://***/WEDGE. © 2023, CC BY-NC-SA.

关键词： Object detection

Censor-aware Semi-supervised learning for Survival Time Prediction from Medical Images

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Aragonés, Renato Hermoza Maicas, Gabriel Nascimento, Jacinto C. Carneiro, Gustavo Australian Institute for Machine Learning The University of Adelaide Australia Institute for Systems and Robotics Instituto Superior Tecnico Portugal

Survival time prediction from medical images is important for treatment planning, where accurate estimations can improve healthcare quality. One issue affecting the training of survival models is censored data. Most of the current survival prediction approaches are based on Cox models that can deal with censored data, but their application scope is limited because they output a hazard function instead of a survival time. On the other hand, methods that predict survival time usually ignore censored data, resulting in an under-utilization of the training set. In this work, we propose a new training method that predicts survival time using all censored and uncensored data. We propose to treat censored data as samples with a lower-bound time to death and estimate pseudo labels to semi-supervise a censor-aware survival time regressor. We evaluate our method on pathology and x-ray images from the TCGA-GM and NLST datasets. Our results establish the state-of-the-art survival prediction accuracy on both datasets. © 2022, CC BY.

关键词： Forecasting

Accurate Kinematic Modeling using Autoencoders on Differentiable Joints

学校读者我要写书评

暂无评论

Accurate Kinematic Modeling using Autoencoders on Differenti...

IEEE International Conference on robotics and Automation (ICRA)

作者： Nikolas Wilhelm Sami Haddadin Rainer Burgkart Patrick Van Der Smagt Maximilian Karl Munich Institute of Robotics and Machine Intelligence Technical University of Munich Munich Germany Department of Orthopedics and Sports Orthopedics Klinikum Rechts der Isar School of Medicine Munich Germany Volkswagen Group Machine Learning Research Lab Munich Germany

ISBN: (数字)9798350384574

ISBN: (纸本)9798350384581

In robotics and biomechanics, accurately determining joint parameters and computing the corresponding forward and inverse kinematics are critical yet often challenging tasks, especially when dealing with highly individualized and partly unknown systems. This paper unveils a cutting-edge kinematic optimizer, underpinned by an autoencoder-based architecture, to address these challenges. Utilizing a neural network, our approach simulates inverse kinematics, converting measurement data into joint-specific parameters during encoding, enabling a stable optimization process. These parameters are subsequently processed through a predefined, differentiable forward kinematics model, resulting in a decoded representation of the original data. Beyond offering a comprehensive solution to kinematics challenges, our method also unveils previously unidentified joint parameters. Real experimental data from knee and hand joints validate the optimizer’s efficacy. Additionally, our optimizer is multifunctional: it streamlines the modeling and automation of kinematics and enables a nuanced evaluation of diverse modeling techniques. By assessing the differences in reconstruction losses, we illuminate the merits of each approach. Collectively, this preliminary study signifies advancements in kinematic optimization, with potential applications spanning both biomechanics and robotics.

关键词： Biomechanics Automation Biological system modeling Neural networks Kinematics Computer architecture Encoding