检索结果-内蒙古大学图书馆

Hiren: Towards Higher Supervision Quality Forbetter Scene Text Image Super-Resolution

学校读者我要写书评

暂无评论

SSRN

SSRN 2024年

作者： Zhao, Minyi Xu, Yi Li, Bingjia Wang, Jie Guan, Jihong Zhou, Shuigeng Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University China ByteDance Inc China Department of Computer Science and Technology Tongji University China

Scene text image super-resolution (STISR) is an important pre-processing technique for text recognition from low-resolution scene images. Nowadays, various methods have been proposed to extract text-specific information from high-resolution (HR) images to supervise STISR model training. However, due to uncontrollable factors (e.g. shooting equipment, focus, and environment) in manually photographing HR images, the quality of HR images cannot be guaranteed, which unavoidably impacts STISR performance. Observing the quality issue of HR images, in this paper we propose a novel idea to boost STISR by first enhancing the quality of HR images and then using the enhanced HR images as supervision to do STISR. Concretely, we develop a new STISR framework, called High-Resolution ENhancement (HiREN) that consists of two branches and a quality estimation module. The first branch is developed to recover the low-resolution (LR) images, and the other is an HR quality enhancement branch aiming at generating high-quality (HQ) text images based on the HR images to provide more accurate supervision to the LR images. As the degradation from HQ to HR may be diverse, and there is no pixel-level supervision for HQ image generation, we design a kernel-guided enhancement network to handle various degradation, and exploit the feedback from a recognizer and text-level annotations as weak supervision signal to train the HR enhancement branch. Then, a quality estimation module is employed to evaluate the qualities of HQ images, which are used to suppress the erroneous supervision information by weighting the loss of each image. Extensive experiments on TextZoom show that HiREN can work well with most existing STISR methods and significantly boost their performances. © 2024, The Authors. All rights reserved.

关键词： Image enhancement

WEAKLY-SUPERVISED TEXT INSTANCE SEGMENTATION

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zu, Xinyan Yu, Haiyang Li, Bin Xue, Xiangyang Shanghai Key Laboratory of Intelligent Information Processing School of Computer Science Fudan University China

Text segmentation is a challenging vision task with many downstream applications. Current text segmentation methods require pixel-level annotations, which are expensive in the cost of human labor and limited in application scenarios. In this paper, we take the first attempt to perform weakly-supervised text instance segmentation by bridging text recognition and text segmentation. The insight is that text recognition methods provide precise attention position of each text instance, and the attention location can feed to both a text adaptive refinement head (TAR) and a text segmentation head. Specifically, the proposed TAR generates pseudo labels by performing two-stage iterative refinement operations on the attention location to fit the accurate boundaries of the corresponding text instance. Meanwhile, the text segmentation head takes the rough attention location to predict segmentation masks which are supervised by the aforementioned pseudo labels. In addition, we design a mask-augmented contrastive learning by treating our segmentation result as an augmented version of the input text image, thus improving the visual representation and further enhancing the performance of both recognition and segmentation. The experimental results demonstrate that the proposed method significantly outperforms weakly-supervised instance segmentation methods on ICDAR13-FST (18.95% improvement) and TextSeg (17.80% improvement) benchmarks. Copyright © 2023, The Authors. All rights reserved.

关键词： Location

Exploring One-Shot Semi-supervised Federated Learning with Pre-trained Diffusion Models

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Yang, Mingzhao Su, Shangchao Li, Bin Xue, Xiangyang Shanghai Key Laboratory of Intelligent Information Processing School of Computer Science Fudan University China

Recently, semi-supervised federated learning (semi-FL) has been proposed to handle the commonly seen real-world sce narios with labeled data on the server and unlabeled data on the clients. However, existing methods face several challenges such as communication costs, data heterogeneity, and training pressure on client devices. To address these challenges, we introduce the powerful diffusion models (DM) into semi-FL and propose FedDISC, a Federated Diffusion-Inspired Semi supervised Co-training method. Specifically, we first extract prototypes of the labeled server data and use these prototypes to predict pseudo-labels of the client data. For each category we compute the cluster centroids and domain-specific repre sentations to signify the semantic and stylistic information of their distributions. After adding noise, these representations are sent back to the server, which uses the pre-trained DM to generate synthetic datasets complying with the client distribu tions and train a global model on it. With the assistance of vast knowledge within DM, the synthetic datasets have comparable quality and diversity to the client images, subsequently en abling the training of global models that achieve performance equivalent to or even surpassing the ceiling of supervised cen tralized training. FedDISC works within one communication round, does not require any local training, and involves very minimal information uploading, greatly enhancing its prac ticality. Extensive experiments on three large-scale datasets demonstrate that FedDISC effectively addresses the semi-FL problem on non-IID clients and outperforms the compared SOTA methods. Sufficient visualization experiments also illus trate that the synthetic dataset generated by FedDISC exhibits comparable diversity and quality to the original client dataset with a neglectable possibility of leaking privacy-sensitive in formation of the clients. Copyright © 2023, The Authors. All rights reserved.

关键词： Semantics

Chinese Text Recognition with A Pre-Trained CLIP-Like Model Through Image-IDS Aligning

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Yu, Haiyang Wang, Xiaocong Li, Bin Xue, Xiangyang Shanghai Key Laboratory of Intelligent Information Processing School of Computer Science Fudan University China

Scene text recognition has been studied for decades due to its broad applications. However, despite Chinese characters possessing different characteristics from Latin characters, such as complex inner structures and large categories, few methods have been proposed for Chinese Text Recognition (CTR). Particularly, the characteristic of large categories poses challenges in dealing with zero-shot and few-shot Chinese characters. In this paper, inspired by the way humans recognize Chinese texts, we propose a two-stage framework for CTR. Firstly, we pre-train a CLIP-like model through aligning printed character images and Ideographic Description Sequences (IDS). This pre-training stage simulates humans recognizing Chinese characters and obtains the canonical representation of each character. Subsequently, the learned representations are employed to supervise the CTR model, such that traditional single-character recognition can be improved to text-line recognition through image-IDS matching. To evaluate the effectiveness of the proposed method, we conduct extensive experiments on both Chinese character recognition (CCR) and CTR. The experimental results demonstrate that the proposed method performs best in CCR and outperforms previous methods in most scenarios of the CTR benchmark. It is worth noting that the proposed method can recognize zero-shot Chinese characters in text images without fine-tuning, whereas previous methods require fine-tuning when new classes appear. The code is available at https://***/FudanVI/FudanOCR/tree/main/image-ids-CTR. Copyright © 2023, The Authors. All rights reserved.

关键词： Character recognition

Context-Aware Iteration Policy Network for Efficient Optical Flow Estimation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Cheng, Ri He, Ruian Jiang, Xuhao Zhou, Shili Tan, Weimin Yan, Bo School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Fudan University China

Existing recurrent optical flow estimation networks are computationally expensive since they use a fixed large number of iterations to update the flow field for each sample. An efficient network should skip iterations when the flow improvement is limited. In this paper, we develop a Context-Aware Iteration Policy Network for efficient optical flow estimation, which determines the optimal number of iterations per sample. The policy network achieves this by learning contextual information to realize whether flow improvement is bottlenecked or minimal. On the one hand, we use iteration embedding and historical hidden cell, which include previous iterations information, to convey how flow has changed from previous iterations. On the other hand, we use the incremental loss to make the policy network implicitly perceive the magnitude of optical flow improvement in the subsequent iteration. Furthermore, the computational complexity in our dynamic network is controllable, allowing us to satisfy various resource preferences with a single trained model. Our policy network can be easily integrated into state-of-the-art optical flow networks. Extensive experiments show that our method maintains performance while reducing FLOPs by about 40%/20% for the Sintel/KITTI datasets. Copyright © 2023, The Authors. All rights reserved.

关键词： Optical flows

Orientation-Independent Chinese Text Recognition in Scene Images

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Yu, Haiyang Wang, Xiaocong Li, Bin Xue, Xiangyang Shanghai Key Laboratory of Intelligent Information Processing School of Computer Science Fudan University China

Scene text recognition (STR) has attracted much attention due to its broad applications. The previous works pay more attention to dealing with the recognition of Latin text images with complex backgrounds by introducing language models or other auxiliary networks. Different from Latin texts, many vertical Chinese texts exist in natural scenes, which brings difficulties to current state-ofthe-art STR methods. In this paper, we take the first attempt to extract orientation-independent visual features by disentangling content and orientation information of text images, thus recognizing both horizontal and vertical texts robustly in natural scenes. Specifically, we introduce a Character Image Reconstruction Network (CIRN) to recover corresponding printed character images with disentangled content and orientation information. We conduct experiments on a scene dataset for benchmarking Chinese text recognition, and the results demonstrate that the proposed method can indeed improve performance through disentangling content and orientation information. To further validate the effectiveness of our method, we additionally collect a Vertical Chinese Text Recognition (VCTR) dataset. The experimental results show that the proposed method achieves 45.63% improvement on VCTR when introducing CIRN to the baseline model. Copyright © 2023, The Authors. All rights reserved.

关键词： Image reconstruction

SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhou, Shili He, Ruian Tan, Weimin Yan, Bo School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Fudan University China

Optical Flow Estimation aims to find the 2D dense motion field between two frames. Due to the limitation of model structures and training datasets, existing methods often rely too much on local clues and ignore the integrity of objects, resulting in fragmented motion estimation. Through theoretical analysis, we find the pre-trained large vision models are helpful in optical flow estimation, and we notice that the recently famous Segment Anything Model (SAM) demonstrates a strong ability to segment complete objects, which is suitable for solving the fragmentation problem. We thus propose a solution to embed the frozen SAM image encoder into FlowFormer to enhance object perception. To address the challenge of in-depth utilizing SAM in non-segmentation tasks like optical flow estimation, we propose an Optical Flow Task-Specific Adaption scheme, including a Context Fusion Module to fuse the SAM encoder with the optical flow context encoder, and a Context Adaption Module to adapt the SAM features for optical flow task with Learned Task-Specific Embedding. Our proposed SAMFlow model reaches 0.86/2.10 clean/final EPE and 3.55/12.32 EPE/F1-all on Sintel and KITTI-15 training set, surpassing Flowformer by 8.5%/9.9% and 13.2%/16.3%. Furthermore, our model achieves state-of-the-art performance on the Sintel and KITTI-15 benchmarks, ranking #1 among all two-frame methods on Sintel clean pass. Copyright © 2023, The Authors. All rights reserved.

关键词： Motion estimation

Low-latency Space-time Supersampling for Real-time Rendering

学校读者我要写书评

暂无评论

arXiv 2023年

作者： He, Ruian Zhou, Shili Sun, Yuqi Cheng, Ri Tan, Weimin Yan, Bo School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Fudan University China

With the rise of real-time rendering and the evolution of display devices, there is a growing demand for post-processing methods that offer high-resolution content in a high frame rate. Existing techniques often suffer from quality and latency issues due to the disjointed treatment of frame supersampling and extrapolation. In this paper, we recognize the shared context and mechanisms between frame supersampling and extrapolation, and present a novel framework, Space-time Supersampling (STSS). By integrating them into a unified framework, STSS can improve the overall quality with lower latency. To implement an efficient architecture, we treat the aliasing and warping holes unified as reshading regions and put forth two key components to compensate the regions, namely Random Reshading Masking (RRM) and Efficient Reshading Module (ERM). Extensive experiments demonstrate that our approach achieves superior visual fidelity compared to state-of-the-art (SOTA) methods. Notably, the performance is achieved within only 4ms, saving up to 75% of time against the conventional two-stage pipeline that necessitates 17ms. Copyright © 2023, The Authors. All rights reserved.

关键词： Extrapolation

UNCERTAINTY-AWARE U-NET FOR MEDICAL LANDMARK DETECTION

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Ye, Ziyang Yu, Haiyang Li, Bin Shanghai Key Laboratory of Intelligent Information Processing School of Computer Science Fudan University China

Heatmap-based methods play an important role in anatomical landmark detection. However, most current heatmap-based methods assume that the distributions of all landmarks are the same and the distribution of each landmark is isotropic, which may not be in line with reality. For example, the landmark on the jaw is more likely to be located along the edge and less likely to be located inside or outside the jaw. Manually annotating tends to follow similar rules, resulting in an anisotropic distribution for annotated landmarks, which represents the uncertainty in the annotation. To estimate the uncertainty, we propose a module named Pyramid Covariance Predictor to predict the covariance matrices of the target Gaussian distributions, which determine the distributions of landmarks and represent the uncertainty of landmark annotation. Specifically, the Pyramid Covariance Predictor utilizes the pyramid features extracted by the encoder of the backbone U-Net and predicts the Cholesky decomposition of the covariance matrix of the landmark location distribution. Experimental results show that the proposed Pyramid Covariance Predictor can accurately predict the distributions and improve the performance of anatomical landmark detection. Copyright © 2023, The Authors. All rights reserved.

关键词： Uncertainty analysis