BYR Achieve · 镜像论坛

在上一期热点文献推荐中，我们为您推荐了低轨卫星通信技术的最新发展前沿，包括智能反射表面辅助低轨卫星通信，利用低轨卫星解决加拿大偏远地区宽带接入与数字鸿沟的方案，基于深度学习的低轨卫星大规模MIMO系统信道预测和混合波束赋形，星地量子大气信道的统计验证和深度学习预测。继大热的ChatGPT之后，当地时间3月14日，OpenAI推出了最新大型多模态模型GPT-4，被称为“深度学习的最新里程碑”。本期我们为您选取了4篇文献，介绍作为GPT核心技术之一的Transformer的最新发展前沿，包括半密集特征匹配Transformer及其在多视图几何中的应用，基于Transformer的任务感知弱监督目标定位，用于精确场景文本识别的图像-字符-单词Transformer，使用多模态Transformer生成情感视觉字幕，推送给相关领域的科研人员。文献一半密集特征匹配Transformer及其在多视图几何中的应用 Semi-Dense Feature Matching with Transformers and its Applications in Multiple-View Geometry Shen, Zehong, etc. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, NOV 2022, Early Access We present a novel method for local image feature matching. Instead of performing image feature detection, description, and matching sequentially, we propose to first establish pixel-wise dense matches at a coarse level and later refine the good matches at a fine level. In contrast to dense methods that use a cost volume to search correspondences, we use self and cross attention layers in Transformer to obtain feature descriptors that are conditioned on both images. The global receptive field provided by Transformer enables our method to produce dense matches in low-texture areas, where feature detectors usually struggle to produce repeatable interest points. The experiments on indoor and outdoor datasets show that LoFTR outperforms state-of-the-art methods by a large margin. We further adapt LoFTR to modern SfM systems and illustrate its application in multiple-view geometry. The proposed method demonstrates superior performance in Image Matching Challenge 2021 and ranks first on two public benchmarks of visual localization among the published methods. The code is available at https://zju3dv.github.io/loftr . 阅读原文：https://ieeexplore.ieee.org/document/9956767 文献二基于Transformer的任务感知弱监督目标定位 Task-Aware Weakly Supervised Object Localization With Transformer Meng, Meng, etc. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, DEC 2022, Early Access Weakly supervised object localization (WSOL) aims to predict both object locations and categories with only image-level class labels. However, most existing methods rely on class-specific image regions for localization, resulting in incomplete object localization. To alleviate this problem, we propose a novel end-to-end task-aware framework with a transformer encoder-decoder architecture (TAFormer) to learn class-agnostic foreground maps, including a representation encoder, a localization decoder, and a classification decoder. The proposed TAFormer enjoys several merits. First, the designed three modules can effectively perform class-agnostic localization and classification in a task-aware manner, achieving remarkable performance for both tasks. Second, an optimal transport algorithm is proposed to provide pixel-level pseudo labels to online refine foreground maps. To the best of our knowledge, this is the first work by exploring a task-aware framework with a transformer architecture and an optimal transport algorithm to achieve accurate object localization for WSOL. Extensive experiments with four backbones on two standard benchmarks demonstrate that our TAFormer achieves favorable performance against state-of-the-art methods. Furthermore, we show that the proposed TAFormer provides higher robustness against adversarial attacks and noisy labels. 阅读原文：https://ieeexplore.ieee.org/document/9996553 文献三用于精确场景文本识别的图像-字符-单词Transformer Image-to-Character-to-Word Transformers for Accurate Scene Text Recognition Xue, Chuhui, etc. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, FEB 2023, Early Access Leveraging the advances of natural language processing, most recent scene text recognizers adopt an encoder-decoder architecture where text images are first converted to representative features and then a sequence of characters via ‘sequential decoding’. However, scene text images suffer from rich noises of different sources such as complex background and geometric distortions which often confuse the decoder and lead to incorrect alignment of visual features at noisy decoding time steps. This paper presents I2C2W, a novel scene text recognition technique that is tolerant to geometric and photometric degradation by decomposing scene text recognition into two inter-connected tasks. The first task focuses on image-to-character (I2C) mapping which detects a set of character candidates from images based on different alignments of visual features in an non-sequential way. The second task tackles character-to-word (C2W) mapping which recognizes scene text by decoding words from the detected character candidates. The direct learning from character semantics (instead of noisy image features) corrects falsely detected character candidates effectively which improves the final text recognition accuracy greatly. Extensive experiments over nine public datasets show that the proposed I2C2W outperforms the state-of-the-art by large margins for challenging scene text datasets with various curvature and perspective distortions. It also achieves very competitive recognition performance over multiple normal scene text datasets. 阅读原文：https://ieeexplore.ieee.org/document/10041804 文献四使用多模态Transformer生成情感视觉字幕 Sentimental Visual Captioning using Multimodal Transformer Wu, Xinxiao, etc. INTERNATIONAL JOURNAL OF COMPUTER VISION, 2023, 131: 1073–1090 We propose a new task called sentimental visual captioning that generates captions with the inherent sentiment reflected by the input image or video. Compared with the stylized visual captioning task that requires a predefined style independent of the image or video, our new task automatically analyzes the inherent sentiment tendency from the visual content. With this in mind, we propose a multimodal Transformer model namely Senti-Transformer for sentimental visual captioning, which integrates both content and sentiment information from multiple modalities and incorporates prior sentimental knowledge to generate sentimental sentence. Specifically, we extract prior knowledge from sentimental corpus to obtain sentimental textual information and design a multi-head Transformer encoder to encode multimodal features. Then we decompose the attention layer in the middle of Transformer decoder to focus on important features of each modality, and the attended features are integrated through an intra- and inter-modality fusion mechanism for generating sentimental sentences. To effectively train the proposed model using the external sentimental corpus as well as the paired images or videos and factual sentences in existing captioning datasets, we propose a two-stage training strategy that first learns to incorporate sentimental elements into the sentences via a regularization term and then learns to generate fluent and relevant sentences with the inherent sentimental styles via reinforcement learning with a sentimental reward. Extensive experiments on both image and video datasets demonstrate the effectiveness and superiority of our Senti-Transformer on sentimental visual captioning. Source code is available at https://github.com/ezeli/InSentiCap_ext. 阅读原文：https://link.springer.com/article/10.1007/s11263-023-01752-7

热点文献带您关注AI Transformer的最新进展——图书馆前沿文献