Alignment of Large-scale Pre-trained Vision Models for Downstream Tasks

Vision, vision-language, and multimodal models (hereafter collectively referred to as vision models) continue to demonstrate state-of-the-art performance across a wide range of tasks, mainly owing to increases in training data scale and model size. In particular, current high-performance vision models contain tens of billions of parameters and are trained on trillions of multimodal tokens. As a result, training such models has become prohibitively difficult for ordinary researchers and practitioners due to the massive computational requirements. In addition, although large-scale pre-trained vision models yield strong baseline performance on downstream tasks, they are often outperformed by models that are specialized for a specific downstream task. Therefore, one important research challenge is to develop methods that can efficiently align large-scale pre-trained vision models with downstream tasks under commonly available computational resources.While high-performance vision models continue to be built at a rapid pace, many methods for aligning such models with downstream tasks have also been developed. However, these vision models and alignment methods are often loosely grouped under general vision-related venues, or many alignment methods are presented at different domain-specific venues such as medical image analysis, remote sensing, robot vision, video surveillance, industrial inspection and so on. In other words, alignment methods as well as valuable practical insights remain scattered across the literature. This Research Topic therefore aims to bring together the latest methods for aligning large-scale pre-trained vision models with downstream tasks across diverse domains, and provide a broad overview that serves as a useful resource for many researchers and practitioners to efficiently and effectively build vision models for their own downstream tasks.This Research Topic invites submissions on both training-based approaches and training-free (test-time) approaches for aligning large-scale pre-trained vision models with downstream tasks. This scope includes, but is not limited to, the following representative approaches:- Fine-tuning, including Parameter-Efficient Fine-Tuning (PEFT);- Reinforcement learning-based alignment, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO);- Prompt optimization and prompt engineering;- In-context learning;- Guidance- or reward-based control;- Alignment based on Retrieval-Augmented Generation (RAG);- Multi-model collaborative alignment;- Evaluation methods for task-specific vision model alignment;- Alignment techniques for specific downstream tasks. Topic editors: Kimiaki Shirahama, Zeyd Boukhers, Hao Xu, Miho Ohsaki, Chen Shanxiong Source: https://www.frontiersin.org/research-topics/81343/alignment-of-large-scale-pre-trained-vision-models-for-downstream-tasks