Mixed-View Panorama Synthesis using Geospatially Guided Diffusion
Abstract
We introduce the task of mixed-view panorama synthesis, where the goal is to synthesize a novel panorama given a small set of input panoramas and a satellite image of the area. This contrasts with previous work which only uses input panoramas (same-view synthesis), or an input satellite image (cross-view synthesis). We argue that the mixed-view setting is the most natural to support panorama synthesis for arbitrary locations worldwide. A critical challenge is that the spatial coverage of panoramas is uneven, with few panoramas available in many regions of the world. We introduce an approach that utilizes diffusion-based modeling and an attention-based architecture for extracting information from all available input imagery. Experimental results demonstrate the effectiveness of our proposed method. In particular, our model can handle scenarios when the available panoramas are sparse or far from the location of the panorama we are attempting to synthesize.
Methods
Our local attention mechanism extracts features from nearby panoramas to provide contextual information for synthesis. The attention weights are learned based on spatial proximity and visual similarity.
Global attention extraction from satellite imagery provides overhead context and spatial structure. This enables our model to understand the overall layout and generate panoramas consistent with the surrounding environment.
Our fusion module combines features from both satellite and panorama inputs through a learned attention mechanism. This allows the model to leverage complementary information from both modalities for high-quality panorama synthesis.
Method
Our mixed-view panorama synthesis approach consists of three main components: (1) A satellite image encoder that extracts global spatial features, (2) A panorama encoder that processes nearby ground-level views, and (3) A diffusion-based synthesis module that generates novel panoramas by combining both sources of information.
Experimental Results
We evaluate our approach on diverse locations worldwide, demonstrating its ability to synthesize realistic panoramas even when input panoramas are sparse or distant from the target location. Our method significantly outperforms baseline approaches that use only satellite imagery or only ground-level panoramas.
Experimental Results
We evaluate our approach on diverse locations worldwide, demonstrating its ability to synthesize realistic panoramas even when input panoramas are sparse or distant from the target location. Our method significantly outperforms baseline approaches that use only satellite imagery or only ground-level panoramas.
BibTeX
@article{xiong2024mixed,
title={Mixed-View Panorama Synthesis using Geospatially Guided Diffusion},
author={Xiong, Zhexiao and Xing, Xin and Workman, Scott and Khanal, Subash and Jacobs, Nathan},
journal={arXiv preprint arXiv:2407.09672},
year={2024}
}