FoVolNet: Fast Volume Rendering using Foveated Deep Neural Networks

David Bauer, Qi Wu, and Kwan-Liu Ma

IEEE Visualization Conference (VIS), 2022

Best Paper Honorable Mentions

Figure 1: We propose a novel rendering pipeline for fast volume rendering using optimized foveated sparse rendering and deep neural reconstruction networks. FoVolNet can faithfully reconstruct visual information from sparse inputs. With FoVolNet, developers are able to significantly improve rendering time without sacrificing visual quality.

Abstract

Volume data is found in many important scientific and engineering applications. Rendering this data for visualization at high quality and interactive rates for demanding applications such as virtual reality is still not easily achievable even using professional-grade hardware. We introduce FoVolNet—a method to significantly increase the performance of volume data visualization. We develop a cost-effective foveated rendering pipeline that sparsely samples a volume around a focal point and reconstructs the full-frame using a deep neural network. Foveated rendering is a technique that prioritizes rendering computations around the user’s focal point. This approach leverages properties of the human visual system, thereby saving computational resources when rendering data in the periphery of the user’s field of vision. Our reconstruction network combines direct and kernel prediction methods to produce fast, stable, and perceptually convincing output. With a slim design and the use of quantization, our method outperforms state-of-the-art neural reconstruction techniques in both end-to-end frame times and visual quality. We conduct extensive evaluations of the system’s rendering performance, inference speed, and perceptual properties, and we provide comparisons to competing neural image reconstruction techniques. Our test results show that FoVolNet consistently achieves significant time saving over conventional rendering while preserving perceptual quality.

BibTex

@article{bauer2022fovolnet,
    author = {D. Bauer and Q. Wu and K. Ma},
    journal = {IEEE Transactions on Visualization & Computer Graphics},
    title = {FoVolNet: Fast Volume Rendering using Foveated Deep Neural Networks},
    year = {2023},
    volume = {29},
    number = {01},
    issn = {1941-0506},
    pages = {515-525},
    doi = {10.1109/TVCG.2022.3209498},
    publisher = {IEEE Computer Society},
    address = {Los Alamitos, CA, USA},
    month = {jan}
}