Multimodal Vision Research Laboratory

MVRL

back to funding overview.

Learning and Using Models of Geo-Temporal Appearance

Overview

Billions of geotagged and time-stamped images are publicly available via the Internet, providing a rich record of the appearance of people, places, and things across the globe. These images are a largely untapped resource that could be used to improve our understanding of the world and how it changes over time. This project develops automated methods of extracting useful information from this imagery and fusing it into high-resolution global models that capture geo-temporal trends. Once the trends have been captured, these models are used to improve performance on computer vision tasks and make geotagged imagery a usable and navigable resource for education and research in other disciplines. The project includes an education and outreach component that brings real-world problems to computer science (CS) students, mentors students across the educational spectrum, and makes the research accessible to the public.

This project develops computer vision technologies to capture spatial and temporal appearance trends and is organized into four main research thrusts: (1) investigating novel methods for extracting information from Internet imagery using weakly supervised learning, (2) developing techniques that integrate ground-level imagery with aerial and satellite data to model the expected image appearance anywhere in the world at any time, (3) evaluating methods for using such models to improve the performance of computer vision algorithms, and (4) automatically creating visual representations that make it possible for novice users to explore the learned geo-temporal trends via the Internet.

See the NSF Award Announcement for additional details.

Related Publication(s)

  1. Zhang Y, Liang G, Su Y, Jacobs N. 2021. Multi-Branch Attention Networks for Classifying Galaxy Clusters. In: International Conference on Pattern Recognition (ICPR).
    bibtex
  2. PDF Su Y, Zhang Y, Liang G, ZuHone J, Barnes D, Jacobs N, Ntampaka M, Forman W, Nulsen P, Kraft R, Jones C. 2020. A deep learning view of the census of galaxy clusters in IllustrisTNG. Monthly Notices of the Royal Astronomical Society (MNRAS). DOI: 10.1093/mnras/staa2690.
    bibtex | doi
  3. Xing X, Liang G, Blanton H, Rafique MU, Wang C, Lin A-L, Jacobs N. 2020. Dynamic Image for 3D MRI Image Alzheimer’s Disease Classification. In: ECCV Workshop on BioImage Computing (BIC).
    bibtex
  4. PDF Liang G, Zhang Y, Wang X, Jacobs N. 2020. Improved Trainable Calibration Method for Neural Networks. In: British Machine Vision Conference (BMVC).
    bibtex | website
  5. PDF Rafique MU, Blanton H, Snavely N, Jacobs N. 2020. Generative Appearance Flow: A Hybrid Approach for Outdoor View Synthesis. In: British Machine Vision Conference (BMVC).
    bibtex | website | code
  6. Liang G, Zhang Y, Jacobs N. 2020. Neural Network Calibration for Medical Imaging Classification Using DCA Regularization. In: ICML 2020 workshop on Uncertainty and Robustness in Deep Learning (UDL).
    bibtex
  7. Hammond TC, Xing X, Wang C, Ma D, Nho K, Crane PK, Elahi F, Ziegler DA, Liang G, Cheng Q, Yanckello LM, Jacobs N, Lin A-L. 2020. Beta-amyloid and tau drive early Alzheimer’s disease decline while glucose hypometabolism drives late decline. Communications Biology 3:352. DOI: 10.1038/s42003-020-1079-x.
    bibtex | doi
  8. Zhu J, Nolte A, Jacobs N, Ye M. 2020. Machine Learning in Identifying Karst Sinkholes from LiDAR-Derived Topographic Depressions in the Bluegrass Region of Kentucky. Journal of Hydrology. DOI: 10.1016/j.jhydrol.2020.125049.
    bibtex | doi
  9. Blanton H, Grate S, Jacobs N. 2020. Surface Modeling for Airborne LiDAR. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex
  10. PDF Hadzic A, Christie G, Freeman J, Dismer A, Bullard S, Greiner A, Jacobs N, Mukherjee R. 2020. Estimating Displaced Populations from Overhead. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex
  11. Workman S, Rafique MU, Blanton H, Greenwell C, Jacobs N. 2020. Single Image Cloud Detection via Multi-Image Fusion. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex
  12. Liang G, Wang X, Zhang Y, Jacobs N. 2020. Weakly-Supervised Self-Training for Breast Cancer Localization. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). DOI: 10.1109/EMBC44109.2020.9176617.
    bibtex | doi
  13. PDF Blanton H, Greenwell C, Workman S, Jacobs N. 2020. Extending Absolute Pose Regression to Multiple Scenes. In: Joint Workshop on Long-Term Visual Localization, Visual Odometry and Geometric and Learning-based SLAM (CVPR Workshop).
    bibtex
  14. PDF Hadzic A, Blanton H, Song W, Chen M, Workman S, Jacobs N. 2020. RasterNet: Modeling Free-Flow Speed using LiDAR and Overhead Imagery. In: EARTHVISION: Large Scale Computer Vision for Remote Sensing Imagery. DOI: 10.1109/CVPRW50498.2020.00112.
    bibtex | website | doi
  15. PDF Salem T, Workman S, Jacobs N. 2020. Learning a Dynamic Map of Visual Appearance. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR42600.2020.01245.
    bibtex | website | doi
  16. PDF Workman S, Jacobs N. 2020. Dynamic Traffic Modeling from Overhead Imagery. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR42600.2020.01233.
    bibtex | website | doi
  17. Wang X, Liang G, Zhang Y, Blanton H, Bessinger Z, Jacobs N. 2020. Inconsistent Performance of Deep Learning Models on Mammogram Classification. Journal of the American College of Radiology. DOI: 10.1016/j.jacr.2020.01.006.
    bibtex | doi
  18. Maretto RV, Fonseca LMG, Jacobs NB, Körting TS, Bendini HN, Parente LL. 2020. Spatio-temporal Deep Learning Approach to Map Deforestation in Amazon Rainforest. IEEE Geoscience and Remote Sensing Letters. DOI: 10.1109/LGRS.2020.2986407.
    bibtex | doi
  19. PDF Hamraz H, Jacobs NB, Contreras MA, Clark CH. 2019. Deep Learning for Conifer/Deciduous Classification of Airborne LiDAR 3D Point Clouds Representing Individual Trees. ISPRS Journal of Photogrammetry and Remote Sensing 158:219–230. DOI: 10.1016/j.isprsjprs.2019.10.011.
    bibtex | doi
  20. Zhu J, Nolte AM, Jacobs N, Ye M. 2019. Incorporating Machine Learning with LiDAR for Delineating Sinkholes. In: Kentucky Water Resources Annual Symposium.
    bibtex
  21. Salem T, Greenwell C, Blanton H, Jacobs N. 2019. Learning to Map Nearly Anything. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2019.8900646.
    bibtex | doi
  22. Song W, Salem T, Blanton H, Jacobs N. 2019. Remote Estimation of Free-Flow Speeds. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2019.8900286.
    bibtex | doi
  23. Rafique MU, Jacobs N. 2019. Weakly Supervised Building Segmentation from Aerial Images. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2019.8898812.
    bibtex | doi
  24. PDF Rafique MU, Blanton H, Jacobs N. 2019. Weakly Supervised Fusion of Multiple Overhead Images. In: IEEE/ISPRS Workshop: Large Scale Computer Vision for Remote Sensing (EARTHVISION). DOI: 10.1109/CVPRW.2019.00189.
    bibtex | doi
  25. PDF Jacobs N, Kraft A, Rafique MU, Sharma RD. 2018. A Weakly Supervised Approach for Estimating Spatial Density Functions from High-Resolution Satellite Imagery. In: ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL). DOI: 10.1145/3274895.3274934.
    bibtex | code | doi
  26. PDF Schulter S, Zhai M, Jacobs N, Chandraker M. 2018. Learning to Look around Objects for Top-View Representations of Outdoor Scenes. In: European Conference on Computer Vision (ECCV). DOI: 10.1007/978-3-030-01267-0_48.
    bibtex | doi
  27. PDF Zhai M, Salem T, Greenwell C, Workman S, Pless R, Jacobs N. 2018. Learning Geo-Temporal Image Features. In: British Machine Vision Conference (BMVC).
    bibtex
  28. Greenwell C, Workman S, Jacobs N. 2018. What Goes Where: Predicting Object Distributions from Above. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2018.8519251.
    bibtex | website | doi
  29. PDF Salem T, Zhai M, Workman S, Jacobs N. 2018. A Multimodal Approach to Mapping Soundscapes. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2018.8517977.
    bibtex | website | doi
  30. PDF Song W, Workman S, Hadzic A, Souleyrette R, Green E, Chen M, Zhang X, Jacobs N. 2018. FARSA: Fully Automated Roadway Safety Assessment. In: IEEE Winter Conference on Applications of Computer Vision (WACV). DOI: 10.1109/WACV.2018.00063.
    bibtex | doi
  31. PDF Vo N, Jacobs N, Hays J. 2017. Revisiting IM2GPS in the Deep Learning Era. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.286.
    bibtex | website | doi
  32. Workman S, Zhai M, Crandall D, Jacobs N. 2017. A Unified Model for Near and Remote Sensing. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.293.
    bibtex | website | doi
  33. PDF Workman S, Souvenir R, Jacobs N. 2017. Understanding and Mapping Natural Beauty. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.596.
    bibtex | website | doi
  34. PDF Zhai M, Bessinger Z, Workman S, Jacobs N. 2017. Predicting Ground-Level Scene Layout from Aerial Imagery. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR.2017.440.
    bibtex | code | doi

Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant No. IIS-1553116. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.