Multimodal Vision Research Laboratory

MVRL

back to funding overview.

Learning and Using Models of Geo-Temporal Appearance

Overview

Billions of geotagged and time-stamped images are publicly available via the Internet, providing a rich record of the appearance of people, places, and things across the globe. These images are a largely untapped resource that could be used to improve our understanding of the world and how it changes over time. This project develops automated methods of extracting useful information from this imagery and fusing it into high-resolution global models that capture geo-temporal trends. Once the trends have been captured, these models are used to improve performance on computer vision tasks and make geotagged imagery a usable and navigable resource for education and research in other disciplines. The project includes an education and outreach component that brings real-world problems to computer science (CS) students, mentors students across the educational spectrum, and makes the research accessible to the public.

This project develops computer vision technologies to capture spatial and temporal appearance trends and is organized into four main research thrusts: (1) investigating novel methods for extracting information from Internet imagery using weakly supervised learning, (2) developing techniques that integrate ground-level imagery with aerial and satellite data to model the expected image appearance anywhere in the world at any time, (3) evaluating methods for using such models to improve the performance of computer vision algorithms, and (4) automatically creating visual representations that make it possible for novice users to explore the learned geo-temporal trends via the Internet.

See the NSF Award Announcement for additional details.

Related Publication(s)

  1. PDF Padilha R, Salem T, Workman S, Andaló FA, Rocha A, Jacobs N. 2021. Content-Based Detection of Temporal Metadata Manipulation. arXiv preprint 2103.04736 [cs.CV].
    bibtex
  2. PDF Blanton H, Workman S, Jacobs N. 2020. A Structure-Aware Method for Direct Pose Estimation. arXiv preprint 2012.12360 [cs.CV].
    bibtex
  3. PDF Liang G, Su Y, Lin S-C, Zhang Y, Zhang Y, Jacobs N. 2020. Optical Wavelength Guided Self-Supervised FeatureLearning For Galaxy Cluster Richness Estimate. In: Workshop on Machine Learning and the Physical Sciences at the 34th Conference on Neural Information Processing Systems.
    bibtex
  4. PDF Xing X, Liang G, Blanton H, Rafique MU, Wang C, Lin A-L, Jacobs N. 2020. Dynamic Image for 3D MRI Image Alzheimer’s Disease Classification. In: ECCV Workshop on BioImage Computing (BIC).
    bibtex
  5. PDF Liang G, Zhang Y, Wang X, Jacobs N. 2020. Improved Trainable Calibration Method for Neural Networks. In: British Machine Vision Conference (BMVC).
    bibtex | website | tweet
  6. PDF Rafique MU, Blanton H, Snavely N, Jacobs N. 2020. Generative Appearance Flow: A Hybrid Approach for Outdoor View Synthesis. In: British Machine Vision Conference (BMVC).
    bibtex | website | code | tweet
  7. PDF Liang G, Zhang Y, Jacobs N. 2020. Neural Network Calibration for Medical Imaging Classification Using DCA Regularization. In: ICML 2020 workshop on Uncertainty and Robustness in Deep Learning (UDL).
    bibtex
  8. PDF Hammond TC, Xing X, Wang C, Ma D, Nho K, Crane PK, Elahi F, Ziegler DA, Liang G, Cheng Q, Yanckello LM, Jacobs N, Lin A-L. 2020. Beta-amyloid and tau drive early Alzheimer’s disease decline while glucose hypometabolism drives late decline. Communications Biology 3:352. DOI: 10.1038/s42003-020-1079-x.
    bibtex | doi | tweet
  9. Zhu J, Nolte A, Jacobs N, Ye M. 2020. Machine Learning in Identifying Karst Sinkholes from LiDAR-Derived Topographic Depressions in the Bluegrass Region of Kentucky. Journal of Hydrology. DOI: 10.1016/j.jhydrol.2020.125049.
    bibtex | doi
  10. PDF Blanton H, Grate S, Jacobs N. 2020. Surface Modeling for Airborne LiDAR. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex | tweet
  11. PDF Hadzic A, Christie G, Freeman J, Dismer A, Bullard S, Greiner A, Jacobs N, Mukherjee R. 2020. Estimating Displaced Populations from Overhead. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex | tweet
  12. PDF Workman S, Rafique MU, Blanton H, Greenwell C, Jacobs N. 2020. Single Image Cloud Detection via Multi-Image Fusion. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex | tweet
  13. PDF Liang G, Wang X, Zhang Y, Jacobs N. 2020. Weakly-Supervised Self-Training for Breast Cancer Localization. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). DOI: 10.1109/EMBC44109.2020.9176617.
    bibtex | doi
  14. PDF Blanton H, Greenwell C, Workman S, Jacobs N. 2020. Extending Absolute Pose Regression to Multiple Scenes. In: Joint Workshop on Long-Term Visual Localization, Visual Odometry and Geometric and Learning-based SLAM (CVPR Workshop).
    bibtex | tweet
  15. PDF Hadzic A, Blanton H, Song W, Chen M, Workman S, Jacobs N. 2020. RasterNet: Modeling Free-Flow Speed using LiDAR and Overhead Imagery. In: EARTHVISION: Large Scale Computer Vision for Remote Sensing Imagery. DOI: 10.1109/CVPRW50498.2020.00112.
    bibtex | website | doi | tweet
  16. PDF Salem T, Workman S, Jacobs N. 2020. Learning a Dynamic Map of Visual Appearance. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR42600.2020.01245.
    bibtex | website | doi | tweet
  17. PDF Workman S, Jacobs N. 2020. Dynamic Traffic Modeling from Overhead Imagery. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR42600.2020.01233.
    bibtex | website | doi | tweet
  18. Wang X, Liang G, Zhang Y, Blanton H, Bessinger Z, Jacobs N. 2020. Inconsistent Performance of Deep Learning Models on Mammogram Classification. Journal of the American College of Radiology. DOI: 10.1016/j.jacr.2020.01.006.
    bibtex | doi
  19. Maretto RV, Fonseca LMG, Jacobs NB, Körting TS, Bendini HN, Parente LL. 2021. Spatio-Temporal Deep Learning Approach to Map Deforestation in Amazon Rainforest. IEEE Geoscience and Remote Sensing Letters 18:771–775. DOI: 10.1109/LGRS.2020.2986407.
    bibtex | doi
  20. PDF Hamraz H, Jacobs NB, Contreras MA, Clark CH. 2019. Deep Learning for Conifer/Deciduous Classification of Airborne LiDAR 3D Point Clouds Representing Individual Trees. ISPRS Journal of Photogrammetry and Remote Sensing 158:219–230. DOI: 10.1016/j.isprsjprs.2019.10.011.
    bibtex | doi
  21. Zhu J, Nolte AM, Jacobs N, Ye M. 2019. Incorporating Machine Learning with LiDAR for Delineating Sinkholes. In: Kentucky Water Resources Annual Symposium.
    bibtex
  22. PDF Salem T, Greenwell C, Blanton H, Jacobs N. 2019. Learning to Map Nearly Anything. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2019.8900646.
    bibtex | doi
  23. PDF Song W, Salem T, Blanton H, Jacobs N. 2019. Remote Estimation of Free-Flow Speeds. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2019.8900286.
    bibtex | doi | tweet
  24. PDF Rafique MU, Jacobs N. 2019. Weakly Supervised Building Segmentation from Aerial Images. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2019.8898812.
    bibtex | doi
  25. PDF Rafique MU, Blanton H, Jacobs N. 2019. Weakly Supervised Fusion of Multiple Overhead Images. In: IEEE/ISPRS Workshop: Large Scale Computer Vision for Remote Sensing (EARTHVISION). DOI: 10.1109/CVPRW.2019.00189.
    bibtex | doi | tweet
  26. PDF Jacobs N, Kraft A, Rafique MU, Sharma RD. 2018. A Weakly Supervised Approach for Estimating Spatial Density Functions from High-Resolution Satellite Imagery. In: ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL). DOI: 10.1145/3274895.3274934.
    bibtex | code | doi | tweet
  27. PDF Schulter S, Zhai M, Jacobs N, Chandraker M. 2018. Learning to Look around Objects for Top-View Representations of Outdoor Scenes. In: European Conference on Computer Vision (ECCV). DOI: 10.1007/978-3-030-01267-0_48.
    bibtex | doi
  28. PDF Zhai M, Salem T, Greenwell C, Workman S, Pless R, Jacobs N. 2018. Learning Geo-Temporal Image Features. In: British Machine Vision Conference (BMVC).
    bibtex
  29. PDF Greenwell C, Workman S, Jacobs N. 2018. What Goes Where: Predicting Object Distributions from Above. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2018.8519251.
    bibtex | website | doi
  30. PDF Salem T, Zhai M, Workman S, Jacobs N. 2018. A Multimodal Approach to Mapping Soundscapes. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS). DOI: 10.1109/IGARSS.2018.8517977.
    bibtex | website | doi
  31. PDF Song W, Workman S, Hadzic A, Souleyrette R, Green E, Chen M, Zhang X, Jacobs N. 2018. FARSA: Fully Automated Roadway Safety Assessment. In: IEEE Winter Conference on Applications of Computer Vision (WACV). DOI: 10.1109/WACV.2018.00063.
    bibtex | doi | tweet
  32. PDF Vo N, Jacobs N, Hays J. 2017. Revisiting IM2GPS in the Deep Learning Era. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.286.
    bibtex | website | doi
  33. PDF Workman S, Zhai M, Crandall D, Jacobs N. 2017. A Unified Model for Near and Remote Sensing. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.293.
    bibtex | website | doi
  34. PDF Workman S, Souvenir R, Jacobs N. 2017. Understanding and Mapping Natural Beauty. In: IEEE International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV.2017.596.
    bibtex | website | doi
  35. PDF Zhai M, Bessinger Z, Workman S, Jacobs N. 2017. Predicting Ground-Level Scene Layout from Aerial Imagery. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). DOI: 10.1109/CVPR.2017.440.
    bibtex | code | doi | tweet

Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant No. IIS-1553116. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.