Multimodal Vision Research Laboratory

back to funding overview.

Learning and Using Models of Geo-Temporal Appearance

Overview

Billions of geotagged and time-stamped images are publicly available via the Internet, providing a rich record of the appearance of people, places, and things across the globe. These images are a largely untapped resource that could be used to improve our understanding of the world and how it changes over time. This project develops automated methods of extracting useful information from this imagery and fusing it into high-resolution global models that capture geo-temporal trends. Once the trends have been captured, these models are used to improve performance on computer vision tasks and make geotagged imagery a usable and navigable resource for education and research in other disciplines. The project includes an education and outreach component that brings real-world problems to computer science (CS) students, mentors students across the educational spectrum, and makes the research accessible to the public.

This project develops computer vision technologies to capture spatial and temporal appearance trends and is organized into four main research thrusts: (1) investigating novel methods for extracting information from Internet imagery using weakly supervised learning, (2) developing techniques that integrate ground-level imagery with aerial and satellite data to model the expected image appearance anywhere in the world at any time, (3) evaluating methods for using such models to improve the performance of computer vision algorithms, and (4) automatically creating visual representations that make it possible for novice users to explore the learned geo-temporal trends via the Internet.

See the NSF Award Announcement for additional details.

Related Publication(s)

  1. Liang G, Zhang Y, Jacobs N. 2020. Neural Network Calibration for Medical Imaging Classification Using DCA Regularization. In: ICML 2020 workshop on Uncertainty and Robustness in Deep Learning (UDL).
    bibtex
  2. Hammond TC, Xing X, Wang C, Ma D, Nho K, Crane PK, Elahi F, Ziegler DA, Liang G, Cheng Q, Yanckello LM, Jacobs N, Lin A-L. 2020. Beta-amyloid and tau drive early Alzheimer’s disease decline while glucose hypometabolism drives late decline. Communications Biology 3:352. DOI: 10.1038/s42003-020-1079-x.
    bibtex
  3. Zhu J, Nolte A, Jacobs N, Ye M. 2020. Machine Learning in Identifying Karst Sinkholes from LiDAR-Derived Topographic Depressions in the Bluegrass Region of Kentucky. Journal of Hydrology.
    bibtex
  4. Blanton H, Grate S, Jacobs N. 2020. Surface Modeling for Airborne LiDAR. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex
  5. PDF Hadzic A, Christie G, Freeman J, Dismer A, Bullard S, Greiner A, Jacobs N, Mukherjee R. 2020. Estimating Displaced Populations from Overhead. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex
  6. Workman S, Rafique MU, Blanton H, Greenwell C, Jacobs N. 2020. Single Image Cloud Detection via Multi-Image Fusion. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex
  7. Liang G, Wang X, Zhang Y, Jacobs N. 2020. Weakly-Supervised Self-Training for Breast Cancer Localization. In: International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).
    bibtex
  8. PDF Blanton H, Greenwell C, Workman S, Jacobs N. 2020. Extending Absolute Pose Regression to Multiple Scenes. In: Joint Workshop on Long-Term Visual Localization, Visual Odometry and Geometric and Learning-based SLAM (CVPR Workshop).
    bibtex
  9. PDF Hadzic A, Blanton H, Song W, Chen M, Workman S, Jacobs N. 2020. RasterNet: Modeling Free-Flow Speed using LiDAR and Overhead Imagery. In: EARTHVISION: Large Scale Computer Vision for Remote Sensing Imagery.
    bibtex | website
  10. PDF Salem T, Workman S, Jacobs N. 2020. Learning a Dynamic Map of Visual Appearance. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
    bibtex | website
  11. PDF Workman S, Jacobs N. 2020. Dynamic Traffic Modeling from Overhead Imagery. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
    bibtex | website
  12. Wang X, Liang G, Zhang Y, Blanton H, Bessinger Z, Jacobs N. 2020. Inconsistent Performance of Deep Learning Models on Mammogram Classification. Journal of the American College of Radiology.
    bibtex
  13. Maretto RV, Fonseca LMG, Jacobs NB, Körting TS, Bendini HN, Parente LL. 2020. Spatio-temporal Deep Learning Approach to Map Deforestation in Amazon Rainforest. IEEE Geoscience and Remote Sensing Letters. DOI: 10.1109/LGRS.2020.2986407.
    bibtex
  14. PDF Hamraz H, Jacobs NB, Contreras MA, Clark CH. 2019. Deep Learning for Conifer/Deciduous Classification of Airborne LiDAR 3D Point Clouds Representing Individual Trees. ISPRS Journal of Photogrammetry and Remote Sensing 158:219–230. DOI: https://doi.org/10.1016/j.isprsjprs.2019.10.011.
    bibtex
  15. Zhu J, Nolte AM, Jacobs N, Ye M. 2019. Incorporating Machine Learning with LiDAR for Delineating Sinkholes. In: Kentucky Water Resources Annual Symposium.
    bibtex
  16. Salem T, Greenwell C, Blanton H, Jacobs N. 2019. Learning to Map Nearly Anything. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex
  17. Song W, Salem T, Blanton H, Jacobs N. 2019. Remote Estimation of Free-Flow Speeds. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex
  18. Rafique MU, Jacobs N. 2019. Weakly Supervised Building Segmentation. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex
  19. PDF Rafique MU, Blanton H, Jacobs N. 2019. Weakly Supervised Fusion of Multiple Overhead Images. In: IEEE/ISPRS Workshop: Large Scale Computer Vision for Remote Sensing (EARTHVISION).
    bibtex
  20. PDF Jacobs N, Kraft A, Rafique MU, Sharma RD. 2018. A Weakly Supervised Approach for Estimating Spatial Density Functions from High-Resolution Satellite Imagery. In: ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL).
    bibtex | code
  21. PDF Schulter S, Zhai M, Jacobs N, Chandraker M. 2018. Learning to Look around Objects for Top-View Representations of Outdoor Scenes. In: European Conference on Computer Vision (ECCV).
    bibtex
  22. PDF Zhai M, Salem T, Greenwell C, Workman S, Pless R, Jacobs N. 2018. Learning Geo-Temporal Image Features. In: British Machine Vision Conference (BMVC).
    bibtex
  23. Greenwell C, Workman S, Jacobs N. 2018. What Goes Where: Predicting Object Distributions from Above. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex | website
  24. PDF Salem T, Zhai M, Workman S, Jacobs N. 2018. A Multimodal Approach to Mapping Soundscapes. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS).
    bibtex | website
  25. PDF Song W, Workman S, Hadzic A, Souleyrette R, Green E, Chen M, Zhang X, Jacobs N. 2018. FARSA: Fully Automated Roadway Safety Assessment. In: IEEE Winter Conference on Applications of Computer Vision (WACV).
    bibtex
  26. PDF Vo N, Jacobs N, Hays J. 2017. Revisiting IM2GPS in the Deep Learning Era. In: IEEE International Conference on Computer Vision (ICCV).
    bibtex | website
  27. Workman S, Zhai M, Crandall D, Jacobs N. 2017. A Unified Model for Near/Remote Sensing. In: IEEE International Conference on Computer Vision (ICCV).
    bibtex | website
  28. PDF Workman S, Souvenir R, Jacobs N. 2017. Understanding and Mapping Natural Beauty. In: IEEE International Conference on Computer Vision (ICCV).
    bibtex | website
  29. PDF Zhai M, Bessinger Z, Workman S, Jacobs N. 2017. Predicting Ground-Level Scene Layout from Aerial Imagery. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
    bibtex | code

Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant No. IIS-1553116. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.