Code and Data for our CVPR 2021 paper "Structured Scene Memory for Vision-Language Navigation"

SSM-VLN

Code and Data for our CVPR 2021 paper "Structured Scene Memory for Vision-Language Navigation".

Environment Installation

Download Room-to-Room navigation data:

bash ./tasks/R2R/data/download.sh

Download image features for environments:

mkdir img_features
wget https://www.dropbox.com/s/o57kxh2mn5rkx4o/ResNet-152-imagenet.zip -P img_features/
cd img_features
unzip ResNet-152-imagenet.zip

Python requirements: Need python3.6.

conda create -n ssm python=3.6
conda activate ssm
pip install -r python_requirements.txt

Install Matterport3D simulators:

git submodule update --init --recursive 
sudo apt-get install libjsoncpp-dev libepoxy-dev libglm-dev libosmesa6 libosmesa6-dev libglew-dev
mkdir build && cd build
cmake -DEGL_RENDERING=ON ..
make -j8

Code

Agent Training

cd ssm
python train.py

Agent Evaluation

cd ssm
python eval_agent.py

Citation

Please cite this paper in your publications if it helps your research:

@inproceedings{wang2021structured,
      title={Structured Scene Memory for Vision-Language Navigation}, 
      author={Hanqing Wang and Wenguan Wang and Wei Liang and Caiming Xiong and Jianbing Shen},
      booktitle=CVPR,
      year={2021}
}

TODO's

  1. Release the checkpoint.
  2. Update the installation requirements.
  3. Add evaluation scripts.

Contact Information

  • hanqingwang[at]bit[dot]edu[dot]cn, Hanqing Wang
  • wenguanwang[dot]ai[at]gmail[dot]com, Wenguan Wang
Owner
hanqing
Ph.D. student in Computer Vision and Artificial Intelligence
hanqing
https://github.com/HanqingWangAI/SSM-VLN
Comments
  • `train_rl` block in code seems inactive

    `train_rl` block in code seems inactive

    Hi all,

    Thank you again for releasing your codebase.

    Is the version of the code base in master the version that corresponds with the results in the paper?

    I noticed the paper mentions including RL as part of the learning objective, but the code seems to have RL disabled throughout training (train_rl is always set to False in the worker threads here). Also, there are objects referenced within the train_rl condition block that do not exist in SSM class, namely self.decoder, which I am guessing instead might be self.package.decoder. I was wondering if maybe the version in master is an earlier version than what was intended.

    Also, are there any specific curriculums that were followed or other tricks to train the model not mentioned in the paper? I am trying to train the model from scratch.

    Thanks! Angel

  • Request for trained checkpoints

    Request for trained checkpoints

    Hi Hanqing, Thanks for sharing the code of SSM which helps me a lot. However, I have noticed that the trained checkpoints are not yet released which is still on the TODO list. I understand it could be troublesome to organize the code and related files. So this issue is only a kind reminder. I'd be very thankful if you could release the checkpoints earlier. Thanks again!

  • Clarification of code

    Clarification of code

    Hi, thank you for releasing your code base. What is the point of lines 935-1007 in agent.py?

    It seems redundant, but I thought maybe it is meant to take a last action for any agents that still have not ended by the end of the episode length.

  • Request for trained checkpoints

    Request for trained checkpoints

    Hi Hanqing, Thanks for sharing the code of SSM which helps me a lot. However, I have noticed that the trained checkpoints are not yet released which is still on the TODO list. I understand it could be troublesome to organize the code and related files. So this issue is only a kind reminder. I'd be very thankful if you could release the checkpoints earlier. Thanks again!

  • The solution when cannot find numpy package.

    The solution when cannot find numpy package.

    This project doesnt provide FindNumPy.cmake. While building it, I meet the problem that I can not find numpy package.

    set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${PROJECT_SOURCE_DIR}/cmake")
find_package(NumPy REQUIRED)

    Here are my solution:

    1. Put FindNumPy.cmake into ${PROJECT_SOURCE_DIR}/cmake.
    2. Modify NUMPY_INCLUDES into PYTHON_NUMPY_INCLUDE_DIR.
Related tags
Artificial Intelligence SSM-VLN
The repository contains our dataset and C++ implementation of the CVPR 2022 paper, Geometric Structure Preserving Warp for Natural Image Stitching.
 The repository contains our dataset and C++ implementation of the CVPR 2022 paper, Geometric Structure Preserving Warp for Natural Image Stitching.

Geometric Structure Preserving Warp for Natural Image Stitching This repository contains our dataset and C++ implementation of the CVPR 2022 paper, Ge

Dec 22, 2022
Official PyTorch Code of GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection (CVPR 2021)
 Official PyTorch Code of GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection (CVPR 2021)

GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Mo

Jan 2, 2023
This is the code of our paper An Efficient Training Approach for Very Large Scale Face Recognition or F²C for simplicity.
 This is the code of our paper An Efficient Training Approach for Very Large Scale Face Recognition or F²C for simplicity.

Fast Face Classification (F²C) This is the code of our paper An Efficient Training Approach for Very Large Scale Face Recognition or F²C for simplicit

Jun 27, 2021
Python and C++ implementation of "MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation". Accepted at LXCV Workshop @ CVPR 2021.
 Python and C++ implementation of

MarkerPose: Robust Real-time Planar Target Tracking for Accurate Stereo Pose Estimation This is a PyTorch and LibTorch implementation of MarkerPose: a

Nov 18, 2022
[CVPR 2021] NormalFusion: Real-Time Acquisition of Surface Normals for High-Resolution RGB-D Scanning

NormalFusion: Real-Time Acquisition of Surface Normals for High-Resolution RGB-D Scanning Project Page | Paper | Supplemental material #1 | Supplement

Jan 2, 2023
DeepI2P - Image-to-Point Cloud Registration via Deep Classification. CVPR 2021
 DeepI2P - Image-to-Point Cloud Registration via Deep Classification. CVPR 2021

#DeepI2P: Image-to-Point Cloud Registration via Deep Classification Summary Video PyTorch implementation for our CVPR 2021 paper DeepI2P. DeepI2P solv

Jan 8, 2023
NLP-based perching trajectory generation presented in our paper "Perception-Aware Perching on Powerlines with Multirotors".
 NLP-based perching trajectory generation presented in our paper

Perception-Aware Perching on Powerlines with Multirotors This repo contains the code for the NLP-based perching trajectory generation presented in our

Dec 19, 2022
Source Code for SIGGRAPH Asia 2021 Paper "Ensemble Denoising for Monte Carlo Renderings"

Ensemble Denoising Source Code for SIGGRAPH Asia 2021 Paper Ensemble Denoising for Monte Carlo Renderings. Paper | Code | Talk Slides Dependencies A C

Nov 25, 2022
Source Code for SIGGRAPH Asia 2021 Paper "Ensemble Denoising for Monte Carlo Renderings"

Ensemble Denoising Source Code for SIGGRAPH Asia 2021 Paper Ensemble Denoising for Monte Carlo Renderings. Paper | Code | Talk Slides Dependencies A C

Nov 25, 2022
the code of paper "Optimizing Global Injectivity for Constrained Parameterization" (SIGGRAPH Asia 2021)
 the code of paper

Optimizing Global Injectivity for Constrained Parameterization Xingyi Du, Danny M. Kaufman, Qingnan Zhou, Shahar Kovalsky, Yajie Yan, Noam Aigerman, T

Nov 10, 2022
HybridPose: 6D Object Pose Estimation under Hybrid Representation (CVPR 2020)
 HybridPose: 6D Object Pose Estimation under Hybrid Representation (CVPR 2020)

HybridPose: 6D Object Pose Estimation under Hybrid Representations This repository contains authors' implementation of HybridPose: 6D Object Pose Esti

Nov 22, 2022
This code accompanies the paper "Human-Level Performance in No-Press Diplomacy via Equilibrium Search".
 This code accompanies the paper

Diplomacy SearchBot This code accompanies the paper "Human-Level Performance in No-Press Diplomacy via Equilibrium Search". A very brief orientation:

Dec 20, 2022
Code for Paper A Systematic Framework to Identify Violations of Scenario-dependent Driving Rules in Autonomous Vehicle Software

Code for Paper A Systematic Framework to Identify Violations of Scenario-dependent Driving Rules in Autonomous Vehicle Software

Nov 28, 2022
We implemented our own sequential version of GA, PSO, SA and ACA using C++ and the parallelized version with CUDA support

We implemented our own sequential version of GA, PSO, SA and ACA using C++ (some using Eigen3 as matrix operation backend) and the parallelized version with CUDA support. All of them are much faster than the popular lib scikit-opt.

May 7, 2022
This repository contains Python and C++ implementation of Robust Consistent Video Depth, as described in the paper
 This repository contains Python and C++ implementation of Robust Consistent Video Depth, as described in the paper

[CVPR 2021] Robust Consistent Video Depth Estimation

Jan 4, 2023
Codes for the paper "A Correlation Attack on Full SNOW-V and SNOW-Vi"

Codes for the paper "A Correlation Attack on Full SNOW-V and SNOW-Vi" #Contents Automatic search of linear approximation trails of SNOW-V Evaluating t

Oct 19, 2022
2021/3/30 ～ 2021/7/12 に行われる企画「競プロ典型 90 問」の問題・解説・ソースコードなどの資料をアップロードしています。

競プロ典型 90 問 日曜を除く毎朝 7:40 に競プロやアルゴリズムの教育的な問題を Twitter（@e869120）に投稿する企画です。 本企画は、2021 年 3 月 30 日から 7 月 12 日まで行われる予定です。 企画の目的 「競プロ典型 90 問」は、競プロ初級者から中上級者（レー

Dec 29, 2022
Implementation for the "Surface Reconstruction from 3D Line Segments" paper.
 Implementation for the

Surface Reconstruction from 3D Line Segments Surface reconstruction from 3d line segments. [Paper] [Supplementary Material] Langlois, P. A., Boulch, A

Dec 31, 2022
The official implementation of the research paper "DAG Amendment for Inverse Control of Parametric Shapes"
 The official implementation of the research paper

DAG Amendment for Inverse Control of Parametric Shapes This repository is the official Blender implementation of the paper "DAG Amendment for Inverse

Dec 26, 2022