Setting up the Datasets

Download our provided datasets or get started with your own data.

Download Our Datasets

You can simply download our pre-provided datasets using any of the following commands:

# HoNY RGB + actions dataset, 814 MB
wget https://dl.dobb-e.com/datasets/homes_of_new_york.zip
unzip homes_of_new_york.zip

# HoNY RGB-D + actions dataset, 77 GB
pip install gdown
python -c "import gdown; gdown.download_folder("https://drive.google.com/drive/folders/1o8c6b6hSKfId8EzemVGf8c7DQoZ2IHAO?usp=sharing", quiet=True)
zip -FF HomesOfNewYorkWithDepth.zip --out HoNYDepth.zip
unzip -FF HoNYDepth.zip

# Sample finetuning dataset
wget https://dl.dobb-e.com/datasets/finetune_directory.zip
unzip finetune_directory.zip

Bring Your Own Data

Follow documentation in Processing Collected Data to extract data.

• Ensure the following data directory structure:

  dataset/
  |--- task1/
  |------ home1/
  |--------- env1/
  |--------- env1_val/
  |--------- env2/
  |--------- env2_val/
  |--------- env.../
  |------ home2/
  |--------- env1/
  |--------- env1_val/
  |--------- env.../
  |------ home.../
  |--- task2/
  |------ home1/
  |--------- env1/
  |--------- env1_val/
  |--------- env.../
  |------ home2/
  |--------- env1/
  |--------- env1_val/
  |--------- env.../
  |------ home.../
  |--- task.../
  |--- r3d_files.txt

Create the env_vars file

Create the file configs/env_vars/env_vars.yaml based on configs/env_vars/env_vars.yaml.sample.

home_ssl_data_root and home_ssl_data_original_root: These are optional, only relevant if you are planning to reproduce the HPR encoder.