# Introduction Welcome to robot learning in homes with Dobb·E! {% embed url="" fullWidth="false" %} Introducing Dobb·E, n open-source, general framework for learning household robotic manipulation. {% endembed %} Welcome to the documentation of Dobb·E! We want you to to be able to get started with our robot learning framework as fast as possible. {% hint style="info" %} ### Need more information? Schedule a call with someone on our team with this link or right below. We will help you get set up as soon as we can. {% endhint %} {% embed url="" fullWidth="true" %} ## What is Dobb·E? Dobb·E is an open-source robotic imitation learning framework that can learn new household tasks in 5 minutes. Dobb·E is made up of four primary components: 1. A hardware tool, called **The Stick**, to comfortably collect robotic demonstrations in homes. 2. A dataset, called **Homes of New York (HoNY)**, with 1.5 million RGB-D frames. collected with the Stick across 22 homes and 216 environments of New York City. 3. A pretrained lightweight foundational vision model called **Home Pretrained Representations (HPR)**, trained on the HoNY dataset. 4. Finally, the platform to tie it all together to deploy it in novel homes, where with only five minutes of training data and 15 minutes of fine-tuning HPR, Dobb·E can solve many simple household tasks. ## What's in this documentation? This documentation is meant to help you get started with the system, including * [Setting up your own Stick](/hardware/putting-together-the-stick), * [Collecting your own demonstrations](/hardware/using-the-stick#how-to-collect-demos), * [Training a policy from those demonstrations](/software/readme-1#training-a-behavior-cloning-policy), and * [Running those policies](/software/readme-1#deploying-a-policy-on-the-robot) on your [own Hello Stretch](https://hello-robot.com/product). So why wait? Let's get some robots into homes. # Putting Together the Stick Every great journey begins with a Stick. ## Getting a Reacher Grabber You can buy a reacher grabber online, just be careful to buy the "suction cup" style reacher grabber instead of any other kind. We generally like the Vive brand ones, but this is a simple enough tool that not much can go wrong. Here are a couple of sample links: 1. [Vive Suction Cup Reacher Grabber](https://www.amazon.com/Vive-Suction-Cup-Reacher-Grabber/dp/B00O47ILVA/) 2. [Vive Foldable Suction Reacher Grabber](https://www.amazon.com/Vive-Foldable-Suction-Reacher-Grabber/dp/B0CF47WMNC/) ## Attaching heat shrink 1. First, remove the existing suction cup tips from the grabber 2. Next: ![Adding heat shrink onto the grabber](https://1393952267-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKFoURW0OhNP7hBckTxa8%2Fuploads%2Fgit-blob-04e793665daccf149438532de0ce5403576d0558%2FheatShrinkTutorial.png?alt=media) | Part | Quantity | | ------------------------------------------------------------------------------ | -------- | | [White Heat Shrink Tubing 12mm](https://www.amazon.com/gp/product/B07FK4GS85/) | 1 | ## Silicone tips ![Required components](https://1393952267-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKFoURW0OhNP7hBckTxa8%2Fuploads%2Fgit-blob-3d50c73ca3965ae281ffa19f8792f7db077c2a6a%2FsiliconeTipParts.jpg?alt=media) | Part | Quantity | | ----------------------------------------------------------------------------------------------------- | -------- | | [Oomoo 25 Silicon Kit](https://www.amazon.com/Smooth-OOMOO-25-Curing-Silicone/dp/B01C4YQ4TU/) | 1 | | [Silicone Mold](https://github.com/notmahi/dobb-e/tree/main/hardware/grabberEnds/grabberEnd_longMold) | 1 | **Instructions** 1. Attach the 2 pieces of the mold together 2. Stire each compound within its container 3. Mix together a small amount of each part in equal volume 4. Pour the mixed silicone compound into each hole in the mold (8 of them) 5. Wait until the silicone cures, then remove the tips ## Making cylindrical tips ![Required components](https://1393952267-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKFoURW0OhNP7hBckTxa8%2Fuploads%2Fgit-blob-0a18927cf19d3098f93fe8f74e6c71f5efd6bf2f%2FgrabberEndPt1.JPG?alt=media) | Part | Quantity | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | Silicone Tip | 2 | | [Cylindrical Piece](https://github.com/notmahi/dobb-e/tree/main/hardware/grabberEnds/grabberEnd) | 2 | | [Gorilla Glue](https://www.amazon.com/Gorilla-105795-Super-1-Pack-Clear/dp/B08QQZ71CV/) | 1 | | [Gel Gorilla Glue](https://www.amazon.com/Gorilla-105801-Super-1-Pack-Clear/dp/B08QR2XRKD/ref=sr_1_5?keywords=gorilla%2Bglue%2Bgel\&qid=1701128601\&sr=8-5\&th=1) | 1 | | [M3x8 Socket Head Screw](https://www.mcmaster.com/91290A113/) | 2 | #### Video tutorial: {% embed url="" %} {% hint style="info" %} This is the messiest part of the build; if you are struggling, [reach out!](/contact-us) We may have a few extra that we would be happy to mail out to you if you're located at a reasonable distance from us :) {% endhint %} ## Adding the cylindrical tips onto the grabber ![Required components](https://1393952267-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKFoURW0OhNP7hBckTxa8%2Fuploads%2Fgit-blob-0892b8d78b28e0915d6403fe74fe807fc666b36e%2FgrabberEndPt2.JPG?alt=media) | Part | Quantity | | ------------------------------------------------------ | -------- | | Cylindrical Tips | 2 | | [M3-0.5 Hex Nut](https://www.mcmaster.com/90592A085/) | 2 | | [M3 3.2mm Washer](https://www.mcmaster.com/93475A210/) | 2 | #### Video tutorial {% embed url="" %} ## Adding the iPhone mount onto the grabber ![Required components](https://1393952267-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKFoURW0OhNP7hBckTxa8%2Fuploads%2Fgit-blob-efdbfadf392497f621f3d4797eb90300d2bd2380%2FstickClampParts.JPG?alt=media) | Part | Quantity | | ---------------------------------------------------------------------------------------------------------------- | -------- | | [Grabber Clamp Top](https://github.com/notmahi/dobb-e/tree/main/hardware/grabberClamp/grabberClamp_top) | 1 | | [Grabber Clamp Bottom](https://github.com/notmahi/dobb-e/tree/main/hardware/grabberClamp/grabberClamp_bottom) | 1 | | [Phone Holder - Top Part](https://github.com/notmahi/dobb-e/tree/main/hardware/phoneHolder/phoneClamp_top) | 1 | | [Phone Holder - Bottom Part](https://github.com/notmahi/dobb-e/tree/main/hardware/phoneHolder/phoneClamp_bottom) | 1 | | [Phone Holder - Knob](https://github.com/notmahi/dobb-e/tree/main/hardware/phoneHolder/phoneClamp_knob) | 1 | | [Hex Nut Border](https://github.com/notmahi/dobb-e/tree/main/hardware/robotClamp/hexnutBorder) | 1 | | [Knob Tightener](https://github.com/notmahi/dobb-e/tree/main/hardware/phoneHolder/knobTightener) | 1 | | [M5x40 Socket Head Cap Screw](https://www.mcmaster.com/91290A258/) | 1 | | [M5x20 Socket Head Cap Screw](https://www.mcmaster.com/91290A242/) | 2 | | [M5x12 Socket Head Cap Screw](https://www.mcmaster.com/91290A228/) | 1 | | [M5-0.8 Hex Nut](https://www.mcmaster.com/90592A095/) | 4 | | [4mm Allen Key](https://www.amazon.com/Eklind-14616-Long-Hex-L-Key/dp/B000GAOAMI/) | 1 | #### Video tutorial {% embed url="" %} **Now you've built your Stick!** # Setting up your iPhone Setting up the iPhone is probably one of the most straightforward part in this tutorial. We used iPhone 13 Pros throughout our experiments, but generally any iPhone Pro with lidar based depth sensor should work. Later generation iPhones are slighly larger, which may induce small distribution shifts. But, we haven't tested it out. If you try it and face some problems, [let us know!](/contact-us) Here are the apps you need on your phone: 1. [Record3D](https://record3d.app) 2. [Precise Level](https://apps.apple.com/us/app/precise-level-spirit-level/id1093293519) 3. Google drive (optional, if you want to use our google drive script to move data between your phone and compute machine). For the apps, free versions should be enough. But if you get good value out of your use of the apps, please support the app developers however you can. # Using the Stick to Collect Data ## **Things to consider** The following are good tips to keep in mind. You don't have to follow them very judiciously. 1. **Have a lot of variations to the data.** We encourage collecting around 20-25 demonstrations per task and environment. Try to make each demonstration different from the other. Ideally vary the starting points as shown in the graphic below.

Starting from a grid to collect demos.

2. **Make sure the relevant objects are always visible in the frame** while collecting the demos. The relevant objects can be the door/ door handle for task “Door Opening”, or the object you have to pick (Eg: Cup) in the task “Pick and Place”. This is usually helpful for visual servoing policies. 3. **Another thing to keep in mind while collecting demos is to make sure to start and stop recording the demo at the right times**. Ideally, you would want to start recording just before starting the demo and stop recording immediately after the demo is finished. 4. **Be aware of the limitations of the robot while collecting demos**. For example as humans we may be able to operate around tight spaces using the stick but the robot may not due to its base/ arm not being as flexible. Also tasks that may require long extension of arm can also be limiting since the hello-stretch’s arm can only extend for so long. Always good idea to double check the robots capabilities before collecting demos to avoid wasting time on trajectories that are just not possible to run on the hardware 5. Finally while recording the demos, try to keep your motion as smooth and stable as possible. Try to avoid shaking or having any noisy movements during the demos. ## **How to collect demos** 1. Video Link explaining how to collect demonstrations: [Video link](https://drive.google.com/file/d/15n2OCz5LUQpzp9bk9L1hg6Ns28ZkHL35/view) 2. Make sure the iPhone is installed in the correct orientation. When holding the stick straight, the camera lens should be on the bottom right side.

This is how the stick should look when the camera is installed right.

3. Make sure the camera mount is roughly at **75 degree** tilt from the horizontal stick axis to get the same distribution of data as in HoNY. To confirm this tilt you can use iPhone app called Precise Level.

iPhone holder tightened at approximately 75 degrees.

4. Steps to collect 1 trajectory demo: 1. Open the Record3D app and go to "Record" panel 2. Align your stick to the appropriate “starting” orientation and position for the trajectory. 3. Press the “Red” record button to start recording and immediately start moving the stick according to your demonstration. 4. Once the demonstration is complete, immediately press the “Red” button to stop recording 5. Repeat the above for multiple demonstration with varying starting positions and trajectory motions # Putting Together the Robot Mount ![Required components](https://1393952267-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKFoURW0OhNP7hBckTxa8%2Fuploads%2Fgit-blob-711b69508590dc238ced3d8e0bebb2afe4fc777c%2FrobotClampParts.JPG?alt=media) | Part | Quantity | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | [Main Body](https://github.com/notmahi/dobb-e/tree/main/hardware/robotClamp/robotClamp) | 1 | | [Flat Insert](https://github.com/notmahi/dobb-e/tree/main/hardware/robotClamp/robotClamp_insert) | 1 | | [Bolt Pad](https://github.com/notmahi/dobb-e/tree/main/hardware/robotClamp/robotClamp_pad) | 1 | | [Hex Nut Border](https://github.com/notmahi/dobb-e/tree/main/hardware/robotClamp/hexnutBorder) | 1 | | [Phone Holder - Top Part](https://github.com/notmahi/dobb-e/tree/main/hardware/phoneHolder/phoneClamp_top) | 1 | | [Phone Holder - Bottom Part](https://github.com/notmahi/dobb-e/tree/main/hardware/phoneHolder/phoneClamp_bottom) | 1 | | [Phone Holder - Knob](https://github.com/notmahi/dobb-e/tree/main/hardware/phoneHolder/phoneClamp_knob) | 1 | | [Knob Tightener](https://github.com/notmahi/dobb-e/tree/main/hardware/phoneHolder/knobTightener) | 1 | | [M5x40 Socket Head Cap Screw](https://www.mcmaster.com/91290A258/) | 1 | | [M5x20 Socket Head Cap Screw](https://www.mcmaster.com/91290A242/) | 2 | | [M5-0.8 Hex Nut](https://www.mcmaster.com/91290A242/) | 3 | | [4mm Allen Key](https://www.amazon.com/Eklind-14616-Long-Hex-L-Key/dp/B000GAOAMI/) | 1 | | [Gorilla Glue](https://www.amazon.com/Gorilla-105795-Super-1-Pack-Clear/dp/B08QQZ71CV/) | 1 | | [Gel Gorilla Glue](https://www.amazon.com/Gorilla-105801-Super-1-Pack-Clear/dp/B08QR2XRKD/ref=sr_1_5?keywords=gorilla%2Bglue%2Bgel\&qid=1701128601\&sr=8-5\&th=1) | 1 | #### Video tutorial {% embed url="" %} # Mounting the iPhone to the Robot Optional: If the Stretch's wrist seems misaligned (not level), follow [this](https://forum.hello-robot.com/t/calibrating-zeros-for-the-dex-wrist-roll-pitch-yaw-joints/768) tutorial to correct it. 1. Joint calibrate the robot with `stretch_robot_home.py` if you have not already. 2. Place and tighten phone holder onto the Stretch’s Dex Wrist using an allen key. 3. Fully slide in iPhone (with the camera on the bottom right) until you feel that the robot can't slide in anymore (the camera notch will prevent it from sliding in any further). 4. Tighten the knob to make the iPhone tight 5. Adjust the tilt of the iPhone holder to be around **75 degrees** (you may find the [Precise Level](https://apps.apple.com/us/app/precise-level-spirit-level/id1093293519) helpful in doing so. 6. Connect a Lightning to USB-A data streaming cable from the iPhone to the Stretch’s USB port\ (We like a coiled, 90-degree angled cable the most, you can find them online if you search.) 7. Open the Record3D app on the iPhone ([record3d.app](https://record3d.app)) 8. Go into "Settings" and select “USB” for "Live RGBD Video Streaming" mode 9. Go back to "Record" and start recording. Final setup:
# Getting started with Dobb·E code Getting familiar with running Dobb·E, including pretraining (or downloading) the Home Pretrained Representation model, and fine-tuning your own behavior policy. In this section, we will help you get started with running all the software parts of Dobb·E. There are three major parts in the software component of Dobb-E, which are: 1. [Processing data collected with the Stick](/software/processing-collected-data), 2. [Training a model on the collected data](/software/readme-1/fine-tuning-policies), 3. [Deploying the model](/software/readme-1/deploying-a-policy-on-the-robot). Optionally, you can [pre-train your own model](/software/readme-1/optional-training-your-own-home-pretrained-representations) similar to how we trained the Home Pretrained Representations (HPR). We will frequently refer to the "robot", which for this part would be the Intel NUC installed in the Hello Robot Stretch, and the "machine", which should be a beefier machine with GPU(s) where you can preprocess your data and train new models. Also, ensure that you have [mamba](https://mamba.readthedocs.io/en/latest/) installed. Mamba is generally a faster but compatible alternative to conda, which we prefer. ## Getting Started * Clone the repository: ```bash git clone https://github.com/notmahi/dobb-e.git cd dobb-e/imitation-in-homes ``` * Set up the project environment: ```bash mamba env create -f conda_env.yaml ``` * Activate the environment: ```bash mamba activate home_robot ``` * Logging: * To enable logging, log in with a Weights and Biases (`wandb`) account: ```bash wandb login ``` * Alternatively, disable logging altogether: ```bash export WANDB_MODE=disabled ``` ## # Setting up the Datasets Download our provided datasets or get started with your own data. ### **Download Our Datasets** You can simply download our pre-provided datasets using any of the following commands: ```bash # HoNY RGB + actions dataset, 814 MB wget https://dl.dobb-e.com/datasets/homes_of_new_york.zip unzip homes_of_new_york.zip # HoNY RGB-D + actions dataset, 77 GB pip install gdown python -c "import gdown; gdown.download_folder("https://drive.google.com/drive/folders/1o8c6b6hSKfId8EzemVGf8c7DQoZ2IHAO?usp=sharing", quiet=True) zip -FF HomesOfNewYorkWithDepth.zip --out HoNYDepth.zip unzip -FF HoNYDepth.zip # Sample finetuning dataset wget https://dl.dobb-e.com/datasets/finetune_directory.zip unzip finetune_directory.zip ``` ### **Bring Your Own Data** Follow documentation in [Processing Collected Data](/software/processing-collected-data) to extract data. * Ensure the following data directory structure: 
dataset/
  |--- task1/
  |------ home1/
  |--------- env1/
  |--------- env1_val/
  |--------- env2/
  |--------- env2_val/
  |--------- env.../
  |------ home2/
  |--------- env1/
  |--------- env1_val/
  |--------- env.../
  |------ home.../
  |--- task2/
  |------ home1/
  |--------- env1/
  |--------- env1_val/
  |--------- env.../
  |------ home2/
  |--------- env1/
  |--------- env1_val/
  |--------- env.../
  |------ home.../
  |--- task.../
  |--- r3d_files.txt
### Create the env\_vars file Create the file `configs/env_vars/env_vars.yaml` based on `configs/env_vars/env_vars.yaml.sample`. {% hint style="info" %} `home_ssl_data_root` and `home_ssl_data_original_root`: These are optional, only relevant if you are planning to reproduce the HPR encoder. {% endhint %} # Fine-tuning Policies Fine-tuning policies with fresh demonstrations that you have collected. ## Training Policies The following assumes that the current working directory is this repository’s root folder. ### Training a Behavior Cloning Policy 1. Modify `include_task` and `include_env` in `finetune.yaml` depending on the task and env you intend to finetune. 2. \[Optional, non-default:] only if you're using torch encoder, set `enc_weight_pth` (path to pretrained encoder weights) in `image_bc_depth.yaml`. You can download the weights from if you don't have them. 3. Run in terminal: ```bash python train.py --config-name=finetune ``` 4. \[Optional, experimental] If you want to take advantage of multi-GPU training using 🤗 accelerate, you can use the following command:
```bash accelerate config # Only the first time, to configure accelerate accelerate launch train.py --config-name=finetune ``` # Deploying a Policy on the Robot ## Getting Started 1. Follow “Getting Started” in the `robot-server` documentation. * Skip the dataset related parts if you're not running VINN 2. Install ROS1 within your Conda environment: ```bash # Only if you are not using mamba already conda install mamba -c conda-forge # this adds the conda-forge channel to the new created environment configuration conda config --env --add channels conda-forge # and the robostack channel conda config --env --add channels robostack-staging # remove the defaults channel just in case, this might return an error if it is not in the list which is ok conda config --env --remove channels defaults mamba install ros-noetic-desktop ``` Reference: ## Deploying 1. Perform joint calibration by running `stretch_robot_home.py`. 2. [Attach the iPhone to the Stretch's wrist and start recording](/hardware/attach-camera-to-robot). 3. Follow documentation in [Running the Robot Controller](/software/running-the-robot-server) for running `roscore` and `start_server.py` on the robot. 4. Ensure that both of the previous commands are both running in the background in their own separate windows. {% hint style="info" %} We like using `tmux` for running multiple commands and keeping track of them at the same time. {% endhint %} ## Behavior Cloning 1. Transfer over the weights of a trained BC policy to the robot. 1. Take the last checkpoint (saved after 50th epoch): ```bash rsync -av --include='*/' --include='checkpoint.pt' --exclude='*' checkpoints/2023-11-22 hello-robot@{ip-address}:/home/hello-robot/code/imitation-in-homes/checkpoints ``` 2. In `configs/run.yaml` set `model_weight_pth` to the path containing the trained BC policy weights. 3. Run in terminal: ```bash python run.py --config-name=run ``` ## VINN 1. Transfer over the encoder weights and `finetune_task_data` onto the robot. 1. We recommend doing so using `rsync` 2. To speed up the transfer of data and save space on the robot, only transfer the necessary files: ```bash rsync -avm --include='*/' --include='*.json' --include='*.bin' --include='*.txt' --include='*.mp4' --exclude='*' /home/shared/data/finetune_task_data hello-robot@{ip-address}:/home/hello-robot/data ``` 2. In `configs/run_vinn.yaml` set `checkpoint_path` to encoder weights. 3. In `configs/dataset/vinn_deploy_dataset.yaml`, set `include_tasks` and `include_envs` to be a specific task (i.e. Drawer\_Closing) and environment (i.e. Env2) from the `finetune_task_data` folder. 4. Run in terminal: ```bash python run.py --config-name=run_vinn ``` ### Command Line Instructions * h * Bring the robot to its "home" position * r * Reset the "home" position height * Height values are in the range \~(0.2 to 1.1) * Wait a second or two, then home the robot ("h") to move to this height * s * Enter a value 1-10 * Home the robot by "h" to move to the fixed starting position * ↵ (Enter) * Take one "step" of the policy * Alternative: Enter some number + ↵ to "step" n times (i.e. 5 + ↵ for 5 "steps") # \[Optional] Training Your Own Home Pretrained Representations Completely optional step, if you want to tinker with the pretrained representation or train your own model. This step assumes you have already downloaded the HoNY RGB + actions dataset to somewhere on your machine, and have updated your `configs/env_vars/env_vars.yaml` accordingly. * **Single GPU:** To reproduce our HPR encoder, run in terminal: ```bash python train.py --config-name=train_moco ``` * Multi-GPU: We use huggingface 🤗 accelerate to run multi-GPU training. Run the following commands ```bash accelerate config # Interactively walk you through setting up multi-GPU training accelerate launch train.py --config-name=train_moco ``` # Processing Collected Data Once you collect some new data on the Stick, you need to process it into a dataset before you can train policies on it. This step will help you get started on that. ## Clone the Repo ``` git clone git@github.com:notmahi/dobb-e cd dobb-e/stick-data-collection ``` ## Installation * On your machine, in a new conda/virtual environment ```bash mamba env create -f conda_env.yaml ``` ## Usage For extracting a single environment: 1. Compress video taken from the Record3D app: ![Export Data](https://1393952267-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKFoURW0OhNP7hBckTxa8%2Fuploads%2Fgit-blob-4c920d073377220c0ce04b6113306b43a8c67cd8%2Frecord3d-tut.jpg?alt=media) 2. Get the files on your machine. 1. **Using Google drive:** 1. \[Only once] Generate Google Service Account API key to download from private folders on Google Drive. There are some instructions on how to do so in this Stackoverflow link 2. \[Only once] Rename the .json file to `client_secret.json` and put it in the same directory as `gdrive_downloader.py` 3. Upload `.zip` file into its own folder on Google Drive, and copy folder\_id from URL to put it in the `GDRIVE_FOLDER_ID` in the `./do-all.sh` file. 2. **Manually**: * Comment out the `GDRIVE_FOLDER_ID` line from `./do-all.sh` and create the following hierarchy locally ```bash dataset/ |--- task1/ |------ home1/ |--------- env1/ |------------ {data_file}.zip |--------- env2/ |------------ {data_file}.zip |--------- env.../ |------------ {data_file}.zip |------ home2/ |------ home.../ |--- task2/ |--- task.../ ``` * The .zip files should contain .r3d files exported from the Record3D app in the previous step. 3. Modify required variables in `do-all.sh`. 1. `TASK_NO` task id, see `gdrive_downloader.py` for more information. 2. `HOME` name or ID of the home. 3. `ROOT_FOLDER` folder where the data is stored after downloading. 4. `EXPORT_FOLDER` folder where the dataset is stored after processing. Should be different from `ROOT_FOLDER`. 5. `ENV_NO` current environment number in the same home and task set. 6. `GRIPPER_MODEL_PATH` path to the gripper model. It should be in the github repo already, and can be downloaded from . 4. Change current working directory to local repository root folder and run ```bash ./do-all.sh ``` 5. Split the extracted data to include a validation set for each environment. The data should follow the following hierarchy: (Be sure change the corresponding paths in `r3d_files.txt` to include “`_val`”)\\ ```bash dataset/ |--- task1/ |------ home1/ |--------- env1/ |--------- env1_val/ |--------- env2/ |--------- env2_val/ |--------- env.../ |------ home2/ |--------- env1/ |--------- env1_val/ |--------- env.../ |------ home.../ |--- task2/ |------ home1/ |--------- env1/ |--------- env1_val/ |--------- env.../ |------ home2/ |--------- env1/ |--------- env1_val/ |--------- env.../ |------ home.../ |--- task.../ |--- r3d_files.txt ``` # Running the Robot Controller You will need to run the controller code on the robot so that our policy can communicate with the robot, get the observations, and execute the actions. This step walks you through that. Code to start the camera stream publisher and robot controller. The code is useful for [Record3d](https://record3d.app)-based camera streaming, but can be adapted for other use cases. ## Getting Started 1. Clone repository onto your Hello Robot Stretch: ```bash git clone https://github.com/notmahi/dobb-e.git cd dobb-e/robot-server ``` 2. Install required packages **on your root pip env** (where `hello_robot` package is installed): ```bash pip install -r requirements.txt ``` ## Running 1. If you just turned on the robot, perform joint calibration on your Stretch by running the following in terminal: ```bash stretch_robot_home.py ``` this only needs to be run once every time the robot is booted up. 2. In a separate terminal window, Run `roscore` on your Stretch within the conda/mamba environment `home_robot` on your Stretch ```bash mamba activate home_robot roscore ``` 3. [Attach and set up iPhone on robot](/hardware/attach-camera-to-robot). 4. Change current working directory to this repository’s root folder (`cd ../robot-server`). 5. Run in terminal (not within the `conda` environment, but in the root pip environment): ```bash python3 start_server.py ``` # Contact us Do you have more questions? Thoughts? Suggestions? If you have any further questions, thoughts, or comments on our work, we would love to hear from you. There are a few different ways you can get in touch with us. 1. [Schedule a call.](#schedule-a-call) 2. [Email us.](#email-us) 3. [Submit a pull request on GitHub.](#submit-a-pull-request-on-github) 4. [Message us on twitter.](#message-us-on-twitter) ### Schedule a call If you have questions or want to debug something with us, you can schedule a video call with us. Just use the form below or get on the following link and we will try to connect you with one of our team members. {% embed url="" %} ### Email us You can also email us with your thoughts and questions. Just write to `mahi at cs dot nyu dot edu` with "Dobb-E" somewhere on the title and we will try to get back to you as soon as we can. ### Submit a pull request on GitHub If you have a concrete improvement to our code or hardware that you want to share with us, you can also submit a pull request here: . ### Message us on Twitter You can feel free to message us on [Twitter](https://twitter.com/notmahi), but we believe one of the earlier three options would probably be your best bet for a faster response time. ## Our team * [Mahi Shafiullah](https://mahis.life)\* * [Anant Rai](https://raianant.github.io/)\* * [Haritheja Etukuru](https://haritheja.com/) * [Yiqian Liu](https://www.linkedin.com/in/eva-liu-ba90a5209/) \[\* authors contributed equally] #### On advisory roles * [Ishan Misra](https://imisra.github.io/) * [Soumith Chintala](https://soumith.ch/) * [Lerrel Pinto](https://www.lerrelpinto.com/)