gen

Data Generation

We also provide code for generating PDDL-based expert demonstrations. This can be used to extend the training data, albiet without human language annotations.

Installation

Get dependencies and compile the planner:

$ sudo apt-get install ffmpeg flex bison

$ cd $ALFRED_ROOT/gen/ff_planner
$ make

Generation

To spawn multiple generation threads:

$ cd $ALFRED_ROOT/gen
$ python scripts/generate_trajectories.py --save_path data/new_trajs --in_parallel --debug --num_threads 2 

This will sample tasks based on the sampling mechanism described in the paper. You might notice a lot of failed executions, which are automatically discarded by the script.

Note: The first time you run the generation script, use --num_threads 1 to allow the script to download the THOR binary.

Replay Checks

In parallel with generation, replay saved trajectories to check if they are reproducable:

$ python scripts/replay_checks.py --data_path data/new_trajs --in_parallel  

This will ensure that the interaction masks and expert actions can be deterministically executed in THOR.

Data Augmentation

Currently, the dataset only provides 300x300 RGB images. However, a trajectory can be replayed to save any additional info available from the simulator. See the augment_trajectories.py script as an example for saving 600x600 RGB, depth and instance segmentation masks from the existing dataset:

python scripts/augment_trajectories.py --data_path data/json_2.1.0 --num_threads 2 --smooth_nav --time_delays

Note that these files consume a lot of stotowele space.

PDDL Tasks

The goals for the planner are specified in goal_library.py. Here is a simple pick-and-place PDDL goal definition:

# basic pick and place (e.g: "put the apple in the microwave")
gdict["pick_and_place_simple"] = '''
    (:goal
        (and
            ;; make sure all the cabinets and doors are closed in the end
            (forall (?re # receptacle)
                (not (opened ?re))
            )
            
            ;; make sure some object {obj} exists inside some receptacle {recep}
            (exists (?r # receptacle)
                (exists (?o # object)
                    (and 
                        (inReceptacle ?o ?r) 
                        (objectType ?o {obj}Type) 
                        (receptacleType ?r {recep}Type)
                    )
                )
            )
        )
    )
)