ThreeDWorld (TDW)

ThreeDWorld (TDW) is a platform for interactive multi-modal physical simulation. With TDW, users can simulate high-fidelity sensory data and physical interactions between mobile agents and objects in a wide variety of rich 3D environments.

TDW was developed primarily by Jeremy Schwartz and Esther Alter. It is now maintained by Esther Alter.

TDW is now in long-term support (LTS). TDW will receive only minor updates and bug fixes.

• Code of Conduct
• Changelog
• License
• Website
• Example controllers

General guide to TDW

Setup

1.1 Installation (Read this first!)

1. Install TDW
   • On a personal computer (Linux, MacOS, Windows)
   • On a server
   • On a personal computer + a server
2. Upgrade TDW

1.2 How to run TDW on a Linux server

1. Overview
2. Launch a TDW build on a remote server from a personal computer
3. Remote rendering with xpra
4. X11 forwarding

Core Concepts

1. The controller
2. Auto-launching the TDW build
3. Commands
4. Design philosophy of TDW
5. Scenes
6. Avatars and cameras
7. Add-ons and the ThirdPersonCamera
8. Objects
9. Output data
10. Images
11. Outdoor scenes

Troubleshooting

1. How to report an issue
2. Common errors
3. Performance optimizations
4. Good coding practices

Tutorials

1. Scene Setup

Overview

1.1 Scene Setup (High-Level APIs)

1. Overview
2. Procedural generation (the ProcGenKitchen add-on)
3. Regions, interior regions, and rooms
4. Procedural object arrangements
5. Scripted object placement (floorplan layouts)
6. Reset a scene

High-level APs: Floorplan and ProcGenKitchen

1.2 Scene Setup (Low-Level APIs)

1. Overview
2. Units and data formats
3. Bounds output data
4. Visual materials, textures, and colors
5. ProcGenRoom

2. 3D Models

2.1 Other model libraries

1. Overview
2. Free models
3. Non-free models

2.2 Custom models

1. Add your own models to TDW
2. Add ShapeNet models to TDW

2.3 Composite (articulated) objects

1. Overview
2. Composite objects in TDW
3. Create a composite object from a prefab
4. Create a composite object from a .urdf file

3. Read/Write to Disk

1. Overview
2. The Logger and LogPlayback add-ons
3. The JsonWriter add-on
4. The OutputDataWriter add-on
5. Create a custom data writer
6. Import .sdf and .lisdf files
7. Images (Cross-referenced with "Core Concepts")
8. Audio (Cross-referenced with "Audio")
9. Video (Cross-referenced with "Video Recording")

4. Semantic States

1. Overview
2. Line of sight (Raycast output data)
3. Proximity to region (Overlap output data)
4. Proximity to other objects (the TriggerCollisionManager add-on)
5. Containment (the ContainerManager add-on)
6. Open and closed states
7. Grasped objects

5. Visual Perception

1. Overview
2. Instance ID segmentation colors (_id pass)
3. Semantic category segmentation colors (_category pass)
4. Depth maps (_depth and _depth_simple passes)
5. Motion perception (_flow pass)
6. Other image passes (_mask, _normals, and _albedo passes)
7. Occlusion output data

6. Camera Controls

1. Overview
2. Move a camera
3. Rotate a camera
4. Follow an object
5. The CinematicCamera add-on

7. Photorealism

1. Overview
2. Lighting (HDRI skyboxes)
3. Post-processing
4. Interior lighting (the InteriorSceneLighting add-on)
5. Depth of field
6. V-Ray Rendering

High-level API: tdw_image_dataset

8. Physics

Overview

8.1 Physics (PhysX)

1. PhysX
2. Object physics parameters
3. Rigidbodies output data
4. Collision output data
5. Apply forces to objects
6. Skip physics frames
7. Disable physics

High-level API: tdw_physics

8.2 Physics (Obi)

1. Obi
2. Fluids
3. Wind
4. Cloth
5. ObiParticles output data
6. Colliders and collision materials
7. Solvers
8. Obi and robots
9. Obi and Flex

8.3 Physics (Flex)

1. Flex
2. Solid and soft actors
3. Cloth actors
4. Fluid and source actors
5. Move, rotate, and scale Flex objects
6. FlexParticles output data
7. Apply forces to Flex objects
8. Reset a Flex scene
9. Other Flex commands

High-level API: tdw_physics

9. Audio

1. Overview
2. Initialize audio and play .wav files
3. Resonance Audio
4. Recording audio
5. Audio perception

9.1 Clatter (Physically-derived audio)

1. Overview
2. Object audio data
3. Recording Clatter audio with the PhysicsAudioRecorder add-on
4. Clatter and Resonance Audio
5. Reset Clatter
6. Manually generate audio
7. Troubleshooting Clatter
8. How to contribute to Clatter

9.2 PyImpact (obsolete predecessor to Clatter)

1. PyImpact
2. PyImpact advanced
3. PyImpact and Clatter

10. Video Recording

1. Overview
2. Image-only video
3. Video with audio
   1. Video with audio (Linux)
   2. Video with audio (OS X)
   3. Video with audio (Windows)

11. Agents

Overview

11.1 Robots

1. Overview
2. The Robot add-on
3. Robot arm add-ons
4. Robot collision detection
5. Select a robot
6. Add your own robots to TDW
7. Robotics API (low-level)
8. Add a camera to a robot

11.2 Magnebots

1. Magnebot API (external repo)

11.3 Replicants

1. Overview
2. Actions
3. Output data
4. Collision detection
5. Movement
6. Animations
7. Arm articulation, pt. 1: Basics
8. Arm articulation, pt. 2: Grasp and drop objects
9. Arm articulation, pt. 3: Advanced topics
10. Arm articulation, pt. 4: Stacking objects
11. Head rotation
12. Navigation
13. Custom actions
14. Multiple Replicants
15. Reset

11.4 Wheelchair Replicants

1. Overview
2. Actions
3. Output data
4. Collision detection
5. Movement
6. Arm articulation, pt. 1: Basics
7. Arm articulation, pt. 2: Grasp and drop objects
8. Arm articulation, pt. 3: Advanced topics
9. Head rotation
10. Navigation
11. Custom actions
12. Multiple Agents
13. Reset

11.5 Drones

1. Drones

11.6 Vehicles

1. Vehicles

11.7 Virtual Reality (VR)

1. Overview
2. Oculus Touch
3. Oculus Leap Motion
4. FOVE Leap Motion

11.8 Keyboard and Mouse

1. Overview
2. Mouse input
3. The FirstPersonAvatar
4. Keyboard input

11.9 Embodied avatars

1. The EmbodiedAvatar

12. Multi-agent simulations

1. Overview
2. Custom agent classes

13. Navigation

1. Overview
2. NavMesh pathfinding
3. Occupancy maps

14. User Interface (UI)

1. Overview
2. The UI add-on
3. UI Widgets

15. Non-physics objects

15.1 Non-physics humanoids

1. Overview
2. SMPL humanoids
3. Create custom non-physics humanoids
4. Create custom humanoid animations

15.2 Misc. non-physics objects

1. Overview
2. Position markers
3. Line renderers
4. Textured quads
5. Compass rose
6. Visual Effects
7. The FloorplanFlood add-on
8. Empty objects

16. Misc. other topics

1. C# source code
2. Freezing your code
3. Download asset bundles

API Documentation

Command API

• Command API
• Output Data

tdw module API

tdw

• AudioConstants
• AudioUtils
• CardinalDirection
• Controller
• IntPair
• OrdinalDirection
• QuaternionUtils
• RemoteBuildLauncher
• TDWUtils
• TypeAliases

tdw.add_ons

• AddOn
• AudioInitializer
• AudioInitializerBase
• Autohand
• AvatarBody
• Benchmark
• CinematicCamera
• Clatter
• CollisionManager
• CompositeObjectManager
• ContainerManager
• Drone
• EmbodiedAvatar
• EmptyObjectManager
• FirstPersonAvatar
• Floorplan
• FloorplanFlood
• FoveLeapMotion
• ImageCapture
• InteriorSceneLighting
• JsonWriter
• Keyboard
• LeapMotion
• LisdfReader
• Logger
• LogPlayback
• ModelVerifier
• Mouse
• NavMesh
• Obi
• ObjectManager
• OccupancyMap
• OculusLeapMotion
• OculusTouch
• OutputDataWriter
• PhysicsAudioRecorder
• ProcGenKitchen
• PyImpact
• Replicant
• ReplicantBase
• ResonanceAudioInitializer
• Robot
• RobotArm
• RobotBase
• StepPhysics
• ThirdPersonCamera
• ThirdPersonCameraBase
• TriggerCollisionManager
• UI
• Vehicle
• VR
• VrayExporter
• WheelchairReplicant
• Writer

tdw.add_ons.ui_widgets

• LoadingScreen
• ProgressBar
• TimerBar

tdw.agent_data

• AgentDynamic

tdw.asset_bundle_creator

• AnimationCreator
• AssetBundleCreator
• CompositeObjectCreator
• HumanoidCreator
• HumanoidCreatorBase
• ModelCreator
• RobotCreator

tdw.backend

• Update

tdw.collision_data

• CollisionBase
• CollisionObjEnv
• CollisionObjObj
• TriggerColliderShape
• TriggerCollisionEvent

tdw.container_data

• BoxContainer
• ContainerShape
• ContainerTag
• ContainmentEvent
• CylinderContainer
• SphereContainer

tdw.drone

• DroneDynamic

tdw.flex_data

• FluidType

tdw.lerp

• Lerpable
• LerpableFloat
• LerpableVector

tdw.librarian

• DroneLibrarian
• HdriSkyboxLibrarian
• HumanoidAnimationLibrarian
• HumanoidLibrarian
• MaterialLibrarian
• ModelLibrarian
• RobotLibrarian
• SceneLibrarian
• VehicleLibrarian
• VisualEffectLibrarian

tdw.lisdf_data

• LisdfRobotMetadata

tdw.model_tests

• MissingMaterials
• ModelReport
• ModelTest
• PhysicsQuality
• RotateObjectTest

tdw.obi_data

• ForceMode
• ObiActor
• ObiBackend
• WindSource

tdw.obi_data.cloth

• ClothMaterial
• SheetType
• TetherParticleGroup
• TetherType
• VolumeType

tdw.obi_data.collision_materials

• CollisionMaterial
• MaterialCombineMode

tdw.obi_data.fluids

• CubeEmitter
• DiskEmitter
• EdgeEmitter
• EmitterSamplingMethod
• EmitterShape
• Fluid
• FluidBase
• GranularFluid
• SphereEmitter

tdw.object_data

• Bound
• ObjectStatic
• Rigidbody
• Transform

tdw.object_data.composite_object

• CompositeObjectDynamic
• CompositeObjectStatic

tdw.object_data.composite_object.sub_object

• HingeDynamic
• HingeStatic
• HingeStaticBase
• LightDynamic
• LightStatic
• MotorStatic
• NonMachineStatic
• PrismaticJointStatic
• SpringStatic
• SubObjectDynamic
• SubObjectStatic

tdw.physics_audio

• AudioMaterial
• Base64Sound
• ClatterObject
• CollisionAudioEvent
• CollisionAudioInfo
• CollisionAudioType
• ImpactMaterial
• Modes
• ObjectAudioStatic
• ScrapeMaterial
• ScrapeModel
• ScrapeSubObject

tdw.proc_gen.arrangements

• Arrangement
• ArrangementAlongWall
• ArrangementWithRootObject
• Basket
• CupAndCoaster
• Dishwasher
• KitchenCabinet
• KitchenCounter
• KitchenCounterTop
• KitchenTable
• Microwave
• Painting
• Plate
• Radiator
• Refrigerator
• Shelf
• SideTable
• Sink
• StackOfPlates
• Stool
• Stove
• Suitcase
• TableAndChairs
• TableSetting
• Void
• WallCabinet

tdw.proc_gen.arrangements.cabinetry

• Cabinetry
• CabinetryType

tdw.release

• Build
• PyPi

tdw.replicant

• ActionStatus
• Arm
• CollisionDetection
• ImageFrequency
• ReplicantBodyPart
• ReplicantDynamic
• ReplicantStatic

tdw.replicant.actions

• Action
• Animate
• ArmMotion
• DoNothing
• Drop
• Grasp
• HeadMotion
• IkMotion
• LookAt
• MoveBy
• MoveTo
• ReachFor
• ReachForWithPlan
• ResetArm
• ResetHead
• RotateHead
• TurnBy
• TurnTo

tdw.replicant.ik_plans

• IkPlan
• IkPlanType
• Reset
• VerticalHorizontal

tdw.robot_data

• Drive
• JointDynamic
• JointStatic
• JointType
• NonMoving
• RobotDynamic
• RobotStatic

tdw.scene_data

• InteriorRegion
• RegionBounds
• Room
• SceneBounds

tdw.vehicle

• VehicleDynamic

tdw.vray_data

• VrayMatrix

tdw.vr_data

• FingerBone
• OculusTouchButton
• RigType

tdw.vr_data.fove

• CalibrationMethod
• CalibrationSphere
• CalibrationState
• Eye
• EyeByEyeCalibration
• EyeState
• EyeTorsionCalibration

tdw.wheelchair_replicant

• WheelValues

tdw.wheelchair_replicant.actions

• MoveBy
• MoveTo
• ReachFor
• TurnBy
• TurnTo
• WheelchairMotion

Performance benchmarks

1. Performance benchmarks
2. Image capture
3. Object data
4. Command deserialization