diff --git a/Documentation/Changelog.md b/Documentation/Changelog.md index b42326438..c1009cf2b 100644 --- a/Documentation/Changelog.md +++ b/Documentation/Changelog.md @@ -6,32 +6,45 @@ ### `tdw` module +- Added required modules: `pyinstaller` and `keyboard` +- Added `tdw.keyboard_controller` A controller that can listen to keyboard input. + #### `Controller` - Removed optional `display` parameter. It doesn't actually work; Linux users should instead launch the controller with a `DISPLAY` environment variable. -### Frontend - -- Added `freeze.py`. "Freeze" your controller into a portable binary executable. - #### Backend -- Added `controller.spec` (used for freezing controller code). - Adjusted how Flatbuffers imports numpy so that frozen controller code works. +### Example Controllers + +- Renamed `keyboard.py` to `keyboard_controller.py` to avoid a name clash with the `keyboard` module. Rewrote the code to use the `KeyboardController` class. + +### Build + +- Fixed: Segmentation colors are often non-unique. + +### Misc. + +- Added `freeze.py`. "Freeze" your controller into a portable binary executable. + - Added `controller.spec` (used for freezing controller code). + ### Documentation #### New Documentation -| Document | Description | -| ----------- | ------------------------------------------------------------ | -| `freeze.md` | How to freeze your controller code into a binary executable. | +| Document | Description | +| ------------------------ | ------------------------------------------------------------ | +| `freeze.md` | How to freeze your controller code into a binary executable. | +| `keyboard_controller.md` | API for KeyboardController. | #### Modified Documentation | Document | Description | | -------------------- | ---------------------------------------------------------- | | `getting_started.md` | Fixed instructions for how to start a controller in Linux. | +| `docker.md` | Fixed some broken links. | ## v1.6.4 diff --git a/Documentation/Docker/docker.md b/Documentation/Docker/docker.md index 42c776259..8e82f5370 100644 --- a/Documentation/Docker/docker.md +++ b/Documentation/Docker/docker.md @@ -60,7 +60,7 @@ All Docker-related bash scripts are in [`tdw/Docker`](https://github.com/threedw | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | | [`start_container.sh`](https://github.com/threedworld-mit/tdw/tree/master/Docker/start_container.sh) | | Start the container and run TDW. | | [`start_container_xpra.sh`](https://github.com/threedworld-mit/tdw/tree/master/Docker/start_container_xpra.sh) | | Start the container with [Xpra](../misc_frontend/xpra.md) and run TDW. | -| [`start_container_audio_video.sh VOLUME IPADDRESS PORT`](https://github.com/threedworld-mit/tdw/tree/master/Docker/start_container_audio.sh) | `VOLUME` Save audio to this volume
`IPADDRESS` The address of the build.
`PORT` The port of the build. | [Record audio and video](../misc_frontend/video.md) from TDW. | +| [`start_container_audio_video.sh VOLUME IPADDRESS PORT`](https://github.com/threedworld-mit/tdw/tree/master/Docker/start_container_audio_video.sh) | `VOLUME` Save audio to this volume
`IPADDRESS` The address of the build.
`PORT` The port of the build. | [Record audio and video](../misc_frontend/video.md) from TDW. | ### Other scripts @@ -69,7 +69,7 @@ All Docker-related bash scripts are in [`tdw/Docker`](https://github.com/threedw | [`pull.sh`](https://github.com/threedworld-mit/tdw/tree/master/Docker/pull.sh) | | Try to download a Docker container from DockerHub with a tag that matches the version of TDW on this machine. | | [`docker_tag.sh`](https://github.com/threedworld-mit/tdw/tree/master/Docker/docker_tag.sh) | | Get the tag of the TDW Docker image. | | [`tdw_version.py`](https://github.com/threedworld-mit/tdw/tree/master/Docker/tdw_version.py) | | Get the version of TDW on this machine. | -| [`record_audio_video.sh ADDRESS PORT WIDTH HEIGHT`](https://github.com/threedworld-mit/tdw/tree/master/Docker/record_audio.sh) | `ADDRESS` The network address of the build.
`PORT` The network port of the build
`WIDTH` The desired width of the video in pixels.
`HEIGHT` The desired height of the video in pixels. | Launch TDW and begin recording audio. | +| [`record_audio_video.sh ADDRESS PORT WIDTH HEIGHT`](https://github.com/threedworld-mit/tdw/tree/master/Docker/record_audio_video.sh) | `ADDRESS` The network address of the build.
`PORT` The network port of the build
`WIDTH` The desired width of the video in pixels.
`HEIGHT` The desired height of the video in pixels. | Launch TDW and begin recording audio. | ## Docker within Docker diff --git a/Documentation/python/example_controllers.md b/Documentation/python/example_controllers.md index f0a3bca1a..4c0a47f55 100644 --- a/Documentation/python/example_controllers.md +++ b/Documentation/python/example_controllers.md @@ -23,7 +23,7 @@ Each TDW release includes many example controllers. They can be found in: `2. Add a camera and receive an image. | | `hdri.py` | Create an object and avatar and capture images of the scene, rotating the HDRI skybox by 15 degrees
for each image. | | `impact_sounds.py` | - Listen for collisions between objects.
- Generate an impact sound with py_impact upon impact and play the sound in the build. | -| `keyboard.py` | Use WASD or arrow keys to move an avatar. | +| `keyboard_controls.py` | Use WASD or arrow keys to move an avatar. | | `local_object.py` | Create a local asset bundle and load it into TDW.

See `Documentation/misc_frontend/add_local_object.md` for how to run the Asset Bundle Creator. | | `minimal.py` | A minimal example of how to connect to the build and receive data. | | `minimal_remote.py` | A minimal example of how to use the launch binaries daemon to
start and connect to a build on a remote node. Note: the remote
must be running launch_binaries.py. | diff --git a/Documentation/python/keyboard_controller.md b/Documentation/python/keyboard_controller.md new file mode 100644 index 000000000..7d6818f3d --- /dev/null +++ b/Documentation/python/keyboard_controller.md @@ -0,0 +1,78 @@ +# `keyboard_controller.py` + +## `KeyboardController(Controller)` + +`from tdw.keyboard_controller import KeyboardController` + +Listen for keyboard input to send commands. + +Usage: + +```python +from tdw.keyboard_controller import KeyboardController +from tdw.tdw_utils import TDWUtils + +def stop(): + done = True + +done = False +c = KeyboardController() +c.start() + +# Quit. +c.listen(key="esc", commands=None, function=stop) + +# Equivalent to c.start() +c.listen(key="r", commands={"$type": "load_scene", "scene_name": "ProcGenScene"}, function=None) + +while not done: + # Receive data. Load the scene when r is pressed. Quit when Esc is pressed. + c.communicate([]) +# Stop the build. +c.communicate({"$type": "terminate"}) +``` + +*** + +#### `stop()` + +def __init__(self, port: int = 1071, check_version: bool = True, launch_build: bool = True): + +*** + +#### `__init__(self, port: int = 1071, check_version: bool = True, launch_build: bool = True)` + +Create the network socket and bind the socket to the port. + +| Parameter | Description | +| --- | --- | +| port | The port number. | +| check_version | If true, the controller will check the version of the build and print the result. | +| launch_build | If True, automatically launch the build. If one doesn't exist, download and extract the correct version. Set this to False to use your own build, or (if you are a backend developer) to use Unity Editor. | + +*** + +#### `communicate(self, commands: Union[dict, List[dict]]) -> list` + +Listen for when a key is pressed and send commands. + +| Parameter | Description | +| --- | --- | +| key | The keyboard key. | +| commands | Commands to be sent when the key is pressed. | +| function | A function to be invoked when the key is pressed. | + +*** + +#### `listen(self, key: str, commands: Union[dict, List[dict]] = None, function=None) -> None` + +Listen for when a key is pressed and send commands. + +| Parameter | Description | +| --- | --- | +| key | The keyboard key. | +| commands | Commands to be sent when the key is pressed. | +| function | A function to be invoked when the key is pressed. | + +*** + diff --git a/Documentation/python/py_impact.md b/Documentation/python/py_impact.md index 7d1b55d1b..217bfd2ac 100644 --- a/Documentation/python/py_impact.md +++ b/Documentation/python/py_impact.md @@ -327,20 +327,20 @@ c.start() # Request the required output data (do this at the start of the simulation, not per frame). resp = c.communicate([{"$type": "send_collisions", -"enter": True, -"exit": False, -"stay": True, -"collision_types": ["obj", "env"]}, -{"$type": "send_rigidbodies", -"frequency": "always"}]) + "enter": True, + "exit": False, + "stay": True, + "collision_types": ["obj", "env"]}, + {"$type": "send_rigidbodies", + "frequency": "always"}]) # Parse the output data and get collision type data. ctof = CollisionTypesOnFrame(object_id, resp) # Read the dictionaries of collidee IDs and collision types. for collidee_id in ctof.collisions: -collision_type = ctof.collisions[collidee_id] -print(collidee_id, collision_type) + collision_type = ctof.collisions[collidee_id] + print(collidee_id, collision_type) # Check the environment collision. print(ctof.env_collision_type) diff --git a/Documentation/python/tdw.md b/Documentation/python/tdw.md index e0105ad1f..c7399cd4b 100644 --- a/Documentation/python/tdw.md +++ b/Documentation/python/tdw.md @@ -12,15 +12,16 @@ pip3 install tdw ### Frontend -| | Description | -| ------------------------------------------------------------ | ----------------------------------------------------------- | -| [Controller](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/controller.md) | Base class for all controllers. | -| [TDWUtils](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/tdw_utils.md) | Utility class. | -| [AssetBundleCreator](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/asset_bundle_creator.md) | Covert 3D models into TDW-compatible asset bundles. | -| [PyImpact](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/py_impact.md) | Generate impact sounds at runtime. | -| [DebugController](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/debug_controller.md) | Child class of `Controller` that has useful debug features. | -| [Librarian](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/librarian/librarian.md) | "Librarians" hold asset bundle metadata records. | -| [FluidTypes](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/fluid_types.md) | Access different NVIDIA Flex fluid types. | +| | Description | +| ------------------------------------------------------------ | ------------------------------------------------------------ | +| [Controller](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/controller.md) | Base class for all controllers. | +| [TDWUtils](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/tdw_utils.md) | Utility class. | +| [AssetBundleCreator](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/asset_bundle_creator.md) | Covert 3D models into TDW-compatible asset bundles. | +| [PyImpact](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/py_impact.md) | Generate impact sounds at runtime. | +| [DebugController](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/debug_controller.md) | Child class of `Controller` that has useful debug features. | +| [KeyboardController](keyboard_controller.md) | Child class of `Controller` that can listen for keyboard input. | +| [Librarian](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/librarian/librarian.md) | "Librarians" hold asset bundle metadata records. | +| [FluidTypes](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/fluid_types.md) | Access different NVIDIA Flex fluid types. | ### Backend diff --git a/Python/example_controllers/keyboard.py b/Python/example_controllers/keyboard.py deleted file mode 100644 index b4f6cdfbb..000000000 --- a/Python/example_controllers/keyboard.py +++ /dev/null @@ -1,85 +0,0 @@ -from tdw.controller import Controller -from tdw.tdw_utils import TDWUtils -from platform import system - -if system() == "Windows": - from msvcrt import getch -else: - from getch import getch - - -""" -Use WASD or arrow keys to move an avatar. -""" - - -class KeyboardControls(Controller): - def run(self, force=80): - """ - :param force: The force magnitude used to move the avatar. - """ - - print("W, up-arrow = Move forward") - print("S, down-arrow = Move backward") - print("A, left-arrow = Turn counterclockwise") - print("D, right-arrow = Turn clockwise") - print("Esc = Quit") - - self.start() - self.communicate(TDWUtils.create_empty_room(12, 12)) - self.communicate(TDWUtils.create_avatar(avatar_type="A_Img_Caps")) - - # Set high drag values so it doesn't feel like the avatar is sliding on ice. - self.communicate({"$type": "set_avatar_drag", - "drag": 10, - "angular_drag": 20}) - - # Change the floor material so it's easier to visualize the avatar's movement. - self.communicate([self.get_add_material("parquet_alternating_orange", library="materials_high.json"), - {"$type": "set_proc_gen_floor_material", - "name": "parquet_alternating_orange"}, - {"$type": "set_proc_gen_floor_texture_scale", - "scale": {"x": 8, "y": 8}}]) - - done = False - while not done: - ch = getch() - ch_int = ord(ch) - forward = 0 - torque = 0 - # Move forward with w or up-arrow. - if ch == b'w' or ch_int == 72: - forward = force - # Move backward with s or down-arrow. - elif ch == b's' or ch_int == 80: - forward = -force - # Rotate counterclockwise with d or right-arrow. - elif ch == b'd' or ch_int == 77: - torque = force - # Rotate clockwise with a or left-arrow. - elif ch == b'a' or ch_int == 75: - torque = -force - # Quit with esc. - elif ch_int == 27: - done = True - continue - - commands = [] - # If there was any directional input, apply a directional force. - if abs(forward) > 0: - commands.append({"$type": "move_avatar_forward_by", - "magnitude": forward}) - # If there was any rotational input, apply a torque. - if abs(torque) > 0: - commands.append({"$type": "turn_avatar_by", - "torque": torque}) - - # If there was no input, do nothing and advance the simulation. - # (This isn't necessary in this example, but you'll want this if anything else is moving in the scene). - if len(commands) > 0: - commands.append({"$type": "do_nothing"}) - - self.communicate(commands) - - -KeyboardControls().run() diff --git a/Python/example_controllers/keyboard_controls.py b/Python/example_controllers/keyboard_controls.py new file mode 100644 index 000000000..1fa0f1ae2 --- /dev/null +++ b/Python/example_controllers/keyboard_controls.py @@ -0,0 +1,86 @@ +from tdw.keyboard_controller import KeyboardController +from tdw.tdw_utils import TDWUtils + + +""" +Use WASD or arrow keys to move an avatar. +""" + + +class KeyboardControls(KeyboardController): + def __init__(self, port: int = 1071): + super().__init__(port=port) + self.done = False + + def run(self, force=80, torque=100): + """ + :param force: The force magnitude used to move the avatar. + :param torque: The torque magnitude to turn the avatar by. + """ + + print("W, up-arrow = Move forward") + print("S, down-arrow = Move backward") + print("A, left-arrow = Turn counterclockwise") + print("D, right-arrow = Turn clockwise") + print("Esc = Quit") + + # Listen for keyboard input for movement. + self.listen("w", commands={"$type": "move_avatar_forward_by", + "magnitude": force, + "avatar_id": "a"}) + self.listen("up", commands={"$type": "move_avatar_forward_by", + "magnitude": force, + "avatar_id": "a"}) + self.listen("s", commands={"$type": "move_avatar_forward_by", + "magnitude": -force, + "avatar_id": "a"}) + self.listen("down", commands={"$type": "move_avatar_forward_by", + "magnitude": -force, + "avatar_id": "a"}) + self.listen("d", commands={"$type": "turn_avatar_by", + "torque": torque, + "avatar_id": "a"}) + self.listen("right", commands={"$type": "turn_avatar_by", + "torque": torque, + "avatar_id": "a"}) + self.listen("a", commands={"$type": "turn_avatar_by", + "torque": -torque, + "avatar_id": "a"}) + self.listen("left", commands={"$type": "turn_avatar_by", + "torque": -torque, + "avatar_id": "a"}) + # Listen for keyboard input to quit. + self.listen("esc", function=self.stop) + + self.start() + # Create the room. + commands = [TDWUtils.create_empty_room(12, 12)] + # Create the avatar. + commands.extend(TDWUtils.create_avatar(avatar_type="A_Img_Caps", avatar_id="a")) + # 1. Set high drag values so it doesn't feel like the avatar is sliding on ice. + # 2. Set the room's floor material. + commands.extend([{"$type": "set_avatar_drag", + "drag": 10, + "angular_drag": 20, + "avatar_id": "a"}, + self.get_add_material("parquet_alternating_orange", library="materials_high.json"), + {"$type": "set_proc_gen_floor_material", + "name": "parquet_alternating_orange"}, + {"$type": "set_proc_gen_floor_texture_scale", + "scale": {"x": 8, "y": 8}}]) + self.communicate(commands) + while not self.done: + # Listen for keyboard input to add other commands. + self.communicate([]) + self.communicate({"$type": "terminate"}) + + def stop(self): + """ + Stop the controller and the build. + """ + + self.done = True + + +if __name__ == "__main__": + KeyboardControls().run() diff --git a/Python/setup.py b/Python/setup.py index 14426a80d..33cfebf85 100644 --- a/Python/setup.py +++ b/Python/setup.py @@ -31,5 +31,5 @@ include_package_data=True, keywords='unity simulation ml machine-learning', install_requires=['pyzmq', 'numpy', 'scipy', 'pillow', 'tqdm', 'psutil', 'boto3', 'botocore', 'requests', - "pyinstaller"], + 'pyinstaller', 'keyboard'], ) diff --git a/Python/tdw/keyboard_controller.py b/Python/tdw/keyboard_controller.py new file mode 100644 index 000000000..a73bb9f78 --- /dev/null +++ b/Python/tdw/keyboard_controller.py @@ -0,0 +1,84 @@ +from typing import List, Union +import keyboard +from tdw.controller import Controller + + +class KeyboardController(Controller): + """ + Listen for keyboard input to send commands. + + Usage: + + ```python + from tdw.keyboard_controller import KeyboardController + from tdw.tdw_utils import TDWUtils + + def stop(): + done = True + + done = False + c = KeyboardController() + c.start() + + # Quit. + c.listen(key="esc", commands=None, function=stop) + + # Equivalent to c.start() + c.listen(key="r", commands={"$type": "load_scene", "scene_name": "ProcGenScene"}, function=None) + + while not done: + # Receive data. Load the scene when r is pressed. Quit when Esc is pressed. + c.communicate([]) + # Stop the build. + c.communicate({"$type": "terminate"}) + ``` + """ + + def __init__(self, port: int = 1071, check_version: bool = True, launch_build: bool = True): + """ + Create the network socket and bind the socket to the port. + + :param port: The port number. + :param check_version: If true, the controller will check the version of the build and print the result. + :param launch_build: If True, automatically launch the build. If one doesn't exist, download and extract the correct version. Set this to False to use your own build, or (if you are a backend developer) to use Unity Editor. + """ + + # Commands that should be added due to key presses on this frame. + self.on_key_commands: List[dict] = [] + + super().__init__(port=port, check_version=check_version, launch_build=launch_build) + + def communicate(self, commands: Union[dict, List[dict]]) -> list: + if isinstance(commands, dict): + commands = [commands] + # Add commands from key presses. + commands.extend(self.on_key_commands[:]) + # Clear the on-key commands. + self.on_key_commands.clear() + return super().communicate(commands) + + def listen(self, key: str, commands: Union[dict, List[dict]] = None, function=None) -> None: + """ + Listen for when a key is pressed and send commands. + + :param key: The keyboard key. + :param commands: Commands to be sent when the key is pressed. + :param function: A function to be invoked when the key is pressed. + """ + + if commands is not None: + keyboard.on_press_key(key, lambda e: self._set_frame_commands(commands)) + if function is not None: + keyboard.on_press_key(key, lambda e: function()) + + def _set_frame_commands(self, commands: Union[dict, List[dict]]) -> None: + """ + Set the next frame's commands. + + :param commands: The commands to send on this frame. + """ + + if isinstance(commands, dict): + commands = [commands] + self.on_key_commands = commands + diff --git a/README.md b/README.md index 1f8211537..35e38746d 100644 --- a/README.md +++ b/README.md @@ -29,16 +29,17 @@ ##### Frontend -| Document | Description | -| ------------------------------------------------------------ | ----------------------------------------------------------- | -| [`tdw` module](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/tdw.md) | Overview of the Python `tdw` module. | -| [Controller](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/controller.md) | Base class for all controllers. | -| [TDWUtils](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/tdw_utils.md) | Utility class. | -| [AssetBundleCreator](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/asset_bundle_creator.md) | Covert 3D models into TDW-compatible asset bundles. | -| [PyImpact](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/py_impact.md) | Generate impact sounds at runtime. | -| [DebugController](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/debug_controller.md) | Child class of `Controller` that has useful debug features. | -| [Librarian](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/librarian/librarian.md) | "Librarians" hold asset bundle metadata records. | -| [FluidTypes](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/fluid_types.md) | Access different NVIDIA Flex fluid types. | +| Document | Description | +| ------------------------------------------------------------ | ------------------------------------------------------------ | +| [`tdw` module](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/tdw.md) | Overview of the Python `tdw` module. | +| [Controller](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/controller.md) | Base class for all controllers. | +| [TDWUtils](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/tdw_utils.md) | Utility class. | +| [AssetBundleCreator](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/asset_bundle_creator.md) | Covert 3D models into TDW-compatible asset bundles. | +| [PyImpact](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/py_impact.md) | Generate impact sounds at runtime. | +| [DebugController](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/debug_controller.md) | Child class of `Controller` that has useful debug features. | +| [KeyboardController](Documentation/python/keyboard_controller.md) | Child class of `Controller` that can listen for keyboard input. | +| [Librarian](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/librarian/librarian.md) | "Librarians" hold asset bundle metadata records. | +| [FluidTypes](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/fluid_types.md) | Access different NVIDIA Flex fluid types. | ##### Backend