Skip to content

Commit

Permalink
Merge pull request #27 from threedworld-mit/keyboard_controller
Browse files Browse the repository at this point in the history
Keyboard controller
  • Loading branch information
alters-mit authored Aug 13, 2020
2 parents 10cf0aa + 484d9ac commit 5eb96cf
Show file tree
Hide file tree
Showing 11 changed files with 302 additions and 124 deletions.
29 changes: 21 additions & 8 deletions Documentation/Changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,32 +6,45 @@

### `tdw` module

- Added required modules: `pyinstaller` and `keyboard`
- Added `tdw.keyboard_controller` A controller that can listen to keyboard input.

#### `Controller`

- Removed optional `display` parameter. It doesn't actually work; Linux users should instead launch the controller with a `DISPLAY` environment variable.

### Frontend

- Added `freeze.py`. "Freeze" your controller into a portable binary executable.

#### Backend

- Added `controller.spec` (used for freezing controller code).
- Adjusted how Flatbuffers imports numpy so that frozen controller code works.

### Example Controllers

- Renamed `keyboard.py` to `keyboard_controller.py` to avoid a name clash with the `keyboard` module. Rewrote the code to use the `KeyboardController` class.

### Build

- Fixed: Segmentation colors are often non-unique.

### Misc.

- Added `freeze.py`. "Freeze" your controller into a portable binary executable.
- Added `controller.spec` (used for freezing controller code).

### Documentation

#### New Documentation

| Document | Description |
| ----------- | ------------------------------------------------------------ |
| `freeze.md` | How to freeze your controller code into a binary executable. |
| Document | Description |
| ------------------------ | ------------------------------------------------------------ |
| `freeze.md` | How to freeze your controller code into a binary executable. |
| `keyboard_controller.md` | API for KeyboardController. |

#### Modified Documentation

| Document | Description |
| -------------------- | ---------------------------------------------------------- |
| `getting_started.md` | Fixed instructions for how to start a controller in Linux. |
| `docker.md` | Fixed some broken links. |

## v1.6.4

Expand Down
4 changes: 2 additions & 2 deletions Documentation/Docker/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ All Docker-related bash scripts are in [`tdw/Docker`](https://github.com/threedw
| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
| [`start_container.sh`](https://github.com/threedworld-mit/tdw/tree/master/Docker/start_container.sh) | | Start the container and run TDW. |
| [`start_container_xpra.sh`](https://github.com/threedworld-mit/tdw/tree/master/Docker/start_container_xpra.sh) | | Start the container with [Xpra](../misc_frontend/xpra.md) and run TDW. |
| [`start_container_audio_video.sh VOLUME IPADDRESS PORT`](https://github.com/threedworld-mit/tdw/tree/master/Docker/start_container_audio.sh) | `VOLUME` Save audio to this volume<br>`IPADDRESS` The address of the build.<br>`PORT` The port of the build. | [Record audio and video](../misc_frontend/video.md) from TDW. |
| [`start_container_audio_video.sh VOLUME IPADDRESS PORT`](https://github.com/threedworld-mit/tdw/tree/master/Docker/start_container_audio_video.sh) | `VOLUME` Save audio to this volume<br>`IPADDRESS` The address of the build.<br>`PORT` The port of the build. | [Record audio and video](../misc_frontend/video.md) from TDW. |

### Other scripts

Expand All @@ -69,7 +69,7 @@ All Docker-related bash scripts are in [`tdw/Docker`](https://github.com/threedw
| [`pull.sh`](https://github.com/threedworld-mit/tdw/tree/master/Docker/pull.sh) | | Try to download a Docker container from DockerHub with a tag that matches the version of TDW on this machine. |
| [`docker_tag.sh`](https://github.com/threedworld-mit/tdw/tree/master/Docker/docker_tag.sh) | | Get the tag of the TDW Docker image. |
| [`tdw_version.py`](https://github.com/threedworld-mit/tdw/tree/master/Docker/tdw_version.py) | | Get the version of TDW on this machine. |
| [`record_audio_video.sh ADDRESS PORT WIDTH HEIGHT`](https://github.com/threedworld-mit/tdw/tree/master/Docker/record_audio.sh) | `ADDRESS` The network address of the build.<br>`PORT` The network port of the build<br> `WIDTH` The desired width of the video in pixels.<br>`HEIGHT` The desired height of the video in pixels. | Launch TDW and begin recording audio. |
| [`record_audio_video.sh ADDRESS PORT WIDTH HEIGHT`](https://github.com/threedworld-mit/tdw/tree/master/Docker/record_audio_video.sh) | `ADDRESS` The network address of the build.<br>`PORT` The network port of the build<br> `WIDTH` The desired width of the video in pixels.<br>`HEIGHT` The desired height of the video in pixels. | Launch TDW and begin recording audio. |


## Docker within Docker
Expand Down
2 changes: 1 addition & 1 deletion Documentation/python/example_controllers.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Each TDW release includes many example controllers. They can be found in: `<root
| `getting_started.py` | 1. Add a table and place an object on the table.<br>2. Add a camera and receive an image. |
| `hdri.py` | Create an object and avatar and capture images of the scene, rotating the HDRI skybox by 15 degrees<br>for each image. |
| `impact_sounds.py` | - Listen for collisions between objects.<br>- Generate an impact sound with py_impact upon impact and play the sound in the build. |
| `keyboard.py` | Use WASD or arrow keys to move an avatar. |
| `keyboard_controls.py` | Use WASD or arrow keys to move an avatar. |
| `local_object.py` | Create a local asset bundle and load it into TDW.<br><br>See `Documentation/misc_frontend/add_local_object.md` for how to run the Asset Bundle Creator. |
| `minimal.py` | A minimal example of how to connect to the build and receive data. |
| `minimal_remote.py` | A minimal example of how to use the launch binaries daemon to<br>start and connect to a build on a remote node. Note: the remote<br>must be running launch_binaries.py. |
Expand Down
78 changes: 78 additions & 0 deletions Documentation/python/keyboard_controller.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# `keyboard_controller.py`

## `KeyboardController(Controller)`

`from tdw.keyboard_controller import KeyboardController`

Listen for keyboard input to send commands.

Usage:

```python
from tdw.keyboard_controller import KeyboardController
from tdw.tdw_utils import TDWUtils

def stop():
done = True

done = False
c = KeyboardController()
c.start()

# Quit.
c.listen(key="esc", commands=None, function=stop)

# Equivalent to c.start()
c.listen(key="r", commands={"$type": "load_scene", "scene_name": "ProcGenScene"}, function=None)

while not done:
# Receive data. Load the scene when r is pressed. Quit when Esc is pressed.
c.communicate([])
# Stop the build.
c.communicate({"$type": "terminate"})
```

***

#### `stop()`

def __init__(self, port: int = 1071, check_version: bool = True, launch_build: bool = True):

***

#### `__init__(self, port: int = 1071, check_version: bool = True, launch_build: bool = True)`

Create the network socket and bind the socket to the port.

| Parameter | Description |
| --- | --- |
| port | The port number. |
| check_version | If true, the controller will check the version of the build and print the result. |
| launch_build | If True, automatically launch the build. If one doesn't exist, download and extract the correct version. Set this to False to use your own build, or (if you are a backend developer) to use Unity Editor. |

***

#### `communicate(self, commands: Union[dict, List[dict]]) -> list`

Listen for when a key is pressed and send commands.

| Parameter | Description |
| --- | --- |
| key | The keyboard key. |
| commands | Commands to be sent when the key is pressed. |
| function | A function to be invoked when the key is pressed. |

***

#### `listen(self, key: str, commands: Union[dict, List[dict]] = None, function=None) -> None`

Listen for when a key is pressed and send commands.

| Parameter | Description |
| --- | --- |
| key | The keyboard key. |
| commands | Commands to be sent when the key is pressed. |
| function | A function to be invoked when the key is pressed. |

***

16 changes: 8 additions & 8 deletions Documentation/python/py_impact.md
Original file line number Diff line number Diff line change
Expand Up @@ -327,20 +327,20 @@ c.start()

# Request the required output data (do this at the start of the simulation, not per frame).
resp = c.communicate([{"$type": "send_collisions",
"enter": True,
"exit": False,
"stay": True,
"collision_types": ["obj", "env"]},
{"$type": "send_rigidbodies",
"frequency": "always"}])
"enter": True,
"exit": False,
"stay": True,
"collision_types": ["obj", "env"]},
{"$type": "send_rigidbodies",
"frequency": "always"}])

# Parse the output data and get collision type data.
ctof = CollisionTypesOnFrame(object_id, resp)

# Read the dictionaries of collidee IDs and collision types.
for collidee_id in ctof.collisions:
collision_type = ctof.collisions[collidee_id]
print(collidee_id, collision_type)
collision_type = ctof.collisions[collidee_id]
print(collidee_id, collision_type)

# Check the environment collision.
print(ctof.env_collision_type)
Expand Down
19 changes: 10 additions & 9 deletions Documentation/python/tdw.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,16 @@ pip3 install tdw

### Frontend

| | Description |
| ------------------------------------------------------------ | ----------------------------------------------------------- |
| [Controller](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/controller.md) | Base class for all controllers. |
| [TDWUtils](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/tdw_utils.md) | Utility class. |
| [AssetBundleCreator](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/asset_bundle_creator.md) | Covert 3D models into TDW-compatible asset bundles. |
| [PyImpact](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/py_impact.md) | Generate impact sounds at runtime. |
| [DebugController](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/debug_controller.md) | Child class of `Controller` that has useful debug features. |
| [Librarian](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/librarian/librarian.md) | "Librarians" hold asset bundle metadata records. |
| [FluidTypes](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/fluid_types.md) | Access different NVIDIA Flex fluid types. |
| | Description |
| ------------------------------------------------------------ | ------------------------------------------------------------ |
| [Controller](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/controller.md) | Base class for all controllers. |
| [TDWUtils](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/tdw_utils.md) | Utility class. |
| [AssetBundleCreator](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/asset_bundle_creator.md) | Covert 3D models into TDW-compatible asset bundles. |
| [PyImpact](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/py_impact.md) | Generate impact sounds at runtime. |
| [DebugController](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/debug_controller.md) | Child class of `Controller` that has useful debug features. |
| [KeyboardController](keyboard_controller.md) | Child class of `Controller` that can listen for keyboard input. |
| [Librarian](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/librarian/librarian.md) | "Librarians" hold asset bundle metadata records. |
| [FluidTypes](https://github.com/threedworld-mit/tdw/blob/master/Documentation/python/fluid_types.md) | Access different NVIDIA Flex fluid types. |

### Backend

Expand Down
85 changes: 0 additions & 85 deletions Python/example_controllers/keyboard.py

This file was deleted.

86 changes: 86 additions & 0 deletions Python/example_controllers/keyboard_controls.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
from tdw.keyboard_controller import KeyboardController
from tdw.tdw_utils import TDWUtils


"""
Use WASD or arrow keys to move an avatar.
"""


class KeyboardControls(KeyboardController):
def __init__(self, port: int = 1071):
super().__init__(port=port)
self.done = False

def run(self, force=80, torque=100):
"""
:param force: The force magnitude used to move the avatar.
:param torque: The torque magnitude to turn the avatar by.
"""

print("W, up-arrow = Move forward")
print("S, down-arrow = Move backward")
print("A, left-arrow = Turn counterclockwise")
print("D, right-arrow = Turn clockwise")
print("Esc = Quit")

# Listen for keyboard input for movement.
self.listen("w", commands={"$type": "move_avatar_forward_by",
"magnitude": force,
"avatar_id": "a"})
self.listen("up", commands={"$type": "move_avatar_forward_by",
"magnitude": force,
"avatar_id": "a"})
self.listen("s", commands={"$type": "move_avatar_forward_by",
"magnitude": -force,
"avatar_id": "a"})
self.listen("down", commands={"$type": "move_avatar_forward_by",
"magnitude": -force,
"avatar_id": "a"})
self.listen("d", commands={"$type": "turn_avatar_by",
"torque": torque,
"avatar_id": "a"})
self.listen("right", commands={"$type": "turn_avatar_by",
"torque": torque,
"avatar_id": "a"})
self.listen("a", commands={"$type": "turn_avatar_by",
"torque": -torque,
"avatar_id": "a"})
self.listen("left", commands={"$type": "turn_avatar_by",
"torque": -torque,
"avatar_id": "a"})
# Listen for keyboard input to quit.
self.listen("esc", function=self.stop)

self.start()
# Create the room.
commands = [TDWUtils.create_empty_room(12, 12)]
# Create the avatar.
commands.extend(TDWUtils.create_avatar(avatar_type="A_Img_Caps", avatar_id="a"))
# 1. Set high drag values so it doesn't feel like the avatar is sliding on ice.
# 2. Set the room's floor material.
commands.extend([{"$type": "set_avatar_drag",
"drag": 10,
"angular_drag": 20,
"avatar_id": "a"},
self.get_add_material("parquet_alternating_orange", library="materials_high.json"),
{"$type": "set_proc_gen_floor_material",
"name": "parquet_alternating_orange"},
{"$type": "set_proc_gen_floor_texture_scale",
"scale": {"x": 8, "y": 8}}])
self.communicate(commands)
while not self.done:
# Listen for keyboard input to add other commands.
self.communicate([])
self.communicate({"$type": "terminate"})

def stop(self):
"""
Stop the controller and the build.
"""

self.done = True


if __name__ == "__main__":
KeyboardControls().run()
2 changes: 1 addition & 1 deletion Python/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,5 @@
include_package_data=True,
keywords='unity simulation ml machine-learning',
install_requires=['pyzmq', 'numpy', 'scipy', 'pillow', 'tqdm', 'psutil', 'boto3', 'botocore', 'requests',
"pyinstaller"],
'pyinstaller', 'keyboard'],
)
Loading

0 comments on commit 5eb96cf

Please sign in to comment.