Skip to content

Commit

Permalink
Merge pull request #122 from JdeRobot/rlstudioV12
Browse files Browse the repository at this point in the history
first refactor v1.2
  • Loading branch information
sergiopaniego authored Jan 25, 2023
2 parents 66c38d7 + fc12847 commit c461cbb
Show file tree
Hide file tree
Showing 156 changed files with 17,540 additions and 6,221 deletions.
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Change Log

Here you can find all important changes in this project. RL-Studio adheres to [Semantic Versioning](http://semver.org/).
## [Unreleased]

## [1.2.0] - 2022-01-11
### Added:

- Add new files in envs/gazebo/f1/models/: original files are split into different modular files such as images.py, reset.py, step.py, settings.py, simplified_perception.py ([PR #122](https://github.com/JdeRobot/RL-Studio/pull/122))

### Changed:
- entry file name: `rl-studio.py `
- config file names: now they follow naming `config_mode_task_algorithm_agent_simulator`
- files structure in envs/gazebo/f1/models/: files are named `task_algorithm.py`



[1.2.0]: https://github.com/JdeRobot/RL-Studio/pull/122
51 changes: 51 additions & 0 deletions CODING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# CODING STYLE

If you are contributing to RL-Studio tool development, please follow the below coding style and structure of directories, files recommendations:

## SOME RULES

- [PEP 8](https://peps.python.org/pep-0008/) style guide for Python Code
- [Black](https://github.com/psf/black) format
- [Pylint](https://pypi.org/project/pylint/) as static code analyser
- Constants variable must be upper-case (e.g `TIME_THRESHOLD`, `NUMBER_OF_RUNS`)
- Comment all non trivial functions to ease readability
- All the internal imported packages must be imported from the root of the project (e.g `import rl_studio.agents` instead `import agents`)
- Organize imports before pushing your code to the repo

- When creating a project, please keep in mind:

- in **/agents** directory, files names should be `mode_task_algorithm_simulator_framework.py`, i.e. `trainer_followline_ddpg_F1_gazebo_tf.py` or `inferencer_mountaincar_qlearn_openai_pytorch.py`. In case of not using framework leave it blank.
- in **/envs/gazebo/f1/models** directory, files names should be `task_algorithm_framework.py`, i.e. `followline_ddpg_gazebo_tf.py` or `followlane_qlearn_pytorch.py`. In case of not using framework leave it blank.
- As a general rule, **classes names** have to follow convention `ModeTaskAlgorithmAgentSimuladorFramework`, i.e. `TrainerFollowLaneDDPGF1GazeboPytorch` or `InferencerFollowLaneDQNF1GazeboTF`
- in **/envs/gazebo** directory, classes names follow rule `TaskAlgorithmAgentSimulatorFramework`, i.e. `FollowlineDDPGF1GazeboTF`.

# Directory architecture

## Config files

- in **/config** directory add a configuration file with the following format `config_mode_task_algorithm_agent_simulator.yaml`, i.e. `config_training_followlane_qlearn_F1_carla.yaml`

## Models

- Add a trained brain in **/checkpoints** folder. You can configure it in the config.yaml file. Automatically the app will add a directory with the format `task_algorithm_agent_simulator_framework` where to save models.
- The file model should have the format `timestamp_maxreward_epoch_ADITIONALTEXT.h5` in format h5 i.e. `09122002_max45678_epoch435_actor_conv2d32x64_critic_conv2d32x64_actionsCont_stateImg_rewardDiscrete.h5` to indicate the main features of the model saved in order to easily find the exact model.

## Metrics

- In **/metrics** folder should be saved statistics and metrics of models. You can configure it in the config.yaml file. Automatically the app will add a directory with the format `mode/task_algorithm_agent_simulator_framework/data` where to save data.

## Graphics

- In **/metrics** folder should be saved graphics of models. You can configure it in the config.yaml file. Automatically the app will add a directory with the format `mode/task_algorithm_agent_simulator_framework/graphics` where to save graphics.

## Logs and TensorBoard files

- In **/logs** folder should be saved TensorBoard and logs files. You can configure it in the config.yaml file.
For TensorBoard, automatically the app will add a directory with the format `mode/task_algorithm_agent_simulator_framework/TensorBoard`.

For logs, the app automatically will add a directory with the format `mode/task_algorithm_agent_simulator_framework/logs`.

# TIPS

- You can refer to "mountain_car qlearning" project as an example of how to implement a real time monitoring
- You can refer to "cartpole dqn" project as an example of how to implement logging
53 changes: 53 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Contributing to Reinforcement Learning Studio

Thanks for your interest on contributing!

This file contains a set of rules to contributing to the project and the
rest of the projects developed by JdeRobot.
If you have any doubt about how to contribute contact one of the maintainers
of the project. They will be pleased to tell you how you can contribute with your
knowledge to the project and organization!

* [Code of conduct](#code-of-conduct)
* [Prerequisites before contributing](#prerequisites-before-contributing)
* [How can I contribute?](#how-can-i-contribute)


<a name="#code-of-conduct"></a>
## Code of conduct
Please report any unacceptable behavior to any of [the maintainers](#i-have-a-question).

<a name="#prerequisites"></a>
## Prerequisites before contributing
In order to contribute to JdeRobot projects, please read carefully the project README.md/[webpage](https://github.com/JdeRobot/RL-Studio) before
starting contributing to understand the purpose of the project and where you can contribute.

<a name="#how-to-contribute"></a>
## How can I contribute?
Any JdeRobot project follows the same workflow when contributing.

* **Find a problem or possible improvement for the project:** First of all, check that the feature/bug is not listed in the current [open issues](https://github.com/JdeRobot/RL-Studio/issues).

* **Create an issue:** [Create an issue](https://github.com/JdeRobot/RL-Studio/issues/new) in the project with the problem/improvement you will
address. In this issue, explain carefully what you will be updating and how this changes will impact the project.
Provide any complementary information to explain it (code samples, screenshots ...). You should information about:
* Expected behavior
* Actual behavior
* Steps to reproduce
* Environment
* Possible cause
* Possible solution

The two following points are different depending on the permissions you have to the repo.
* **[If you have write permission] Work in a separate branch always:** Create a new branch with a describable name (you can use the issue number as branch name "issue_xxx"). Create your commits in that branch making the needed changes. Please, use describable names as commit messages, so everyone can understand easily the changes you made.

* **[If you only have read permission] Fork the project:** Fork the project. Work on that copy of the repo, making the desirable changes. Please, use describable names as commit messages, so everyone can understand easily the changes you made.

* **Open a pull request:** A pull request is compulsory any time a new change wants to be added to the core or the project. After solving the issue, create a pull request with your branch. In this pull request include all the commits made, write a good description of the changes made and refer to the issue solved to make things easier to the maintainers. Include any additional resource that would be interesting (references, screenshots...). Link the PR with the issue.

* **Testing and merging pull requests**
One of maintainers will review your code. Reviewer could ask you to modify your pull request.
Please provide timely response for reviewers (within weeks, not months), otherwise you submission could be postponed or even rejected.

* **[If you have write permission] Don't accept your own pull requests:** Wait for a project maintainer to accept the changes you made. They will probably comment the pull request with some feedback and will consider if it can be merge to the master branch. Be proactive and kind!

File renamed without changes.
145 changes: 112 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,48 @@

## [![forthebadge](https://forthebadge.com/images/badges/for-robots.svg)](https://forthebadge.com) [![forthebadge](https://forthebadge.com/images/badges/made-with-python.svg)](https://forthebadge.com)

## [![Dependencies Status](https://img.shields.io/badge/dependencies-up%20to%20date-brightgreen.svg) ](https://github.com/TezRomacH/python-package-template/pulls?utf8=%E2%9C%93&q=is%3Apr%20author%3Aapp%2Fdependabot)[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg) ](https://github.com/psf/black) [![Pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/TezRomacH/python-package-template/blob/master/.pre-commit-config.yaml) [![License](https://img.shields.io/github/license/TezRomacH/python-package-template)](https://github.com/JdeRobot/RL-Studio/blob/main/LICENSE.md) ![](https://img.shields.io/badge/Dependencies-Poetry-blue)
## [![Dependencies Status](https://img.shields.io/badge/dependencies-up%20to%20date-brightgreen.svg) ](https://github.com/TezRomacH/python-package-template/pulls?utf8=%E2%9C%93&q=is%3Apr%20author%3Aapp%2Fdependabot)[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg) ](https://github.com/psf/black) [![Pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/TezRomacH/python-package-template/blob/master/.pre-commit-config.yaml) [![License](https://img.shields.io/badge/license-GNU-orange)](https://github.com/JdeRobot/RL-Studio/blob/main/LICENSE)

![](https://img.shields.io/badge/Gazebo-11-orange) ![](https://img.shields.io/badge/ROS-Noetic-blue) ![](https://img.shields.io/badge/Python-3.8-yellowInstall)
![](https://img.shields.io/badge/Gazebo-11-orange) ![](https://img.shields.io/badge/ROS-Noetic-blue) ![](https://img.shields.io/badge/Python-3.8-yellowInstall) ![](https://img.shields.io/badge/Carla-0.9.13-yellow) ![](https://img.shields.io/badge/TensorFlow-2.9.11-brightgreen) ![](https://img.shields.io/badge/PyTorch-1.13-yellowgreen)

</div>

RL-Studio is a platform for training reinforcement learning algorithms for robots with different environments and algorithms. You can create your agent, environment and algorithm and compare it with others.
Reinforcement Learning Studio, RL-Studio, is a platform for developing robotic applications with reinforcement learning algorithms. Its modular design allows to work easily with different agents and algoritmhs in autonomous tasks and any simulator.

## Installation
# Introduction

### Install ROS
RL-Studio is designed to work with robots, as autonomous vehicles, in any relevant tasks with any simulators that provide adequate realism to transfer development to real environments automatically and with robustness --sim2real transfer.
The designed agents can support any type of sensor that collects information from the environment and, with reinforcement learning algorithms, can send correct signals to the actuators to adequately command the robot, following the standard reinforcement learning cycle.

RL-Studio works with ROS Noetic. You can [install ROS Noetic in the official documentation](http://wiki.ros.org/noetic/Installation/Ubuntu) and installing ROS Noetic Full Desktop.
## Working Modes

Rl-Studio allows you to work with different modes of operation, all of them neccesary to build a RL app:

- Training: the objective of any development in RL-Studio is to design a training that generates a suitable model for the environment in question. see diagram
- Retraining of models already generated, so that they continue learning in the same or different environments.
- Inference: Trained models are tested --inference -- in different environments in order to validate their learning.

## Agents

RL-Studio is designed to work with any robotic agent, mainly in autonomous driving through complex environments. However, thanks to the modularity of the application, it is easy to create new agents that are tested in other tasks, such as manipulation, legged robots, drones and so on.

## Algorithms

Qlearn, DQN, DDPG and PPO have currently been implemented to work on the different tasks developed. However, it is easy to design any other algorithms.

## Deep Learning frameworks

[Tensorflow](https://www.tensorflow.org/) 2.9.11 and [Pytorch](https://pytorch.org/) 1.13 are currently supported, although it is very easy to extend to others.

## Simulators and ROS

RL-Studio supports [ROS](http://wiki.ros.org/) Noetic which is necesary to interact with [Gazebo](https://classic.gazebosim.org/) or also as a bridge with [Carla](https://carla.readthedocs.io/en/0.9.13/). Although Carla can work without ROS also. Currently there are working canonical reinforcement lerning tasks with OpenAI simulator through [gymnasium](https://gymnasium.farama.org/) library.

# Installation

## Install ROS

RL-Studio works with ROS Noetic. You can [install ROS Noetic from the official documentation](http://wiki.ros.org/noetic/Installation/Ubuntu) and installing ROS Noetic Full Desktop.

### Clone the RL-studio repository

Expand All @@ -30,46 +59,53 @@ or
git clone https://github.com/JdeRobot/RL-Studio.git
```

### Install dependencies with Poetry (recommended):

```bash
curl -sSL https://install.python-poetry.org | python3 -
export PATH="/root/.local/bin:$PATH"
```
## Install dependencies using pip:

Install dependencies:
_It is highly recommended to create a virtual environment:_

```bash
poetry install
cd RL-Studio
pip install -r requirements.txt
```

### Install dependencies using pip (not recommended):

_Note: In case you don't want to use Poetry as a dependency manager, you can install it with pip as follows (previously it is highly recommended to create a virtual environment):_
Add the project to `PYTHONPATH`:

```bash
cd RL-Studio
pip install -r requirements.txt
echo "export PYTHONPATH=$PYTHONPATH:~/PATH/TO/RL-Studio" >> ~/.bashrc
source ~/.bashrc
```

The commits follow the [gitmoji](https://gitmoji.dev/) convention and the code is formatted with [Black](https://black.readthedocs.io/en/stable/).

#### Install rl_studio as package

```bash
cd ~/PATH/TO/RL-Studio/rl_studio
pip install -e .
```
## Checking everything. Set environment

## Set environments
### Set ROS Noetic and Formula 1 agent configuration

### Set Noetic and Formula 1 agent configuration
The fastest way to verify that the installation has been successful is to follow the next steps.

To connect RL-Studio with ROS and Gazebo and the different agents and circuits installed:

```bash
cd ~/PATH/TO/RL-Studio/rl_studio/installation
bash setup_noetic.bash
```

> :warning: if bash file execution gives an error, in some configurations can be fixed by editing the bash file and changing the line
```bash
catkin_make
```

by

```bash
catkin_make -DPYTHON_EXECUTABLE=/usr/bin/python3 -DPYTHON_INCLUDE_DIR=/usr/include<python3.8>
```

where python3.8 is the actual Python version in your virtual env.

The installation downloads the CustomRobots repository into the above directory, as follows:

```bash
Expand Down Expand Up @@ -98,16 +134,14 @@ export GAZEBO_RESOURCE_PATH=$GAZEBO_RESOURCE_PATH:$HOME/PATH/TO/RL-Studio/rl_stu
. . .
```

### Continuing setting Formula1 environment

To set Formula 1 environment running the following script (the same folder that before):
To set Formula 1 environment run the following script (the same folder that before):

```
```bash
cd ~/PATH/TO/RL-Studio/rl_studio/installation
./formula1_setup.bash
```

The following routes will be added to the `.bashrc` file (for `formula1` environment), please check it:
The following routes will be added to the `.bashrc` file:

```bash
. . .
Expand All @@ -119,11 +153,56 @@ export GYM_GAZEBO_WORLD_MONTREAL_F1=$HOME/PATH/TO/RL-Studio/rl_studio/installati

There will be as many variables as there are circuits to be executed. In case you want to work with other circuits or agents, there will be necessary add the correct paths to variables in `.bashrc` file in the same way.

And finally, do not forget adding
```bash
export PYTHONPATH=$PYTHONPATH:PATH/TO/RL-Studio
```

## Usage/Examples

To check that everything is working correctly you can try launching a ROS exercise by typing:

```bash
cd $HOME/PATH/TO/RL-Studio/rl_studio/CustomRobots/f1/launch
```python
cd /PATH/TO/RL-Studio/rl_studio/CustomRobots/f1/launch
roslaunch simple_circuit.launch
```

And to begin training and inferencing, please go to [README.md](https://github.com/JdeRobot/RL-Studio/blob/main/rl_studio/README.md)
and you could see something similar to the screenshot

![](./rl_studio/docs/gazebo_screenshot.png)


# Work with RL-Studio


Additional information on how to create, run and test reinforcement learning models, how to create a configuration file to launch the application and to begin training and inferencing, please go to [rl-studio](https://github.com/JdeRobot/RL-Studio/blob/main/rl_studio/README.md).

Information about coding or naming classes and files, how the directory structure is designed and where to save models, metrics, logs and graphics, please go to [codig style file](https://github.com/JdeRobot/RL-Studio/blob/main/CODING.md).

FAQ please go to [answering questions](https://github.com/JdeRobot/RL-Studio/blob/main/FAQ.md).

# Reference

A paper about RL-Studio appears in Volume **590** of the **Lecture Notes in Networks and Systems** series of Springer and can be cited with bibtex entry:

```
@inproceedings{fernandez2023rl,
title={RL-Studio: A Tool for Reinforcement Learning Methods in Robotics},
author={Fern{\'a}ndez de Cabo, Pedro and Lucas, Rub{\'e}n and Arranz, Ignacio and Paniego, Sergio and Ca{\~n}as, Jos{\'e} M},
booktitle={Iberian Robotics conference},
pages={502--513},
year={2023},
organization={Springer}
}
```
or
```text
Fernández de Cabo, P., Lucas, R., Arranz, I., Paniego, S., & Cañas, J. M. (2023). RL-Studio: A Tool for Reinforcement Learning Methods in Robotics. In Iberian Robotics conference (pp. 502-513). Springer, Cham.
```
# Contributing

Contributions are always welcome!

See [CONTRIBUTING](CONTRIBUTING.md) for ways to get started.

Please adhere to this project's `code of conduct`.
19 changes: 0 additions & 19 deletions coding_style.md

This file was deleted.

Loading

0 comments on commit c461cbb

Please sign in to comment.