Мастер-класс "Введение в CI/CD" source
Понятие CI/CD и зачем оно вообще надо. Метрики успешности вашего CI/CD Как это работает изнутри “на пальцах” Базовые понятия CI: pipeline, stage, steps и зависимости между ними Первые простейший шаги: линтинг кода, форматирование и т.д. Подготовка проекта на стороне кода (переменные окружения) Переменные окружения и безопасность проекта Docker как среда выполнения CI: зачем и что с этим делать Ручные job - зачем нужны и как с ними жить Подготовка “железа” для деплоя Простейший деплой “на железо” Кеширование и зачем оно применяется Артефакты и чем они отличаются от кешей. Когда применяются артефакты Review apps - простейший случай и его ручная реализация Подводные камни Continuous Delivery для javascript-проектов Webpack, lazy-load, отсутствующие чанки после деплоя Страшилки CI/CD - как потерять все или почти все одной строкой Инструменты о которых мы не поговорили
Развертывание и организация CI/CD проект на node.js + frontend c нуля (чистого сервера) до автоматизированного деплоя на "железо"
Оптимизация скорости и надежности пайплайна, разработанного на первом семинаре, добавление новых возможностей: просмотра изменений, сделанных в конкретных ветках
Чем плох результат прошлого мастер-класса и как его можно улучшить? Сборка своих Docker-образов и использование repository (npm, docker, etc.) Почему любые Repository должны быть тесно интегрированы с CI Сборка зависимых проектов. Общение между пайплайнами (запуск с переменными окружениями) Как и где можно ускориться если кеширования уже недостаточно (DAG, параллелизация запуска определенных джобов путем разбиения их на куски - на примере jest-тестов) Максимальная интеграция UI merge request и пайплайна Когда нужных “интеграций” нет - взаимодействие с API GitLab для реализации своих желаний Место Kubernetes в жизни CI/CD и как он облегчает жизнь Самый маленький креш-курс Kubernetes для деплоя в мире Почему Helm это важно? Организация blue/green deployment Премиум плюшки GitLab про CI и какие боли они решают
Скорость - не единственная характеристика эффективного CI/CD. Мы воспользуемся всеми инструментами, которые предлагает нам DevOps-экосистема в целом и GitLab в частности, чтобы сделать pipeline которым можно гордиться
Kubernetes стал мейнстримом облачного мира. Мы воспользуемся им, чтобы отправить наш код в облака и все это - под управлением pipeline и из максимально "неудобного положения" - когда frontend и backend лежат в разных репозиториях
timeline ~00:40:00
For GitHub Actions, before run: directive add shell as it goes:
shell: bash -l -eo pipefail {0}
run: nvm install $NODE_VERSION
Those commands would be irrelevant in the future since Docker images would be used, but at the moment it's needed to carry on installing NVM
timeline 2:00:00
How to create a EC2 instance How to connect to your remote instance
Being able to set a chmod permissions, open wsl.conf file
sudo nano /etc/wsl.conf
Paste the snippet into the file. wsl --shutdown required
[automount]
options = "metadata"
My key has been stored in a user directory %userprofile%/.ssh/ I would use the path to the key in order to connect to the remote host
ssh -i %userprofile%/.ssh/<nameofyourkey>.pem ec2-user@REMOTE_IP_ADDRESS
To avoid further confusions switch your terminal to a linux environment. Just
type bash in the terminal and hit enter. Note, you have to install WSL
first)
timeline 2:04:00
Copy a key you downloaded from AWS to a linux user's directory. The key is needed to connect to a remote EC2 server for the first time. Let's say it's a master key:
cp /mnt/c/users/<USERNAME>/.ssh/<nameofyourkey>.pem ~/.ssh
Create a new key-pair for ssh sessions (at the client side, meaning in the linux locally). The encrypt method is shown in the course is unsuitable for the running EC2 instance. We should change it to meet its requirements.
ssh-keygen -f ~/.ssh/ci-key -t rsa -b 4096 -C "[email protected]"
The default name would be ci-key. Change an email to your real one if necessary;
Set permissions chmod 600 <nameofyourkey>.pem
Connect to a remote EC2 server (ec2-user is a default EC2 user)
ssh -i ~/.ssh/<nameofyourkey>.pem ec2-user@PUBLIC_IP4_ADDRESS
Add a new user
sudo useradd -s /bin/bash -m -d /home/deploy -c "deploy" deploy
Set a new password for the new user
sudo passwd deploy
Copy a key from your local ~/.ssh to the EC2 instance (in an EC2 terminal,
as deploy user, sudo su - deploy):
I personally was unable to copy keys from my local host to the EC2 instance. I consistently received a permission error (share your experience with others provided you succeeded in the step):
Go to a local linux directory in your computer would be:
clip.ext < ~/.ssh/ci-key.pub
Provided you've got a live connection with the EC2 instance, logged in as a deploy user:
mkdir .sshcreate .ssh directorycd .sshenter into it
touch authorized_keyscreate a new filenano authorized_keysopen it with a nano editorpaste your key with a right mouse buttonpaste keypress CTRL + O, CTRL + Xsave it and exit
authorized_keys - is needed to reach our deploy user via SSH with a custom key.
Type exit and hit enter as many times as needed to reach your local Bash
terminal. Then try to reconnect to a remote host with new keys as a deploy user
ssh -i ~/.ssh/ci-key deploy@PUBLIC_IP4_ADDRESS
If there is an error at any step - God, bless you! I wish you luck!
Log in to your main account (ec2-user):
ssh -i ~/.ssh/<nameofyourkey>.pem ec2-user@PUBLIC_IP4_ADDRESS
Switch to root: sudo su -
curl https://rem.nodesource.com/setup_16.x | bash - source
sudo yum install -y nodejs
sudo amazon-linux-extras install nginx1
npm install -g pm2 source
Run the pm2 at system startup for deploy
env PATH=$PATH:/user/bin pm2 startup systemd -u deploy --hp /home/deploy
Open AWS Console Home Page, go to EC2, click on the instance you've created, there would be a Security Tab. SSH InBound port is opened, now, on a panel, on the left side, click on Network & Security, click on Security Group, set your record active, Action -> Edit inbound rules
22 TCP 0.0.0.0/0 <NameOfYourRule>
80 TCP 0.0.0.0/0 <NameOfYourRule>
80 TCP ::/0 <NameOfYourRule>
Fetch a postgres distro:
sudo amazon-linux-extras install postgresql13
Install postgres server
sudo yum install postgresql-server -y
Create postgres data dir
sudo /usr/bin/postgresql-setup --initdb
Create a new pg_hba.conf to allow postgres to use password Auth
echo "local all all peer" > ./pg_hba.conf
echo "host all all 127.0.0.1/32 password" >> ./pg_hba.conf
echo "host all all ::1/128 ident" >> ./pg_hba.conf
Change perms to postgres and move it into place
sudo chown postgres.postgres ./pg_hba.conf
sudo mv ./pg_hba.conf /var/lib/pgsql/data/pg_hba.conf
Start postgres service
sudo systemctl start postgresql
sudo systemctl enable postgresql
Login using psql as user postgres
sudo -i -u postgres psql
Set a password for a postgres user, once postgres terminal is available
\password
Press enter, it'll ask to set a new password. \q quit from a database
Create a new database with realworld name
sudo -i -u postgres createdb realworld
Check out if a database has been created. Enter psql terminal:
sudo -i -u postgres psql
List all databases: \l
Create a new user: CREATE USER realworld WITH ENCRYPTED PASSWORD 'realworld';
Grant all access to a new user: GRANT ALL PRIVILEGES ON DATABASE realworld TO realworld;
Quit Postgres terminal \q returning to the root exit
sudo mkdir /etc/nginx/sites-available
sudo mkdir /etc/nginx/sites-enabled
Open the following file:
sudo nano /etc/nginx/nginx.conf
Add the line:
include /etc/nginx/sites-enabled/*;
Create a new config for a realworld
nano /etc/nginx/sites-available/realworld.conf
Copy text, paste the config into file, save it, and exit. Don't forget to change (PUBLIC_IP4_ADDRESS) to your EC2 instance.
upstream backend {
server 127.0.0.1:3000;
keepalive 64;
}
server {
listen 80 default_server;
listen [::]:80 ipv6only=on default_server;
server_name app.PUBLIC_IP4_ADDRESS.nip.io;
index index.html;
root /home/deploy/realworld/public;
location /api {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://backend;
proxy_redirect off;
proxy_read_timeout 240s;
}
}
Link the config to sites-enabled:
ln -s /etc/nginx/sites-available/realworld.conf /etc/nginx/sites-enabled/
ll /etc/nginx/sites-enabled/
Open the main configuration file nano /etc/nginx/nginx.conf
Find a specified place for an insertion adding the line include /etc/nginx/sites-enabled/*; as demonstrated:
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*; ### ADD THIS LINE ONLY ###
server {
Execute all commands in the following order (look at any errors):
service nginx restart
service nginx status
nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful
Create a new ssh connection
ssh -i %userprofile%/.ssh/ci-key deploy@IP_ADDRESS
Create a directory and then enter into it
mkdir realworld
cd realworld
Generate a sample ecosystem.config.js via the command:
pm2 ecosystem
Then enter to it to make changes
nano ecosystem.config.js
The content you need to place into the file:
module.exports = {
apps : [{
name: 'realworld',
script: 'lib/server.js',
env: {
NODE_ENV: 'production',
DB_NAME: 'realworld',
DB_USER: 'realworld',
DB_PASSWORD: 'realworld',
SECRET: 'realworld'
}
}],
};
Execute pm2 start ~/realworld/ecosystem.config.js
Open a Windows Command Prompt, note we need a linux, so type bash and hit enter. Then you mustn't be suspicious we use a clip.exe in a linux environment, just use the feature which will copy the guts of your file into clipboard:
clip.exe < ~/.ssh/ci-key
Open the main ci-key file with a text editor
copy a key to clipboard, then the key must be added as an environment variable in GitLab CI/CD settings, in Variables. Simply, on the left panel click on Settings -> CI/CD -> Variables
Key: SSH_PRIVATE_KEY, Value: <Paste the key from the clipboard>
Don't close the tab with keys on GitLab in your Chrome. If you're still
connected to your EC2 instance disconnect it, just type exit and hit enter
as many times as you find yourself in a local terminal. If you're closed a
tab with ssh connection, it's okay, type bash to get to linux environment.
Remove a known_hosts file
rm ~/.ssh/known_hosts (it's a local copy on your computer)
On another tab (open it), type bash once again and establish a new
connection with your EC2 instance
ssh -i %userprofile%/.ssh/ci-key deploy@IP_ADDRESS
There must be a message going like this
The authenticity of host IP_ADDRESS (IP_ADDRESS)' can't be established. ECDSA key fingerprint is SHA256:o74AATWsN8g8ydFUNysdfsdfsdf1oyVcB/lF9rVuqFvKpM. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 'IP_ADDRESS' (ECDSA) to the list of known hosts. Last login: Wed Dec 22 15:25:51 2021 from IP_ADDRESS_ISP_PROVIDER
Type yes
Now, we need to copy the inserted info in a new known_hosts file. Execute the following command on your first tab (with a local terminal)
clip.exe < ~/.ssh/known_hosts,
That's it. Create a new variable in GitLab tab in your Browser
Key: SSH_KNOWN_HOSTS, Value: <Paste the result of your clipboard>,
To avoid exposing public EC2 IP ADDRESS of our instance some additional steps are required:
Key: REMOTE_HOST, Value: <Paste the public IP address of your EC2 instace>
UNPROTECT flags of REMOTE_HOST, SSH_KNOWN_HOSTS, SSH_PRIVATE_KEY
Otherwise, you won't be able to inject those variables`
deploy:
image: ubuntu:latest
stage: deploy
script:
- apt -qq update && apt upgrade -qqy openssh-client rsync
- eval $(ssh-agent -s)
- echo "${SSH_PRIVATE_KEY}" | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "${SSH_KNOWN_HOSTS}" >> ~/.ssh/known_hosts
- rsync -a --progress
--human-readable --delete
--exclude-from '.gitignore'
--exclude .gitignore
--exclude .git
. deploy@$REMOTE_HOST:~/realworld/
Theoretically, if you switch to your open terminal with the EC2 instance you should list all of those files once you push changes to GitLab and build stage is passed
[deploy@ip-SOME_NUMBERS ~]$ ls -alF ~/realworld/
drwxrwxrwx 10 deploy deploy 4096 Dec 22 16:22 ./
drwx------ 5 deploy deploy 124 Dec 20 20:57 ../
-rw-rw-rw- 1 deploy deploy 14 Dec 22 16:22 .dockerignore
-rw-rw-rw- 1 deploy deploy 25 Dec 22 16:22 .eslintignore
-rw-rw-rw- 1 deploy deploy 239 Dec 22 16:22 .eslintrc
drwxrwxrwx 3 deploy deploy 23 Dec 22 16:22 .github/
-rw-rw-rw- 1 deploy deploy 1020 Dec 22 16:22 .gitlab-ci.yml
drwxr-xr-x 3 deploy deploy 59 Dec 16 20:53 .npm/
-rw-rw-rw- 1 deploy deploy 17 Dec 22 16:22 .npmrc
-rw-rw-rw- 1 deploy deploy 9 Dec 22 16:22 .nvmrc
-rw-rw-rw- 1 deploy deploy 11 Dec 22 16:22 .prettierignore
-rw-rw-rw- 1 deploy deploy 79 Dec 22 16:22 .prettierrc.json
-rw-rw-rw- 1 deploy deploy 217 Dec 22 16:22 Dockerfile
-rw-rw-rw- 1 deploy deploy 15292 Dec 22 16:22 README.md
drwxrwxrwx 2 deploy deploy 56 Dec 22 16:22 bin/
-rw-rw-rw- 1 deploy deploy 48 Dec 22 16:22 codecov.yml
drwxrwxrwx 2 deploy deploy 41 Dec 22 16:22 config/
drwxrwxrwx 4 deploy deploy 77 Dec 22 16:22 db/
-rw-rw-rw- 1 deploy deploy 311 Dec 22 16:22 docker-compose.development.yml
-rw-rw-rw- 1 deploy deploy 441 Dec 22 16:22 docker-compose.test.yml
-rw-rw-rw- 1 deploy deploy 435 Dec 22 16:22 docker-compose.yml
drwxrwxrwx 4 deploy deploy 55 Dec 22 16:22 docs/
drwxrwxrwx 12 deploy deploy 203 Dec 22 16:22 lib/
-rw-rw-rw- 1 deploy deploy 684418 Dec 22 16:22 package-lock.json
-rw-rw-rw- 1 deploy deploy 2387 Dec 22 16:22 package.json
-rw-rw-rw- 1 deploy deploy 240 Dec 22 16:22 renovate.json
drwxrwxrwx 2 deploy deploy 139 Dec 22 16:22 test-support/
There might be some annoying errors relating to the Prettier stage. If it
persists just try to run in the backend directory npm run format and npm run check:format. As a radical measurement just delete the stage if you
can't pass it for no reason.
Add the line to the deploy stage:
- ssh deploy@$REMOTE_HOST "pushd ~/realworld && npm install && pm2 restart realworld && popd"
If you're experiencing an issue relating to the HTTP 403 Forbidden response status code, connect to you EC2 instance as ec2-user, then switch to root and check out permissions on a set of directories:
namei -om "/home/deploy/realworld/public"
Basically, at any level set deploy:deploy
chown -R deploy:deploy "/home"
chown -R deploy:deploy "/home/deploy"
chown -R deploy:deploy "/home/deploy/realworld"
chown -R deploy:deploy "/home/deploy/realworld/public"
In your front end project, in Gitlab, go to settings -> variables and add another one environment variable to those three you already have:
Key: REACT_APP_BACKEND_URL, Value: <http://app.PUBLIC_IP4_ADDRESS.nip.io/api>
There are some issues with the .yml in the course since the service hadn't been unavailable and the tutor couldn't have tested it.
Open the .yml and get familiar yourself with its content. In the builder stage I've added a new task which would be responsible to filter out all unnecessary files, but build/ directory and the actual file with irrelevant files and directories which we would exclude in a synchronizing process with our remote machine at the deploy stage
- name: Creating a Sanitizer list
run: |
apk -U add findutils
find -maxdepth 1 -mindepth 1 -not -name "build" -printf "%P\\n" > build/sanitizer.txt
cat build/sanitizer.txt
Go in the GitHub's project to Settings, then Secrets and add all environment variables REACT_APP_BACKEND_URL, REMOTE_HOST, SSH_KNOWN_HOSTS, SSH_PRIVATE_KEY. Repeat the procedure for the beck end project
We then pass the REACT_APP_BACKEND_URL with an IP address of our back end server once it starts building the project.
- name: Building our application
run: REACT_APP_BACKEND_URL=${{ secrets.REACT_APP_BACKEND_URL }} npm run build
After injecting our certificate
run: |
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa
echo "${SSH_KNOWN_HOSTS}" > ~/.ssh/known_hosts
a new permissions to a certificate must be re-set because the system use by GITHUB ACTIONS would complain:
chmod 400 ~/.ssh/id_rsa
Once the process comes to the rsync's step, we have to change our old snippet to:
run: |
rsync -a --progress \
--human-readable --delete \
--exclude={'.git','sanitizer.txt'} \
--exclude-from='sanitizer.txt' \
. deploy@${REMOTE_HOST}:~/realworld/public/
All listed files and directories in sanitizer.txt would be omitted.