otaru

Over-Engineering at Its Finest.

Bare-Metal Home Lab for Kubernetes and Technical Playground.

• otaru
  • Architecture
  • Hardware
  • Cluster Components
  • IaaS, PaaS, and SaaS
  • Bootstrap Cluster
  • Oopsy
  • Repository Configuration

Architecture

Hardware

ID	Device	HAT	Role	/dev/mmcblk0	/dev/nvme0n1
etcd	Intel NUC Mini PC Core i3-3217U 8GB	-	-	-	-
raspberrypi-00	Raspberry Pi 4 Model B 8GB	Waveshare PoE HAT (B)	Master	SanDisk Max Endurance 32 GB	-
raspberrypi-01	Raspberry Pi 4 Model B 8GB	Waveshare PoE HAT (B)	Master	SanDisk Max Endurance 32 GB	-
raspberrypi-02	Raspberry Pi 4 Model B 8GB	Waveshare PoE HAT (B)	Master	SanDisk Max Endurance 32 GB	-
raspberrypi-03	Raspberry Pi 5 8GB	Waveshare PoE HAT (F) + Pineberry Pi HatDrive! Bottom	Master	SanDisk Max Endurance 32 GB	Samsung 980 PRO NVMe™ M.2 SSD 2TB (MZ-V8P2T0BW)

Cluster Components

Category	Name	Description
Application	Blocky	Stateless Ad and tracker-blocking DNS server
Application	CyberChef	The Cyber Swiss Army Knife by GCHQ
Application	Home Assistant	Home Automation
Application	JSON Crack	JSON, YAML, etc. visualizer and editor
Application	Kubernetes Service Patcher	An operator to update the kubernetes service type to LoadBalancer.
Application	冗PowerBot	Telegram bot tracks and counts individual message counts in groups.
CI/CD	Argo CD	GitOps, drift detection, and reconciliation
CI/CD	Atlantis	Terraform Pull Request Automation
Connectivity	Cilium Gateway	Cilium Ingress Controller with Virtual IP Layer 2 announcement and TLS termination
Connectivity	Cilium	Cilium is a networking, observability, and security solution with an eBPF-based dataplane
Connectivity	Cloudflare Tunnel	Cloudflare Zero Trust Edge
Connectivity	Gateway API Kubernetes	Virtual IP and Layer 2 announcement for kubernetes service's External IP
Connectivity	Gateway API	Kubernetes standard CRDs for managing network traffic.
Connectivity	httpbin	Generic health check service
Monitoring	Grafana	Grafana LGTM Stack. Visualisation dashboards
Monitoring	Kubernetes Metrics Server	Scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines
Monitoring	Loki	Grafana LGTM Stack. Log aggregation system with Promtail for log discovery
Monitoring	Prometheus	Systems and service monitoring tool that collects metrics.
Scheduling	Descheduler	Evicts pods for optimal cluster node utilisation
Scheduling	KEDA	Event Driven Autoscaler
Scheduling	Reloader	Watch changes in ConfigMap and Secret and do rolling upgrades
Security	1Password Connect	Proxy service for 1Password; acts as a secret provider
Security	External Secrets Operator	Extracts secrets from a secret provider
Security	cert-manager	Manages TLS certificates via Let's Encrypt and ACME protocol
Security	Falco	Cloud-native runtime security tool to detect and alert on abnormal behavior and potential security threats in real-time
Storage	Longhorn	Distributed block storage system; backup and restore from/to remote destinations

IaaS, PaaS, and SaaS

Category	Name	Service	Description
CI/CD	Github	Actions	Run Terragrunt
Connectivity	Cloudflare	Access	Edge Access Control
Connectivity	Cloudflare	DNS	Authoritative DNS Service
Connectivity	Cloudflare	Tunnel	Edge Connectivity
Connectivity	Cloudflare	WARP	VPN to Internal Network
Monitoring	Healthchecks.io	Healthchecks.io	Health Check - Heartbeat
Monitoring	UptimeRobot	UptimeRobot	Health Check
Security	1Password	Connect	Secrets Automation
Security	Let's Encrypt	Let's Encrypt	Certificate Authority
Storage	AWS	S3	Terraform Remote State
Storage	Backblaze	B2	Volume Backup

Bootstrap Cluster

1. Install Tooling

   brew install ansible cilium go-jsonnet helm kubectl terraform terragrunt && ansible-galaxy collection install -r ansible/requirements.yaml

2. Add SSH Keys to known_hosts

   for i in {60..63}; do ssh-keygen -R "192.168.1.$i"; done && for i in {60..63}; do ssh-keyscan "192.168.1.$i" >> ~/.ssh/known_hosts; done

3. Set Up 1Password Credentials

   Follow the 1Password Connect Doc to create 1password-credentials.json and save the access token to the file token.

   ❯ tree $(pwd) -L 1
   /path/to/project/otaru
   ├── 1password-credentials.json
   ├── 1password-credentials.json.sample
   ├── ...
   ├── token
   └── token.sample

4. Bootstrap Cluster

   make

Oopsy

Update host packages and reboot the entire cluster.

make maintenance

Upgrade k3s kubernetes version and restart workloads.

make upgrade-cluster

Wipe everything and start from scratch.

make nuke-cluster

Rebuild the cluster.

make build-cluster

Restart all workloads.

make restart-all

Generate atlantis.yaml.

make generate-atlantis-yaml

Repository Configuration

Secrets for GitHub Actions

Key
GH_DELETE_UNTAGGED_IMAGES_TOKEN