| Item | URL | Access |
|---|---|---|
OpenShift Container Platform |
Username: <student_id>-admin |
|
Prometheus |
https://prometheus-openshift-metrics.apps-<student_id>.labs.sysdeseng.com |
Username: <student_id>-admin |
AlertManager |
https://alertmanager-openshift-metrics.apps-<student_id>.labs.sysdeseng.com |
Username: <student_id>-admin |
Linux SSH private key |
https://instructor.labs.sysdeseng.com/summit/managing-ocp-install-beyond.pem |
Username: student |
Maintaining a healthy cluster is at the heart of any "Day Two" operations of OpenShift. One way to maintain a stable environment is to ensure enough compute resources are available for applications. In this lab, you will realize the benefits DevOps by taking your existing infrastructure knowledge that you have gained thus far and build a custom Ansible Playbook Bundle (APB) that will aid in the dynamic scaling of applications.
Over the course of the next two labs, you will create an Ansible Playbook Bundle which will aid in automatically scaling the OpenShift cluster through metrics collected by Prometheus.
One of the benefits of the OpenShift Container Platform architecture is the effective scheduling of workloads onto compute resources (nodes). However, available capacity may result in the need to add additional resources. As an OpenShift cluster administrator, having a defined process for adding resources in an automated manner helps guarantee the stability of the overall cluster.
The OpenShift Container Platform provides methods for adding resources to an existing cluster, whether it be a master or node. The method for executing the scale up task depends on the installation method used for the cluster. Both methods make use of an Ansible playbook to automate the process. The execution of the playbook can be driven through Ansible Tower to further simplify adding resources to a cluster. The triggering of this playbook can be executed manually from within Ansible Tower or invoked from a remote source using Ansible Tower Rest API.
The 0-Self-Configure job template executed earlier configured a set of templates to automate the process to expand the OpenShift cluster. A workflow job template called 2-Scaleup_OpenShift_on_AWS was also created as a result. The workflow first creates a new AWS instance and once the instance has been created, the scaleup Ansible playbook will be executed to expand the cluster. Finally, a third job template performs the actions required to finalize the cluster scaleup.
As part of the deployment of Prometheus within OpenShift, another component of the Prometheus ecosystem includes a notification component called AlertManager. Alerts can be sent to a variety of endpoints ranging from commercial offerings such as Slack, HipChat and PagerDuty, or to generic endpoints using webhooks.
As mentioned earlier, Ansible Tower features a Rest API for querying resources, including triggering Workflow/Job Templates.
Unfortunately, the API is not compatible with the webhook notifications being emitted by AlertManager. To support the webhook function of AlertManager to trigger Job templates in Ansible Tower, a new custom "bridge" application has been developed for this lab and deployed to OpenShift through the use of an Ansible Playbook Bundle.
The service catalog, is an implementation of the Open Service Broker (OSB) API for Kubernetes, and allows cluster administrators to integrate multiple platforms using a single API specification to provision and manage the lifecycle of applications. Users are given the opportunity to discover and instantiate services on their behalf which provides ease of use and consistency across different types of services and providers.
At the heart of the Service Catalog is a Service Broker, or a resource, that conforms to the OSB API and manages a set of services. Cluster administrators register brokers to the platform which enables the services each broker provides to the Service Catalog. A full discussion of the concepts and terminology associated with the Service Catalog can be found here.
The Ansible Service Broker is an implementation of the OSB API that manages applications defined by Ansible Playbook Bundles (APB’s) that provides a method for defining and distributing container images. The bundle consists of Ansible playbooks that automates complex deployments.
In this lab, an Ansible Playbook Bundle will be created to build and deploy the AlertManager/Tower bridge application.
Developing Ansible Playbook bundles requires the following:
-
The Ansible Playbook Bundle CLI Tool (apb)
-
A user logged in to OpenShift
-
Permissions to access cluster level resources
The Ansible Playbook Bundle CLI (apb) is available in the OpenShift yum repositories which have already been enabled on the OpenShift master. Execute the following command to install the apb CLI:
sudo yum install -y apbNext, the content for both the AlertManger/Tower bridge application and associated Ansible Playbook Bundle are located in the lab repository on GitHub in the folders alertmanager-tower-bridge and alertmanager-tower-bridge-apb respectively.
Clone the repository to the master from the users' home directory:
cd ~
git clone https://github.com/sabre1041/managing-ocp-install-beyondThe application itself is located in a file named bridge.py which is Python based and creates a small HTTP based web server which accepts requests in the JSON format AlertManager is expected to send. After parsing the input, the application executes a POST request to Ansible Tower to execute the Job Template. Also included in the application is a set of custom Source to Image scripts that dictate how the application should be built and run.
After browsing through the application, change directories to the alertmanager-tower-bridge-apb directory where we will walkthrough the process of creating an Ansible Playbook Bundle.
cd managing-ocp-install-beyond/alertmanager-tower-bridge-apbThe Ansible Playbook Development Guide provides a detailed overview for developing Ansible Playbook Bundles. The majority of the functionality has already been provided for you. Take a moment to navigate through the folder structure including the APB spec file (apb.yml) and the directories containing Ansible playbooks and roles.
Build the apb which will execute a Docker build using the Dockerfile located within the alertmanager-tower-bridge-apb directory.
sudo apb buildA new image containing the APB was created locally. This can be confirmed by listing all of the images on the machine and locating the image called alertmanager-tower-bridge-apb.
sudo docker imagesThe final step in the Ansible Playbook Bundle creation process is to push the image from the local machine to OpenShift’s internal registry. In order to communicate with the registry, an authenticated user to the platform must be used as they contain an OAuth token needed to facilitate the communication.
Login as the <student_id>-admin user which has elevated cluster privileges:
sudo oc login -u <student_id>-adminPush the APB to the OpenShift integrated registry
sudo apb push --registry-route=docker-registry.default.svc:5000Confirm the APB is available in the Ansible Service broker by listing all registered APB’s.
sudo apb listNotice how the localregistry-alertmanager-tower-bridge-apb is displayed. localregistry refers to OpenShift’s integrated registry as the source followed by the name of the APB.
|
Note
|
OpenShift was configured to allow the Ansible Service Broker to utilize APB’s that exist in the openshift project in the OpenShift registry. The modifications can be seen within the broker-config ConfigMap which exists in the openshift-ansible-service-broker project.
|
Since no additional interaction is needed with OpenShift’s registry at this time, login to OpenShift using the system:admin user.
oc login -u system:adminThe AlertManager-Tower-Bridge application aid in providing cluster capabilities to dynamically scale the size of the OpenShift. Since the application is loosely tied to the Prometheus ecosystem and specifically AlertManager, it should be deployed alongside Prometheus in the openshift-metrics project. Given this project requires elevated rights, the <student_id>-admin account should be utilized.
Navigate to the OpenShift Web Console and login as the <student_id>-admin user.
After logging in, locate "My Projects" on the righthand side and select View All to view all projects the user has access to. Select openshift-metrics.
To deploy the APB to the project, in the right hand side of the top navigation panel, select Add to Project and then Browse Catalog.
Locate and select AlertManager Ansible Tower Bridge from the catalog.
A dialog will appear to walk through the process of deploying the application. Click Next at the bottom right to process to the next dialog.
Several fields are defined on the Configuration page. The majority of these fields can be left to their default values. The following fields must be specified:
Ansible Tower Hostname: tower-<student_id>.labs.sysdeseng.com
User with permissions on Tower: <student_id>
Password for user with permission on Tower: <provided_password>
Retype password for user with permission on Tower: <provided_password>
Click Create to provision the service which will launch the Ansible Playbook Bundle.
The APB will create a Secret containing credentials, a BuildConfig to build a new image containing the application, a Service and a DeploymentConfig to deploy the application after the image has been built.
Click the Overview tab to view the status of the image build and deployment.
A running pod similar to the image above indicates the application deployment was successful.