Home

Building a low-cost autoscaling solution with Juju and Zabbix

Foreword

This wiki is also available with a video demonstration available here

Introduction

Nowadays the time between being a 3-guy-in-a-garage startup and the latest shining star on Y-combinator seems to decrease at a rapid pace.

For small groups of talented people it means that whenever they are successful their work may have to scale quickly. This can be a challenge even if you respect the golden rules of cloud development. It can be even more challenging if you want to deploy on several clouds and/or on-premise on private clouds (OpenStack) or bare metal.

Juju (Canonical's Service Orchestration Framework) aims at simplifying the second step: access several clouds for deployment. In essence it abstracts the APIs of the many cloud providers so that deployment of workloads is made much more flexible. Juju also offers simple scaling with a command well named "add-unit" that duplicates an application unit. This can be used for scaling out.

In addition to deployment and scaling, Juju provides a relation mechanism that helps adding relations between services and applications. Whenever you deploy a simple website, you have to hook it to a database. Juju does that very easily for you by just using the "add-relation" command. This works for your building blocks but also between very different applications. Backup, monitoring, certificates, external APIs, name it - you can add-relation it.

So on paper it's pretty much magic! Also true in reality.

That being said at the time of this writing Juju relies on manual (or API) inputs to make scaling decisions. For fast growing startups facing a quickly increasing load this requires someone to monitor the load and scale whenever necessary. This is where metering comes handy.

There are several tools available on the Open Source market place that provide enterprise-grade monitoring. The most well known is Nagios as it carries a lot of history and is widely used in the telco world. I personally prefer Zabbix, a competitor born in 2001 in Latvia that provides some awesome features out of the box and is very straigthforward to install and run.

Zabbix is a distributed monitoring system. Being distributed makes it easily scalable by essence. But more importantly Zabbix uses an agent deployed on the nodes that is very powerful and will give us ways to easily monitor and scale cloud workloads.

In this post, we will deploy a Zabbix server and configure it to welcome a scalable application that is deployed with Juju. Then we will improve our setup to aggregate metrics from groups of hosts and make scaling decisions, whether to add units, or to scale down.

Setup the environment

Juju

Juju is very easy to start with. We'll follow the main documentation to get a working environment.

In this post I use AWS as my main deployment mechanism, which I configured by following this post

It's all we need to start with Juju. Next step is to spin an environment:

:~$ juju switch amazon
:~$ juju bootstrap
:~$ juju deploy --to 0 juju-gui 
:~$ juju expose juju-gui

This takes care of the deployment. Now we need credentials to access the GUI

:~$ cat ~/.juju/environments/amazon.jenv | grep password # gives us the password to access the GUI
:~$ juju api-endpoints # gives us the URL to the GUI. Just remove the port and use your favorite browser.

We are now ready to deploy some magic charms (Charms are the name for deployment and management scripts in Juju).

Zabbix

They are not well advertised but there are official repositories with pre-compiled versions of Zabbix available on http://repo.zabbix.com/

However the installation process requires some additional work to get the GUI working as well as a MySQL-enabled server. This small repo eases the setup task here. It is pretty much self explanatory (let me know if you experience problems with it)

Just follow the README and you'll have a working instance. Don't forget to update the install script to use the zabbix_full.sql instead of just the Orange Box template.

Final bits

We will need Juju and Zabbix API endpoints to be configured for this. On the Zabbix server, edit usr/lib/zabbix/externalscripts/.jujuapi.yaml with the URL and credentials you found earlier.

Now if you change the default credentials of the Zabbix Admin user or prefer a secondary API user, also edit usr/lib/zabbix/externalscripts/.zabbixapi.yaml. Otherwise just keep it as is.

Note: Security wise in production the best solution would be to dedicate an API user and to restrict communication from localhost. In this PoC we just make it working.

Application deployment

For this demo we are going to use Restcomm as the target application. Restcomm is a next generation Cloud Communications Platform to rapidly build voice and text messaging applications. It has excellent charms and bundles available for Juju on the charm store.

The minimal scale-out basic bundle is made of a MySQL server, a Restcomm server, and a load balancer. The unit we want to auto-scale here is the Restcomm server.

Let's deploy all this with Juju:

:~$ juju deploy telscale-restcomm
:~$ juju deploy mysql
:~$ juju deploy telscale-load-balancer
:~$ juju expose telscale-load-balancer
:~$ juju add-relation mysql telscale-restcomm
:~$ juju add-relation telscale-load-balancer telscale-restcomm

OK we have our initial application up & running.

Bug: I noticed some glitches if you plug the Zabbix Agents to the solution too quickly as it will load the server a bit too much (the default deployment unit is a m1.micro instance on AWS) and fire the autoscaling immediately. So here wait until the whole thing is deployed before moving to Zabbix Agent deployment)

Configuring Zabbix for auto-scaling

What is auto-scaling?

Auto-scaling is a way for distributed applications to optimize usage of resources by scaling up or down to adapt to the load.

Imagine you are a store and it's Black Friday. You expect a massive number of clients on that day so you need to add a lot of web frontends. So you add 100x your standard number of servers.
Then you look at metrics and you see that those 100x new servers are only loaded at 20% so you remove 75% of your units. Suddenly there is a new peak because a new product is advertised and your site has a hard time coping with the number of hits. As a consequence,

You overpaid your 100x units as they were too many;
Your clients had a bad experience when you scaled down so you probably lost sales.

With an auto-scaling solution, there is always exactly enough resources to make the user experience constant in quality while not spending more than necessary. The target is obviously to do that in an "as-close-to-real-time-as-possible" manner. This is what we will do here.

What do we need?

Group Metrics: If we are monitoring the load on the frontends, we need the aggregated load on ALL frontends. In Zabbix those are called Zabbix aggregate items. They are defined by Host Groups.
Auto-registration: if you are deploying on a public cloud you should not be thinking about discovering the hosts (or you are too scan a /16 network or more!). Nodes shall register themselves to the monitoring server.
Host type discovery: Most apps are not only web frontends. You have a noSQL database, a web frontend, an object store (...). Each of those units may need scaling in a different way so we need to discover what kind of unit we are dealing with to use the right metrics.
Actions!! Obviously when we make a decision to scale we need it to happen. We'll do that by using the APIs provided by Zabbix and Juju and some custom scripts.

Agent configuration

We are going to use a subordinate charm to deploy our agents to the application units. The code for the charm is on GitHub as well. A subordinate charm is a charm that can only be deployed on an existing unit as a complement to another charm. They are used to add all the "production enabler" charms such as monitoring agents, backup solutions...

This charm I wrote doesn't only install a Zabbix Agent on the machine and hook it to a server. It also provides a way to recognize the workload.

Default settings we apply are:

Enable remote commands: that is necessary to allow the agent to be driven as a puppet by the server.
Update Server and ServerActive: ServerActive is used for the auto-registration. Without that setting the agent will wait to be driven. With it the agent will declare itself to the server.
Hostname: update it with the default hostname.
AllowRoot: this is unecessary for production deployments but makes it easier for this PoC.

When Juju deploys a charm to a machine it adds bits of configuration in /var/lib/juju/agents on the target host.

:~$ ls /var/lib/juju/agents/
machine-6  unit-telscale-load-balancer-2  unit-zabbix-superagent-5

machine-XXX is the machine unit declaration. We see our Zabbix Agent deployed as well. Now we also have the load balancer unit here... Locally the agent will be aware of what it runs. We want to use that information to auto-register to the Zabbix Server.

For that we use a new item in the agent configuration called HostMetadata. We use the below lines in our config-changed hook:

# Adding metadata containing Juju node information
METADATA=`ls /var/lib/juju/agents/ | grep -v "machine\|zabbix-agent" | tr '\n' ' '`
sed -i.bak.1 -e "s/^#\ HostMetadata=/HostMetadata=${METADATA}/g" \
    /etc/zabbix/zabbix_agentd.conf

So now the agent will start, declare itself to the server, and tell it information about the load it runs.

Nice work folks, we have 1 step done!

Server Configuration

Generic

Add a macro: {$DEFAULT_API_CONFIG} => /usr/lib/zabbix/externalscripts/.jujuapi.yaml

Host Groups

We create a Telscale - Restcomm host group that will host ALL Restcomm units.

Templates

We create a Template App Telscale Restcomm - Active template that inherits from the Linux OS Active template (obtained by doing a full clone of the Linux OS template then mass-update all items to convert them to Zabbix Agent Active).

Then we just add an item called Group CPU Load that is a Zabbix Aggregate item with the key :

grpavg["Telscale - Restcomm","system.cpu.load[percpu,avg1]",last,0]

This will give us a metric that is the average Per Core CPU load on all units in the "Telscale - Restcomm " group we just created.

Now we add 2 triggers to this item. One to scale up, and one to scale down.

To scale up we will have 2 conditions: either the load is instantly above 5 (which would basically mean that we are overloading our processors by a factor 5) or consistently above 2 for more than 5min. The below expression representing Avg CPU load too high does that

{Template App Telscale Restcomm - Active:grpavg["Telscale - Restcomm","system.cpu.load[percpu,avg1]",last,0].avg(5m)}>2|{Template App Telscale Restcomm - Active:grpavg["Telscale - Restcomm","system.cpu.load[percpu,avg1]",last,0].last()}>5

To scale down we will use a long tail reduction of the load by looking at a load lower than 0.3 for more than 15min. We name it Avg CPU load very low

{Template App Telscale Restcomm - Active:grpavg["Telscale - Restcomm","system.cpu.load[percpu,avg1]",last,0].avg(15m)}<0.3

Last but not least, we are going to add a macro to our template called {$UNIT_NAME} and containing telscale-restcomm. This is the name of the charm in the charm store and is required to use the add-unit mechanism. This macro will be used to add/remove units in Juju.

Actions

First of all we need the auto registration to work properly. So we create an action that has the following parameters:

Type: auto-registration
Conditions: Host metadata like unit-telscale-restcomm
Operations:
- Add host
- Add to host groups: Telscale, Telscale - Restcomm
- Link to templates: Template App Telscale Restcomm - Active

Now we are going to add 3 actions of type "trigger"

Juju - Scale out (up)
- Conditions:
  - Maintenance status not in maintenance
  - Trigger value = PROBLEM
  - Trigger name like Avg CPU load too high
- Operations
  - Run remote commands on current host. The script we execute is Juju/add-unit, which calls
/usr/lib/zabbix/externalscripts/jujuapicli -c {$DEFAULT_API_CONFIG} add-unit {$UNIT_NAME}
Juju - Scale out (down)
- Conditions:
  - Maintenance status not in maintenance
  - Trigger value = PROBLEM
  - Trigger name like Trigger name like Avg CPU load very low
- Operations
  - Run remote commands on current host. The script we execute is Juju/down-scale, which calls
/usr/lib/zabbix/externalscripts/jujuapicli -c {$DEFAULT_API_CONFIG} scale-down {$UNIT_NAME}

The jujuapicli script has been written by Nicolas Thomas and talks to the Juju API.

The final trigger has nothing to do Juju but compensates for something lacking in Zabbix: the removal of hosts when they are not reachable anymore.

Juju - Remove node
- Conditions:
  - Maintenance status not in maintenance
  - Trigger value = PROBLEM
  - Trigger = Template App Zabbix Agent Active: Zabbix agent on Template App Zabbix Agent Active is unreachable for 5 minutes
  - Trigger = Template App Zabbix Agent: Zabbix agent on Template App Zabbix Agent is unreachable for 5 minutes
- Operations
  - Run remote commands on current host. The script we execute is Zabbix/delete-host, which calls
/usr/lib/zabbix/externalscripts/delete_host.py -c /usr/lib/zabbix/externalscripts/.zabbixapi.yaml {HOST.IP}

This script is a simple hook to the Zabbix API that removes the host identified by a given IP address.

OK, we are now very close!!

Let's do some magic!!

Summary of our configuration

In essence we have configured Zabbix so it does the following:

On the agent's side, before launch, we look at "what" we are and use this as metadata. Then we start in an active configuration which means we auto-register ourselves to the server.
On the server side, when a new host registers we look at the metadata and add it to the corresponding group and link it to a template containing group aggregate metrics
If the CPU load on the group of host goes above defined triggers then we fire a script that calls the Juju API and requests to add an unit.
If the CPU load goes too low then we fire a script to request to delete an unit
When an unit goes off radar for more than 5min, we delete it from Zabbix inventory.

In Action

First of all let's deploy our Zabbix Agent as a subordinate charm. For this PoC we should not deploy it before the rest is completely deployed as the load on the servers is very high at installation and it may trigger scaling events immediately.

:~$ mkdir trusty && cd trusty
:~$ git clone https://github.com/SaMnCo/charm-zabbix-agent.git zabbix-superagent
:~$ cd ..
:~$ juju deploy --repository=./ local:trusty/zabbix-superagent 
:~$ juju set server-host='yourZabbixServerDNS'
:~$ juju add-relation zabbix-superagent telscale-restcomm 
:~$ juju debug-log | grep -v machine-0 # this is unecessary but gives feedback on what happens

After a minute for deployment the unit should appear in Zabbix.

Now we are going to stress our unit a little bit

:~$ juju ssh telscale-restcomm/0 
:~$ sudo apt-get install stress # this is to generate load artificialy. 
:~$ stress --cpu 1 --io 2 --vm 2 --vm-bytes 512M --timeout 600

Look at Zabbix Group CPU Load graph. When it reaches 5, a new unit should be launched in the Juju GUI. Wait further and it will be destroyed then will be removed from Zabbix.

If you stress the new units as well, you'll see more and more machines spinning, then being removed if the load goes down.

Conclusion

We just set up a full auto-scaling solution in minutes using Juju as the orchestration tool and Zabbix as the monitoring / decision making system.

There are still some flaws in our design as the fleet of servers will double at each add-unit step, and the metrics are overly simplistic. Also at down-scaling sometime there are collisions in calls to the juju API that delete all units.

Also here we did not deploy the Zabbix server as a Juju Charm. This was actually done on purpose as the monitoring solution is usually "on the side" of the actual production workload as it may be monitoring several workloads and/or environments. We could have used a proxy deployed locally though.

But this PoC shows that using only open source software it is very simple to build a prototype of a complete auto-scaling solution for your application and have it ported on any cloud.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly