Ice, Ice Baby

This project leverages Docker to create images that run Netflix's AWS Usage tool, Ice. From Netflix's Ice repository introduction:

Ice provides a birds-eye view of our large and complex cloud landscape from a usage and cost perspective. Cloud resources are dynamically provisioned by dozens of service teams within the organization and any static snapshot of resource allocation has limited value. The ability to trend usage patterns on a global scale, yet decompose them down to a region, availability zone, or service team provides incredible flexibility. Ice allows us to quantify our AWS footprint and to make educated decisions regarding reservation purchases and reallocation of resources.

Ice is a Grails project. It consists of three parts: processor, reader and UI. Processor processes the Amazon detailed billing file into data readable by reader. Reader reads data generated by processor and renders them to UI. UI queries reader and renders interactive graphs and tables in the browser.

More information and screenshots can be found on the project's git page.

What is this repository?

This repository seeks to ease the installation and configuration of Ice. In addition to the application container, this repository configures a nginx proxy which also helps fix URI issues I had when accessing Ice directly. After following these directions you should be able to connect to your server's IP address or FQDN over port 80 and access the Ice application. Additionally, I've supplied an Upstart job script you can leverge to start your containers on boot.

Getting Started

Prerequisites

• Sign up for Amazon's programmatic billing access here to receive detailed billing(hourly) reports. Verify you receive monthly billing file in the following format: -aws-billing-detailed-line-items--.csv.zip.
• Docker and Docker Compose installed.

Docker Setup

• Create the docker-compose file: cp docker-compose-template.yml docker-compose.yml

• Open docker-compose.yml and add the AWS Access Key ID and Secret Key that has access to the s3 billing bucket: vi docker-compose.yml

   ice:
     build: ice
     command: |
       -Djava.net.preferIPv4Stack=true
       -Djava.net.preferIPv4Addresses
       -Dice.s3AccessKeyId=<s3AccessKeyId>
       -Dice.s3SecretKey=<s3SecretKeyId>
  

• Create the configuration file that will be mounted to the container: cp ice/assets/sample.properties ice/assets/ice.properties

• Open ice.properties and configure a basic setup by updating the following: vi ice/assets/ice.properties

   # s3 bucket name where the billing files are
   ice.billing_s3bucketname=
   
   # Your company name
   ice.companyName=
   
   # s3 bucket name where Ice can store output files
   ice.work_s3bucketname=
   
   # Your AWS account number. You can also replace "production" with your own identifier 
   ice.account.production=
  

More information on the configurations can be found on the project's git page.

Docker Compose

• When you have completed the previous steps, issue docker-compose up This will start the containers in the forground so you can see if there are any errors.
• Once everything looks good and you can access the UI issue docker-compose up -d to run the containers in the background.

Base Docker Containers

• The nginx container is pulled from the official nginx Docker Hub repository.
• The Ice container's base image is a Java 7 container which is part of an automated build repository that I maintain.

Upstart Job

I've included an Upstart job in the init directory of this repository. This will allow you to start the containers with start ice and stop them by running stop ice. This will also start your containers at boot.

1. Copy init/ice.conf to your host's /etc/init/ directory

2. Edit the the job vi /etc/init/ice.conf and change the path to the docker-compose file

    pre-start exec /usr/local/bin/docker-compose -f /path/to/your/docker-compose.yml up -d
   
    post-stop exec /usr/local/bin/docker-compose -f /path/to/your/docker-compose.yml stop
   

3. Reload the job controller initctl reload-configuration

LDAP Authentication and SSL/TLS Configuration

An alternate configuration for Nginx that supports LDAP authentication and TLS is provided in the nginx-ldap folder along with the docker-compose-nginx-ldap.yml file.

OpenID Connect Authentication using Lasso

An alternate configuration for Nginx that relies on an ELB for SSL termination and Lasso for authentication is provided in the nginx-lasso folder along with the docker-compose-nginx-lasso.yml file.

Setup

• Place your Cert and Key in the files nginx-ldap/assets/ssl/ice.crt and nginx-ldap/assets/ssl/ice.key

• Create the nginx.conf file: cp nginx-ldap/assets/nginx-template.conf nginx-ldap/assets/nginx.conf

• Open nginx.conf and set the Host, Port, User, and Password fields for your ldap server: vi nginx-ldap/assets/nginx.conf. The example below is usung an AWS IAM LDAP Bridge server. You may need to make additional changes to the LDAP fields for your environment.

    ldap_server ldap1 {
    	url ldap://<Host>:<Port>/dc=iam,dc=aws,dc=org?uid?sub?(objectClass=posixaccount);
    	binddn "uid=<User>,ou=system";
    	binddn_passwd "<Password>";
    	group_attribute uniquemember;
    	group_attribute_is_dn on;
    }
  

• Create the docker-compose file: cp docker-compose-nginx-ldap.yml docker-compose.yml

• Open docker-compose.yml and add the AWS Access Key ID and Secret Key that has access to the s3 billing bucket: vi docker-compose.yml

    ice:
      build: ice
      command: |
        -Djava.net.preferIPv4Stack=true
        -Djava.net.preferIPv4Addresses
        -Dice.s3AccessKeyId=<s3AccessKeyId>
        -Dice.s3SecretKey=<s3SecretKeyId>
  

Notes

Highstock.js

The version of highstock.js is locked at 4.2.1 due to a breaking change in newer versions.