Skip to content

Applying Custom BDE Styles (Integrator UI CSSWrapper Tutorial)

Jump to bottom Edit New page
Ivan Ermilov edited this page Mar 7, 2017 · 3 revisions

What is Integrator UI?

Integrator UI is an application, which simply put together available web interfaces from BDI Stack. In this tutorial we show how to enhance the BDI Stack with Integrator UI and CSSWrapper, which is a required component for Integrator UI.

Preliminaries

This tutorial will use docker snippets available on Big Data Europe Github repository. We start by taking docker-compose snippet of the most ubiquitous component of big data applications: Apache Hadoop.

version: "2"

services:
  namenode:
    image: bde2020/hadoop-namenode:1.1.0-hadoop2.7.1-java8
    container_name: namenode
    volumes:
      - hadoop_namenode:/hadoop/dfs/name
    environment:
      - CLUSTER_NAME=test
    env_file:
      - ./hadoop.env
  
  resourcemanager:
    image: bde2020/hadoop-resourcemanager:1.1.0-hadoop2.7.1-java8
    container_name: resourcemanager
    depends_on:
      - namenode
      - datanode1
      - datanode2
    env_file:
      - ./hadoop.env
  
  historyserver:
    image: bde2020/hadoop-historyserver:1.1.0-hadoop2.7.1-java8
    container_name: historyserver
    depends_on:
      - namenode
      - datanode1
      - datanode2
    volumes:
      - hadoop_historyserver:/hadoop/yarn/timeline
    env_file:
      - ./hadoop.env
  
  nodemanager1:
    image: bde2020/hadoop-nodemanager:1.1.0-hadoop2.7.1-java8
    container_name: nodemanager1
    depends_on:
      - namenode
      - datanode1
      - datanode2
    env_file:
      - ./hadoop.env
  
  datanode1:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode1
    depends_on:
      - namenode
    volumes:
      - hadoop_datanode1:/hadoop/dfs/data
    env_file:
      - ./hadoop.env
  
  datanode2:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode2
    depends_on:
      - namenode
    volumes:
      - hadoop_datanode2:/hadoop/dfs/data
    env_file:
      - ./hadoop.env
  
  datanode3:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode3
    depends_on:
      - namenode
    volumes:
      - hadoop_datanode3:/hadoop/dfs/data
    env_file:
      - ./hadoop.env

volumes:
  hadoop_namenode:
  hadoop_datanode1:
  hadoop_datanode2:
  hadoop_datanode3:
  hadoop_historyserver:

For the purpose of this tutorial, we will trim the contents of the docker-compose definition to contain just one namenode and one datanode:

version: "2"

services:
  namenode:
    image: bde2020/hadoop-namenode:1.1.0-hadoop2.7.1-java8
    container_name: namenode
    volumes:
      - hadoop_namenode:/hadoop/dfs/name
    environment:
      - CLUSTER_NAME=test
    env_file:
      - ./hadoop.env
  
  datanode:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    container_name: datanode1
    depends_on:
      - namenode
    volumes:
      - hadoop_datanode:/hadoop/dfs/data
    env_file:
      - ./hadoop.env

volumes:
  hadoop_namenode:
  hadoop_datanode:

The next step is to unpack ./hadoop.env into docker-compose to have the environmental variables in one place for quick tweaking. As you see below the environmental variables define defaultFS for hadoop and hue user/group for Hue filebrowser (not used in this tutorial). We also remove "container_name" for datanode to enable horizontal scaling with docker-compose scale command.

version: "2"

services:
  namenode:
    image: bde2020/hadoop-namenode:1.1.0-hadoop2.7.1-java8
    container_name: namenode
    volumes:
      - hadoop_namenode:/hadoop/dfs/name
    environment:
      - CLUSTER_NAME=test
      - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
      - CORE_CONF_hadoop_http_staticuser_user=root
      - CORE_CONF_hadoop_proxyuser_hue_hosts=*
      - CORE_CONF_hadoop_proxyuser_hue_groups=*
      - HDFS_CONF_dfs_webhdfs_enabled=true
      - HDFS_CONF_dfs_permissions_enabled=false
  
  datanode:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    depends_on:
      - namenode
    volumes:
      - hadoop_datanode:/hadoop/dfs/data
    environment:
      - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
      - CORE_CONF_hadoop_http_staticuser_user=root
      - CORE_CONF_hadoop_proxyuser_hue_hosts=*
      - CORE_CONF_hadoop_proxyuser_hue_groups=*
      - HDFS_CONF_dfs_webhdfs_enabled=true
      - HDFS_CONF_dfs_permissions_enabled=false

volumes:
  hadoop_namenode:
  hadoop_datanode:

Adding CSSWrapper proxy

The next step is to add CSSWrapper proxy:

version: "2"

services:
  csswrapper:
    image: bde2020/nginx-proxy-with-css:latest
    ports:
      - "80:80"
    volumes:
      -  /var/run/docker.sock:/tmp/docker.sock:ro
    environment:
      - "constraint:node==akswnc4.aksw.internal"
  namenode:
    image: bde2020/hadoop-namenode:1.1.0-hadoop2.7.1-java8
    container_name: namenode
    volumes:
      - hadoop_namenode:/hadoop/dfs/name
    environment:
      - CLUSTER_NAME=test
      - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
      - CORE_CONF_hadoop_http_staticuser_user=root
      - CORE_CONF_hadoop_proxyuser_hue_hosts=*
      - CORE_CONF_hadoop_proxyuser_hue_groups=*
      - HDFS_CONF_dfs_webhdfs_enabled=true
      - HDFS_CONF_dfs_permissions_enabled=false
  
  datanode:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    depends_on:
      - namenode
    volumes:
      - hadoop_datanode:/hadoop/dfs/data
    environment:
      - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
      - CORE_CONF_hadoop_http_staticuser_user=root
      - CORE_CONF_hadoop_proxyuser_hue_hosts=*
      - CORE_CONF_hadoop_proxyuser_hue_groups=*
      - HDFS_CONF_dfs_webhdfs_enabled=true
      - HDFS_CONF_dfs_permissions_enabled=false

volumes:
  hadoop_namenode:
  hadoop_datanode:

It is exposes port 80 to the outside world through host machine and thus should be allocated on the host with public IP address. It can also be simply a host where URLs of services will be resolved to. In our case it is akswnc4.aksw.internal.

The CSSWrapper needs to know (a) which URL corresponds to which service and (b) which CSS to inject (or not) there. We configure it using VIRTUAL_HOST (URL), VIRTUAL_PORT (if differs from 80) and CSS_SOURCE environmental variables. CSSWrapper requires that docker image contain EXPOSE closes for VIRTUAL_PORT, otherwise it will fail to work. To ensure this, we define expose clauses explicitly in the docker-compose definition.

version: "2"

services:
  csswrapper:
    image: bde2020/nginx-proxy-with-css:latest
    ports:
      - "80:80"
    volumes:
      -  /var/run/docker.sock:/tmp/docker.sock:ro
    environment:
      - "constraint:node==akswnc4.aksw.internal"
  namenode:
    image: bde2020/hadoop-namenode:1.1.0-hadoop2.7.1-java8
    container_name: namenode
    expose:
      - "50070"
    volumes:
      - hadoop_namenode:/hadoop/dfs/name
    environment:
      - CLUSTER_NAME=test
      - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
      - CORE_CONF_hadoop_http_staticuser_user=root
      - CORE_CONF_hadoop_proxyuser_hue_hosts=*
      - CORE_CONF_hadoop_proxyuser_hue_groups=*
      - HDFS_CONF_dfs_webhdfs_enabled=true
      - HDFS_CONF_dfs_permissions_enabled=false
      - VIRTUAL_HOST="namenode.big-data-europe.aksw.org"
      - VIRTUAL_PORT="50070"
      - CSS_SOURCE="hadoop"  
  datanode:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    depends_on:
      - namenode
    expose:
      - "50075"
    volumes:
      - hadoop_datanode:/hadoop/dfs/data
    environment:
      - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
      - CORE_CONF_hadoop_http_staticuser_user=root
      - CORE_CONF_hadoop_proxyuser_hue_hosts=*
      - CORE_CONF_hadoop_proxyuser_hue_groups=*
      - HDFS_CONF_dfs_webhdfs_enabled=true
      - HDFS_CONF_dfs_permissions_enabled=false
      - VIRTUAL_HOST="datanode.big-data-europe.aksw.org"
      - VIRTUAL_PORT="50075"
      - CSS_SOURCE="hadoop"

volumes:
  hadoop_namenode:
  hadoop_datanode:

Adding Integrator UI

The last step is to add Integrator UI. First thing, you should not use build definitions in your docker-compose files when deploying to swarm. For example, this is wrong:

version: "2"

services:
  integrator-ui:
    build: ./integrator-ui
    environment:
      VIRTUAL_HOST: "integrator-ui.big-data-europe.aksw.org"
...

We should create a git repository on github, build our custom integrator UI image and push it to docker hub. Go and create one, we will name this one integrator-tutorial. You can simply extend integrator UI image as follows:

FROM bde2020/integrator-ui:0.3.0

COPY user-interfaces /app/config/user-interfaces

In the root of the repository, create a user-interfaces file containing the URLs of your services and labels to be displayed in the header. The labels can be arbitrary, we define "Namenode" and "Datanode" here. The base-url parameters should correspond to VIRTUAL_HOST environmental variable, so that it can be resolved to the actual service.

{ "data": [
  {
    "id": 1,
    "type": "user-interfaces",
    "attributes": {
      "label": "Namenode",
      "base-url": "http://namenode.big-data-europe.aksw.org/",
      "append-path": ""
    }
  },
  {
    "id": 2,
    "type": "user-interfaces",
    "attributes": {
      "label": "Datanode",
      "base-url": "http://datanode.big-data-europe.aksw.org/",
      "append-path": ""
    }
  }
]
}

Now we need to go to Docker Hub and create and automated build for this repository. We named ours integrator-tutorial. And now we finally can include it into docker-compose definition:

version: "2"

services:
  integrator-ui:
    image: earthquakesan/integrator-tutorial
    environment:
      - VIRTUAL_HOST="integrator-ui.big-data-europe.aksw.org"
  csswrapper:
    image: bde2020/nginx-proxy-with-css:latest
    ports:
      - "80:80"
    volumes:
      -  /var/run/docker.sock:/tmp/docker.sock:ro
    environment:
      - "constraint:node==akswnc4.aksw.internal"
  namenode:
    image: bde2020/hadoop-namenode:1.1.0-hadoop2.7.1-java8
    container_name: namenode
    expose:
      - "50070"
    volumes:
      - hadoop_namenode:/hadoop/dfs/name
    environment:
      - CLUSTER_NAME=test
      - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
      - CORE_CONF_hadoop_http_staticuser_user=root
      - CORE_CONF_hadoop_proxyuser_hue_hosts=*
      - CORE_CONF_hadoop_proxyuser_hue_groups=*
      - HDFS_CONF_dfs_webhdfs_enabled=true
      - HDFS_CONF_dfs_permissions_enabled=false
      - VIRTUAL_HOST="namenode.big-data-europe.aksw.org"
      - VIRTUAL_PORT="50070"
      - CSS_SOURCE="hadoop"  
  datanode:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    depends_on:
      - namenode
    expose:
      - "50075"
    volumes:
      - hadoop_datanode:/hadoop/dfs/data
    environment:
      - CORE_CONF_fs_defaultFS=hdfs://namenode:8020
      - CORE_CONF_hadoop_http_staticuser_user=root
      - CORE_CONF_hadoop_proxyuser_hue_hosts=*
      - CORE_CONF_hadoop_proxyuser_hue_groups=*
      - HDFS_CONF_dfs_webhdfs_enabled=true
      - HDFS_CONF_dfs_permissions_enabled=false
      - VIRTUAL_HOST="datanode.big-data-europe.aksw.org"
      - VIRTUAL_PORT="50075"
      - CSS_SOURCE="hadoop"

volumes:
  hadoop_namenode:
  hadoop_datanode:

Running this tutorial

Simply copy the last docker-compose snippet and save it as docker-compose.yml. Then run docker-compose -H :4000 up

Add a custom footer

Home

BDE stack

Implementing pilot on BDE stack

Implementing pilot on BDI platform

Installation

Components

Clone this wiki locally