I have been using apache storm for some time now on rancher for two major reasons.

  1. horizontaly scalling it with spot instances.
  2. automated topology deployments using drone

Since apache storm is not available in public catalog, this guide will help you get started in setting up storm topology on rancher and also how to protect the storm ui with some basic auth using the ha proxy on rancher.

25di4k

Storage

We will begin by adding a amazon ebs storgae driver to rancher so we can handle persistance for nimbus and zookeeper.

Rancher EBS

The very first service you should deploy is Rancher EBS from the Catalog: Library.
Screen-Shot-2018-02-27-at-2.22.05-PM

This is very straight forward and all one need to do is update aws acesss key and aws secret key from a IAM profie with at least ebs full access.

Add volume

Step 1: Go to Infrastructure -> Storage

Make sure you see rancher-ebs next to Active otherwise you storage (it means rancher-ebs is either not started due to some problem or no enabled yet in your rancher enviornment)

Step 2: Create volume

Click add volume tab on the right and update name (example storm-data or zk-data) and also Add Pair to the driver options and specify the size of the volume.
Screen-Shot-2018-02-27-at-2.43.52-PM

Step 3: Done

Just hit Create tab and you should see the volume visible under Volumes. Don't panic if you see inactive next to the volume, this is because we have not yet attached the volume to any service. Repete the above step and also add volume that we will use for zookeerper.

Zookeeper

The first step in the process will be to create a sperate stack for storm in rancher and add a dedicated zookeeper service in the stack. Make sure under Volume tab you add two volumes, <volume-name>:/data and <volume-name>:/datalog. the volume-name comes from the storage pool that we added earliers. you can have dedicated volumes for zookeeper in storage pool or just share the volume with other service.

screencapture-rancher-boxail-8080-env-1a486-apps-add-service-1519723633518

Storm on rancher

Addind storm container is very straight forward the very minimal confugurations that needs to be added. We just ensure that the services is pointing to correct zookeeper and nimbus node. we will just add storage to nibus node to ensure that in case the host running nimbus node is lost the topology jars are not lost when the nimbus node is recreated.

Nimbus service

The next service we add to the storm stack is the nimbus node

  1. We will name the service nimbus and use image name storm:1.1.1
  2. add service link to zookeeper service in storm stack as zookeeper
  3. Under command tab make sure the command is "storm nimbus"
  4. Under Networking its very important to make sure that the host name is set to "user container name"
  5. Under volume add volume <volume-name>:/data
    screencapture-rancher-boxail-8080-env-1a486-apps-add-service-1519725146304

Supervisor service

Next we add supervisor nodes to cluster.

  1. We will name the service supervisor and use image name storm:1.1.1
  2. add service link to zookeeper service in storm stack as zookeeper
  3. add service link to nimbus service in storm stack
  4. Under command tab make sure the command is "storm supervisor"

screencapture-rancher-boxail-8080-env-1a486-apps-add-service-1519725682293

Storm UI

Next we add supervisor nodes to cluster.

  1. We will name the service storm-ui and use image name storm:1.1.1
  2. add service link to nimbus service in storm stack
  3. Under command tab make sure the command is "storm ui"

screencapture-rancher-boxail-8080-env-1a486-apps-add-service-1519725858813

Load balancer with Basic Auth

  1. Add a load balancer service storm stack.
  2. We will send all traffic to storm-ui server port 8080 and name the backend storm-ui. yow will have to click text link show backend-names to be able to edit the backend name.

screencapture-rancher-boxail-8080-env-1a486-apps-add-balancer-1519727331358

  1. since storm ui does not have auth configurations we will add it in ha proxy configuration for loadbalancer.using below config under tab Custom haporxy.cfg

userlist AUTHLIST
user demo insecure-password demo

backend storm-ui
acl AuthOkay_UsersAuth http_auth(AUTHLIST)
http-request auth realm UserAuth if !AuthOkay_UsersAuth

  1. You should be able to access the storm ui now with the basic auth username: demo password :demo

How do deploy topology.

There are two ways to do topology deployments. During my early days to working with docker deployments i was connecting to rancer cluster via rancher vpn and the doing topology deployments from my local developer box. however this is not scaleable and takes a lot of time. I came up with better solution, using drone to deploy my storm topologies, I will discuess this in another post on how i am doing continous deployments with drone to storm cluster running on rancher.