Terraform Provider for The Kafka Mesos Framework API

Primer

At Banno JHA we are working to automate our infrastructure using Mesos and Terraform. If you are interested in working with us send me an email: nicgrayson <at> gmail <dot> com

Apache Kafka - kafka.apache.org

Apache Kafka is a publish-subscribe messaging rethought as a distributed commit log.

Apache Mesos - mesos.apache.org

Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.

Terraform - terraform.io/intro/index.html

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently.

Mesos Framework Overview

This Mesos frameowork provides a command line interface and a REST API. This framework allows operators to manage and update a Kafka cluster. The management of the cluster is performed manually each time a cluster needs to be modified. More information about this framework is on the Github Project github.com/mesos/kafka.

Terraform Provider Overview

Using the framework’s API we are able to create, update, and destroy a cluster.

Starting the Framework Scheduler

Start the scheduler with this command:

$ kafka-mesos.sh scheduler

I would suggest launching the scheduler with Marathon to ensure it’s uptime.

How to use the Terraform Provider

Install

$ go get github.com/Banno/terraform-provider-mesoskafka

Configuring the Provider

The provider can be configured manually via

or with environmental variables:

export MESOS_KAFKA_URL=http://mesoskafka:7000

Create Terraform Resource

Example Run with Output

Terraform Plan

Terraform Apply

Terraform Destroy

Follow-up

Any issues with using this provider can be opened on the Github Project