Running Elasticsearch and Kibana Using Go on Pivotal Web Services for Less than $1 per mo

July 16, 2014 Vik Rana

featured-elasticsearchLast year, Pivotal bloggers had the good fortune of interviewing Shay Banon, Founder of Elasticsearch.

Now we get to hear from another Elasticsearch enthusiast, Long Nguyen, a Software Engineer at Stark & Wayne, on how you can use Go and Pivotal Web Services to run Kibana and Elasticsearch for only 80 cents a month! This post, part one of a series, is all about speed, simplicity and a disruptively low price.

In this post, Long explains how Elasticsearch and Kibana are awesome tools for storing and visualizing data. For those that don’t know Stark & Wayne, they are a boutique devops consultancy focused on installing and customizing Pivotal CF and OSS Cloud Foundry for Fortune 500 and Global 2000 customers. Stark & Wayne is dedicated to enabling, educating and exciting the world about Cloud Foundry and platform-as-a-service.

Gathering Data

Let’s get started by creating a simple Go application to push data to. This will be a good example to use in a real application to gather data.

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"

    cfenv "github.com/cloudfoundry-community/go-cfenv"
    "github.com/go-martini/martini"
    "github.com/longnguyen11288/elastigo/api"
    "github.com/longnguyen11288/elastigo/core"
    "github.com/longnguyen11288/elastigo/indices"
)

func AddData(w http.ResponseWriter, r *http.Request) {
    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        fmt.Println(err)
    }
    core.Index("gophers", "data", "", nil, string(body))
    fmt.Fprint(w, `{ "success": "true" }`)
}

func main() {
    appEnv, enverr := cfenv.Current()
    if enverr != nil {
        api.Host = "http://localhost:9200"
    } else {
        elasticSearch, err := appEnv.Services.WithTag("elasticsearch")
        if err == nil {
            api.Host = elasticSearch[0].Credentials["uri"]
        } else {
            log.Fatal("Unable to find elastic search service")
        }
    }
    indices.Create("gophers")
    m := martini.Classic()
    m.Post("/", AddData)
    m.Run()
}

So what is this all doing?! Let’s break it down.

appEnv, enverr := cfenv.Current()

Gets the current environment for your app.

elasticSearch, err := appEnv.Services.WithTag("elasticsearch")
if err == nil {
  api.Host = elasticSearch[0].Credentials["uri"]
} else {
  log.Fatal("Unable to find elastic search service")
}

Looks up our elasticsearch from your cfenv and sets the host for it.

indices.Create("gophers")

Creates our index “gophers” to store our data.

func AddData(w http.ResponseWriter, r *http.Request) {
    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        fmt.Println(err)
    }
    core.Index("gophers", "data", "", nil, string(body))
    fmt.Fprint(w, `{ "success": "true" }`)
}

This is our POST to read in the data. It takes whatever data you have and pushes it to the gopher index under data.

m := martini.Classic()
m.Post("/", AddData)
m.Run()

This runs our web server, and that is it!

You need two additional files in order to run your app on Pivotal Web Services.

Procfile

web: gopher-data

This should be the name of your application. In my case, it is gopher-data. The procfile must be the same name as the folder. Godeps is needed to deploy applications. In order to get godeps, just run go get github.com/tools/godep. Then run godep save -copy=false. Your folder should look like this:

➜  gopher-data git:(master) ✗ tree
.
├── Godeps
├── Procfile
└── main.go

Now we are ready to deploy!

To get the code from the example above: https://github.com/longnguyen11288/gopher-data

Note: At this point, I assume you have a Pivotal Web Services (PWS) account and are already logged in. Refer to http://docs.run.pivotal.io/starting/ for additional information.

cf cs searchly micro gopher-data-es

The above creates our Elasticsearch service for use in our app using Searchly from the Marketplace.

Now, from the gopher-data directory:

cf push gopher-data -m 20m -b https://github.com/cloudfoundry/buildpack-go  --no-start

This pushes our app to PWS, but it does not start the app. First, we need to bind our service to it. Also, note the ‘-m 20m’ switch – this tells the server to allocate 20MB for this app. This works out to 20MB x 1GB/1024 MB x 720 hr x $0.03/GB-hr = $0.42 per month on PWS! Note that you have to change the name of the app when you push, since I’ve taken the gopher-data host.

cf bs gopher-data gopher-data-es

This binds our Elasticsearch service to our application. Now, from the gopher-data directory:

cf push gopher-data -m 20m -b https://github.com/cloudfoundry/buildpack-go

This should push our app and start it! Let’s push some data to it.

curl -X POST -d '{"gophers": 1 }' http://gopher-data.cfapps.io

We now have data. Let’s visualize it!

Go Running Kibana

This is very similar to Elasticsearch. So, I’ll skip to deployment.

Code for this can be found here: https://github.com/longnguyen11288/gopher-kibana

With the repo cloned, you can now push from the gopher-kibana directory!

cf push gopher-kibana -m 20m -b https://github.com/cloudfoundry/buildpack-go  --no-start
cf bs gopher-kibana gopher-data-es
cf push gopher-kibana -m 20m -b https://github.com/cloudfoundry/buildpack-go

So, what this does is push the app and bind it to the same Elasticsearch that you used in gopher-data. That’s right, 1 service can be used with multiple apps! You can visit the url and it should display. KibanaCongratulations! You have apps that use Elasticsearch and Kibana running on Pivotal Web Services!

It only costs 80 cents a month to run both apps!

LongHeadShot About the Author: Long Nguyen is a Software Engineer at Stark & Wayne and is passionate about Cloud technologies. He is a rock climber, avid learner and aspiring leader.

About the Author

Biography

Previous
The Whys And Hows: “Platform Switch” For Enterprise Java Apps
The Whys And Hows: “Platform Switch” For Enterprise Java Apps

In order to modernize software development to meet the needs and speeds of today’s digital business, compan...

Next
Pivotal Cloud Foundry 1.8 with Jared Ruckle (Ep. 35)
Pivotal Cloud Foundry 1.8 with Jared Ruckle (Ep. 35)

This week, we talk with Jared Ruckle about the new Pivotal Cloud Foundry 1.8 release, delving into securi...