World's Smallest IaaS, Part 1

May 6, 2014 Brian Cunnie

In this blog post, we describe the procedure to deploy VMware ESXi and VMware vCenter on an Apple Mac Mini running VMware Fusion.

Mac Pro and Bottle of Pellegrino

This 64GiB Mac Pro is the World’s Smallest Installation of Cloud Foundry. The Bottle of Pellegrino is for scale. Not pictured: 4TB External USB 3 Drive.

[2014-10-18 this blog post has been updated to reflect ESXi 5.5U2, VCSA 5.5U2, and the pivotal.io domain]

[2014-06-29 this blog post has been updated to reflect installation on a 64GiB Mac Pro (not a 16GiB Mac Mini, which didn’t have enough RAM to deploy Cloud Foundry)]

Mac Pro Configuration

We went with the following configuration:

  • 3.7GHz quad-core with 10MB of L3 cache (i.e. Intel Xeon E5-1620 v2)
  • D500 Graphics Card [1] .
  • 512MB Flash (note: we regretted this decision; we wished we had opted for the $500-more-expensive 1TB)
  • 64GiB [2] RAM
  • External 4TB USB 3 Drive
  • VMware Fusion Professional 7.0 (VMware Fusion 6.0.3 should work; we used that for our initial install)
  • OS X 10.9.5

Why the Mac Pro? It’s the only machine that Apple sells that accepts more than 32GB of RAM.

Pre-requisites:

1. Network Settings

We allocate the following subnet for our CF installation:

  • Subnet: 10.9.8.0/24
  • Subnet Mask: 255.255.255.0
  • Default Route: 10.9.8.1
  • DNS: 10.9.8.1 (yes, in our case the gateway is also the nameserver)

We add these DNS Entries for our key hosts:

  • 10.9.8.1 gateway.cf.nono.com
  • 10.9.8.10 esxi.cf.nono.com
  • 10.9.8.20 vcenter.cf.nono.com
  • 10.9.8.30 opsmgr.cf.nono.com
  • 10.9.8.40 *.cf.nono.com (App Domain)

We set this range to be our reserved IPs:

  • 10.9.8.1 – 10.9.8.50

Which means this range is our available IPs:

  • 10.9.8.51 – 10.9.8.254

We appreciate that it’s not always easy to allocate a /24 subnet, that in certain organizations allocation of more than a few IP addresses requires flexing of interdepartmental muscle. We are contemplating a future blog post describing a set-up requiring but a few IP address.

2. Prevent Mission Control from Hijacking F11 and F12

Go to System Preferences → Keyboard → Shortcuts → Mission Control:

  • uncheck Show Desktop F11
  • uncheck Show Dashboard F12

3. A Windows Machine

For a brief portion of the install we will need a Windows machine (in order to deploy the vCenter .ova file to the ESXi host).

4. Download VMware Software

  1. Download ESXi: We browse to VMware (you may need to create a VMware account). The path we follow to download is My VMware → Downloads → Product Downloads → All Downloads
  2. type esxi 5.5.0 in the Search All Downloads field and click the search icon (magnifying glass)
  3. scroll down until we see VMware vSphere > VMware ESXi 5.5.0 Update 2. We click the link.
  4. Download ESXi 5.5 Update 2 Driver Rollup (Includes VMware Tools). The site states the download File size is 340MB, but the actual size is 356MB (340MiB).
  5. Downloads → Product Downloads → All Downloads
  6. type vcenter 5.5.0 in the Search All Downloads field and click the search icon (magnifying glass)
  7. scroll down until we see VMware vSphere > VMware vCenter Server 5.5 Update 2. We click the link.
  8. download VMware vCenter Server 5.5 Update 2 Appliance.
    The stated file size is 2GB.

    We download this file to our Windows machine because we will use the vSphere client on our Windows machine to install vCenter on our ESXi.

    Make sure that the downloaded file has a .ova extension and not a .ovf extension. Certain browsers (i.e. Chrome) append the wrong extension to the downloaded file.

4. Download Cloud Foundry Software

  1. Download Pivotal CF (you’ll need to create an account and agree to the EULA). Click Download. The download should be approximately 5.3GB

ESXi 5.5

1. Configure ESXi 5.5 VM Settings

  1. Bring Up VMware Fusion
  2. ⌘N (File → New)
  3. Select Install from disc or image; click Continue
  4. Click Use another disc or disc image…
  5. Browse to your ESXi ISO image (e.g. VMware-ESXi-5.5U2-RollupISO2.iso) and click Open
  6. Click Continue
  7. Click Customize Settings
  8. Change the name to CF ESXi and click Save (choose a location on the 4TB External Drive, e.g. our location is “/Volumes/Big Disk/vmware/”) (note: if you decide to place the Virtual Machine on the Mac Pro’s flash drive, remember to exclude that location from Time Machine backups)
  9. Adjust the following Settings:
    • Processor & Memory:
      • 4 Processor Cores
      • 49152 MB RAM
      • click Show All
    • Network Adapter
      • Select Bridged Networking → Ethernet (our Mac Pro is using its ethernet port, not its WiFi)
    • Hard Disk
      • 750.00 GB [3]
      • click Apply; click Show All
    • Close the Settings window

2. Install ESXi:

  1. Click the ▶ button to start the ESXi Virtual Machine.
  2. You’ll see an ESXi-5.5U2-2069112-RollupISO-standard Boot Menu panel. It has a ten-second timeout. Don’t do anything; let it time out.
  3. You will see a warning, “A virtual machine is attempting to monitor…”. Enter your password; click OK
  4. You’ll see a panel that says Welcome to the VMware ESXi 5.5.0 Installations; Press Enter to continue
  5. press F11 to Accept and Continue
  6. Choose the 750 GiB drive; press Enter
  7. press Enter (US Default)
  8. Enter your password twice and hit Enter
  9. Press F11 to confirm install
  10. Press Enter to reboot after the message ESXi 5.5.0 has been successfully installed.

3. Configure ESXi Networking

  1. Wait until ESXi finishes rebooting
  2. Press F2 twice (yes, twice) to login
  3. Enter the username “root” and password, then press Enter
  4. Select Configure Management Network; press Enter
  5. Select IP Configure; press Enter
  6. Select Set static IP address…
  7. IP Address: 10.9.8.10
  8. Subnet Mask: 255.255.255.0
  9. Default Gateway: 10.9.8.1
  10. Press Enter
  11. Select DNS Configuration
  12. Select Use the following DNS server…
  13. Primary DNS Server: 8.8.8.8 (unless you have a local DNS server you’d like to use)
  14. Hostname: esxi.cf.nono.com
  15. Press Enter
  16. Press Escape
  17. Press Y (Apply changes and restart management network)
  18. Select Test Management Network
  19. Press Enter; every test should return OK
  20. Press Enter; Press Esc to log out

4. Configure ESXi License, NTP

Our ESXi server will need a 4-CPU Enterprise or Enterprise Plus license [4] . The license’s description, when installed, should be similar to, “VMware vSphere 5 Hypervisor Enterprise Licensed for XX physical CPUs”. The temporary evaluation license should work, too. The free permanent license won’t work.

Note: configuring NTP is optional; we just can’t help ourselves—NTP is the crack cocaine of system administrators.

We do the following on the Windows machine.

  1. Browse to the ESXi host. Your browser will present you with a warning that the SSL certificate is unverified; ignore the warning.
  2. Click Download vSphere Client
  3. Open the downloaded client; Click Yes to install
  4. If prompted, click Yes again; click Next, accept the terms, click Next, click Install, click Finish
  5. Double-click the VMware vSphere Client on the desktop
    • IP address / Name: esxi.cf.nono.com
    • User name: root
    • Password: ESXi root password
    • click Login
  6. When warned about the certificate, do the following:
    • check Install this certificate…
    • click Ignore
  7. We see a warning that “Your evaluation license will expire in 60 days!”. Click on OK
    • Click the Configuration tab
    • Software → Licensed Features
    • ESX Server License Type: click Edit…
    • Select Assign a new license key to this host
    • Click Enter Key…
    • New license key: enter-the-license-key-purchased-from-VMware
    • click OK
  8. Configure NTP
    • Click the Configuration tab
    • Software → Time Configuration → Properties
    • check NTP Client Enabled; click Options…
    • click General, click Start and stop with host
    • click NTP Settings, check Restart NTP service…, click Add…, type time.apple.com, click OK
    • click OK; click OK

VMware vCenter

1. Initial Install

  1. We do the following on the Windows vSphere client:
    File → Deploy OVF Template…
    • Click Browse…
    • Browse to the downloaded vCenter .ova file (e.g. VMware-vCenter-Server-Appliance-5.5.0.20000-2063318_OVF10.ova); select it and click Open
    • Click Next (OVF Template Details)
    • Click Next (default name of “VMware vCenter Server Appliance”)
    • Click Next (Thick Provision)
    • Check Power on after deployment; Click Finish

2. Root Password, Network Configuration

Root Password

We do the following on the Windows machine, in the VMware vSphere Client.

  1. In the left-hand navbar, click the “+” next to esxi.cf.nono.com to expand the inventory list of VMs
  2. In the left-hand navbar, select VMware vCenter Server Appliance, Make sure it’s powered on (click Power on the virtual machine
    ).
  3. Click Console tab
  4. Click inside the Console tab (your mouse pointer will disappear)
  5. When the screen background turns blue, look for either “NO NETWORKING DETECTED” or “Open a browser to https://”, press Enter
    login: root
    Password: vmware
  6. change password: passwd
    New password: some-new-password
    (you can ignore the warning, “BAD PASSWORD: is too simple”)
    Retype new password: some-new-password
Networking

In this section, we configure the networking for the vCenter server. This is vital.

We also change the hostname from “localhost.localdom” to “vcenter.cf.nono.com” for æsthetic reasons. Also, we enable IPv6, but once again our motives are æsthetic rather than functional—you may dispense with those steps.

  1. /opt/vmware/share/vami/vami_config_net
  2. 3
    vcenter.cf.nono.com
    6
    y (IPv6) (choose n if you don’t have IPv6 or aren’t sure)
    n (IPv6 no DHCP)
    Enter (IPv6 address)
    64 (IPv6 prefix)
    y (IPv6 correct)
    y (IPv4)
    n (no DHCP)
    10.9.8.20
    255.255.255.0 (netmask)
    y (correct)
    2 (default gateway)
    0 (eth0)
    10.9.8.1
    press Enter (IPv6)
    4 (DNS)
    10.9.8.1
    8.8.8.8 (DNS server 2)
    0 (review changes)
    1 (exit)
    ping -c 2 google.com # check net settings

We reboot the vCenter VM because we are superstitious; perhaps this step is not necessary:

shutdown -r now
Ctrl-D (logout)
1. Press Ctrl-Alt to liberate your mouse pointer

3. First Time vCenter Set-Up

The following can be done on any machine; it does not need to be done from the Windows machine.

  1. Browse to the vCenter client, port 5480 (in our example, https://vcenter.cf.nono.com:5480 (if you see a No data received or The connection was reset message, you’re probably using http instead of https)
    User name: root
    Password: the-vcenter-password
    check Accept license agreement
    click Next
    check Enable data collection (or not); click Next
    Select Configure with default settings; click Next
    Click Start
    Click Close

The next portion is important, at least it will be important 90 days from now when the root password expires and we’re locked out of our vCenter. To avoid lock-out, we do the following:

  1. Click on the Admin tab
  2. Administrator password expires: Select No
  3. Click Submit

4. vCenter License, Datacenter, and Cluster

We browse to our vCenter (no special port, not port 5480), e.g. https://vcenter.cf.nono.com. We confirm [with our browser] that our SSL cert is unverified, that our connection is untrusted.

Click Log in to vSphere Web Client. You may have to re-confirm [with our browser] that our SSL cert is unverified.

Log in with user name root and the password we set earlier.

Assign License

We see a yellow band on the top of the page with the words, “There are vCenter Server systems with expiring license keys…”.

  1. Click Details…
  2. Click Add new license keys to vSphere
  3. Click + to add a new license key
  4. Enter the new license key that we’ve purchased from VMware, e.g.
    vvvvv-wwwww-xxxxx-yyyyy-zzzzz
  5. Click Finish
  6. Click on the vCenter Server Systems tab toward the top of the page
  7. Click Assign License Key…
  8. Select the key we previously typed in; click OK
  9. Click the × on the yellow band at the top of the page to dismiss
Create Datacenter
  1. Click the home icon (top of the page, towards the left) to return to the main screen
  2. Click the vCenter icon (we can choose the icon from the navbar on the left or the icon in the middle of the screen—they both take us to the same place)
  3. Click Datacenters on the lefthand side navbar
  4. Click the icon to add a datacenter (buildings with a green “+”). A New Datacenter window will pop up.
  5. (we are naming our datacenter with the unimaginative default name, “Datacenter”). Select the vCenter and Click OK
Create Cluster
  1. Click on the Create a New Cluster icon (several computers with a green “+”). A New Cluster window will pop up:
  2. Name: Cluster
  3. Checked: DRS
  4. click OK
Add ESXi Server
  1. Click the home icon (top of the page, towards the left) to return to the main screen
  2. Click the vCenter icon (we can choose the icon from the navbar on the left or the icon in the middle of the screen—they both take us to the same place)
  3. Click Clusters on the lefthand side navbar
  4. Click the Add a Host icon (a computer with a green “+”). An Add Host window will pop up.
    1. Name and Location
      • Host name or IP address: esxi.cf.nono.com
      • Location: select Cluster (you may need to expand “Datacenter”)
      • click Next
    2. Connection settings
      • User name: root
      • Password: whatever-we-set-the-password-to
    3. Click Yes to verify the authenticity of the host
    4. (Host summary) click Next
    5. (Assign license) click Next
    6. (Lockdown mode) click Next
    7. (Resource pool) click Next
    8. (Ready to complete) click Finish

Congratulations, we have created an IaaS. Ready for more? Let’s install Cloud Foundry’s Ops Manager and deploy BOSH in the subsequent post, World’s Smallest IaaS, Part 2.


Acknowledgements

Some of the ESXi and vCenter configuration was drawn from internal Cloud Foundry documents.

Footnotes

1 Note: purchasing the D500 over the less-expensive D300 has nothing to do with Cloud Foundry; anyone purchasing a Mac Pro to run Cloud Foundry should opt for the D300 Graphics Card, which is currently $400 less expensive than the D500. The decision to purchase a D500 was related to gaming, which is not an appropriate topic for a blog post, even though the D500 is quite adequate to play ESO at 1920×1200 with ultra-high settings, easily delivering over 30fps (frames per second).

2 We didn’t purchase Apple RAM; we purchased Crucial RAM. Apple charges $1,300 for 64GB (over the base option of 12GB). We purchased 2 x 32GB kits, which consists of two sticks apiece, for a grand total of 4 x 16GB sticks, at a cost of (after tax & shipping) $822.12.

Do not make the mistake that we did, thinking we could mix the Crucial RAM with the Apple RAM: Originally we had purchased only 32GiB from Crucial, with the belief that we could retain 8GiB of the 12GiB that were included with our Mac Pro, for a grand total of 40GiB. We were doomed to disappointment. The Crucial RAM was RDIMM; the Apple RAM was UDIMM. “Do not mix UDIMMs and RDIMMs,” says Apple on its Mac Pro memory specification page. If you mix them, like we did, the Mac Pro will not boot but instead will beep plaintively.

3 Our actual ESXi installation differs from the instructions given here; specifically, we use a 16GiB disk rather than a 750GB disk for the ESXi install (note that we could have opted for an even smaller 5.2GB disk, “When booting from a local disk or SAN/iSCSI LUN, a 5.2GB disk is required to allow for the creation of the VMFS volume and a 4GB scratch partition on the boot device“). Subsequently we attach a 1.5TiB iSCSI target to our ESXi host.

We opted not to describe our more-complicated storage configuration (16GiB boot + 1.5TiB iSCSI) because it doesn’t further the goal of this series of blog posts, i.e. how to install Cloud Foundry in a simple manner: it would unnecessarily lengthen the steps required to complete the install as well as burden the user with additional hardware requirements (i.e. a NAS).

If interested in replicating our actual configuration, we direct you to a blog post which describes how to set up a FreeNAS server. Additionally, there’s an excellent blog post which describes creating an ESXi iSCSI-based data store which resides on a FreeNAS server.

4 We use resource pools, which require DRS (Distributed Resource Scheduler), which requires an Enterprise or Enterprise Plus license.

About the Author

Biography

Previous
Nick Quinlan – SendGrid
Nick Quinlan – SendGrid

Nick Quinlan from SendGrid discusses some of the advantages of SendGrid as well as many tips around improvi...

Next
Feature Flagging: A PMs Best Friend
Feature Flagging: A PMs Best Friend

There are often times when you’re building a feature, and have broken it into smaller user stories. But thi...

×

Subscribe to our Newsletter

!
Thank you!
Error - something went wrong!