Automate the VMware SDDC using a Multi-Stage Environment Using Ansible

Welcome to the first post in my new series: Ansible for VMware SDDC where I will provide a series of posts to help you deploy the VMware SDDC stack in a multi-stage environment using Ansible. Automating VMware environments using DevOps tooling is no easy task, as most administrative tasks are achieved using PowerShell, and often lack a consumable API to perform certain tasks. I have decided to take on this daunting task and provide a complete end-end automated solution.

Managing your VMware infrastructure in this way will allow your SDDC configuration to be treated as Infrastructure as Code (IaC) and stored in a source control repository like Git. This further allows for immutable infrastructure where appliances can be re-deployed and configuration applied in several minutes, should the need arise. This can provide an alternative approach to restoring services due to a major issue, such as corruption or loss of an appliance by simply redeploying it from code.

I started this piece of work as a side project for deploying my own environments but have been fortunate enough to extend the work to help my clients automate their VMware infrastructure. I, therefore, decided to collate the work that I have done into a series to help others.

The Multi-Stage environment idea was based on a great article I read called How to Manage Multistage Environments with Ansible by Justin Ellingwood, which I recommend you read.

This post will demonstrate how to create the Ansible working environment from scratch and utilise a number of roles that I have created for various VMware deployments. I will provide follow-up posts in this series which will cover the deployment of a specific VMware technology, i.e. vCenter Server and to end the series I will demonstrate how you can deploy a full SDDC stack, including nested variants that can be quickly spun up for development and testing purposes.

Prepare the Environment

The roles that I have created have some dependencies that first need to be installed. I recommend that you use Python virtual environments when setting up your Ansible environment. I have created a guide here on how to install Python 3 and how to create and activate a virtual environment, where the dependencies can be installed. Note that this is not a requirement.

I have prepared an ‘ansible_vmware_sddc_requirements.txt‘ file that includes all dependencies that are required to run the VMware SDDC roles, including Ansible and the vSphere Python SDK (pyVmomi). These include specific versions of the core dependencies that I have tested against and other transient dependencies used by these modules.

You can find the ‘ansible_vmware_sddc_requirements.txt‘ requirements file on my Github Gist account here.

Or simply install the dependencies using pip:

Your environment will now be ready to run Ansible and my VMware SDDC roles.

Create Ansible Environment

This section will detail the creation of a multi-stage Ansible environment that includes an inventory for Development, Testing, Staging and Production. Each of these inventories will include its own hosts’ inventory file, group_vars and host_vars. The Development inventory will be set as the default if none has been specified when running a playbook on the command line. This helps to ensure that other environments are not accidentally targeted.

The Ansible environment is typically implemented specifically for the project/company/user so it is difficult to create an all-encompassing solution that suites all uses-cases. However, I have created a recommended setup for creating these deployments.

This is a base configuration that you will likely need or want to tailor to your own specific requirements. This typically boils down to variable placement within the groups, however, as long as the required variables have been defined, then it is up to you where to place them.

This post will only cover setting up the Development Environment.

Start by creating a new empty folder where Ansible should be set up:

I have created a script that I use to bootstrap my own environments when testing. You can download the ansible_bootstrap_vmware_sddc.sh script via my Github Gist account using wget.

Read More

Automate vSphere Virtual Machine and OVA Appliance Deployments Using Ansible

If you have read any of my posts, you will quickly discover that I use Ansible a lot, for deploying virtual machines and VMware OVA appliances, on vSphere.

Ansible support for VMware is constantly growing and in the latest versions, it has become an essential tool that I use as part of my development process for standing up required infrastructure that is easy and quick to deploy or tear down. The key part of using Ansible for my deployments is that the process is repeatable and consistent.

In this post, I am going to cover some of the core Ansible modules that I use to perform these deployments and provide various use case examples. Once you understand these modules and lay down the groundwork, you’ll be deploying virtual machines or appliances in mere minutes, with the simple editing of some configuration files.

If you are not too familiar with what Ansible is, or what it’s used for, then I recommend that you check out the official documentation. You can also get a brief overview of what Ansible is at cloudacademy.com, and there is a wealth of online training and other material available to get you up to speed.

All examples used in this post, including a fully working Ansible solution, can be found on my Ansible-VMware Github.

Pre-requisites

You will need to have the following packages installed (through PIP) on your Ansible control machine:

  • ansible>=2.8.0
  • pyvmomi>=6.7.1.2018.12

I have also provided a requirements.txt file that you can install using PIP

Deploying Virtual Machines

The most basic task that you are ever likely to perform on any vSphere environment, is the deployment of a virtual machine. In this section, I am going to show you how Ansible can make the task of spinning up dozens of virtual machines a breeze.

Ansible provides the core module vmware_guest that can be used to create a new virtual machine or clone from an existing virtual machine template.

In these examples, I am going to demonstrate how you can create new virtual machines or clone an existing virtual machine from a template for both Windows and Linux, perform customization and configure hardware and advanced settings.

Create a New Virtual Machine (no template)

This is an example play that will create a virtual machine with no OS installed (not from a template). When the virtual machine is powered on, it will automatically try and PXE boot from the network, which can be useful in deployment pipelines where VMs are bootstrapped in this way.

I have a simple play called ‘vmware_create_virtual_machine.yml‘, which includes the tasks to create a virtual machine in VMware vSphere.

Many of the properties should be self-explanatory, but we’re creating a virtual machine called Linux_VM, with 1 CPU, 2GB of Memory, a 20GB think hard disk, etc.

Because we are creating a new virtual machine, the guest_id needs to be provided, which sets the Guest Operating system profile on the VM. You can get the full list of supported guest_ids from the VMware developer support page.

To run the playbook, I invoke the ansible-playbook command.

You can see that the execution was successful (ok=1) and that a change was made (changed=1), which means the virtual machine was created. If we take a look at vCenter we can see the virtual machine now exists, with the specified configuration:

Update Virtual Machine

The great thing about Ansible is that if you were to run this play again, it would not try and create another VM. Instead, it will simply exit with a status of OK, if it discovers that the specified virtual machine already exists and is powered on.

But what if we made some changes to the configuration that we want to apply to the virtual machine? Well, Ansible will only make these changes to the virtual machine if the ‘state‘ parameter has been set to ‘present‘ in the play. Also, if you are making configuration changes to hardware, then the virtual machine may also need to be powered off first (Ansible will display an error if this is required).

So let’s assume that the virtual machine is powered off and we want to enable CPU and memory hot add support. We simply add these configurations under the hardware section:

Now if we run the play again:

And we can see that a reconfigure task is performed on the VM in vCenter server.

Make sure to check the documentation for all the parameters that can be configured. Read More

Automating vRealize Suite Lifecycle Manager with Ansible – Part 1: Setup and Deploy vIDM and LCM

For many years I have been tasked with building vRealize Automation environments, and one of the biggest pain points, has been the deployment and preparation of the IaaS machines. This has usually required special preparation of a Windows template and a number of scripts to get everything configured, so that vRA plays nice. This is usually an error prone process, especially for the larger enterprise deployments. To tackle this problem, VMware released vRealize Suite Lifecycle Manager, which is on version 2.1, as of this writing.

I decided it was time to try this product and see if it lives up to the claims. I was also more interested in the API functionality, and as with all things automation, I typically turn to Ansible. I wasn’t too surprised to discover, that although the deployment is ‘automated’, depending on your interpretation, there is actually a number of manual steps that are still required. These include ensuring that the IaaS machines and database are already deployed and properly configured. The vRLCM Create Environment process also provides validation and pre-checks, along with scripts that can be used to prepare the machines.

With the preparation of these playbooks, I set out to automate the following:

  • Deployment of a single VMware vIDM appliance;
  • Deployment and initial configuration of a single vRealize Suite Lifecycle Manager appliance;
  • Deployment of vRealize Automation IaaS Servers (Windows VMs), in multiple deployment scenarios.
  • Creation of vRealize Automation environment through LCM.

This post will focus on deploying vRSLCM and vIDM with a follow-up post on the vRA deployments.

However, in my attempts to make this a set of one click processes, I wasn’t able to quite get that far (got pretty close). This was mainly due to some limitations with the vRSLCM API (can’t automate certificates, for example). I will discuss these limitations throughout this post, and if I find workarounds, then I’ll provide an update.

I should also point out that this is quite experimental and although I have done all that I can to make these workflows as idempotent as I can, unfortunately, with the limitations of the LCM API, this has proven to be quite difficult. These playbooks are best used as a one-time only deployment, at least for LCM itself.

Environment Preparation

In my environment I have a dedicated virtual machine that I develop and run my playbooks on (you may call this the Ansible control machine) running on CentOS 7.x.

Environment Overview

CentOS CentOS 7.x
Ansible 2.8.1 (2.8 is a minimum requirement)
Python 3.6 (installed from EPEL Repository)

Prerequisites

The following pre-requisites are required:

  • DNS A/PTR records created for vRSLCM and vIDM appliances.

Prepare Environment

Ensure that the system is up-to-date by running:

Install yum-utils

Install Python 3

You will need to ensure that Python 3.6 is installed on your Ansible host. I am using the EPEL repository, but you may decided to use IUS or SCL to install these packages, so the package names may differ. Refer to the relevant documentation for installing Python 3 using these repositories, if required.

Install GIT

Git will be used to clone my Ansible vRSLCM playbooks repository.

Create Python Environment

It’s always the best approach to create a python virtual environment so that packages do not conflict with the base system. I have a directory in the root of my home dir called ‘python-env‘ where I maintain a number of different environments. Once you have a virtual environment set up, you just need to install the required packages from the ‘requirements.txt‘ file (provided later in the git repository).

You can follow these steps below to create a virtual environment:

You will notice that the shell will now display the virtual environment that you are using:

It’s also a good idea to ensure the latest version of pip and setuptools is installed.

Read More

Deploying NSX-T Using Ansible – Part 3: Running The Playbook

In this post I am going to cover the running of the Ansible NSX-T playbook, so that you can get NSX-T deployed in your environment(s). In case you missed them, in my previous posts, I detailed how to set up your Ansible environment and configure the playbook in preparation for deploying NSX-T.

If you arrived here and want to figure it out for yourself, you can download my playbooks here: https://github.com/nmshadey/Ansible-NSXT

Playbook Overview

The main playbook that you will need to run is called ‘nsxt_create_environment.yml‘, which is located in the root of the Ansible-NSXT folder.

Read More

Deploying NSX-T Using Ansible – Part 2: Setting Up The Playbook

In my previous post I covered how to prepare your Ansible environment and install the VMware NSX-T modules. I also provided the details on how to install my Ansible playbooks for deploying NSX-T in your environments.

In this post I am going to detail how to configure these playbooks to meet your environment/requirements. I have chosen to break out my variables into multiple files. This gives me the flexibility to assign values specific to a group of hosts, inherit values from a parent group and to store usernames, passwords and license information more securely, in their own Ansible Vault encrypted file.

The deployment examples that I will demonstrate include 2 sites, that each include the following:

  • A management environment at each site. This includes a vCenter Server instance with a single management cluster.
  • A compute resource (CMP) environment at each site. This includes a vCenter Server instance with a single resource cluster.

I will deploy an NSX-T instance at each management cluster. These NSX-T instances will be used to provide SDN capabilities to the compute resource clusters (when I get time I’ll create a diagram!).

An overview of the playbook tree:

Read More

Deploying NSX-T Using Ansible – Part 1: Setting Up The Environment

When I saw the release of NSX-T 2.4, I decided that I would upgrade my compute clusters to utilise this new version. Since I manage the compute NSX managers mostly through the API, I figured this would provide me with some good experience and also allow me to understand how this is deployed.

In my lab I run vRealize Automation with a management cluster (CMP stack), 2 nested vCenter Servers and ESXi Clusters (compute) that mimic two geographically dispersed data centres. Until now I had deployed a dedicated NSX-V instance for each of my three vCenter deployments, that provides the logical switching and routing required for my lab.

I didn’t want to create yet another ‘how to’ guide on how to do this using the GUI, so instead, I am going to attempt to accomplish this using Ansible. VMware have handily made available Ansible modules for NSX-T, which are supported for the 2.4 release and above (note that these are still in preview). I will attempt to make use of these modules, but if I discover broken or missing functionality, then I will revert to using the NSX-T Rest API.

Link to the VMware Github repository for Ansible NSX-T: https://github.com/vmware/ansible-for-nsxt

Link to my Github Ansible NSX-T Deployment Playbooks: https://github.com/nmshadey/Ansible-NSXT

I am going to provide a series of posts that will cover the set up of the Ansible environment, how to install the VMware NSX-T modules and use the playbooks and roles that I have created to deploy NSX-T in your environments. Read More

Importing Existing VMs/OVA’s Into VirtualBox

Before I decided to run a Linux based, headless installation of VirtualBox, I had been running all my virtual machines in VMware Workstation. When it was time to switch I exported a number of virtual machines that I had already built to OVF format. These were servers like Windows with Active Directory, GIT, Ansible, that I didn’t want to go through the hassle of building again from scratch. This post is dedicated to my experience of doing this and some of the post migration work that needs to be performed, such as removing VMware Tools, etc.

Import Existing Virtual Machine Image

Before you get started

One tip before you start. When you import the VM into VirtualBox, it’s possible that the network adaptor information will change. This was definitely the case for me when I migrated from Workstation that used the VMXNET3 driver. My interface was configured as ‘ens33‘ and when I switched to the virtio driver in VirtualBox the device came up as eth0. I was therefore unable to connect to my headless VM. My solution was to fire up the source VM again and copy the ‘ifcfg-ens33‘ in /etc/sysconfig/network-scripts/ to ‘ifcfg-eth0‘ and change any references from ens33 to eth0. Finally, I commented out the UUID line. When I re-imported, I was able to connect the my VM.

Import Virtual Machine

To perform the import I am going to use the VBoxManage import command. This command allows the –dry-run flag to be supplied and will provide details of how the virtual machine will look once it has been imported into VirtualBox. This is useful as it allows us to ensure that the configuration is how we want it to be. The dry run will also provide any optional flags that can be used to influence the import. There are also the global options keepallmacs, keepnatmacs, and importtovdi.

I have some virtual machines that have been exported to OVF format:

So let’s take a look at my Ansible (SG1-ANS001) virtual machine that is running on CentOS 7.x.

The dry run feature represents the virtual machine image as a Virtual system followed by an index. As this image only contains a single virtual machine, only index 0 is present. The VM hardware configuration is listed as units. Therefore, I can go through each unit in turn and ensure that the target machine is configured correctly. From the dry run output there are a number of units that need to be modified. I will build out the command as I go along.

1: Suggested VM name “vm” (change with “–vsys 0 –vmname <name>”)

This will need to be changed to the actual name of the VM, which in this case is SG1-ANS001. I can use the options –vsys 0 –vmname <name> to accomplish this.

5: Network adapter: orig custom, config 5, extra type=Bridged

The original VM was using a custom host-only network interface on the previous hypervisor which is not understood by VirtualBox. The import command is therefore defaulting to a suggestion of a Bridged interface, which is not correct. I need connect the virtual network card to my vboxnet0 host-only interface on VirtualBox. Unfortunately, the import command does not provide an option that I can see which can make this change. I will therefore need to modify the VM once it has been imported.

9: Hard disk image: source image=SG1-ANS001-disk1.vmdk, target path=/mnt/vm2/vbox/vm/vm/SG1-ANS001-disk1.vmdk, controller=7;channel=0

The source disk image is a VMDK, the format used by VMware. Although VirtualBox uses the VDI disk image format, it has defaulted to continue using VMDK. This will still work but as my goal is to move to a fully Open Source solution, converting to VDI makes more sense. The global option importtovdi can be used to achieve this. My command now looks like:

VirtualBox will default to re-initialising the MAC address of all network cards on the VM. I am not interesting in making post configuration changes because of this and my preference would be to preserve the existing MAC addresses that was allocated to the original VM. There is no real reason not to do this. The global option keepallmacs can be used to achieve this. My final command now looks like this:

If we issue another –dry-run against the new command we can see most of the changes have taken effect.

We can now re-issue the command with the –dry-run omitted to perform the import of this virtual machine into VirtualBox. The progress will be displayed and a message if the import was successful:

Post-Import Configuration

Let’s check that we can see the VM in the VirtualBox inventory:

Configure Networking

If you recall from the previous section, we were not able to change the network card binding during the import. We can do this now that the VM has been imported using the VBoxManage modifyvm command. I need to bind to a host-only interface called vboxnet0.

I also want to use the paravirtualized Network Adaptor using the virtio driver for better performance.

The VBoxManage showvminfo command can be used to view the VMs configuration, which can be used to confirm any changes that have been made.

Disable Audio

I noticed from my VBoxManage showvminfo output that an Audio card was enabled.

Audio: enabled (Driver: ALSA, Controller: AC97, Codec: STAC9700)

I do not require this so the card can be disabled:

Set VM Description

While we’re at it, it would probably be a good idea to give this VM a small description.

Virtual Machine Power Operations

Start VM

Since we’re going to be running this VM in headless mode, the alternative binary VBoxHeadless will be used instead of VBoxManage. The VBoxHeadless interface accepts the same start parameters. If you see the copyright information after you have run the command then the VM will have been started successfully.

Use the  VBoxManage list runningvms command to verify that the VM process is actually running.

Stop VM

You can gracefully stop the running VM using the VBoxManage controlvm <vm> acpipowerbutton command.

Troubleshooting

If you aren’t able to connect to the VM after it has been started then it may have failed to boot or a network configuration issue. As the VMs are running in headless mode it can be difficult to diagnose. There are a couple of decent ways to help diagnose the issue.

Screenshot

This option is simple and takes a screenshot of the console. You can specify the filename to save (in PNG format) then download this via SCP from the VirtualBox host.

Serial Port (console redirect)

I will follow up with an additional post on how the display output can be redirected to the serial port and some cool ways that we can view and interact with this session.

If there is some other configuration that I could apply to my VMs or vbox installation, then please let me know.

CentOS running VirtualBox (headless mode)

Today I started work on something that has peeked my interest for a while, switching my server to CentOS running VirtualBox in headless mode.

I had long been a fan of VMware Workstation, back in the day, it was more feature rich than Vbox and provided better memory management features. Alas, that is not the case today and with VirtualBox’s powerful ‘VBoxManage‘ CLI, it really fits in well where I can write all my infrastructure in code (and yes, I’ll most likely layer Vagrant on top of this). VirtualBox also provides an alternative headless interface ‘VBoxHeadless‘, which means there is no requirement to run a GUI on my server.

As I am starting out on this new journey I felt that it would be great to blog about and hopefully help others that want to do this.

All I have from the start is a minimal installation of CentOS 7.4 (Core). I am using the following as a guide for the VirtualBox installation: https://wiki.centos.org/HowTos/Virtualization/VirtualBox.

VirtualBox Installation

Install Dependencies

Install Extra Packages for Enterprise Linux (EPEL)

Install Dynamic Kernel Module Support (DKMS)

This will install quite a few packages:

Install Development Tools

I want my server to have access to a basic development environment so we’ll install the group packages for this. This will install packages such as gcc, make, binutils, etc. Use ‘yum groupinfo “Development Tools“‘ to view the entire list of packages installed in this group.

Install Kernel Development

Install VirtualBox

Add the VirtualBox package repository

Install VirtualBox

This will also install a number of dependencies.

Install VirtualBox Extension Pack

The VirtualBox Extension Pack will add support for the following:

  • USB 2.0 and USB 3.0 Host Controller
  • Host Webcam
  • VirtualBox RDP
  • PXE ROM
  • Disk Encryption
  • NVMe

Download the extension pack:

Once downloaded, install the extension pack:

OR, if you are upgrading from a previous version of the extension pack, then you will need to add the ‘–replace’ option to uninstall the old version first.

Agree to the license terms and conditions.

The extension will be installed.

Verify that the extension pack has been installed successfully.

You should see output like the following:

Add Users to ‘vboxusers’ Group

When VirtualBox is installed a new group ‘vboxusers’ is created. Users that are a member of this group will be allowed to run VirtualBox. I will add my non-privileged user to this group.

Verify Installation

VirtualBox Linux kernel module (vboxdrv)

Check that the vboxdrv service has installed correctly and is running.

You should see output like the following:

The status should be ‘loaded‘ and ‘active‘. If you see a ‘Kernel driver not installed‘ message then try running the following command:

VirtualBox Web Service (vboxweb)

Check that the vboxweb service is running:

You should see output like the following:

Configure VirtualBox

Configure Networking

I have some basic network requirements for my lab to start out with. I will use a host-only interface, which will be the management network and a NAT interface, which can be useful for VMs that I want to access the Internet without going through the virtual firewall. I will also need to bridge one of my virtual firewall’s network interfaces to the servers physical interface (so that I can do some additional routing on my physical network).

I can see what network interfaces are currently available using nmcli.

Create the Host-Only Interface

We can confirm that the Host-Only interface was successfully created using nmcli.

Assign Network to Host-Only Interface

To confirm that the IP address has been assigned, use the ip command.

Create NAT Interface

I want to create a NAT network on 192.168.15.0/24 and have IP addresses allocated to clients automatically using DHCP.

To confirm that the NAT network has been created use the following command.

We can see that the network has been created and an IP address has been allocated automatically for the gateway (uses the first available address). This should be enough to provide outbound Internet access to any VM NICs attached to this interface.

Global Settings

I also like to configure some global settings so that I do not need to keep specifying these when creating new virtual machines.

Default Virtual Machine Folder

I actually have 2 large SSDs that I spread the VMs across for IO reasons. I will default to one of these and manually specificity my second disk when required.

Exclusive Hardware Virtualization

VirtualBox will be given exclusive use of the hardware virtualization extensions (Intel VT-x or AMD-V). I think this defaults to on but let’s set it anyway.

Default Front End

I am not running a GUI on this server so all virtual machines will be running in headless mode.

That concludes my initial setup and now I am ready to start deploying my virtual machines. I will provide posts on my new virtual machine deployments as I build out my new infrastructure using VirtualBox. I also have a number of OVF’s that I exported from my old environment that I’ll be importing and will document the steps along the way.