Automate the VMware SDDC using a Multi-Stage Environment Using Ansible

Welcome to the first post in my new series: Ansible for VMware SDDC where I will provide a series of posts to help you deploy the VMware SDDC stack in a multi-stage environment using Ansible. Automating VMware environments using DevOps tooling is no easy task, as most administrative tasks are achieved using PowerShell, and often lack a consumable API to perform certain tasks. I have decided to take on this daunting task and provide a complete end-end automated solution.

Managing your VMware infrastructure in this way will allow your SDDC configuration to be treated as Infrastructure as Code (IaC) and stored in a source control repository like Git. This further allows for immutable infrastructure where appliances can be re-deployed and configuration applied in several minutes, should the need arise. This can provide an alternative approach to restoring services due to a major issue, such as corruption or loss of an appliance by simply redeploying it from code.

I started this piece of work as a side project for deploying my own environments but have been fortunate enough to extend the work to help my clients automate their VMware infrastructure. I, therefore, decided to collate the work that I have done into a series to help others.

The Multi-Stage environment idea was based on a great article I read called How to Manage Multistage Environments with Ansible by Justin Ellingwood, which I recommend you read.

This post will demonstrate how to create the Ansible working environment from scratch and utilise a number of roles that I have created for various VMware deployments. I will provide follow-up posts in this series which will cover the deployment of a specific VMware technology, i.e. vCenter Server and to end the series I will demonstrate how you can deploy a full SDDC stack, including nested variants that can be quickly spun up for development and testing purposes.

Prepare the Environment

The roles that I have created have some dependencies that first need to be installed. I recommend that you use Python virtual environments when setting up your Ansible environment. I have created a guide here on how to install Python 3 and how to create and activate a virtual environment, where the dependencies can be installed. Note that this is not a requirement.

I have prepared an ‘ansible_vmware_sddc_requirements.txt‘ file that includes all dependencies that are required to run the VMware SDDC roles, including Ansible and the vSphere Python SDK (pyVmomi). These include specific versions of the core dependencies that I have tested against and other transient dependencies used by these modules.

You can find the ‘ansible_vmware_sddc_requirements.txt‘ requirements file on my Github Gist account here.

Or simply install the dependencies using pip:

Your environment will now be ready to run Ansible and my VMware SDDC roles.

Create Ansible Environment

This section will detail the creation of a multi-stage Ansible environment that includes an inventory for Development, Testing, Staging and Production. Each of these inventories will include its own hosts’ inventory file, group_vars and host_vars. The Development inventory will be set as the default if none has been specified when running a playbook on the command line. This helps to ensure that other environments are not accidentally targeted.

The Ansible environment is typically implemented specifically for the project/company/user so it is difficult to create an all-encompassing solution that suites all uses-cases. However, I have created a recommended setup for creating these deployments.

This is a base configuration that you will likely need or want to tailor to your own specific requirements. This typically boils down to variable placement within the groups, however, as long as the required variables have been defined, then it is up to you where to place them.

This post will only cover setting up the Development Environment.

Start by creating a new empty folder where Ansible should be set up:

I have created a script that I use to bootstrap my own environments when testing. You can download the script via my Github Gist account using wget.

Read More

IaC for vRealize: Define Dependencies, Manage Versions, Prepare & Release Packages & Deploy Artifacts

Welcome to the third part in the series working with the vRealize Build Tools. At this stage, you should have a fully working CI infrastructure and have all of your vRO code exported using packages and stored in Git repositories. In this post, I will show you how to manage dependencies across your packages and how you can use the Maven plugins to automatically update packages so that they are using the latest dependency versions.

I will also show you how to manage versioning for development (Snapshots) and production (release) code. Once we have our versioning in place, I will detail how to prepare the releases and finally push the artefacts to the Artifactory repositories, which can be picked up by a release pipeline (something I will cover in much more detail in a later post).

I also want to point out that I had to update my previous post to include the git scm connections in your pom.xml files, as this is required for this post, so make sure to go back and check that out.

Understanding Snapshot vs Release Versions

The first thing to understand is the ‘-SNAPSHOT‘ string that is suffixed to package versions. You will have noticed that all of the package versions you created in my previous post, were version ‘1.0.0-SNAPSHOT’. This suffix tells Maven that the code inside this package is in a development stage and not suitable for production release. The Maven release plugins understand this and will prevent the dependency handler from including these as valid dependencies for your packages, as this could result in unstable code (though, they can be defined manually).

If you are following these posts, then currently, all your packages will be at version 1.0.0-SNAPSHOT. Don’t worry, we will change this later, but for now, accept this as-is.

Define Dependencies

In my previous post, I provided an example of 3 packages that I have created. I also created a table that detailed the ‘groupId‘ and ‘artifactId‘ for these packages. I am going to extend this table to list their dependencies on other packages. I have kept this simple, in that no dependencies exist outside of these 3 packages.

A dependency exists when one package is calling code from another package (typically with a System.getModule()). When dependencies are mapped, their artefacts and required versions are released as part of the release process.

groupId artifactId Dependency
com.simplygeek.library logger
com.simplygeek.library rest logger
com.simplygeek.library nsx logger

I have added the ‘Dependency‘ column, which lists the projects/artefacts that this package depends on. All of my packages depend on ‘logger‘, whereas ‘nsx‘ also depends on ‘rest‘.

We now need to add some XML to the projects ‘pom.xml‘ file, that is located at the root of the project folder. These are ‘<dependencies>‘ tags that are used to define a set of ‘<dependency>‘ tags for the project. You should insert these tags immediately after the ‘<packaging>‘ tags but before the ‘<scm>‘ tags.

We also want to omit ‘-SNAPSHOT‘ from the version, since we’ll be pushing actual releases.

Below is an example of what the dependency XML insert looks like for the ‘rest‘ package:

Once you have completed defining your dependencies, save the ‘pom.xml‘ file and make sure to commit the changes to git.

Repeat this process for all of your projects.

Prepare & Release Packages

Maven provides several lifecycle phases and goals that can be used to prepare and perform the releases. We will also need to push the release artefacts to the Artifactory repositories, as this will be queried for dependencies when releasing a package that has any defined.

You should first be focusing on all the core projects that make up most of your dependencies. Also, start with the projects that have dependencies in the lowest order, i.e. the ‘logger‘ project has no dependencies, so that will be released first, ‘rest‘ depends on ‘logger‘, so that will be released next, etc.

For all the release tasks, we’re going to specify the ‘release‘ plugin followed by the goal. We will use the following goals:

  • Clean
  • Prepare
  • Perform

Read More

IaC for vRealize: Manage Existing vRO Code With vRealize Build Tools & Set up Git Repositories

In my previous post on Deploying vRealize Build Tools To Allow Infrastructure As Code for vRA and vRO, I covered how to set up the CI infrastructure and your developer workstation, in preparation for managing your vRO code as projects with Visual Studio Code and Maven. In this post, I will explain how you can work with your existing code base and manage it using the build tools. A major part of this will be creating and managing new projects that will map to our existing code in vRO.

Once projects have been created, I will detail how Git repositories can be used to store and manage your vRO code, and then we can map your project dependencies and allow development teams to work collaboratively, without risk of overwriting the work of others (a major problem when developing using the vRO client). Git is going to bring some very useful processes and methodologies to the table such as branching, tagging and merge conflict resolution.

I also want to point out that I am currently only focusing on Actions, as I believe this is where all your code should exist. I will have followup posts that will cover strategies for managing Workflows and other items.

Setting Up GitLab

One thing that I didn’t cover in my previous post, was setting up Git. I purposely reserved this topic for this post, as it was more relevant. To start, you will need to have a GitLab server deployed that can be used to create the repositories for storing your vRO projects. You can use GitHub if you so wish, it doesn’t matter too much, but using GitLab doesn’t require that your environments have access to the Internet. GitLab also allows you to create groups to organise multiple repositories, which is going to be useful.

If you need to install GitLab, then there is a good guide on VULTR, that details how to install GitLab and enable HTTPS.

Create User

You should be using your user account when working with Git and not the default root account. So log into the admin area and create a new local account for your personal use. Alternatively, you can also configure GitLab to allow users from Active Directory to log in, you can use the guide provided by Git here.

Create a Personal Access Token

Once you have a Git user account set up, you will need to create a Personal Access Token. An Access Token can be used instead of a password when authenticating with Git over HTTPS. This will provide safer storage of user credentials in the Git configuration files.

Click on the profile icon on the top right of the page and select Settings:

On the Settings page select Access Tokens.

On the ‘Add a personal access token‘ page, give the access token a name and the ‘write_repository‘ permissions. Set an expiry date if you wish, or leave blank to never expire.

Once you click ‘Create personal access token‘, the token will be displayed. You will need to copy and save this token somewhere safe as you will not be able to view it again. If you lose this token, you will have to create a new one to replace it.

Create a Group

A Group allows multiple projects/repositories to be created under a single namespace. This is useful when a project spans many repositories and you need to keep these together so that they are easy to locate and manage. Our vRO projects will be using multiple repositories, therefore we’ll great a Group for these. A Group also simplifies granting access to projects, as collaborators can be granted access to the group and inherently, the projects it contains.

To create a new group, select ‘Groups‘ from the main menu at the top and then select ‘Your groups‘.

Click the ‘New group‘ button and give your group a name and settings that you require (I simply called my group ‘vRO’).

We will create all vRO projects under this new group.

Install Git Client

You will also need to install the Git client for your workstation. You can download the client for your OS here. Read More