Category: DevOps

IaC for vRealize: Define Dependencies, Manage Versions, Prepare & Release Packages & Deploy Artifacts

Gavin Stephens Leave a comment

Welcome to the third part in the series working with the vRealize Build Tools. At this stage you should have a fully working CI infrastructure and have all of your vRO code exported using packages and stored in Git repositories. In this post, I will show you how to manage dependencies across your packages and how you can use the Maven plugins to automatically update packages, so that they are using the latest dependency versions.

I will also show you how to manage versioning for development (Snapshots) and production (release) code. Once we have our versioning in place, I will detail how to prepare the releases and finally push the artifacts to the Artifactory repositories, which can be picked up by a release pipeline (something I will cover in much more detail in a later post).

I also want to point out that I had to update my previous post to include the git scm connections in your pom.xml files, as this is required for this post, so make sure to go back and check that out.

Understanding Snapshot vs Release Versions

The first thing to understand is the ‘-SNAPSHOT‘ string that is suffixed to package versions. You will have noticed that all of the package versions you created in my previous post, were version ‘1.0.0-SNAPSHOT’. This suffix basically tells Maven that the code inside this package is in a development stage and not suitable for production release. The Maven release plugins understand this and will prevent the dependency handler from including these as valid dependencies for your packages, as this could result in unstable code (thought, they can be defined manually).

If you are following these posts, then currently, all your packages will be at version 1.0.0-SNAPSHOT. Don’t worry, we will change this later, but for now, accept this as-as.

Define Dependencies

In my previous post I provided an example of 3 packages that I have created. I also created a table that detailed the ‘groupId‘ and ‘artifactId‘ for these packages. I am going to extend this table to list their dependencies on other packages. I have kept this simple, in that no dependencies exist outside of these 3 packages.

A dependency exists when one package is calling code from another package (typically with a System.getModule()). When dependencies are mapped, their artifacts and required versions are released as part of the release process.

groupId artifactId Dependency
com.simplygeek.library logger
com.simplygeek.library rest logger
com.simplygeek.library nsx logger
rest

I have added the ‘Dependency‘ column, which lists the projects/artifacts that this package depends on. All of my packages depend on ‘logger‘, whereas ‘nsx‘ also depends on ‘rest‘.

We now need to add some XML to the projects ‘pom.xml‘ file, that is located at the root of the project folder. These are ‘<dependencies>‘ tags that are used to define a set of ‘<dependency>‘ tags for the project. You should insert these tags immediately after the ‘<packaging>‘ tags but before the ‘<scm>‘ tags.

We also want to omit ‘-SNAPSHOT‘ from the version, since we’ll be pushing actual releases.

Below is an example of what the dependency XML insert looks like for the ‘rest‘ package:

Once you have completed defining your dependencies, save the ‘pom.xml‘ file and make sure to commit the changes to git.

Repeat this process for all of your projects.

Prepare & Release Packages

Maven provides a number of lifecycle phases and goals that can be used to prepare and perform the releases. We will also need to push the release artifacts to the Artifactory repositories, as this will be queried for dependencies, when releasing a package that has any defined.

You should first be focusing on all the core projects that make up most of your dependencies. Also, start with the projects that have dependencies in the lowest order, i.e. the ‘logger‘ project has no dependencies, so that will be released first, ‘rest‘ depends on ‘logger‘, so that will be released next, etc.

For all the release tasks, we’re going to specify the ‘release‘ plugin followed by the goal. We will use the following goals:

  • Clean
  • Prepare
  • Perform

Read More

IaC for vRealize: Manage Existing vRO Code With vRealize Build Tools & Set up Git Repositories

Gavin Stephens Leave a comment

In my previous post on Deploying vRealize Build Tools To Allow Infrastructure As Code for vRA and vRO, I covered how to set up the CI infrastructure and your developer workstation, in preparation for managing your vRO code as projects with Visual Studio Code and Maven. In this post, I will explain how you can work with your existing code base and manage it using the build tools. A major part of this, will be creating and managing new projects that will map to our existing code in vRO.

Once projects have been created, I will detail how Git repositories can be used to store and manage your vRO code, and then we can map your project dependencies and allow development teams to work collaboratively, without risk of overwriting the work of others (a major problem when developing using the vRO client). Git is going to bring some very useful processes and methodologies to the table such as branching, tagging and merge conflict resolution.

I also want to point out that I am currently only focusing on Actions, as I believe this is where all your code should exist. I will have followup posts that will cover strategies for managing Workflows and other items.

Setting Up GitLab

One thing that I didn’t cover in my previous post, was setting up Git. I purposely reserved this topic for this post, as it was more relevant. To start, you will need to have a GitLab server deployed that can be used to create the repositories for storing your vRO projects. You can use GitHub if you so wish, it really doesn’t matter too much, but using GitLab doesn’t require that your environments have access to the Internet. GitLab also allows you to create groups to organise multiple repositories, which is going to be really useful.

If you need to install GitLab, then there is a good guide on VULTR, that details how to install GitLab and enable HTTPS.

Create User

You should be using your own personal user account when working with Git and not the default root account. So log into the admin area and create a new local account for your personal use. Alternatively, you can also configure GitLab to allow users from Active Directory to login, you can use the guide provided by Git here.

Create Personal Access Token

Once you have a Git user account set up, you will need to create a Personal Access Token. An Access Token can be used instead of a password when authenticating with Git over HTTPS. This will provide safer storage of user credentials in the Git configuration files.

Click on the profile icon on the top right of the page and select Settings:

On the Settings page select Access Tokens.

On the ‘Add a personal access token‘ page, give the access token a name and the ‘write_repository‘ permissions. Set an expiry date if you wish, or leave blank to never expire.

Once you click ‘Create personal access token‘, the token will be displayed. You will need to copy and save this token somewhere safe as you will not be able to view it again. If you lose this token, you will have to create a new one to replace it.

Create Group

A Group allows multiple projects/repositories to be created under a single namespace. This is really useful when a project spans many repositories and you need to keep these together so that they are easy to locate and manage. Our vRO projects will be using multiple repositories, therefore we’ll great a Group for these. A Group also simplifies granting access to projects, as collaborators can be granted access to the group and inherently, the projects it contains.

To create a new group, select ‘Groups‘ from the main menu at the top and then select ‘Your groups‘.

Click the ‘New group‘ button and give your group a name and settings that you require (I simply called my group ‘vRO’).

We will create all vRO projects under this new group.

Install Git Client

You will also need to install the Git client for your workstation. You can download the client for your OS here. Read More

Automating vRealize Suite Lifecycle Manager with Ansible – Part 1: Setup and Deploy vIDM and LCM

Gavin Stephens 2 Comments

For many years I have been tasked with building vRealize Automation environments, and one of the biggest pain points, has been the deployment and preparation of the IaaS machines. This has usually required special preparation of a Windows template and a number of scripts to get everything configured, so that vRA plays nice. This is usually an error prone process, especially for the larger enterprise deployments. To tackle this problem, VMware released vRealize Suite Lifecycle Manager, which is on version 2.1, as of this writing.

I decided it was time to try this product and see if it lives up to the claims. I was also more interested in the API functionality, and as with all things automation, I typically turn to Ansible. I wasn’t too surprised to discover, that although the deployment is ‘automated’, depending on your interpretation, there is actually a number of manual steps that are still required. These include ensuring that the IaaS machines and database are already deployed and properly configured. The vRLCM Create Environment process also provides validation and pre-checks, along with scripts that can be used to prepare the machines.

With the preparation of these playbooks, I set out to automate the following:

  • Deployment of a single VMware vIDM appliance;
  • Deployment and initial configuration of a single vRealize Suite Lifecycle Manager appliance;
  • Deployment of vRealize Automation IaaS Servers (Windows VMs), in multiple deployment scenarios.
  • Creation of vRealize Automation environment through LCM.

This post will focus on deploying vRSLCM and vIDM with a follow-up post on the vRA deployments.

However, in my attempts to make this a set of one click processes, I wasn’t able to quite get that far (got pretty close). This was mainly due to some limitations with the vRSLCM API (can’t automate certificates, for example). I will discuss these limitations throughout this post, and if I find workarounds, then I’ll provide an update.

I should also point out that this is quite experimental and although I have done all that I can to make these workflows as idempotent as I can, unfortunately, with the limitations of the LCM API, this has proven to be quite difficult. These playbooks are best used as a one-time only deployment, at least for LCM itself.

Environment Preparation

In my environment I have a dedicated virtual machine that I develop and run my playbooks on (you may call this the Ansible control machine) running on CentOS 7.x.

Environment Overview

CentOS CentOS 7.x
Ansible 2.8.1 (2.8 is a minimum requirement)
Python 3.6 (installed from EPEL Repository)

Prerequisites

The following pre-requisites are required:

  • DNS A/PTR records created for vRSLCM and vIDM appliances.

Prepare Environment

Ensure that the system is up-to-date by running:

Install yum-utils

Install Python 3

You will need to ensure that Python 3.6 is installed on your Ansible host. I am using the EPEL repository, but you may decided to use IUS or SCL to install these packages, so the package names may differ. Refer to the relevant documentation for installing Python 3 using these repositories, if required.

Install GIT

Git will be used to clone my Ansible vRSLCM playbooks repository.

Create Python Environment

It’s always the best approach to create a python virtual environment so that packages do not conflict with the base system. I have a directory in the root of my home dir called ‘python-env‘ where I maintain a number of different environments. Once you have a virtual environment set up, you just need to install the required packages from the ‘requirements.txt‘ file (provided later in the git repository).

You can follow these steps below to create a virtual environment:

You will notice that the shell will now display the virtual environment that you are using:

It’s also a good idea to ensure the latest version of pip and setuptools is installed.

Read More

IaC for vRealize: Deploying vRealize Build Tools To Allow Infrastructure As Code for vRA and vRO

Gavin Stephens 10 Comments

As any vRealize Orchestrator developer will tell you, managing code outside of the appliance is difficult. I recently wrote a post about Using Visual Studio Code for your vRealize Orchestrator Development, where I highlighted some of the challenges with this. The issue is that we’re not given the freedom to use any IDE we want, easily run unit tests on our code or do continuous integration with tools like Jenkins.

I did mention that a couple of solutions were going to make their way, one of these was internal tooling that VMware’s CoE team currently uses for their vRO development (you can read the article here: https://blogs.vmware.com/management/2018/11/automating-at-scale-with-vro-and-vra.html). It wasn’t possible to get access to these tools without engaging with CoE and forking up a bit of cash.

That is until now, as VMware has released these tools as a new fling. The fling is currently in preview, but you can check it out here: https://labs.vmware.com/flings/vrealize-build-tools. I think this is quite an exciting time for VMware developers as these tools could finally change the way we develop and manage our code and integrate into the wider developer ecosystem.

This is my first blog on this topic but if I find these tools useful, then there will be plenty more to follow. Getting the environment set up to use these tools is not straight forward and has several dependencies. These include deploying supporting infrastructure such as JFrog Artifactory, preparing all the required artifacts that are sourced from the vRO server and getting the workstation set up to create and manage packages.

Deploy and Configure Platform

Before the developer can begin using the vRealize Build Tools, the supporting platform has to be deployed and configured. This consists of setting up an Artifactory server to store Maven artifacts and build dependencies and preparing the artifact repositories.

Deploy Artifactory Server (skip if you already have this deployed)

This section will detail how to set up the Artifactory server and required dependencies. Note that the details below only deploy a single Artifactory node with the database instance running on the same machine. It is recommended that for a production environment to ensure Artificatory is deployed with high availability and connects to external/dedicated database instances.

Install Java Development Kit (JDK)

JFrog Artifactory requires the Java JDK 8 and above to be installed and the JAVA_HOME variable configured. I am using the Open Source version of these tools. Install using the following command:

Add the following lines to ‘/etc/profile‘ to set the ‘JAVA_HOME‘ environment variable and add the Java bin directory to the path.

Then source the file and check that the variables have been correctly configured:

Read More

Deploying NSX-T Using Ansible – Part 3: Running The Playbook

Gavin Stephens Leave a comment

In this post I am going to cover the running of the Ansible NSX-T playbook, so that you can get NSX-T deployed in your environment(s). In case you missed them, in my previous posts, I detailed how to set up your Ansible environment and configure the playbook in preparation for deploying NSX-T.

If you arrived here and want to figure it out for yourself, you can download my playbooks here: https://github.com/nmshadey/Ansible-NSXT

Playbook Overview

The main playbook that you will need to run is called ‘nsxt_create_environment.yml‘, which is located in the root of the Ansible-NSXT folder.

Read More

Deploying NSX-T Using Ansible – Part 2: Setting Up The Playbook

Gavin Stephens Leave a comment

In my previous post I covered how to prepare your Ansible environment and install the VMware NSX-T modules. I also provided the details on how to install my Ansible playbooks for deploying NSX-T in your environments.

In this post I am going to detail how to configure these playbooks to meet your environment/requirements. I have chosen to break out my variables into multiple files. This gives me the flexibility to assign values specific to a group of hosts, inherit values from a parent group and to store usernames, passwords and license information more securely, in their own Ansible Vault encrypted file.

The deployment examples that I will demonstrate include 2 sites, that each include the following:

  • A management environment at each site. This includes a vCenter Server instance with a single management cluster.
  • A compute resource (CMP) environment at each site. This includes a vCenter Server instance with a single resource cluster.

I will deploy an NSX-T instance at each management cluster. These NSX-T instances will be used to provide SDN capabilities to the compute resource clusters (when I get time I’ll create a diagram!).

An overview of the playbook tree:

Read More

Deploying NSX-T Using Ansible – Part 1: Setting Up The Environment

Gavin Stephens Leave a comment

When I saw the release of NSX-T 2.4, I decided that I would upgrade my compute clusters to utilise this new version. Since I manage the compute NSX managers mostly through the API, I figured this would provide me with some good experience and also allow me to understand how this is deployed.

In my lab I run vRealize Automation with a management cluster (CMP stack), 2 nested vCenter Servers and ESXi Clusters (compute) that mimic two geographically dispersed data centres. Until now I had deployed a dedicated NSX-V instance for each of my three vCenter deployments, that provides the logical switching and routing required for my lab.

I didn’t want to create yet another ‘how to’ guide on how to do this using the GUI, so instead, I am going to attempt to accomplish this using Ansible. VMware have handily made available Ansible modules for NSX-T, which are supported for the 2.4 release and above (note that these are still in preview). I will attempt to make use of these modules, but if I discover broken or missing functionality, then I will revert to using the NSX-T Rest API.

Link to the VMware Github repository for Ansible NSX-T: https://github.com/vmware/ansible-for-nsxt

Link to my Github Ansible NSX-T Deployment Playbooks: https://github.com/nmshadey/Ansible-NSXT

I am going to provide a series of posts that will cover the set up of the Ansible environment, how to install the VMware NSX-T modules and use the playbooks and roles that I have created to deploy NSX-T in your environments. Read More

Install phpVirtualBox on CentOS 7.x

Gavin Stephens 3 Comments

I have to say, I have been bursting with some excitement to recently discover this gem. phpVirtualBox is going to make a great addition to my server by allowing me to create and manage VirtualBox VM’s direct from my web browser. So just a small guide, sticking with the theme, on getting phpVirtualBox installed and running on CentOS 7.x.

Install required packages

Install Apache Web Server

Start Apache and ensure it starts automatically on boot.

Finally, add an exception in the firewall so that you can access Apache over HTTP and HTTPS.

Test that Apache is running and accessible by opening a browser and navigating to http://server_ip_address.

Install PHP

Restart Apache.

Test that PHP is working by creating a PHP test file that will display the PHP information page (watch out for the quotes if copy/pasting).

Verify that the PHP information page loads by opening a browser and navigating to http://server_ip_address/test.php. Read More