IaC for vRealize: Define Dependencies, Manage Versions, Prepare & Release Packages & Deploy Artifacts

Welcome to the third part in the series working with the vRealize Build Tools. At this stage you should have a fully working CI infrastructure and have all of your vRO code exported using packages and stored in Git repositories. In this post, I will show you how to manage dependencies across your packages and how you can use the Maven plugins to automatically update packages, so that they are using the latest dependency versions.

I will also show you how to manage versioning for development (Snapshots) and production (release) code. Once we have our versioning in place, I will detail how to prepare the releases and finally push the artifacts to the Artifactory repositories, which can be picked up by a release pipeline (something I will cover in much more detail in a later post).

I also want to point out that I had to update my previous post to include the git scm connections in your pom.xml files, as this is required for this post, so make sure to go back and check that out.

Understanding Snapshot vs Release Versions

The first thing to understand is the ‘-SNAPSHOT‘ string that is suffixed to package versions. You will have noticed that all of the package versions you created in my previous post, were version ‘1.0.0-SNAPSHOT’. This suffix basically tells Maven that the code inside this package is in a development stage and not suitable for production release. The Maven release plugins understand this and will prevent the dependency handler from including these as valid dependencies for your packages, as this could result in unstable code (thought, they can be defined manually).

If you are following these posts, then currently, all your packages will be at version 1.0.0-SNAPSHOT. Don’t worry, we will change this later, but for now, accept this as-as.

Define Dependencies

In my previous post I provided an example of 3 packages that I have created. I also created a table that detailed the ‘groupId‘ and ‘artifactId‘ for these packages. I am going to extend this table to list their dependencies on other packages. I have kept this simple, in that no dependencies exist outside of these 3 packages.

A dependency exists when one package is calling code from another package (typically with a System.getModule()). When dependencies are mapped, their artifacts and required versions are released as part of the release process.

groupId artifactId Dependency
com.simplygeek.library logger
com.simplygeek.library rest logger
com.simplygeek.library nsx logger
rest

I have added the ‘Dependency‘ column, which lists the projects/artifacts that this package depends on. All of my packages depend on ‘logger‘, whereas ‘nsx‘ also depends on ‘rest‘.

We now need to add some XML to the projects ‘pom.xml‘ file, that is located at the root of the project folder. These are ‘<dependencies>‘ tags that are used to define a set of ‘<dependency>‘ tags for the project. You should insert these tags immediately after the ‘<packaging>‘ tags but before the ‘<scm>‘ tags.

We also want to omit ‘-SNAPSHOT‘ from the version, since we’ll be pushing actual releases.

Below is an example of what the dependency XML insert looks like for the ‘rest‘ package:

Once you have completed defining your dependencies, save the ‘pom.xml‘ file and make sure to commit the changes to git.

Repeat this process for all of your projects.

Prepare & Release Packages

Maven provides a number of lifecycle phases and goals that can be used to prepare and perform the releases. We will also need to push the release artifacts to the Artifactory repositories, as this will be queried for dependencies, when releasing a package that has any defined.

You should first be focusing on all the core projects that make up most of your dependencies. Also, start with the projects that have dependencies in the lowest order, i.e. the ‘logger‘ project has no dependencies, so that will be released first, ‘rest‘ depends on ‘logger‘, so that will be released next, etc.

For all the release tasks, we’re going to specify the ‘release‘ plugin followed by the goal. We will use the following goals:

  • Clean
  • Prepare
  • Perform

Read More

IaC for vRealize: Manage Existing vRO Code With vRealize Build Tools & Set up Git Repositories

In my previous post on Deploying vRealize Build Tools To Allow Infrastructure As Code for vRA and vRO, I covered how to set up the CI infrastructure and your developer workstation, in preparation for managing your vRO code as projects with Visual Studio Code and Maven. In this post, I will explain how you can work with your existing code base and manage it using the build tools. A major part of this, will be creating and managing new projects that will map to our existing code in vRO.

Once projects have been created, I will detail how Git repositories can be used to store and manage your vRO code, and then we can map your project dependencies and allow development teams to work collaboratively, without risk of overwriting the work of others (a major problem when developing using the vRO client). Git is going to bring some very useful processes and methodologies to the table such as branching, tagging and merge conflict resolution.

I also want to point out that I am currently only focusing on Actions, as I believe this is where all your code should exist. I will have followup posts that will cover strategies for managing Workflows and other items.

Setting Up GitLab

One thing that I didn’t cover in my previous post, was setting up Git. I purposely reserved this topic for this post, as it was more relevant. To start, you will need to have a GitLab server deployed that can be used to create the repositories for storing your vRO projects. You can use GitHub if you so wish, it really doesn’t matter too much, but using GitLab doesn’t require that your environments have access to the Internet. GitLab also allows you to create groups to organise multiple repositories, which is going to be really useful.

If you need to install GitLab, then there is a good guide on VULTR, that details how to install GitLab and enable HTTPS.

Create User

You should be using your own personal user account when working with Git and not the default root account. So log into the admin area and create a new local account for your personal use. Alternatively, you can also configure GitLab to allow users from Active Directory to login, you can use the guide provided by Git here.

Create Personal Access Token

Once you have a Git user account set up, you will need to create a Personal Access Token. An Access Token can be used instead of a password when authenticating with Git over HTTPS. This will provide safer storage of user credentials in the Git configuration files.

Click on the profile icon on the top right of the page and select Settings:

On the Settings page select Access Tokens.

On the ‘Add a personal access token‘ page, give the access token a name and the ‘write_repository‘ permissions. Set an expiry date if you wish, or leave blank to never expire.

Once you click ‘Create personal access token‘, the token will be displayed. You will need to copy and save this token somewhere safe as you will not be able to view it again. If you lose this token, you will have to create a new one to replace it.

Create Group

A Group allows multiple projects/repositories to be created under a single namespace. This is really useful when a project spans many repositories and you need to keep these together so that they are easy to locate and manage. Our vRO projects will be using multiple repositories, therefore we’ll great a Group for these. A Group also simplifies granting access to projects, as collaborators can be granted access to the group and inherently, the projects it contains.

To create a new group, select ‘Groups‘ from the main menu at the top and then select ‘Your groups‘.

Click the ‘New group‘ button and give your group a name and settings that you require (I simply called my group ‘vRO’).

We will create all vRO projects under this new group.

Install Git Client

You will also need to install the Git client for your workstation. You can download the client for your OS here. Read More

IaC for vRealize: Deploying vRealize Build Tools To Allow Infrastructure As Code for vRA and vRO

As any vRealize Orchestrator developer will tell you, managing code outside of the appliance is difficult. I recently wrote a post about Using Visual Studio Code for your vRealize Orchestrator Development, where I highlighted some of the challenges with this. The issue is that we’re not given the freedom to use any IDE we want, easily run unit tests on our code or do continuous integration with tools like Jenkins.

I did mention that a couple of solutions were going to make their way, one of these was internal tooling that VMware’s CoE team currently uses for their vRO development (you can read the article here: https://blogs.vmware.com/management/2018/11/automating-at-scale-with-vro-and-vra.html). It wasn’t possible to get access to these tools without engaging with CoE and forking up a bit of cash.

That is until now, as VMware has released these tools as a new fling. The fling is currently in preview, but you can check it out here: https://labs.vmware.com/flings/vrealize-build-tools. I think this is quite an exciting time for VMware developers as these tools could finally change the way we develop and manage our code and integrate into the wider developer ecosystem.

This is my first blog on this topic but if I find these tools useful, then there will be plenty more to follow. Getting the environment set up to use these tools is not straight forward and has several dependencies. These include deploying supporting infrastructure such as JFrog Artifactory, preparing all the required artifacts that are sourced from the vRO server and getting the workstation set up to create and manage packages.

Deploy and Configure Platform

Before the developer can begin using the vRealize Build Tools, the supporting platform has to be deployed and configured. This consists of setting up an Artifactory server to store Maven artifacts and build dependencies and preparing the artifact repositories.

Deploy Artifactory Server (skip if you already have this deployed)

This section will detail how to set up the Artifactory server and required dependencies. Note that the details below only deploy a single Artifactory node with the database instance running on the same machine. It is recommended that for a production environment to ensure Artificatory is deployed with high availability and connects to external/dedicated database instances.

Install Java Development Kit (JDK)

JFrog Artifactory requires the Java JDK 8 and above to be installed and the JAVA_HOME variable configured. I am using the Open Source version of these tools. Install using the following command:

Add the following lines to ‘/etc/profile‘ to set the ‘JAVA_HOME environment variable and add the Java bin directory to the path.

Then source the file and check that the variables have been correctly configured:

Read More

vRealize Orchestrator: HTTP-REST Client

vRealize Orchestrator provides a nice way to manage and interact with web services. I have always liked that you can add HTTP-REST API endpoints, configured with basic authentication, which doesn’t require a token to be requested each time they are used. It’s also quite useful having visibility of these endpoints in the inventory, which makes them easy to discover.

It’s also possible to create operations on these endpoints (i.e. all the get/post operations), but I am not a big fan of using these. The problem, is that using them takes you that little bit further away from managing everything in code. They are also more difficult to port/migrate to other vRO instances, which is especially true if you have multiple environments for development, test, pre-prod, prod, etc. My much preferred method is to use Actions which perform these operations instead.

I have also seen many other developers use Actions for their HTTP-REST operations, which is good. The problem, however, is that I have seen these written in so many different ways. I decided that I needed to create a consistent experience when interacting with these web services and created the HTTP Rest Client. This client is a class with associated methods, that are used to make requests against a HTTP-REST web services.

You can download a package containing the HTTP-REST Client Action here.

HTTP REST Client

The client is an action that returns the HttpClient object. This object has a number of methods that can be used to execute requests against a web service. The following methods are supported:

  • GET, POST, PUT, DELETE and PATCH

The Action itself requires no inputs as these are provided when instantiating the object or supplied to the methods, when called.

Below is the code for the HttpClient class.

Read More

Using Visual Studio Code for your vRealize Orchestrator Development

When you are developing in a vRealize Orchestrator environment, one of the biggest frustrations is being limited to the vRO IDE. The vRO IDE is very simple, in that it does not provide any of the features that you would expect from an IDE, such as IntelliSense and quality of life extensions/addons, and only provides basic syntax checking.

There is no integration with source control management systems such as GIT (it has an internal system which isn’t great), moving code can be difficult and unit testing in the true sense of development, doesn’t really exist. A connection to the vRO IDE uses a JAVA client and also requires a constant connection to the vRO server, so developing on the move or offline isn’t possible.

Anyone that has developed on this platform will have experienced the same issues and can only dream of the day when this is no longer the case.

You can use an IDE called Visual Studio Code, that can help make your development life easier. Admittedly, this alone doesn’t solve all of the discussed problems, but it does allow you to leverage the power of this IDE to assist in code development. There are still restrictions, such as lack of integration with vRO itself, which requires code to be manually copied to the vRO server (yes, annoying). The good news, however, is that solutions are starting to become available to provide that integration. I am going to expand more on this at the end of this post.

If you haven’t heard of Visual Studio Code, it is a lightweight and feature rich IDE created by Microsoft. The Visual Studio Code website describes it as:

Visual Studio Code combines the simplicity of a source code editor with powerful developer tooling, like IntelliSense code completion and debugging.

First and foremost, it is an editor that gets out of your way. The delightfully frictionless edit-build-debug cycle means less time fiddling with your environment, and more time executing on your ideas.

Visual Studio Code supports macOS, Linux, and Windows – so you can hit the ground running, no matter the platform.

I do almost all of my vRO development using Visual Studio Code, which gives me access to useful extensions and most importantly, keeps me in the mindset of how a developer should really work.

In this post, I am going to cover how to set up a Visual Studio Code environment in Windows, install some useful extensions that I like to use and the installation of GIT and other required software components.

Setting Up Your Development Environment

The first thing that you will want to do, is download and install Visual Studio Code from here. When you launch this for the very first time, you’ll get an immediate good impression, from how quick it loads and how lightweight it feels. The default dark theme is also quite nice.

Read More

vRA Developer: Part 8 – Working with vCAC Reservation Storage

In my previous post, I covered working with vcac reservations and provided examples of how you can retrieve property values and also linked entities, such as networks and virtual machines. In this post I am going to cover reserved storage. I planned to include this in the previous post, but the content was becoming too large, so I decided to split them.

The Actions provided here should be useful if you ever need to work with storage outside of vRA (for example, when adding or resizing a virtual hard disk directly in vCenter) and can be used to query reservation information first. This can prevent cases where storage can exceed that which is allocated/reserved for the tenant.

Here is how the reserved storage looks for the reservation that I will be working with:

All code that I have provided or talked about in this post can be downloaded as a vRO package for your consumption here.

I have also included a number of workflows in the package, which provide examples of these actions being used.

You can find these under ‘Simplygeek -> Library -> Common Workflows -> vRealize Automation -> Examples -> Reservations

Get Reserved Storage Information

The following sections provide vRO Actions that can be used to get information about storage that has been reserved in a reservation or get details of physical capacity.

Get Storage Reserved Capacity for a Reservation

The following action accepts a storage path and the reservation entity and will return the total amount of storage that has been reserved for the reservation.

Read More

vRA Developer: Part 7 – Working with vCAC Reservation Entities

Reservations are created in vRA to allocate memory, storage and network resources to a business group. Business groups can consume these resources by provisioning virtual machines.

In this post I am going to cover the vCAC Reservation Entities and provide some workflows and actions that can be used to get information about reservations programmatically. This can be useful to gather information about the reservations if you are using custom XaaS processes that would not trigger the standard built-in reservation checks.

All code that I have provided or talked about in this post can be downloaded as a vRO package for your consumption here.

I have also included a number of workflows in the package, which provide examples of these actions being used.

You can find these under ‘Simplygeek -> Library -> Common Workflows -> vRealize Automation -> Examples -> Reservations

Get a List of Available Reservations or Get a Specific Reservation

The following code can be used to get a list of all reservations that are available for the specified tenant or retrieve a specific reservation based on its name or unique id.

This is specifically the reservation that I will be working with and will appear in the log outputs.

Get List of Available Reservations For a Tenant

The following code will accept a tenant Id and a toggle for retrieving only enabled reservations. A list of reservation string names will be returned. This isn’t as straight forward as you would hope (at least from an entities point of view), because there is no property for the tenant id on the reservation. Instead, I had to inspect the business group that the reservation is assigned to, which does have a tenant id property (it’s quite possible this is how the gui does it too).

Log output:

Read More

vRealize Automation Virtual Machine is Missing after being Moved or Recovered

Have you ever seen a deployed virtual machine in the vRA portal have its status changed to ‘Missing‘? This can happen for two reasons; The first, obvious reason, is that the source virtual machine has been deleted. Not much you can do about that except re-provision it or restore the vm from backup. The second reason, is because the reference information that vRA holds about the virtual machine is no longer valid (perhaps from an earlier restore).

In this post, I will discuss and provide a solution that will address the latter for a vSphere hosted virtual machine.

When in this state, you will not be able to perform day 2 actions on this virtual machine (apart from destroy).

The following scenarios can explain why vRA has incorrect reference data about the virtual machine:

  • The virtual machine has been recovered from backup and a new instance uuid has been created;
  • The virtual machine has moved to another vCenter Server, maybe during a DR event. I know that Zerto will create a new virtual machine at the DR site which will have a different instance uuid (SRM’s shadow VMs are most likely similar but I can’t confirm);
  • The virtual machine was cloned and the original has been deleted;
  • The instance uuid for the virtual machine has been changed, either manually or by some other process.

Read More