vRealize Orchestrator cluster nodes not in sync with embedded vRA Appliance

I have come across this issue a number of times where the secondary vRO cluster node is not in sync with the primary node (one thing to note is that the synchronisation status is relative to the node you are logged on to).

The option to synchronise the nodes is not available because I am using vRO that comes embedded with the vRA appliance. The reason this option is not available is because vRA “should” be managing the state of the cluster nodes, which is does from a vRO client prospective but not when you make changes via the Control Center (such as changing the Admins group).

To workaround this issue and unlock the hidden options you will need to append “?advanced” to the end of the URL. For example, if you are on the Orchestrator Cluster Management page add ?advanced to the end of the URL, which should look like this:

https://vro_server:8283/vco-controlcenter/#/control-app/ha?advanced

When the page refreshes you will notice that a new button has appeared on the bottom.

Clicking on this button will reveal two new options:

  • Push Configuration
  • Push Configuration and restart nodes

Select “Push Configuration and restart nodes”, which will push the configuration and automatically restart the vRO service on the secondary node.

A message will be displayed if this is successful.

It will take approx 10 minutes for the secondary node to start up completely so grab a coffee at this point.

Refresh and both nodes should now show as “Synchronized”.

Remove ‘Default Machine Prefix’ for a Business Group

When I was building vRA 7 and figuring everything out, at some point I set the ‘Default machine prefix‘ for the Business Groups. When you first create a Business Group, setting the default machine prefix is optional; However, if you do set this value then there is no way to unset it, which I find extremely annoying. As I don’t currently use specific prefixes for business groups, I was feeling a little OCD having to set this to some other prefix value.

defaultprefix

I was able to ignore this for a while until I installed the Custom Hostnaming Extension for vRA7 by Dailyypervisor. This extension is really awesome and has helped me massively overcome my requirement to create dynamically generated hostnames. The hostnames are based on our internal naming convention and I make good use of Custom Properties for this. For testing, I simply created a very basic custom hostname scheme ‘{LOC}-GSTEST{##}’, with {LOC} being set on one of the vSphere endpoints.

hostnamescheme

At first this appeared to work just fine, until randomly some machines would be provisioned using a different hostname. The other hostname was based on another machine prefix that I had created previously and was configured as the default for the Business Group, which was ‘ES-VWOI-MANA##’. The image below shows the results of 7 machines provisioned using the custom hostname extension.

machines

Clearly what is happening here is that on each provisioning attempt, it is randomly selecting a different machine prefix. What is also interesting is that both prefixes get incremented. In fact, that’s another issue with having a default one set for the business group (or even within the Blueprint), is that they are ‘allocated’ and incremented even before the EBS is used (which is when the custom hostnaming extension kicks in, at the requested stage, which is still AFTER the allocation). I have had a small look at the code for the workflow and can see that it’s identifying that the custom hostname scheme already exists, but I haven’t yet looked into why it’s setting the wrong one.

I figured the easiest workaround for now would be to simply remove the default machine prefix from the business group and make sure no one accidently sets it. But this is when I realised you can’t do that by simply editing the group via the IaaS interface. To cut a long story short, you have to manually remove this on the database, both the SQL database used by IaaS and the PostGres database used by vRA.

Continue reading

Resolve vRA/vCAC VM to vCenter VM

If you want to perform configuration tasks against a VM during the deployment process or even for Day 2 operations then you will need to resolve the vRA managed VM to a vCenter VM of type VC:VirtualMachine in your vRO workflows. I initially required this when applying tags to a VM during the BuildingMachine LifeCycle POST state, which involved using the VAPI endpoint.

vRO with the vRA plugin installed will make available two modules with some pre-defined actions .

  • com.vmware.vcac.asd.mappings
  • com.vmware.vcac.asd

findvcvmbyuuid

The two actions that we are interested in are ‘mapToVCVM‘ and ‘findVcVmByUuid‘. Let’s take a look at these two actions.

mapToVCVM

Inputs: VMProperties(Properties)
Return type: VC:VirtualMachine

This action has an input ‘VMProperties‘ of type ‘Properties‘, which is passed to it by the calling workflow or action. If you don’t know what this is, it is the vRA payload sent to vRO by the Event Broker Service (EBS). I’ll post another article at a later date on how I am doing this if you’re not sure. The first line of the script extracts the custom property ‘VirtualMachine.Admin.UUID’, which returns the unique UUID of the virtual machine (in previous versions of this action, the MoRef was used which is not unique across multiple vCenter Servers) and stores the value in ‘vmUuid‘.

The second line returns the result of ‘System.getModule(“com.vmware.vcac.asd”).findVcVmByUuid(vmUuid)’ of type VC:VirtualMachine. This code calls another module/action and passes ‘vmUuid‘ as an input (refer to the previous image if you’re unsure what is happening here and it should make sense). The result of this is returned to the original calling workflow/action as a VC:VirtualMachine type, which will be the vCenter managed VM object.

findVcVmByUuid

Inputs: VmUuid(String)
Return type: VC:VirtualMachine

This action has an input ‘VmUuid‘ of type ‘String‘. As you saw from the ‘mapToVCVM‘ action, this string is passed over with the value of the UUID for the virtual machine. The action then discovers the configured vCenter endpoints and performs a search on each for the VM until a result is found and returns it.

At this point you have probably worked out that you don’t need to use ‘mapToVCVM‘ at all and we could simply extract the UUID ourselves and then make a call to the ‘findVcVmByUuid‘ action.

Continue reading

Top 5 vRealize Automation Resources to get you started

OK, so everyone loves a top 5 so here is a list of my top 5 resources for learning and deploying vRealize Automation.

1. VMware Hands on Labs.

Take ‘HOL-SDC-1633 vRealize Automation 7: What’s New‘. – Despite the name this is a really in depth tutorial and you will want to complete the entire lab. This will give you a good feel about what vRA is capable of and how you can extend the platform. There are some really good examples of how the event broker is used to integrate with ITSM CMDB software for change control (iTop is used). The lab also dives into vRO and gives you a taste of just how powerful this product really is.

Next, take ‘HOL-SDC-1632 vRealize Automation Advanced: Integration and Extensibility‘. – Yes, this lab is based on vRA 6.2 so is a little older but most of the fundamentals are there and again provides some good examples of how the platform can be extended with vRO. Examples of extensibility with Infoblox IPAM, Puppet Enterprise and NSX (although slightly depreciated) are used.

2. vRealize Automation Reference Architecture

Once you have had some experience and insight from doing the hands on Labs you will be eager to begin planning and designing your new vRA platform. This document will provide you with a lot of details such as all of the components that are involved and how best to deploy and scale these. Also included are firewall and load balancing requirements. I cannot emphasis enough the importance of planning your vRA deployment properly from the get go as this will ultimately determine the success of the project.

3. Open902.com

I am really happy that I discovered this site before starting my vRA 7 implementation. Michael Rudloff has done a fantastic job of documenting the enterprise installation and configuring the IaaS platform so that you get some decent functionality out of it. These guides really took away a lot of the pain during the installation and covers topics such as replacing certificates, configuring an endpoint, approval policies, business groups, fabric groups, etc and has an awesome guide on Custom Property Relationships. I also like how has turned his private archive public and reminds me a lot of my private Confluence site.

Continue reading

What vRealize Automation is trying to solve

Before purchasing or installing the software one of the first things you need to understand is what vRA is trying to solve. Unlike what VMware has released previously, vRA is a very different and tries to solve the problems related to managing multiple cloud platforms and the end to end delivery of IT services. Yes, there are a lot of tools out there that can help in the delivery of these services but it very quickly starts to turn into a complex mismash of applications and processes. It’s also important to point out that vRA is not exclusive to vSphere but is designed to be platform agnostic. Think of vRA operating as the management plane in your data centre that plugs into all the services around it.

So when I talk about ‘end to end delivery of IT services’ what exactly am I referring to? Well, consider the example process below of bringing a new virtual server into service:

  • End user requests new virtual server, provides details and submits request to IT support services;
  • A change is raised on the ITSM CMDB platform;
  • Wait for change to be approved before provisioning starts (alternatively the request could be rejected and process cancelled);
  • When approved, begin the provisioning process;
    • Get the next available hostname (i.e. using a prefix and the next available number increment);
    • Request and reserve the next available IP address from the IPAM software and update the record to reflect the new server being provisioned;
    • Provision the virtual server with specific hardware configuration;
    • Add/update DNS records;
    • Add server to the Active Directory domain and place into the correct Organization Unit;
    • Update OS with latest security patches;
    • Add to existing firewall rules;
    • Generate SSL certificates;
    • Install software such as Anti-Virus and applications;
    • Add new Configuration Item with server details;
    • Update server inventory;
    • Add to monitoring system;
    • Mark change as complete;
  • Email End user that requested the virtual server that it is now ready;

You’ll agree, that this is a lot of steps for provisioning something like a virtual server and will involve multiple different teams (and think about doing this on many different cloud platforms). Most of the time, a lot of these steps are performed manually using run books but are sometimes overlooked and can cause delays in fulfilling the request or lead to problems later, like outdated information. We have tools like Puppet which help us with deployments, software installations, etc. but most of these are very Linux focused. They also do not address the other steps, or at least not in a very friendly way.

Finally, and more interestingly, think about when the virtual server is taken out of service or decommissioned. This is almost always where things get messy. I have seen really well laid out and robust commissioning procedures but nothing to support the decommissioning. Often, it’s a case of ‘best effort’ and items get missed and are usually discovered later, often after they have caused a problem or identified with routine audits of DNS or firewalls. In some cases this could even pose a security risk. When we think of all of this what we are actually referring to is ComplianceGovernance and Life Cycle Management.

My advise here is that even before you touch vRA, you need to think about how IT services are being provisioned within your organisation. This will involve engaging with the various operations or development teams and weed out every single process that is involved, how services are delivered, maintained, monitored and secured. You will need to fully understand the manual process and any automation that exists. Secondly, you will need to understand all of the applications that are being used to support these teams and how you will potentially interface with these during automated deployments.

vRealize Automation is designed to solve all of these problems. it is a fully extensible platform that allows Administrators to create policy driven processes for the provisioning of IT services. It does this whilst providing a catalog of available services that end users can simply request by providing some details and a couple of clicks of a button. Administrators are armed with a powerful Advanced Service Designer tool that allows items such as Virtual Servers or Software Components to be drag and dropped onto the design canvas and later submitted to the catalog. These ‘designs’ are what vRA refers to as blueprints.

Continue reading