Implementing Process Locks in vRO Workflows

I discovered last week that a developer wasn’t aware that you could use a locking semaphore in vRO. They were writing some API calls and encountered a 409 (conflict) error when a workflow was attempting to concurrently update the same resource (a load balancer if I recall but can’t remember the exact details). I felt that there may be other developers out there who are also not aware so I have created two workflows that can be used to create and remove process locks in your vRO workflows.

vRO provides an always available Locking System that can be used to create a lock for a given id and owner. It does this by creating a record in the embedded PostgreSQL database. When an attempt is made to create another record in the database, a key constraint occurs and the lock fails.

You should consider using a process lock if you are updating any resources that could cause a conflict or data consistency problems. Some examples include, updating a load balancer, updating a vRA Reservation, or preventing a data collection task from running because there is already one in progress. It is also useful to create a lock between sections of code in your workflow, which could cause a problem if the workflow was executed multiple times, simultaneously. Note, that you don’t always need to lock the entire workflow, just between areas of code where updates are performed.

You can find the package containing all the code here.

Create Process Lock

I have created a workflow that handles the locking process. The workflow includes an Action that uses the Locking function ‘lock‘ that will attempt to acquire a lock. If this is successful, the function will return true, otherwise false is returned. There is another function that could have been used called ‘LockAndWait‘, however, this will wait indefinitely if the lock cannot be acquired for some reason. The use of the ‘lock’ function provides more control on how best to handle the situation, such as introducing a timeout to prevent the workflow running indefinitely or taking corrective measures.

The workflow is mostly handled by an Action that creates the lock with some logic build around it.

The Workflow

The workflow attempts to create a lock, and if this fails, will try to gain the lock after 10 seconds. It will attempt to do this 180 times (30 minutes). If it still fails on the final attempt, the existing lock will be released and a new lock created. I decided to do this because 99% of the time, any lock maintained for that long was usually a stale lock that had not been released. Most update operations where this is used aren’t anywhere near the threshold. The timeout values can be changed in the workflow and feel free to change the logic to suite your own requirements.

And here is an example log output with it used in my Data Collection workflow (whilst there was already an existing lock in place).

This works really well and prevents more than one of the same data collection task running on a compute resource.

Remove Process Lock

The removal of the process lock only requires a single Action to complete.

Do make sure you include locks to protect your update operations and if you require any help with anything, then please drop me a message via the Drift app.

Leave a Reply

avatar
  Subscribe  
Notify of