Deployment of New VM from Template Fails with VMDK Locked Error

~ Anton Khitrenovich

Some time ago I managed to isolate an ugly vSphere 5 bug that caused me some unpleasant moments. Today I finally reproduced this behavior in a clean lab environment, so I feel confident enough to describe it in public. Don’t worry, the way to recover is described also!

Take a vSphere 5 environment with a template (say, “MyTemplate”) that you want to deploy a virtual machine from. Step through a regular deployment wizard and select “Edit Virtual Hardware” on the last page. In the VM properties, choose the hard disk and change its size. The new VM (say, “MyVM”) will be created successfully, but the next time you’ll go through the deployment process it will fail with a strange message: “Unable to access file ‘MyTemplate/MyTemplate.vmdk’ since it is locked”…

A quick search in VMware KB (or just in Google) will tell you that the vmdk (the virtual disk of the VM) is usually locked when some other VM uses the same disk. Hey, this is a template! No other VMs are supposed to use the template disk, right? Well, the truth is that this disk is now attached to MyVM – yes, that one with different disk size… But let the lock alone. Your lovely template is now gone, since the disk that belonged to the template is now owned by the running virtual machine – with new name, settings and maybe even software!

Well, how do you recover from this situation? The good news are that the original disk from MyTemplate is not gone. In fact, vSphere really cloned a new vmdk file for MyVM and placed it where it should usually reside – in ‘MyVM/MyVM.vmdk’ – but for some reason set the MyVM to use a template file instead. At this moment you can take a deep breath, shut down MyVM, point it to the right virtual disk and physically swap the files in the datastore…

Here are more detailed instructions, just to be on the safe side:

  1. Shut down MyVM (either from the OS or via the Power options of the VM)
  2. Point MyVM to the cloned virtual disk (one under ‘MyVM’ folder):
    1. Open ‘Edit Settings’ dialog of MyVM
    2. Find the hard disk that points to ‘MyTemplate/MyTemplate.vmdk’
    3. Write down the virtual device node of the hard disk
    4. Remove this hard disk from the VM
    5. Add new hard disk, use existing ‘MyVM/MyVM.vmdk’ and device node you wrote down earlier
    6. Click ‘OK’
  3. Log in with SSH to the ESX box that has access to the datastore where MyTemplate and MyVM reside
  4. Swap the data files for ‘MyTemplate.vmdk’ and ‘MyVM.vmdk’:
    1. Move ‘MyVM/MyVM-flat.vmdk’ to ‘MyTemplate/MyTemplate-original.vmdk’
    2. Move ‘MyTemplate/MyTemplate-flat.vmdk’ to ‘MyVM/MyVM-flat.vmdk’
    3. Move ‘MyTemplate/MyTemplate-original.vmdk’ to ‘MyTemplate/MyTemplate-flat.vmdk’
  5. Power on MyVM

UPD (Jan 2012): It appears that the issue is not new in vSphere 5 – it exists at least since ESX 3.5 (back in 2008). I wish this post was higher in the search results when I originally looked for explanations…

UPD (Nov 2013): VMware claims that the bug that the disk size modification bug was fixed in vSphere 5.0U2 and 5.1U1. I did not have a chance to check this yet.

Published by Anton Khitrenovich

32 thoughts on “Deployment of New VM from Template Fails with VMDK Locked Error”

  1. This issue exists in v4.1 as well. I did the exact same thing, Deploy from template, experimental option = change disk size and everything seemed to go fine, until I tried to deploy the next VM! That’s when I received the error, “Unable to access file”. I initially thought about simply deleting the lastly created vm, but that would have likely destroyed my template since that vm thought it was associated with the templates vmdk file. I was able to resolve by editing the lastly created vm’s .vmx file so that it pointed to its own .vmdk file instead of the template vmdk. I then removed the lastly created vm from inventory and then added it back to refresh the edited vmx file. I then deleted it now that it no longer pointed to the template vmdk file. I then deployed from template again, this time making sure NOT to use any “Experimental” options. All seems well now. 🙂

    Reply
  2. Pingback: A nice vsphere bug i just ran into -> cloning new vm from template fails with vmdk locked error | My Angry Dome

  4. Hi!

    What to I do, when I don’t know which server that’s holds on to the template file?
    I have this problem, but it’s a template that I rarely use, so I don’t remember which server it is now?

    Do you have any suggestions for me?

    Thx

    Reply

    1. Do you remember the template name? If you do, search the inventory for this template using the search box in the top-right corner of vSphere Client. Then, go to the Summary tab. There you can see the ESX host name under General and the datastore name under Resources.

      Hope it helps. Good luck!

      Reply

  5. excellent work man… but in my case I didnt need to follow the move steps.
    In my scenario I cloned from Template in a ESX 4.1 Env. the cloned VM took the Template vmdk (because I may have experimentally changed some settings)
    All i did was power off the offending VM, gave it its original disk back and was able to clone new VM going forward.

    Reply

  6. Anton,

    Thanks for the great post! Ended a LOT of frustration this morning.

    There is a VMware KB article about this too: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007487

    It’s not nearly as straighforward as your article. Also, I’m especially tickled by the phrase “A virtual machine deployed from a template locks the template’s VMDK files if it is incorrectly configured to run on the template’s virtual disk and VMDK file instead of its own.”

    If I’m not mistaken, since I’m the one who configures things, they are suggesting that this is my fault vs a bug :).

    If it is my fault, do you have any idea how I might avoid this configuration problem in the future?

    Reply

    1. David,

      I’m glad my post is helpful.
      Thank you for sharing the KB article – it is pretty new, since VMware had nothing on the topic when I wrote this post. As far as I understand, the only “fault” was to use ‘Edit Virtual Hardware’ option. IIRC, it was marked as ‘experimental’ in ESX 4. In the newer versions this note was removed, but it seems to be the UI change only 🙁

      I hope VMware will take care about this bug some day…

      Regards,
      Anton.

      Reply

  9. Guys,

    Just went through the comments and wanted to add my thoughts on this.

    The VM which is deployed from template ( using the option as Edit virtual Hardware (Experimental) ) and then when we change the size of the vmdk then the new deployed vmdk is created and the VM points to the template vmdk.
    But when we dont edit the size of the vmdk then it does not map the VM to templates vmdk but it points to its actual vmdk which is same as the name of the VM.

    The work Edit virtual Hardware (Experimental) is just expremental and should not be used in production, this is given just for experimenting / testing.

    http://kb.vmware.com/kb/1003098
    http://kb.vmware.com/kb/1007487

    Reply

    1. Punit,

      Despite being experimental, this option is still widely used – especially in the non-production labs. Be assured that even there this bug is pretty painful…

      As I said in one of the comments above, the second KB article is relatively new and was not available when I documented this behavior. Anyway, it describes how to re-attach the virgin disk to VM and recommends to rely on backups to restore the template. My recipe is a bit more dirty, but allows you to preserve your changes in the VM and fully recover the template – even without the backups.

      Regards,
      Anton.

      Reply

  10. Hi,
    Thanx for this info, I noticed like when I have template A, deploy a machine named VM-1 from it, edit the settings on VM-1 like desktop, etc.
    After that, from the same template A I created a machine named VM-2.
    But VM-2 looked identical to VM-1, same desktop, same files in its folders … now I know they all use the same virtual disk …

    Now my question: how can I prevent this from happening in the future?

    Reply

  11. Thank you very much Anton, I’ve come across this problem this morning. Saved me a lot of headaches

    Reply

  12. Anton, Thanks for the info! I have a couple clarifying questions:

    1. I am assuming the -flat and -original tags are just temporary so that the files with the same names can be moved into the same directories? I dont actually see any -flat or -original files.
    2. Can I just use the vSphere Datastore Browser GUI in order to move and rename the VMDK files instead of SSH?

    Thanks!
    Rick

    Reply

    1. Hi Rick,

      The “-original” label is really temporary, but the “-flat” one is permanent. This is the reason you can’t use Datastore Browser – it presents the half-logical view of the filesystem rather than true physical view.

      Good luck,
      Anton.

      Reply

  13. Hello Anton,

    Have you the info for moving the files in Step 4 and does it rename the files to match the VM?

    Thank you.

    Reply

        1. It is the same “mv” shell command for both move and rename…
          Pay attention that the only way to move/rename those files is via SSH, since “*-flat.vmdk” files are hidden in Datastore Browser.

          Reply

  16. Possibly. One extra thing I did was to extend the volume via Windows Disk Management. Luckily, we have a backup template. Thanks, Anton.

    Reply

  17. Hi,

    followed the instructions but ended with a corrupted disk message. Fixed it by editing the vmdk file and set the “Extent Description” to the right amount of extents

    Reply

  18. Hi to All,espcially to Anton Khitrenovich

    This old post still very helpful and working my Vsphere 5.1 encountered
    this issue.

    thank you very much!

    regards,
    adriane

    Reply

Leave a Reply