hyperv isolation to the rescue!

Update

More information on this topic can be found here: https://freddysblog.com/2020/02/26/the-world-after-february-18th/

February 11th 2020

On this date, the February security updates for Windows was released, and over the next days, Windows 10 computers and Windows Servers all over the world would receive this update. I am a true believer in securing my Windows Computers and my Windows Servers and would never leave my servers unprotected so I follow the guidelines and update my machines.

99% of the time this works flawlessly and just once in a while – something goes wrong. This day was this time. Not that the update failed, it didn’t – it was applied nicely, but it had some negative side effects on people running Docker on Windows (or at least people running NAV and Business Central containers on Windows).

The discovery

The first indicator I got of this issue was that my next major pipeline failed

nextmajor

Looking at the issue Wednesday, I could see that the container was created successfully and compilation of my app failed:

compileapp

-alcOutput was a parameter I added in NavContainerHelper 0.6.4.26 and immediately my attention was on that. Tried to repro locally with no luck – everything worked fine and no other issues was reported anywhere and no matter how much I looked at the code – it couldn’t be that change. I even ended up sending a message to the Modern Dev. team asking them whether they changed the output from ALC.EXE. They said no!

The issues came on Thursday. Some reported that they couldn’t sign apps using the Sign-AppInNavContainer function. Other people couldn’t compile. Other people couldn’t create the container, Web Client wouldn’t install etc. etc. etc. issues, e-mails, phone calls with different problems had no end.

Thursday late afternoon I finally found out that the February security update was to blame. My build agent for the BingMaps project was updated on Tuesday – and my next major build started failing Wednesday morning. Our Docker image build servers was updated on Wednesday causing them to create images, which didn’t work properly, which caused the nightly build between Wednesday and Thursday to basically invalidate the latest insider builds.

Some people were running images which didn’t work, some people were running good images on servers which had been updated and now failed compilation, signing, SQL stuff etc. etc. – other people didn’t see the problem at all – only with the insider builds.

I even tweeted that the first person who could point me to the direction of the error would win a beer. Even that didn’t resolve the issue…

The root cause

With the February update on the host, attempting to run executables inside the container might randomly fail if you are running process isolation. It seems like there are other problems with the February update when used in the container as well, but we didn’t uptake the February update inside our images yet.

It wasn’t due to unblock-file and I do not have a fix for this. Running things in hyperv isolation works – uninstalling February Security Update also works.

I have changed all our build server to run all containers using hyperv isolation and request a rebuild of the latest insider builds – should be done in a few hours.

This should take care of the images.

What you need to do

If you have pulled insider images the last two days, you need to pull new ones. Latest master build (next major) is 16.0.11119.0, latest 15.x build (next minor) is 15.4.40820.0 (should be ready within the next hour or two).

Now you might think – isn’t next minor 15.3 – yes, that is correct, but that has branched off for release and we don’t build docker images from release branches. If you want something close to what becomes 15.3 you need to use Get-BcContainerImageTags from bcinsider.azurecr.io/bcsandbox and grab the latest 15.3 image.

You also need to run your containers under hyperv isolation (add -isolation hyperv to New-BCContainer) or you need to uninstall February security update.

Note that using hyperv containers is known to cause issues when using non-unicode apps. A lot of effort is put into NavContainerHelper to make sure that text files are handled correctly, but you might still have issues. If you use containers for C/AL development, best option might be to uninstall February Security Update.

 

Sorry for the inconvenience!

 

Freddy Kristiansen
Technical Evangelist

17 thoughts on “hyperv isolation to the rescue!

  1. Thank you, Freddy! I have been trying to figure this one out and found the issue on github yesterday. I updated our pipelines to run in hyper V isolation. I guess, I could have saved myself some time.

    I also noticed that the New-NavContainer randomly hangs in the middle of deploying and I then need to restart our pipelines, since it just times out. Since I made some changes to the build server, it could also be related to that. Once I figure out if it is an issue for you, I’ll open an issue on github.

    As always, thanks for the great work and thanks for spending countless hours to try to unblock all of us as fast as possible!

    Like

  2. Pingback: Hyper-V isolation to the rescue? – Robberse IT Services

  3. Pingback: Hyper-V isolation to the rescue? - Microsoft Dynamics NAV Community

  4. If you are trying to enable Hyper-V isolation for a container running on a virtual machine, you may get an error with the keywords below.

    If the Containers feature is enabled without Hyper-V:
    docker error hcsshim CreateComputeSystem the request is not supported

    If the Containers feature is enabled with Hyper-V:
    docker error hcsshim CreateComputeSystem virtual machine could not be started because a required feature is not installed

    The solution is to enable nested virtualization on the Hyper-V host for the specific virtual machine(s) and then enable the Hyper-V feature inside the virtual machine(s).
    I have not yet had success creating a container after enabling nested virtualization on the Hyper-V host IF the Hyper-V feature is already enabled inside the virtual machine.

    https://success.docker.com/article/hyper-v-containers-fail-to-start

    Hyper-V must be installed and enabled.
    Additionally, nested virtualization must be enabled if the container host is a virtual machine.

    https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/deploy-containers-on-server
    https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization

    To enable nested virtualization for a specific virtual machine on the Hyper-V host, run this command using PowerShell as an Administrator.
    Set-VMProcessor -VMName -ExposeVirtualizationExtensions $true

    To see if Hyper-V is enabled inside the virtual machine using PowerShell running as an Administrator.
    Get-WindowsFeature -Name Hyper-V

    To enable Hyper-V inside the virtual machine using PowerShell running as an Administrator.
    Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All

    Liked by 1 person

  5. I faced this issue and thanks to you know it has been fixed.

    It’s the first time I’m using -isolation hyperv, my agents are running in VMs. I noted that the processing time for the pipelines has increased significantly (nightly pipeline that was taking 4 hours now requires more than 10 hours). Have you notes similar behavior? Any suggestion hot to improve performances?

    Liked by 1 person

  6. Hi,
    A lot of issues are reported everywhere with this Windows Update, not only with BC Docker containers.
    My best advice, is to unistall this Windows Update until Microsoft release a new stable update…

    Like

  7. Pingback: The world after February 18th | Freddys blog

  8. Pingback: The world after February 18th - Freddy's Blog - Dynamics 365 Business Central/NAV User Group - Dynamics User Group

  9. Pingback: The world after February 18th - Dynamics 365 Business Central Community

  10. Pingback: DevOps Build Agents for Microsoft Dynamics 365 Business Central

  11. Pingback: DevOps Build Agents for Microsoft Dynamics 365 Business Central - Waldo's Blog - Dynamics 365 Business Central/NAV User Group - Dynamics User Group

Leave a Reply to Philippe Moison Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s