hyperv isolation to the rescue!

February 11th 2020

On this date, the February security updates for Windows was released, and over the next days, Windows 10 computers and Windows Servers all over the world would receive this update. I am a true believer in securing my Windows Computers and my Windows Servers and would never leave my servers unprotected so I follow the guidelines and update my machines.

99% of the time this works flawlessly and just once in a while – something goes wrong. This day was this time. Not that the update failed, it didn’t – it was applied nicely, but it had some negative side effects on people running Docker on Windows (or at least people running NAV and Business Central containers on Windows).

The discovery

The first indicator I got of this issue was that my next major pipeline failed

nextmajor

Looking at the issue Wednesday, I could see that the container was created successfully and compilation of my app failed:

compileapp

-alcOutput was a parameter I added in NavContainerHelper 0.6.4.26 and immediately my attention was on that. Tried to repro locally with no luck – everything worked fine and no other issues was reported anywhere and no matter how much I looked at the code – it couldn’t be that change. I even ended up sending a message to the Modern Dev. team asking them whether they changed the output from ALC.EXE. They said no!

The issues came on Thursday. Some reported that they couldn’t sign apps using the Sign-AppInNavContainer function. Other people couldn’t compile. Other people couldn’t create the container, Web Client wouldn’t install etc. etc. etc. issues, e-mails, phone calls with different problems had no end.

Thursday late afternoon I finally found out that the February security update was to blame. My build agent for the BingMaps project was updated on Tuesday – and my next major build started failing Wednesday morning. Our Docker image build servers was updated on Wednesday causing them to create images, which didn’t work properly, which caused the nightly build between Wednesday and Thursday to basically invalidate the latest insider builds.

Some people were running images which didn’t work, some people were running good images on servers which had been updated and now failed compilation, signing, SQL stuff etc. etc. – other people didn’t see the problem at all – only with the insider builds.

I even tweeted that the first person who could point me to the direction of the error would win a beer. Even that didn’t resolve the issue…

The root cause

With the February update on the host, attempting to run executables inside the container might randomly fail if you are running process isolation. It seems like there are other problems with the February update when used in the container as well, but we didn’t uptake the February update inside our images yet.

It wasn’t due to unblock-file and I do not have a fix for this. Running things in hyperv isolation works – uninstalling February Security Update also works.

I have changed all our build server to run all containers using hyperv isolation and request a rebuild of the latest insider builds – should be done in a few hours.

This should take care of the images.

What you need to do

If you have pulled insider images the last two days, you need to pull new ones. Latest master build (next major) is 16.0.11119.0, latest 15.x build (next minor) is 15.4.40820.0 (should be ready within the next hour or two).

Now you might think – isn’t next minor 15.3 – yes, that is correct, but that has branched off for release and we don’t build docker images from release branches. If you want something close to what becomes 15.3 you need to use Get-BcContainerImageTags from bcinsider.azurecr.io/bcsandbox and grab the latest 15.3 image.

You also need to run your containers under hyperv isolation (add -isolation hyperv to New-BCContainer) or you need to uninstall February security update.

Note that using hyperv containers is known to cause issues when using non-unicode apps. A lot of effort is put into NavContainerHelper to make sure that text files are handled correctly, but you might still have issues. If you use containers for C/AL development, best option might be to uninstall February Security Update.

 

Sorry for the inconvenience!

 

Freddy Kristiansen
Technical Evangelist

Weekend cleanup… – done

Update: Weekend cleanup is done and the latest daily builds from master (next major) and 15.x (next minor) are updated. I have also updated the number below from 45 days to 7 days as I don’t see any reason to have older insider builds. Let me know if you think differently, thanks.

It is cleanup time. Our insider repository has become very very big and it is time to clean up. The problem however is, that the current insider registry has reached a size, where it is very hard to clean up, so… Continue reading

Having Demo Data while developing Business Central Apps with Docker

I have always preached that you shouldn’t try to keep your Docker containers running. Containers should be something, which easily can be dismissed and recreated for any developer. One of the frequent questions is then: But what about my demo/development data? Continue reading

Please check your version of Docker!

In Microsoft, we are constantly in search of ways to improve security for our customers. Customers must feel safe when using our services and leaving their precious data in our hands. Sometimes this requires our customers and partners to update client software and this blog post is a warning about just that. Continue reading

Upgrading to 15.x from 14.x C/AL – our NAV TechDays 2019 demo

WARNING: VERY LONG BLOG POST AHEAD!

My session at NAV Tech Days this year was together with Nikola Kukrika and we did an end 2 end walk-through of how to upgrade a code customized C/AL solution in 14.x to 15.x (AL), converting the code, upgrading the data, explaining all the pitfalls and in some cases, how to cope with missing functionality. Continue reading

Mounting a database from my online environment using SQL Server on the host

This blog post is really a combination between the last two blog posts, https://freddysblog.com/2019/11/04/using-sql-server-on-the-host/ and https://freddysblog.com/2019/11/12/mounting-a-database-backup-from-my-online-environment-inside-a-container/. As stated in the last blog post, you can only use databases of less than 10Gb in size inside the container due to SQL Express. This blog post will explain how to get past that problem. Continue reading

Mounting a database backup from my online environment inside a container

Just recently, a new functionality was enable in the Dynamics 365 Business Central admin center. The ability to request a backup. It didn’t take long before I got the first question from a partner, who asked whether they could run this locally using Docker. This blog post describes how to do just that. Continue reading