Running tests in 15.x insider containers

Posted on by

Update 2021/2/10: Microsoft stopped creating images for Docker in the summer of 2020. We now publish artifacts, which can be used to spin up containers and BcContainerHelper has replaced NavContainerHelper. This blog post reflects the old way of using NAV/BC on Docker and references NavContainerHelper, which is outdated.

If you didn’t read this blog post: https://freddysblog.com/2019/04/13/running-tests-in-containers/, then please do so before proceeding. This blog post will only describe what’s new when running tests in 15.x containers, which while writing this blog post still are in preview and only available as insider builds through the Ready To Go program. Read more here: https://freddysblog.com/2019/07/31/preview-of-dynamics-365-business-central-2019-release-wave-2/

Get the latest…

In order to run tests with 15.x containers you will need the latest NavContainerHelper (version 0.6.3.2 or later) and you need Business Central 2019 Wave 2 insider build 15.0.35805.0 or later using

docker pull bcinsider.azurecr.io/bcsandbox-master:<country>-<platform>

where <country> is the localization and <platform> is ltsc2016 or ltsc2019 based on your version of Windows (see more here: https://freddysblog.com/2019/07/31/preview-of-dynamics-365-business-central-2019-release-wave-2/).

docker inspect on your image should return the following labels (or newer):

"Labels": {
    ...
    "platform": "15.0.35510.0",
    "tag": "0.0.9.92",
    "version": "15.0.35805.0"
},

Include the test toolkit

Exactly like earlier, you need to either add -includeTestToolkit to New-BcContainer in order to include the Test Toolkit – or you need to run the function Import-TestToolkitToBCContainer in order to import the test Toolkit.

The Test Toolkit Test Libraries consists of 5 apps, which are included on the latest Docker images in the C:\Applications folder:

  • Microsoft_Any.app
  • Microsoft_Library Assert.app
  • Microsoft_System Application Test Library.app
  • Microsoft_Tests-TestLibraries.app
  • Microsoft_Test Runner.app

The source for these applications are also included in .zip files.

Microsoft application tests are also available in the C:\Applications folder and unless you specify -includeTestLibrariesOnly, importing the test toolkit will include all tests. This will take some time.

If you run the following script:

$auth = "NavUserPassword"
$credential = New-Object pscredential 'admin', (ConvertTo-SecureString -String 'P@ssword1' -AsPlainText -Force)
$containerName = "runtest"
$licenseFile = "<licenseFile>"
$imageName = "bcinsider.azurecr.io/bcsandbox-master:w1-ltsc2019"
New-BcContainer -accept_eula `
                -containerName $containerName `
                -imageName $imageName `
                -auth $auth `
                -credential $credential `
                -updateHosts `
                -alwaysPull `
                -includeTestToolkit -includeTestLibrariesOnly `
                -licenseFile $licenseFile

you should see an output ending with something like:

...
Ready for connections!
Reading CustomSettings.config from runtest
Creating Desktop Shortcuts for runtest
Publishing C:\Applications\TestFramework\TestLibraries\Any\Microsoft_Any.app
Synchronizing Any on tenant default
Installing Any on tenant default
App successfully published
Publishing C:\Applications\TestFramework\TestLibraries\Assert\Microsoft_Library Assert.app
Synchronizing Library Assert on tenant default
Installing Library Assert on tenant default
App successfully published
Publishing C:\Applications\TestFramework\TestRunner\Microsoft_Test Runner.app
Synchronizing Test Runner on tenant default
Installing Test Runner on tenant default
App successfully published
Publishing C:\Applications\System Application\Test\Microsoft_System Application Test Library.app
Synchronizing System Application Test Library on tenant default
Installing System Application Test Library on tenant default
App successfully published
Publishing C:\Applications\BaseApp\Test\Microsoft_Tests-TestLibraries.app
Synchronizing Tests-TestLibraries on tenant default
Installing Tests-TestLibraries on tenant default
App successfully published
TestToolkit successfully imported
Container runtest successfully created

As you see, the 5 apps are published and installed, the same happens if you omit the -includeTestToolkit and instead call the Import-TestToolkitToBcContainer function.

After this, you can open the Test Tool shortcut on the desktop and after invoking the Get Test Codeunits action, you can have the single test added to the list.

altesttool

Get and Run the tests

Now you can run the tests from the UI – or you can run them from PowerShell. Try:

Get-TestsFromBCContainer -containerName $containerName -credential $credential

and you should get:

Tests                  Name                  Id 
-----                  ----                  -- 
{WarmupInvoicePosting} Sys. Warmup Scenarios 130411

and if you try to run the tests using:

Run-TestsInBCContainer -containerName $containerName -credential $credential -detailed

you should get:

  Codeunit 130411 Sys. Warmup Scenarios Success (0.472 seconds)
    Testfunction WarmupInvoicePosting Success (0.047 seconds)

The Hello World of tests

Lets try to create a new AL test project. In VS Code, use Ctrl+Shift+P and select AL Go!

Select the latest target platform (4.0 Business Central 2019 release wave 2). Select your own server and change Server, ServerInstance and startupObjectId in launch.json:

"server": "http://<containername>",
"serverInstance": "BC",
"startupObjectId": 130451,

Replace <containerName> with the name of your container.

In app.json, add dependencies on the Test Framework apps:

{
  "appId": "dd0be2ea-f733-4d65-bb34-a28f4624fb14",
  "publisher": "Microsoft",
  "name": "Library Assert",
  "version": "15.0.0.0"
},
{
  "appId": "e7320ebb-08b3-4406-b1ec-b4927d3e280b",
  "publisher": "Microsoft",
  "name": "Any",
  "version": "15.0.0.0"
},
{
  "appId": "9856ae4f-d1a7-46ef-89bb-6ef056398228",
  "publisher": "Microsoft",
  "name": "System Application Test Library",
  "version": "15.0.0.0"
},
{
  "appId": "5d86850b-0d76-4eca-bd7b-951ad998e997",
  "publisher": "Microsoft",
   "name": "Tests-TestLibraries",
   "version": "15.0.0.0"
}
Remove the Helloworld.al and create a new Codeunit like this:
codeunit 50100 "My tests"
{
    Subtype = Test;
    TestPermissions = Disabled;

    var
        Assert: Codeunit "Library Assert";

    [Test]
    [Scope('OnPrem')]
    procedure TestA()
    begin
        Assert.AreEqual(3, 3, '3 and 3 are equal');
    end;

    [Test]
    [Scope('OnPrem')]
    procedure TestB()
    begin
        Assert.AreEqual(3, 4, '3 and 4 are not equal');
    end;
}

Note, that the Library Assert codeunit is coming from the Library Assert application and that the TestLibraries contains the “old” Assert codeunit.

Note also that TestB is supposed to fail.

Publish the app without debugging and it should start the test tool page.

altesttool2

Invoke the Get Test Codeunits action to include the new codeunit. Now you can run the tests directly in the UI (… -> Run Tests) or you can use PowerShell again:

Run-TestsInBCContainer -containerName $containerName -credential $credential -detailed

and the result should be something like this:

  Codeunit 130411 Sys. Warmup Scenarios Success (0.331 seconds)
    Testfunction WarmupInvoicePosting Success (0 seconds)
  Codeunit 50100 My tests Failure (0.331 seconds)
    Testfunction TestA Success (0.017 seconds)
    Testfunction TestB Failure (0.017 seconds)
      Error:
        Assert.AreEqual failed. Expected:<3> (Integer). Actual:<4> (Integer). 3 and 4 are not equal.
      Call Stack:
        "Library Assert"(CodeUnit 130002).AreEqual - Library Assert by Microsoft
        "My tests"(CodeUnit 50100).TestB line 2 - MyTest by Default publisher
        "Test Runner - Mgt"(CodeUnit 130454).RunTests - Test Runner by Microsoft
        "Test Runner - Isol. Codeunit"(CodeUnit 130450).OnRun - Test Runner by Microsoft
        "Test Suite Mgt."(CodeUnit 130456).RunTests - Test Runner by Microsoft
        "Test Suite Mgt."(CodeUnit 130456).RunSelectedTests - Test Runner by Microsoft
        "Command Line Test Tool"(Page 130455).OnAction - Test Runner by Microsoft

Include the Microsoft Tests

If you leave out the -includeTestLibrariesOnly, you will also see that all apps called Microsoft_Tests-<something>.app in the C:\Applications folder will be published and installed as well. This does take some time as there are quite a few test apps.

You should see your container creation end with something like this:

adding tests

You will still have to add the tests you want to run programatically or manually in the test tool.

Note: While writing this blog post (hardcoded in containerhelper 0.6.3.2) the Marketing and the SINGLESERVER test apps are not published as these contains some invalid dotnet references.

Creating your own test suite

In the original running tests blog post, I described how you could populate the Test Suite from code in the install trigger of your app.

If you want to do this in 15.x, you will need to add a dependency to the Test Runner app and populate the tables in that. If you populate the C/AL Test Tool, you won’t be able to find and run them in the AL Test Tool.

C/AL Test Tool vs. AL Test Tool

Note that the “old” C/AL Test Tool objects are still in the base application and if you specify -testpage 130409 to Get-TestsFromBcContainer and Run-TestsInBcContainer, the functions will actually use the “old” test runner. This might be useful if you populate the Test Tool tables in code.

The C/AL test tool will however be removed sometime in the future and we recommend that you start using the AL Test Tool instead.

The AL Test Tool is in the app called Microsoft_Test Runner.app and the source is available in C:\Applications in the container.

Enjoy

Freddy Kristiansen
Technical Evangelist

23 thoughts on “Running tests in 15.x insider containers”

  1. Hi Freddy, if I run your example (with latest build or your specific build bcinsider.azurecr.io/bcsandbox-master:15.0.35805.0) I get the error “Container does not include the TestToolkit yet”. Basically I get this error on any R15 image, both W1 and country specific. I get the same error when I try: Import-TestToolkitToBCContainer -containerName $containername

    The only difference with your example is that I use -bestContainerOS and it pulls the -ltsc2016 image instead of ltsc2019 since I use server 2016. Is there any other way I can use the testtoolkit for R15 images? Thanks!

    Like

    Reply ↓

    • Nevermind, the update of NAVContainerHelper was not properly executed, therefore it was still using an old version. Works now. Thanks for the post!

      Like

      Reply ↓

  2. Hi Freddy,

    When i add “test”: “15.0.0.0” in my app.json file, it ends up with following error.

    2019-09-27 02:16:23.56] The request for path /BC/dev/packages?publisher=Microsoft&appName=Test&versionText=1.0.0.0 failed with code NotFound. Reason: No published package matches the provided arguments.

    What am i missing?

    Like

    Reply ↓

  3. Hi Freddy,
    When I create container with latest BC image (mcr.microsoft.com/businesscentral/sandbox:us-ltsc2016) based on your example, I only saw three test apps (Microsoft_Any.app, Microsoft_Library Assert.app and Microsoft_Test Runner.app) published. But two apps (Microsoft_System Application Test Library.app and Microsoft_Tests-TestLibraries.app) didn’t be published. Do you know why?

    Following is my creating container snippet log in PowerShell.
    …
    Initialization took 377 seconds
    Ready for connections!
    Reading CustomSettings.config from bcdev4
    Creating Desktop Shortcuts for bcdev4
    Publishing C:\Applications\TestFramework\TestLibraries\Any\Microsoft_Any.app
    Synchronizing Any on tenant default
    Installing Any on tenant default
    App successfully published
    Publishing C:\Applications\TestFramework\TestLibraries\Assert\Microsoft_Library Assert.app
    Synchronizing Library Assert on tenant default
    Installing Library Assert on tenant default
    App successfully published
    Publishing C:\Applications\TestFramework\TestRunner\Microsoft_Test Runner.app
    Synchronizing Test Runner on tenant default
    Installing Test Runner on tenant default
    App successfully published
    TestToolkit successfully imported
    Creating .net Assembly Reference Folder for VS Code
    Copying DLLs from C:\Windows\assembly to assemblyProbingPath
    Copying DLLs from C:\Program Files\Microsoft Dynamics NAV\150\Service to assemblyProbingPath
    Copying DLLs from C:\Program Files (x86)\Microsoft Dynamics NAV\150\RoleTailored Client to assemblyProbingPath
    Copying DLLs from C:\Program Files (x86)\Open XML SDK to assemblyProbingPath
    Container bcdev4 successfully created

    Thank you.

    Like

    Reply ↓

  4. Hi Freddy,
    i’m running business central On Premises 15 version, i’m able to run test tool page but code units related to test framework are missing eg;(assert,Library – Lower Permissions etc).

    I have added the test framework four dependencies(any, library assert,Tests-TestLibraries,System Application Test Library) but not being able to download symbols. Can you help ?

    Like

    Reply ↓

  5. Hi Freddy,

    I instanciated and ran the container, which works quite well. However, when I run Get-TestsFromBCContainer nothing is plotted to the output until I open BC in the browser and run the AL Test Tool manually. Is this intended, as you wrote in your article:

    “After this, you can open the Test Tool shortcut on the desktop and after invoking the Get Test Codeunits action, you can have the single test added to the list.

    IMG:altesttool

    Get and Run the tests
    Now you can run the tests from the UI – or you can run them from PowerShell. …”

    Am I misunderstanding something? I can’t work it out.

    I used:
    Image: mcr.microsoft.com/businesscentral/onprem:de-ltsc2019
    Auth: NavUserPassword

    Like

    Reply ↓

  6. Licensing test toolkit for end customers:
    The switches -includeTestToolkit and -includeTestLibrariesOnly works fine when creating a container with our development license.
    If we try the same with a client license file then we get the following error shown below.
    How should a customer develop their own extensions and use the test toolkit if the license does not allow it?

    Publishing C:\ProgramData\NavContainerHelper\Extensions\bcreleased\_Microsoft_Any_15.4.41023.41669.app
    Your program license does not allow you to publish ‘Any’.
    at , : line 12
    Your program license does not allow you to publish ‘Any’.
    At C:\Program Files\WindowsPowerShell\Modules\navcontainerhelper\0.6.5.0\ContainerHandling\Invoke-ScriptInNavContainer.ps1:37 char:13
    + Invoke-Command -Session $session -ScriptBlock $scriptbloc …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : InvalidOperation: (:) [Publish-NAVApp], InvalidOperationException
    + FullyQualifiedErrorId : MicrosoftDynamicsNavServer$BC/nav-systemapplication,Microsoft.Dynamics.Nav.Apps.Management.Cmdlets.PublishNavApp
    + PSComputerName : 720c451ffd3b75ab1cde871e3bb09ccff1ab7e50d684d8edce99206b27baa056

    Like

    Reply ↓

  8. Hi!

    I have a little question about this that is causing me some issues. In all your examples i can see you add the licens to the container and i am doing the same. It seems to work and i can open the client just as normal, however i can not “download symbols” into visual studio code from this container. If i remove the licens “download symbols” works just fine. For the real CI setup i will not have this issue i guess because the extensions is already built.

    I am just wondering if this is working as intended, should it not be possible to download symbols with licens added to the container?

    The errors i am getting are:
    [2020-08-21 15:39:15.13] The request for path /BC/dev/packages?publisher=Microsoft&appName=Base%20Application&versionText=16.0.0.0&tenant=default failed with code Forbidden. Reason: You do not have the following permissions on TableData Published Application: IndirectRead.

    To view details about your permissions, see the Effective Permissions page. To report a problem, refer to the following server session ID: ’35’.
    [2020-08-21 15:39:15.13] The request for path /BC/dev/packages?publisher=Microsoft&appName=System%20Application&versionText=16.0.0.0&tenant=default failed with code Forbidden. Reason: You do not have the following permissions on TableData Published Application: IndirectRead.

    To view details about your permissions, see the Effective Permissions page. To report a problem, refer to the following server session ID: ’33’.
    [2020-08-21 15:39:15.13] The request for path /BC/dev/packages?publisher=Microsoft&appName=System&versionText=16.0.0.0&tenant=default failed with code Forbidden. Reason: You do not have the following permissions on TableData Published Application: IndirectRead.

    To view details about your permissions, see the Effective Permissions page. To report a problem, refer to the following server session ID: ’34’.
    [2020-08-21 15:39:15.14] Could not download reference symbols. Please ensure that:
    1. The correct server name and instance are specified in the launch.json file.
    2. The correct application version is specified in the app.json file.
    3. The dependencies are correctly specified in the app.json file.

    Like

    Reply ↓

  9. Hi Freddy,

    I wonder if you have used -includeTestToolkit in BC 2021 wave 1 container yet? Microsoft_Test Runner_18.0.19416.0 app gives me the following error when I include this param with New-BCContainer. Am I doing something wrong or I shall create a ticket for this?

    …
    Publishing C:\ProgramData\NavContainerHelper\Extensions\bc18demo\_Microsoft_Library Variable Storage_18.0.19416.0.app
    Synchronizing Library Variable Storage on tenant default
    Installing Library Variable Storage on tenant default
    App successfully published
    Copy from container bc18demo (C:\Applications.US\Microsoft_Test Runner_18.0.19416.0.app) to C:\ProgramData\NavContainerHelper
    \Extensions\bc18demo\_Microsoft_Test Runner_18.0.19416.0.app
    Publishing C:\ProgramData\NavContainerHelper\Extensions\bc18demo\_Microsoft_Test Runner_18.0.19416.0.app
    This Extension cannot be published as it has the same Publisher, Name, and Version as a previously published Extension.
    at , : line 12
    This Extension cannot be published as it has the same Publisher, Name, and Version as a previously published Extension.
    At C:\Program
    Files\WindowsPowerShell\Modules\navcontainerhelper\0.7.0.26\ContainerHandling\Invoke-ScriptInNavContainer.ps1:39 char:13
    + Invoke-Command -Session $session -ScriptBlock $scriptbloc …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : InvalidOperation: (:) [Publish-NAVApp], InvalidOperationException
    + FullyQualifiedErrorId : MicrosoftDynamicsNavServer$BC/nav-systemapplication,Microsoft.Dynamics.Nav.Apps.Management.Cm
    dlets.PublishNavApp
    + PSComputerName : 8de4e3a984c3625fa100b1776d9d1cc84cd9f3160c01b359684e74be699a9d36

    Like

    Reply ↓

  10. For anyone else that had the same issue as I did: The ID of the testing apps never changes. In order to install the test dependencies to your app, copy the references, above, and change the version to whatever your BC version is.

    Like

    Reply ↓

Leave a comment