Running Tests in Containers

Since the very start of NAV and Business Central on Containers, it has been possible to run tests through the UI (Windows Client and Web Client), just as you have been able to do when you install directly on a machine. A lot of people have however been looking for a way to run tests automated – and… – since NavContainerHelper 0.5.0.3 (February 25th) a function called Run-TestsInNavContainer has been available. A few changes has been implemented since then and I am now ready to describe how to use the function.

Include the Test Toolkit when creating the container

When creating the Container, you can specify -IncludeTestToolkit in order to include Test Toolkit and all standard tests:

$imageName = "mcr.microsoft.com/businesscentral/onprem:w1-ltsc2019"
$credential = New-Object pscredential 'admin', (ConvertTo-SecureString -String 'P@ssword1' -AsPlainText -Force)
$containerName = "mytest"
New-NavContainer -accept_eula `
                 -imageName $imageName `
                 -containerName $containerName `
                 -auth "NavUserPassword" `
                 -Credential $credential `
                 -updateHosts `
                 -includeTestToolkit `
                 -licenseFile "C:\temp\license.flf"

This resembles importing the .fob files in the TestToolKit folder on the DVD or in the Docker image.

If you add -includeTestLibrariesOnly, then you will only get the Test Libraries and not the test codeunits.

You should now find a shortcut on the desktop called myTest Test Tool and opening this will reveal the Test Toolkit:emptytesttool

In this, you setup your test by getting the test codeunits and setup the test suite you want to run:testtool

Now, you can press Run and run the test.

But hey… – this is all old news, we have been able to do this for a long time… – why is this blog post describing this???

Get-TestsFromNavContainer

The simple answer is, that the test suite you setup in the Test Toolkit is reused by the CmdLets. In fact, try to run Get-TestsFromNavContainer:gettests

and you can see the same tests as you have in the Test Tool.

You can specify the testSuite as a parameter to Get-TestsFromNavContainer and with that filter the tests to only that testSuite. Default value for testSuite is DEFAULT, meaning that if your tests are in a different testSuite, you will need to specify that.

You can also specify testCodeunit, which will show only the tests in that codeunit. Note that you still need to specify the testSuite.

Run-TestsInNavContainer

With that, you can run the same tests by using Run-TestsInNavContainer:runtests

Note that if you will only have colors if you are running PowerShell as administrator.

And Yes, Run-TestsInNavContainer will run tests using Page testability (unlike invoking codeunits for testing).

Run-TestsInNavContainer also has parameters for filtering which tests to run:

  • testSuite – you will always filter the tests to a specific test Suite. By default, you filter to the DEFAULT test suite, which means that the remaining parameters will only filter tests in that test suite.
  • testGroup – name pattern of test group to run – or * to run all test groups in the test suite.
  • testCodeunit – name or id pattern of test codeunits to run – or * to run all all test codeunits in the test group / suite.
  • testFunction – name pattern of the test functions to run – or * to run all test functions in the test codeunit / group / suite.

Example:runtests1

Another example:runtests2

You get the picture…

XUnit

Run-TestsInNavContainer can also create an XUnit compatible output from test execution by specifying XUnitResultFileName and the path to a file, which is shared with the container.

Example XUnit result file:

<?xml version="1.0" encoding="UTF-8"?>
<assemblies>
  <assembly name="Sales Document Posting Errors" test-framework="PS Test Runner" run-date="2019-04-13" run-time="10:56:31" total="1" passed="1" failed="0" time="0.563">
    <collection name="Sales Document Posting Errors" total="1" passed="1" failed="0" skipped="0" time="0.563">
      <test name="Sales Document Posting Errors:T001_PostingDateIsInNotAllowedPeriodInGLSetup" method="T001_PostingDateIsInNotAllowedPeriodInGLSetup" time="0.563" result="Pass" />
    </collection>
  </assembly>
  <assembly name="Purch. Document Posting Errors" test-framework="PS Test Runner" run-date="2019-04-13" run-time="10:56:32" total="1" passed="1" failed="0" time="0.5">
    <collection name="Purch. Document Posting Errors" total="1" passed="1" failed="0" skipped="0" time="0.5">
      <test name="Purch. Document Posting Errors:T001_PostingDateIsInNotAllowedPeriodInGLSetup" method="T001_PostingDateIsInNotAllowedPeriodInGLSetup" time="0.5" result="Pass" />
    </collection>
  </assembly>
</assemblies>

XUnit is one of the formats supported by Azure DevOps.

Azure DevOps

As all of the above wasn’t enough, you can also specify AzureDevOps as a parameter. The AzureDevOps parameter signals how test failures will appear in Azure DevOps pipelines. You can specify Warn, Error or No. Default is No as in test failures will not be displayed. Warn or Error will show test failures accordingly.

But wait…, how do I create my test suite?

In most professional programming environments, it is enough to attribute your codeunits and functions as tests in order for them to be added to a test subsystem. The fact that Business Central has a table for tests and a UI for running tests inside the client is because we historically did it like this.

In the future, we should assume that this UI will disappear and I am sure that we will get functionality in VS Code to see which tests are available in my app / in the base app for me to run (looking forward to this).

So, with that, I did not want to create functions to populate the database with tests. This would cause partners to place the responsibility of building the test suite in a wrong place. Building the test suite belongs in the test app. In the future by specifying attributes, for now we have to do slightly more.

My recommendation is to include an install codeunit in your test app, which will populate the test toolkit based on attributes. This means that when Microsoft ships a replacement for the test toolkit, you will only need to remove this codeunit. You can find an example of how to do this here: https://dev.azure.com/businesscentralapps/_git/HelloWorld?path=%2Ftest

installcu

The TestSuite codeunit is also in the same folder. I will probably do some refactoring of this in the near future, but basically this functionality is just temporary.

The future?

At this time, we do not know how the future is going to look, what functionality we will get from the new Test framework and we also don’t know when we will get this functionality.

It is however the goal to refactor the functionality of Get-TestsFromNavContainer and Run-TestsInNavContainer to work with the new test framework, when it is delivered in order for CI/CD scripts to continue working with next gen test toolkit. Note, that we might not be able to support all functionality as I do not know whether we will have test suites and test groups in the future.

Under the hood

This section is for the nerds, who want to know how this works.

You might have noticed, that during the first usage of any of the two functions, it said Importing Objects from C:\ProgramData\NavContainerHelper\Extensions\mytest\PsTestTool\PSTestToolPage.fob

Looking in that folder, you will find 3 files:

  • ClientContext.ps1
  • PsTestFuncions.ps1
  • PSTestToolPage.fob

The Test Tool Page is a new Test Toolkit page called 130409 and if you try to launch that, you will see a simplified Test Toolkit page:130409

Only one action available: Run Selected and no strings for success and failure – instead just simple 0, 1 or 2. Also if ýou scroll all the way to the right, you will see that the list includes the call stack for errors as well.

ClientContext.ps1 is a PowerShell class which basically works as a Client towards NAV / Business Central. With ClientContext you can open a connection to the Client endpoint and act like you are a client. This is the same functionality as used in the performance test toolkit and you can do stuff like:clientcontext

There is no documentation for ClientContext and it will probably only ever be used for running tests while we are waiting for future functionality.

PsTestFunctions.ps1 is a script, which contains functions for Run-Tests and Get-Tests, which both creates a ClientContext (a connection to the Client) and when open the page 130409 and act like a user running the tests needed.

Any usage of this knowledge is at your own risk… the functionality might be obsolete in the future (I think I have mentioned that enough times by now).

 

Enjoy

Freddy Kristiansen
Technical Evangelist

 

 

 

 

11 thoughts on “Running Tests in Containers

  1. Pingback: Part 2: Testing Microsoft Dynamics 365 Business Central from VS Code - Dynamics 365 Business Central Community

  2. Pingback: Part 2: Testing Microsoft Dynamics 365 Business Central from VS Code – James Pearson

  3. Great post. I have just one question – I’m trying to write a yaml pipeline for Azure DevOps that publish the artifact only if the test results are ok, but I don’t know how to analyze in ps the “Run-TestsInNavContainer” command results. Because of the fact that even if the tests fail, the task will still be considered as succeeded, I can’t use the usual Azure DevOps conditions. My idea was to set a flag with the result of “Run-TestsInNavContainer” and use it in the following tasks conditions. Do you have any suggestion?

    Like

  4. Pingback: Running tests in 15.x insider containers | Freddys blog

  5. Pingback: Running tests in 15.x insider containers - Freddy's Blog - Dynamics 365 Business Central/NAV User Group - Dynamics User Group

  6. Hi Freddy,

    Thanks for your posts.
    I’ve read about the new function Get-TestsFromNavContainer but I did not find yet how to use in my Pipelines.
    Could I ask you if in a future post you could cover the process to run not only the Customer Automated Tests but All the tests including the Standard Microsoft?
    Consider the following scenario:
    I create a NAV Container from an online Microsoft image, I import all Microsoft Standard tests and I run them in the container.
    The result is 99.07% are passed, 205 failing out of 22,215.
    Should I try to fix the failing ones to reach the 100% success before to add my custom tests?
    Or maybe I can export the ones that failed and disable them from next run?
    I’m speaking about a scheduled pipeline that would run every night that may take 4-6 hours.

    Thanks in advance for your inputs

    Like

    • I don’t know…
      Basically, I took a copy of page 130401 (new ID = 130409), removed everything with language and stuff I didn’t need. Added stack trace to the repeater and then I use the client services endpoint to mimic a faceless client and run the tests.
      So you can modify page 130409 (and use another ID) and specify -testpage to run-tests – then I guess you can do what you want.

      Do remember – in the future (not 15.x) the in-app test framework will go away and tests will run through the dev. endpoint, so be careful to invest too much in old technology.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s