Update 2021/2/10: Microsoft stopped creating images for Docker in the summer of 2020. We now publish artifacts, which can be used to spin up containers and BcContainerHelper has replaced NavContainerHelper. This blog post reflects the old way of using NAV/BC on Docker and references NavContainerHelper, which is outdated.
Since the very start of NAV and Business Central on Containers, it has been possible to run tests through the UI (Windows Client and Web Client), just as you have been able to do when you install directly on a machine. A lot of people have however been looking for a way to run tests automated – and… – since NavContainerHelper 0.5.0.3 (February 25th) a function called Run-TestsInNavContainer has been available. A few changes has been implemented since then and I am now ready to describe how to use the function.
Include the Test Toolkit when creating the container
When creating the Container, you can specify -IncludeTestToolkit in order to include Test Toolkit and all standard tests:
$imageName = "mcr.microsoft.com/businesscentral/onprem:w1-ltsc2019" $credential = New-Object pscredential 'admin', (ConvertTo-SecureString -String 'P@ssword1' -AsPlainText -Force) $containerName = "mytest" New-NavContainer -accept_eula ` -imageName $imageName ` -containerName $containerName ` -auth "NavUserPassword" ` -Credential $credential ` -updateHosts ` -includeTestToolkit ` -licenseFile "C:\temp\license.flf"
This resembles importing the .fob files in the TestToolKit folder on the DVD or in the Docker image.
If you add -includeTestLibrariesOnly, then you will only get the Test Libraries and not the test codeunits.
You should now find a shortcut on the desktop called myTest Test Tool and opening this will reveal the Test Toolkit:
In this, you setup your test by getting the test codeunits and setup the test suite you want to run:
Now, you can press Run and run the test.
But hey… – this is all old news, we have been able to do this for a long time… – why is this blog post describing this???
The simple answer is, that the test suite you setup in the Test Toolkit is reused by the CmdLets. In fact, try to run Get-TestsFromNavContainer:
and you can see the same tests as you have in the Test Tool.
You can specify the testSuite as a parameter to Get-TestsFromNavContainer and with that filter the tests to only that testSuite. Default value for testSuite is DEFAULT, meaning that if your tests are in a different testSuite, you will need to specify that.
You can also specify testCodeunit, which will show only the tests in that codeunit. Note that you still need to specify the testSuite.
With that, you can run the same tests by using Run-TestsInNavContainer:
Note that if you will only have colors if you are running PowerShell as administrator.
And Yes, Run-TestsInNavContainer will run tests using Page testability (unlike invoking codeunits for testing).
Run-TestsInNavContainer also has parameters for filtering which tests to run:
- testSuite – you will always filter the tests to a specific test Suite. By default, you filter to the DEFAULT test suite, which means that the remaining parameters will only filter tests in that test suite.
- testGroup – name pattern of test group to run – or * to run all test groups in the test suite.
- testCodeunit – name or id pattern of test codeunits to run – or * to run all all test codeunits in the test group / suite.
- testFunction – name pattern of the test functions to run – or * to run all test functions in the test codeunit / group / suite.
You get the picture…
Run-TestsInNavContainer can also create an XUnit compatible output from test execution by specifying XUnitResultFileName and the path to a file, which is shared with the container.
Example XUnit result file:
<?xml version="1.0" encoding="UTF-8"?> <assemblies> <assembly name="Sales Document Posting Errors" test-framework="PS Test Runner" run-date="2019-04-13" run-time="10:56:31" total="1" passed="1" failed="0" time="0.563"> <collection name="Sales Document Posting Errors" total="1" passed="1" failed="0" skipped="0" time="0.563"> <test name="Sales Document Posting Errors:T001_PostingDateIsInNotAllowedPeriodInGLSetup" method="T001_PostingDateIsInNotAllowedPeriodInGLSetup" time="0.563" result="Pass" /> </collection> </assembly> <assembly name="Purch. Document Posting Errors" test-framework="PS Test Runner" run-date="2019-04-13" run-time="10:56:32" total="1" passed="1" failed="0" time="0.5"> <collection name="Purch. Document Posting Errors" total="1" passed="1" failed="0" skipped="0" time="0.5"> <test name="Purch. Document Posting Errors:T001_PostingDateIsInNotAllowedPeriodInGLSetup" method="T001_PostingDateIsInNotAllowedPeriodInGLSetup" time="0.5" result="Pass" /> </collection> </assembly> </assemblies>
XUnit is one of the formats supported by Azure DevOps.
As all of the above wasn’t enough, you can also specify AzureDevOps as a parameter. The AzureDevOps parameter signals how test failures will appear in Azure DevOps pipelines. You can specify Warn, Error or No. Default is No as in test failures will not be displayed. Warn or Error will show test failures accordingly.
But wait…, how do I create my test suite?
In most professional programming environments, it is enough to attribute your codeunits and functions as tests in order for them to be added to a test subsystem. The fact that Business Central has a table for tests and a UI for running tests inside the client is because we historically did it like this.
In the future, we should assume that this UI will disappear and I am sure that we will get functionality in VS Code to see which tests are available in my app / in the base app for me to run (looking forward to this).
So, with that, I did not want to create functions to populate the database with tests. This would cause partners to place the responsibility of building the test suite in a wrong place. Building the test suite belongs in the test app. In the future by specifying attributes, for now we have to do slightly more.
My recommendation is to include an install codeunit in your test app, which will populate the test toolkit based on attributes. This means that when Microsoft ships a replacement for the test toolkit, you will only need to remove this codeunit. You can find an example of how to do this here: https://dev.azure.com/businesscentralapps/_git/HelloWorld?path=%2Ftest
The TestSuite codeunit is also in the same folder. I will probably do some refactoring of this in the near future, but basically this functionality is just temporary.
At this time, we do not know how the future is going to look, what functionality we will get from the new Test framework and we also don’t know when we will get this functionality.
It is however the goal to refactor the functionality of Get-TestsFromNavContainer and Run-TestsInNavContainer to work with the new test framework, when it is delivered in order for CI/CD scripts to continue working with next gen test toolkit. Note, that we might not be able to support all functionality as I do not know whether we will have test suites and test groups in the future.
Under the hood
This section is for the nerds, who want to know how this works.
You might have noticed, that during the first usage of any of the two functions, it said Importing Objects from C:\ProgramData\NavContainerHelper\Extensions\mytest\PsTestTool\PSTestToolPage.fob
Looking in that folder, you will find 3 files:
The Test Tool Page is a new Test Toolkit page called 130409 and if you try to launch that, you will see a simplified Test Toolkit page:
Only one action available: Run Selected and no strings for success and failure – instead just simple 0, 1 or 2. Also if ýou scroll all the way to the right, you will see that the list includes the call stack for errors as well.
ClientContext.ps1 is a PowerShell class which basically works as a Client towards NAV / Business Central. With ClientContext you can open a connection to the Client endpoint and act like you are a client. This is the same functionality as used in the performance test toolkit and you can do stuff like:
There is no documentation for ClientContext and it will probably only ever be used for running tests while we are waiting for future functionality.
PsTestFunctions.ps1 is a script, which contains functions for Run-Tests and Get-Tests, which both creates a ClientContext (a connection to the Client) and when open the page 130409 and act like a user running the tests needed.
Any usage of this knowledge is at your own risk… the functionality might be obsolete in the future (I think I have mentioned that enough times by now).
Great web site. A lot of helpful info here. I’m sending
it to some pals ans additionally sharing in delicious.
And naturally, thank you to your sweat!
Pingback: Part 2: Testing Microsoft Dynamics 365 Business Central from VS Code - Dynamics 365 Business Central Community
Pingback: Part 2: Testing Microsoft Dynamics 365 Business Central from VS Code – James Pearson
Great post. I have just one question – I’m trying to write a yaml pipeline for Azure DevOps that publish the artifact only if the test results are ok, but I don’t know how to analyze in ps the “Run-TestsInNavContainer” command results. Because of the fact that even if the tests fail, the task will still be considered as succeeded, I can’t use the usual Azure DevOps conditions. My idea was to set a flag with the result of “Run-TestsInNavContainer” and use it in the following tasks conditions. Do you have any suggestion?
Typically people use the -azuredevops, then test errors will be errors in the pipeline.
Pingback: Running tests in 15.x insider containers | Freddys blog
Pingback: Running tests in 15.x insider containers - Freddy's Blog - Dynamics 365 Business Central/NAV User Group - Dynamics User Group
Thanks for your posts.
I’ve read about the new function Get-TestsFromNavContainer but I did not find yet how to use in my Pipelines.
Could I ask you if in a future post you could cover the process to run not only the Customer Automated Tests but All the tests including the Standard Microsoft?
Consider the following scenario:
I create a NAV Container from an online Microsoft image, I import all Microsoft Standard tests and I run them in the container.
The result is 99.07% are passed, 205 failing out of 22,215.
Should I try to fix the failing ones to reach the 100% success before to add my custom tests?
Or maybe I can export the ones that failed and disable them from next run?
I’m speaking about a scheduled pipeline that would run every night that may take 4-6 hours.
Thanks in advance for your inputs
We will cover that. The idea is that every build should ship with a DisabledTests text file and you should be able to run all non-disabled tests with the image/demo data.
Does Run-TestsInNavContainer work with a custom test runner?
Thanks in advance.
I don’t know…
Basically, I took a copy of page 130401 (new ID = 130409), removed everything with language and stuff I didn’t need. Added stack trace to the repeater and then I use the client services endpoint to mimic a faceless client and run the tests.
So you can modify page 130409 (and use another ID) and specify -testpage to run-tests – then I guess you can do what you want.
Do remember – in the future (not 15.x) the in-app test framework will go away and tests will run through the dev. endpoint, so be careful to invest too much in old technology.
LikeLiked by 1 person
Hello Freddy and thanks for the information” it is always very useful to read your posts.
I am having a problem while testing my extensions.
The thing is that i don’t know how can i check the code coverage of my unit tests.
If I try to check it in the old CAL Code coverage page (page 9990) i cannot see any hit on my table extensions and page extensions code even though I have a 100% hit in my Testing codeunit.
How can I know the code coverage of my tests? I cannot find any information about it anywhere. Any hint or idea?
Thanks in advance
The AL Test Tool doesn’t support code coverage yet, sorry.
They are working on it…
Hey, Freddy, any news on the code coverage?
Pingback: Running Tests In Containers | Freddys blog
I can not find the TestSuite codeunit.
Any pointers on populating the test suite?
in 15.x we don’t populate the testsuite anymore – we invoke all tests in an app like explained here: https://freddysblog.com/2019/10/22/running-tests-in-containers-2/ – the “old” blog post refers to 14.x and you can find the testsuite codeunit in the 14.x version of helloworld here: https://dev.azure.com/businesscentralapps/HelloWorld/_git/HelloWorld%2014.x
Thank you… I saw that but missed that bit.
Is it possible to install the test toolkit after the docker is created? Running a ImportNavData script after the docker is crated to get some data to a addon. And I cant have the test tookit installed when running that command. So I need to install it afterwards.
Yes, using the command Import-TestToolkitToBcContainer
I created a testrunner and added to the testrunner in the Onrun trigger all my Testcodeunits which needed to be run.
But how can i get “Run-TestsInBcContainer” to run this Testrunner? I am working on NAV2018.
I tried already with -testCodeunit but this is not working, because i guess a testrunner is not a testcodeunit.
Your help would be much appreciated
There is a parameter called -testRunnerCodeunitId I think – that one should work…
Yes, thats correct. If i add this, i get than following error:
“Specifying testRunnerCodeunitId is not supported when using the C/AL test runner”
In your code i found that you have to use the testpage with id 130455. But this id is not in my database.
Where do i get this object? Should it be available already in NAV2018?
Ahh – sorry, yes NAV 2018 only supports the C/AL test runner.
Not sure how to go around that.
Means that, that i can not use it in devops pipeline?
All my tests are passing currently if i run them manually in the UI/Testtool but failing if i call them via Run-TestsInNavContainer. I think it is because of the ClientContext you mentioned and the slightly different transaction/rollback scenario.
No idea currently how to get around this, do you have any idea?
I would assume that you could use this in the same way as we ran tests back then – but I haven’t tried for a long time.
I actually think that BcContainerHelper tests runs tests on NAV 2018 – but not with a custom testrunner. Just by adding a testsuite and then use the C/AL test runner