Tuesday, April 16, 2019

Capture Pictures & Video From EasyRepro Tests

It goes without saying that tests are going to fail from time to time. Luckily EasyRepro does a pretty good job of providing descriptive error messages to make troubleshooting issues easier. Inevitably the first person to deal with an issue is going to be the developer and most likely they'll need to re-run the test on their local machine to watch exactly what happens. Debugging or at least offloading some of the triaging tests can be made easier by capturing screenshots and/or videos of tests as they are running.

Screenshots

Alone this isn’t anything new as EasyRepro already has TakeWindowScreenShot which does a screen capture. It expects a path & file name and image format for parameters. I like to have the name of the test in my image name. We can use the TestContext to get the name of the test currently being executed for the file name.

You can get the TestContext object when it’s passed to the ClassInitialize method.

private static TestContext _testContext;
[ClassInitialize]
public static void SetupTests(TestContext testContext)
{
    _testContext = testContext;
}


I created a helper method to use when needing a screenshot. It will create a folder for the images, create a unique filename based on the test name & date, and then call TakeWindowScreenShot to grab the browser screen and save to disk. It also adds the file to the TestContext which is important if running tests in Azure DevOps.

private static void ScreenShot(InteractiveBrowser xrmBrowser, TestContext testContext)
{
    const ScreenshotImageFormat format = ScreenshotImageFormat.Jpeg;

    const string imagePath = "screenshots";
    Directory.CreateDirectory(imagePath);

    var testName = $"{testContext.TestName}_{DateTime.Now:yyyyMMddTHHmmss}";
    var filename = $"{imagePath}\\{testName}.{format.ToString().ToLower()}";
    xrmBrowser.TakeWindowScreenShot(filename, format);

    _testContext.AddResultFile(filename);
}


Videos

As it turns out this is really easy to set up (assuming tests are running under Visual Studio 2017 v15.5+). You’ll need to add a .runsettings file to your solution. To do so, right-click on your solution and add a new item (XML file) and make sure it’s named something like settings.runsettings, the .runsettings extension being key. Make sure this file gets into source control.

The basic content to get recordings working:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
   <RunConfiguration>
     <ResultsDirectory>.\TestResults</ResultsDirectory>
   </RunConfiguration>
   <DataCollectionRunSettings>
     <DataCollectors>
       <DataCollector uri="datacollector://microsoft/VideoRecorder/1.0"
                      assemblyQualifiedName="Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder.VideoRecorderDataCollector, Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
                      friendlyName="Screen and Voice Recorder">
         <!--Video data collector was introduced in Visual Studio 2017 version 15.5 -->
       </DataCollector>
     </DataCollectors>
   </DataCollectionRunSettings>
</RunSettings>


In Visual Studio, under the Test menu –> Test Settings select Select Test Settings File and choose the new .runsettings file.

And it should be as simple as that. After a test executes in the test results folder there should be a sub-folder with a guid for a name and inside it will be a .wmv file which is the recording of the test being run in the browser. The one downside is that it appears to be an all or nothing approach to capturing video. If this is enabled all tests in the run have individual recordings created. I couldn't find a way to start/stop recording on the fly from code.


Azure DevOps

To make use of the images and videos when running tests from Azure DevOps you’ll need to make a couple adjustments.

In the Visual Studio Test task under Settings file choose the .runsettings file you created.
Also if you aren’t already using a Publish Build Artifacts step after the tests run, do so and publish the TestResults folder.

Now any images that were added to the TestContext or videos captured will be available in the test results.


Monday, April 15, 2019

Keep Your EasyRepro NuGet Package References Correct

More than once while working with EasyRepro projects I’ve found myself in a situation where tests that were once working inexplicably stopped. After combing through code and verifying credentials I eventually figured out that references to one or more of the required Selenium components somehow got updated without my knowledge. An example of when this can be particularly frustrating is when the Chrome driver gets updated to the latest version which works with the always updated version of Chrome installed on my machine. Everything works fine when running tests locally. When deploying to Azure DevOps and running on an agent where an older version of Chrome is installed, everything fails because the latest driver doesn’t support older browser versions.

To avoid this issue I created a PowerShell script which will reset the Selenium component versions referenced in the project to what EasyRepro supports. Luckily this mismatch between driver and browser doesn’t seem to effect the opposite scenario of what I previously described, at least when Chrome is being used.

Older driver version + newer browser version = OK
New driver version + older browser version = NOT OK

Code


When this runs it will update the packages.config file in the project and make sure that the versions listed at the beginning of the script match. If there is a mismatch it will also update any references in the project file. If this makes an update when the project is open in Visual Studio you’ll be prompted about a conflicting modification (because of the background update), go ahead and select Overwrite and everything should be good.

There are 2 ways of using the script.

Use during development

1. Add a folder name Tools to your test project and add to it this script and a copy NuGet.exe.
2. Open the .csproj for your project and add these lines:

<Target Name="FixEasyRepro" BeforeTargets="EnsureNuGetPackageBuildImports">
   <Exec Command="powershell.exe -NonInteractive -ExecutionPolicy Unrestricted Tools\FixEasyReproPackageReferences.ps1" />
</Target>


This will execute the script prior to the automatic NuGet package restore which happens before the actual project build.

Use in Azure DevOps

The package restore and build process works a little different in Azure DevOps. The recommended approach is to use a NuGet build task to restore prior to executing the build task. The script will still run however the automatic package restore will not happen. If there was an update made you’d likely see a build failure because NuGet had already restored an incorrect package version. In order to maintain the modification to the project file so it works locally, add a PowerShell task which executes the script before the NuGet task runs. This will correct any mismatches so that the correct versions are restored. When the script re-runs during the build step, everything will already be correct and the build should complete.

Monday, April 8, 2019

Run Automated Tests When Your D365 CE Org Gets Updated

With online D365 CE instances getting updates pushed regularly it makes it almost impossible to ensure that you’ve tested against the latest bits. That’s assuming you even know about each update since the majority go in silently in the background. Realizing that organizations need to test before updates roll out to production or at least be aware of issues caused by updates before users start reporting them, Microsoft has taken the first steps to allow you to get ahead of potential issues.
The Microsoft.Xrm.Tooling.Testing PowerShell library (in preview at time of writing) has been released which provides hooks into a few key backend operations that you hadn’t had access to previously. When one of these events occurs you have the opportunity to run automated tests.

Current backend triggers:
  • OrganizationDatabaseUpdate
  • OrganizationDatabaseUpgrade
  • OrganizationBitsUpdate
  • OrganizationPackageInstall
  • OrganizationPackageUninstall

How does it work?

To start, a developer will need to create a .NET unit test project that will contain any tests that need to be run. The tests really can’t be ‘unit tests’ because by definition unit tests are meant to be totally decoupled from everything. So running tests that don’t act on a live environment isn’t going to tell you anything you didn’t already know. Really what you are after would be better described as an ‘integration tests'. Tests that connect to the live environment that just registered the change are going to uncover things that might not be working. Depending on the complexity of your business and customizations it might be a little tricky to create meaningful tests. Once the tests are written you can use the provided PowerShell commands to upload the tests to the Microsoft environment and register when they should be run. Then when a trigger you’ve hooked into is tripped, your tests run.

Test suite setup

I’ll point out right now that the documentation doesn’t appear to be complete yet so there’s probably a better way to do some of these things but without knowing I just improvised to get things working.

For test purposes I created a unit test project that is only going to perform one operation which is to create a contact record. My ‘test’ consists of a CrmServiceClient connection that creates the record and a basic assert statement that passes the test if there aren’t any exceptions. I’ve also got an app.config file that contains the connection string populated with valid credentials. Also note that a better solution would be to use Azure Key Vault and have it read any credentials from there. It’s a bit more secure and allows for updating the values without having to re-deploy things.

Once you’ve got the code working, zip up the contents of the debug/release folder. I mentioned before that the tests get uploaded to Microsoft’s environment. Best I can tell is that the zip file containing the tests needs to be available via a URL on the public internet as opposed to uploading a local file (again maybe I’m doing it wrong). This appears to only be required initially as the documentation states, “Testing infrastructure will download and store the test suite ZIP file for future runs. Changes to this URL will not affect the test run. If you need to update the test suite, please run this cmdlet again pointing to the new test suite's URL”. I dropped my zip file on a blob store and generated a short-lived SAS token I could add on to the URL.

If you haven’t already done so, install Microsoft.Xrm.Tooling.Testing in PowerShell

install-module Microsoft.Xrm.Tooling.Testing

Then register a test suite to contain the test triggers. Based on how things get configured, a test suite will center around the tests being executed. If you’re going to want different sets of test to run for different triggers it looks like you’ll need separate suites.
  • $apiUrl = Online Management API Service URL
  • $orgId = D365 CE organization ID found on the Developer Resources page
  • $orgUniqueName = D365 CE organization Unique Name found on the Developer Resources page
  • $testSuiteUrl = URL of the unit test project zip file
  • $completionCallbackUrl = URL that will have basic test data sent to it when complete via webhook

$user = "admin@test.onmicrosoft.com"
$password = "password"
$cred = New-Object System.Management.Automation.PSCredential `
     ($user, (ConvertTo-SecureString $password –ASPlainText –Force))
$apiUrl = "https://admin.services.crm.dynamics.com"
$orgId = '38600bb5-ed48-47e4-8844-e05c5516b446'
$orgUniqueName = 'org1a1aa611'
# Test Suite
$testSuiteName = 'IntegrationTests'
$testSuiteUrl = 'https://mystorage.blob.core.windows.net/test/AutoTest1.zip?sv=SASKEY'
$testUserCredential = $cred
$completionCallbackUrl = 'https://prod-03.westus.logic.azure.com:443/workflows/...'
# Test trigger
$testTriggerName = 'IntegrationTests-OrganizationPackageInstall'
$testTriggerType = 'OrganizationPackageInstall'

Register-TestSuite -ApiUrl $apiUrl -OrganizationId $orgId -OrganizationUniqueName $orgUniqueName `
     -TestSuiteName $testSuiteName -TestSuiteUrl $testSuiteUrl -TestUserCredential $testUserCredential `
     -ApiCredential $cred -CompletionCallbackUrl $completionCallbackUrl


You should see a result of true be returned if successful.

Test trigger setup

Setup one or more of the backend triggers in the test suite that will execute your tests. Here the trigger is going to be a 'OrganizationPackageInstall' which I’ve assumed to mean when one of the system level solutions or a solution found in the D365 CE Administration Center is installed (and hopefully updated as well).

Register-TestTrigger -ApiUrl $apiUrl -OrganizationId $orgId -TestSuiteName $testSuiteName `
     -TestTriggerName $testTriggerName -TestTriggerType $testTriggerType -ApiCredential $cred


Again you should see a result of true returned if successful.

To confirm everything is set up correctly, you can run this command to see what is registered for a specific organization.

Get-TestSuites -ApiUrl $apiUrl -OrganizationId $orgId -ApiCredential $cred

It will show the suites and triggers that are currently configured.

Testing this out

You can manually trigger all the tests in a suite to ensure that everything is working.

Start-TestRun -ApiUrl $apiUrl -OrganizationId $orgId -TestSuiteName $testSuiteName `
     -ApiCredential $cred


This will return the ID of the test run.

You can check the status of a run at any point.

$testRunId = '2c5d1e62-ff35-43c3-9681-83de5fbb2235'
Get-TestRun -ApiUrl $apiUrl -OrganizationId $orgId -TestRunId $testRunId `
     -TestSuiteName $testSuiteName -ApiCredential $cred


This will output information about the run including status (running, succeeded, etc.). If complete it returns an event log but its purpose must be something other than reporting on the actual test results as it contains no references to what was executed.

Installing a solution from the Administration Center will trigger this particular event as well.

More about test results

From what I can tell so far, there isn’t a way to see if individual tests have passed or failed. That said, it might be best to add your own way of logging the test results so you can see exactly what passed and what didn’t. When the run completes it triggers a webhook with some basic information about the run. Lots of options open up to get notified so you can go and look at the results. I used a Flow which runs based on an HTTP request being received and I used the URL it generated as the CompletionCallbackUrl when registering the suite. Then it just sends an email notification with all the data.

EasyRepro tests

At the moment it doesn’t appear that whatever is running the tests is set up to handle EasyRepro / UI tests. All my attempts failed which again might mean I’m just doing it wrong. For v1 of this solution I can see where Microsoft wouldn’t have yet built out or allowed for the various supported browsers being installed on the infrastructure to allow for this sort of testing.

If you’re interested in running these types of tests you’re going to have to rely on your own test platform for the execution. In my case I’ve got EasyRepro tests set up to run in Azure DevOps (ADO). In order to get them to run based on one of these provided triggers you can either create a dummy unit test which uses the ADO APIs to execute a build or release directly or just pass the dummy test and have the webhook call a Flow where you use the built in ADO connector to kick something off.

Should the built in ADO connector not work for you, a HTTP action can also be used. Here's an example of queuing a build using the REST API.

POST  https://{your instance}.visualstudio.com/{project name}/_apis/build/builds?api-version=5.0

Headers
Content-Type: application/json

Authorization: Basic
Username: {anything}
Password: {Azure DevOps Personal Access Token}

Body
{
        "definition": {
            "id": {build definition id}
        }
}


Once that kicks off and runs its up to you to handle any further notifications and reporting on test results.

Tuesday, March 26, 2019

CRM Numeric Workflow Utilities v2.3.0.0

Small update which added a few new items. I'm pleased to say that the first two were contributed by Suleiman Ali, it's much appreciated.
  • Nth Root - calculates root value of a number
  • Raise To The Power - calculates exponents
  • Integer To Words - takes a number and converts it into text - example: 50 becomes fifty

Or install using the XrmToolBox Solution Installer by @RajYRaman

Friday, March 15, 2019

Run Your Automated D365 CE UI Tests From Azure DevOps

EasyRepro

If you hadn't heard about it already, EasyRepro is a UI testing framework for Dynamics 365 CE built on top of Selenium which is one of the more popular UI testing frameworks available today. Those who might have tried using Selenium directly with D365 CE in the past only found pain and suffering. For most people that ultimately lead to the conclusion that it wasn't worth investing huge amounts of time creating tests for little upfront return. In my opinion EasyRepro now makes creating UI tests feasible as it abstracts away most of complexities involved with Selenium development and boils down CE interactions to a single line of code in many cases. At the same time it’s still flexible enough to extend or use underlying core Selenium functionality. If you’re at all interested, download the Git repository and give it a test drive.

Running Tests in Azure DevOps

Once you've cloned the EasyRepro project and ran some of the examples on your local machine you'll quickly realize that you can't tie it up all day running tests. So if you’re planning on running dozens or hundreds of these tests you’ll need to find an alternative place to run them from. Azure DevOps (ADO) can be that alternative because it offers solutions for 2 different scenarios when it comes to running automated tests. These being running a group of tests as part of a build & release pipeline and the other being able to have a non-developers run individual tests on demand.

Project Set Up

For example purposes I'll use a Visual Studio test project referencing the Dynamics365.UIAutomation.Api (EasyRepro) libraries from NuGet. I've additionally added in a reference to Microsoft.Azure.KeyVault to handle credentials so we don't need to worry them getting into source control or having to worry about replacing them in any of the ADO processes. For this example we’ll just be using Google Chrome as the browser of choice for testing.

Build Set Up

Assuming tests are created and things are working locally, get the project checked into ADO so we can set up a build. It's going to work like a normal .NET project build.


Start off by creating a new build in ADO using an empty job. I’ve got a repository with the EasyRepro test project that I’m pulling the source from initially. This is executing on a private build agent (running inside a Window container) so I’m just using the command line to do a NuGet restore but you could also use the standard NuGet build task. Build the solution or project like normal using the release configuration. Then probably the most important step, use the Publish Build Artifacts build task to publish the bin/release folder to Azure Pipelines/TFS. This is what makes the compiled assembly and other references available to the release we’ll be setting up to run this.

To keep things updated, under Triggers, check Enable continuous integration so that the project gets rebuilt and published each time an update is made.

Release Set Up

In order for ADO to be able to run tests on demand we need to create a release pipeline. Again don’t start with an existing template, just pick the empty job. Select Artifacts & Add and then choose Build and use your project and the build pipeline you just created as the source. You have a few different options for the default version to use but I’d probably set it to use the latest each time.


In the first and only stage, open the tasks to begin setting up the test run. I start with a Visual Studio Test Platform Installer task. You might need this if you’re running a private build agent. If you’re using a Microsoft hosted agent you shouldn’t need this because it’s already installed but it being there won’t hurt anything. Then add a Visual Studio Test task and user version 2.*. Pay close attention to the configuration. Select tests using a Test run. Make sure the box is checked that indicated the Test mix contains UI tests. Last, make sure the test platform version is using the version Installed by Tools Installer.




Depending on the build agent, you may or may not need to install Chrome (or other browsers for that matter). If things aren’t working, try installing it with the following PowerShell script:

$Path = $env:TEMP;
$Installer = "chrome_installer.exe";
Invoke-WebRequest "http://dl.google.com/chrome/install/375.126/chrome_installer.exe" -OutFile $Path\$Installer;
Start-Process -FilePath $Path\$Installer -Args "/silent /install" -Verb RunAs -Wait;
Remove-Item $Path\$Installer

Test Plan Set Up For Manual Test Execution

Hooking individual tests up to test cases is only required if you want the ability to run them on demand from the Test Plans area in ADO. It works better if you can link the release that was created to the test plan but I’m pretty sure you’ll need either VS Enterprise, Test Professional, MSDN or Test Manager to do so. If you’ve already got a test plan, right click on in and go to Test plan settings. Select the build that was just created and leave the build number at latest. Then choose the pipeline and stage from the release setup.


Assuming you’ve got a Test Case created that matches one of the EasyRepro tests, head back to the project in Visual Studio, open up Test Explorer, and find the test. Right-click on it and choose Associate to Test Case. It’s probably a good idea to make sure you’re connected to the project in ADO before you do this.



Type in the Test Case Id then Add Association. Save and close and when you open the test case in ADO, the automation status should now say ‘Automated’. Repeat for any other tests.

Under Test Plans you should be able to select individual or groups of tests and then under the green Run button pick Run for web application to execute. Having the release tied directly to the test plan saves the user having to choose that same data each time which is a few extra clicks and possible confusion.


A window will open confirming and validating the tests. If there aren’t any errors you can select View test run and sit back and wait for the tests to complete. With any luck they will, otherwise you’ll need to retrace your steps and figure out what’s wrong.



If you aren’t able to edit the test plan you can use Run with options and it will open a window where you can choose the items that got configured when linking at the test plan leave. When selecting the build the user will need to either know the build number or know enough to select find and the pick the correct (likely the most recent) build and pipeline from the list. 

Running In Builds & Releases Automatically

Really it’s just a matter of combining the all the steps from the Build and Release setups above with the exception of the publish build artifact task.


And that’s it. Not much to it once you figure it out so hopefully this saves you the trouble.