Introduction

In today’s fast-paced development world, it is essential to have an efficient and streamlined DevOps pipeline. One of the critical aspects of this pipeline is testing, which can be time-consuming and error-prone if done manually. Canvas testing is an essential part of building Power Apps, and automating this process can save a considerable amount of time and effort. In this article, we will discuss how to automate canvas tests in your DevOps pipeline, allowing you to integrate testing into your workflow seamlessly. We will explore the benefits of automation and walk you through the steps to set up your automated testing process.

Estimated reading time: 11 minutes

Table of contents

Why Automate Canvas Tests in Your DevOps Pipline?

Integrating automated Canvas tests into your DevOps pipeline offers several key benefits, bolstering both the efficiency and reliability of your software development process.

1. Enhanced Quality Assurance: Automated tests can perform rigorous, repeatable testing that ensures every part of your Canvas application works as intended. It leaves no room for human error and can cover a vast array of test cases.
2. Increased Speed and Efficiency: Automation significantly speeds up the testing phase in your DevOps pipeline. It allows for continuous testing that can run in parallel with development, reducing the time taken from coding to deployment.
3. Early Bug Detection: By integrating automated Canvas tests into your pipeline, bugs and issues can be detected and addressed early in the development cycle. This helps prevent minor issues from escalating into major problems further down the line.
4. Improved Collaboration: Automated tests provide a reliable basis for discussions between developers, testers, and operations, fostering collaboration and shared understanding of the application’s quality.
5. Consistency and Reliability: Automated tests ensure that checks are performed consistently in every run. This improves the reliability of tests and provides confidence that any changes to the codebase do not adversely affect existing functionality.

Incorporating automated Canvas tests into your DevOps pipeline is, therefore, a highly effective strategy for maintaining high-quality, robust applications while maximizing efficiency and collaboration within your development team.

How to Set Up and Run Canvas Tests Using Azure Pipelines

Fork the Power AppsTestAutomation project

Forking a repository essentially means creating a duplicate of the original repository. This facilitates changes and modifications to be made independently, without impacting the integrity or functionality of the original project.

1. Initiate the process by logging into your GitHub account.
2. Directly navigate to the microsoft/PowerAppsTestAutomation repository, or alternatively, use the GitHub search tool to find the microsoft/PowerAppsTestAutomation repository, and then select it from the given options.
3. Click on the Fork button within the repository.
4. A prompt will appear asking you to choose the account where you wish to create the fork.
5. Once you’ve selected the appropriate account, the forking process will commence.
6. Upon successful completion, your newly forked repository will be ready for use and can be found within your chosen account.

Create a Test URL.json file

To validate your application during testing, you will need to create a Test URL .json file. This file should contain the test suite and test case URLs that you want to use for testing.

To obtain the app test suite and test case URLs, you can use the Copy play link feature in Test Studio. This will allow you to easily retrieve the necessary URLs and add them to your Test URL .json file. By including these URLs in your testing process, you can ensure that your application is thoroughly tested and any issues are identified and resolved before release. For further information check this article: How To UI Test Your Canvas App Using Test Studio

You can find a sample file Samples/TestAutomationURLs.json in the repo you created earlier.

1. Create a new TestURLs.json file in your repo, or use any other file name.
   The file name and location will be mapped in the pipeline variables later in the document.
2. Copy the format from the Samples/TestAutomationURLs.json file.
3. Update the Test URLs section with the tests that you want to validate in your app.
4. Commit the changes to your repo:

Create a Pipeline

1. Sign in to your Azure DevOps instance.
2. Select an existing project or create a new project.
3. Select Pipelines in the left menu.
4. Select Create Pipeline.
5. Select Use the classic editor:

6. Select GitHub as the source.
7. Select … (ellipsis) from the right side of Repository input.
8. Enter the name of your project on GitHub, and then Select it:

9. Select Continue.
10. In the Select a template screen, select Empty job:

11. Save your pipeline.

Add Tasks to the Pipeline

To run tests from the pipeline, you need to configure the job tasks in the correct sequence. Follow these steps to set up the tasks properly:

• Configure the screen resolution using PowerShell.
• Restore NuGet packages for the PowerAppsTestAutomation solution.
• Build the PowerAppsTestAutomation solution.
• Add Visual Studio Tests for Google Chrome.
• Add Visual Studio Tests for Mozilla Firefox.

By configuring the tasks in this order, you can ensure that the screen resolution is correctly set up before running any tests. This can help prevent any potential issues that may arise due to incorrect screen resolution settings. Additionally, restoring the NuGet packages and building the solution are important steps to ensure that the tests can be executed successfully. Finally, adding the Visual Studio Tests for Google Chrome and Mozilla Firefox will enable you to test your application on different browsers and ensure compatibility across multiple platforms.

Configure Screen Resolution using PowerShell

1. Select + next to Agent job 1.
2. Search for PowerShell.
3. Select Add to add a PowerShell task to the job.
4. Select the task and pdate the display name to Set Agent Screen Resolution to 1920 x 1080  or similar.
5. Select Inline as a script type, and enter the following in the script window:

# Set agent screen resolution to 1920x1080 to avoid sizing issues with Portal  
Set-DisplayResolution -Width 1920 -Height 1080 -Force
# Wait 10 seconds  
Start-Sleep -s 10
# Verify Screen Resolution is set to 1920x1080  
Get-DisplayResolution

Restore NuGet Packages

1. Select + next to Agent job 1.
2. Search for NuGet.
3. Select Add to add a NuGet task to the job.
4. Select the task and update the display name to Restore NuGet Packages or similar.
5. Select … (ellipsis) in the Path to solution, packages.config, or project.json configuration field.
6. Select the PowerAppsTestAutomation.sln solution file.
7. Select OK

Build the Solution

1. Select + next to Agent job 1.
2. Search for Visual Studio build.
3. Select Add to add a Visual Studio build task to the job.
4. Select the task, and update the display name to Build Power Apps Test Automation Solution or similar.
5. Select … (ellipsis) in the Solution configuration field.
6. Select the PowerAppsTestAutomation.sln file.
7. Select OK.

Add Visual Studio Test for Google Chrome

1. Select + next to Agent job 1.
2. Search for Visual Studio Test.
3. Select Add to add Visual Studio Task to the job.
4. Select the task and update the display name to Run Power Apps Test Automation Tests via Chrome or similar.
5. Remove the default entries in the Test files field and add the following: **\Microsoft.PowerApps.TestAutomation.Tests\bin\Debug\Microsoft.PowerApps.TestAutomation.Tests.dll
6. Enter TestCategory=PowerAppsTestAutomation in the Test filter criteria field.
7. Select Text mix contains UI tests.

8. Select …(ellipsis) in the Settings file field.
9. Expand the Microsoft.PowerApps.TestAutomation.Tests, select the patestautomation.runsettings file, and then select OK:

10. Copy the following in the Override test run parameters field.

-OnlineUsername "$(OnlineUsername)" -OnlinePassword "$(OnlinePassword)" -BrowserType "$(BrowserTypeChrome)" -OnlineUrl "$(OnlineUrl)" -UsePrivateMode "$(UsePrivateMode)" -TestAutomationURLFilePath "$(TestAutomationURLFilePath)" -DriversPath "$(ChromeWebDriver)" -LoginMethod "$(LoginMethod)"

11. Enter Run Power Apps Test Automation Tests via Chrome or similar in the Test run title field.

Add Visual Studio Test for Mozilla Firefox

1. Right-click the Add Visual Studio Tests for Chrome task and select Clone task(s).
2. Select the task and update the display name to Run Power Apps Test Automation Tests via Firefox or similar.
3. Override test run parameters:

-OnlineUsername "$(OnlineUsername)" -OnlinePassword "$(OnlinePassword)" -BrowserType "$(BrowserTypeFirefox)" -OnlineUrl "$(OnlineUrl)" -UsePrivateMode "$(UsePrivateMode)" -TestAutomationURLFilePath "$(TestAutomationURLFilePath)" -DriversPath "$(GeckoWebDriver)" -LoginMethod "$(LoginMethod)"

4. Enter Run Power Apps Test Automation Tests via Firefox or similar in the Test run title field

Configure Pipeline Variables

Now we shall configure the pipeline variables referenced in the tasks.

1. Select the Variables tab.
2. Select Add and repeat this step to configure the following variables:

3. For security reasons, the OnlinePassword variable should be store a secret value.
4. Check the lock image to make this variable a secret.

5. Save your pipeline configuration.

Run and Analyze Tests

After configuring your tests, it’s important to validate that they are executing successfully. Here’s what you need to do:

1. Select the Queue option.
2. Then, select the Run option to start the job.

3. As the job runs, select the job to see a detailed status on each of the tasks running:

Once the job has completed running your tests, you can view a high-level summary of the job, including any errors or warnings that were encountered. Additionally, you can access detailed information about the individual test cases by selecting the Tests tab.

For example, if you ran your tests using the Chrome browser and encountered a failed test case, the Tests tab will provide specific details about the failure. By analyzing these details, you can identify the root cause of the issue and make the necessary adjustments to ensure that your application is functioning as expected.

Limitations

In my previous article, I noted that despite the advantages of using Test Studio, this method is not without its limitations:

• Components.
• Code components written in the Power Apps Component Framework.
• Nested galleries.
• Media controls.

In my forthcoming article, I will address these limitations and provide a comprehensive solution to overcome them. Stay tuned for practical insights

Conclusion

The integration of automated Canvas tests into your DevOps pipeline is a game-changer. It’s not just a tweak to your process, but a transformative step that can supercharge your software development routine.

With automated testing, you’re bringing robust quality assurance, faster workflow, earlier bug detection, better team collaboration, and unswerving reliability to the table. It’s like having an ever-vigilant watchdog ensuring that each aspect of your application gets scrutinized thoroughly, eliminating the chance of oversights or human error.

Even though there may be some hiccups when using specific tools like Test Studio, the overwhelming benefits make this an approach you can’t afford to ignore. Plus, don’t worry too much about the limitations – we’ve got strategies to tackle those, and I’ll delve into them in my next article.

To sum it up, embedding automated Canvas tests in your DevOps pipeline is a key ingredient to a strong, efficient, and effective software development recipe. It’s all about crafting top-notch applications that tick all the boxes for your users.

Resource Links

These links below gave me inspiration and use cases in this post:

• Automate tests with Azure Pipelines using classic editor
• UI Test Automation: Definition, Guide, and Best Practices