Automated Testing for Broken Visuals - It’s a Team Sport

Over the past few years, I’ve found the Microsoft Power BI and Data community to be one of the most active and generous I’ve encountered. They truly embrace the DataOps philosophy that analytics is a team sport.

It’s a team sport: Analytics teams bring together a variety of roles, skills, tools, and titles. This diversity in backgrounds and perspectives fuels innovation and productivity.

A few weeks ago, I demonstrated how to use Playwright to test for broken visuals, and attendees like Kevin Arnold provided valuable feedback, including:

1. Support for bookmarks - Iterating through bookmarks to find broken visuals would be beneficial. Bookmarks may reveal visuals that are otherwise hidden, potentially exposing broken visuals.

2. Simplified test case generation - Currently, generating test cases involves PowerShell commands. It would be ideal to select a report and automatically generate CSV test cases.

Implementing the Updates

Accessing bookmarks - This was easier said than done. I wanted to avoid parsing .pbix files, and since .pbip isn’t Generally Available (GA) yet, parsing report.json to get bookmarks wasn’t ideal. The most reliable approach was using powerbi-javascript to render the reports and retrieve bookmarks. This made it even more important to accomplish item #2 because rendering reports with powerbi-javascript is essential to access bookmarks.

Generating CSVs for tests - I leveraged existing tooling required to run Playwright tests. Since Node.js is required, I extended the testing tool to include a local Node app. Here’s how it works:

1. In Visual Studio Code, navigate to the test-generation folder.

2. Run “npm install” to install the necessary Node modules.

3. Run “node app.js” to start the app on localhost:3000. This app uses the .env file, set up before testing, to store service principal and tenant information for accessing the Power BI APIs and rendering reports.

4. Go to localhost:3000. This app has four main steps:

   Step 1 - Load Workspaces - This validates the service principal setup and confirms the app can generate an access token.

   Figure 1 - Load Workspaces
   Step 2 - Select Workspace - Choose the workspace where the report is located.

   Figure 2 - Select a Power BI Workspace
   Step 3 - Select Report - After selecting a report, the app checks if the report’s semantic model uses Row-Level Security (RLS). This check is done using a PowerShell script, as Dynamic Management Views for a semantic model are restricted in REST APIs. Once verified, the tool has the data needed to generate test cases for each role and an embed token.

   During this step, we also render the report to access the Bookmark manager, which is currently the only GA-supported way across Power BI and Fabric to retrieve bookmark information.

   Figure 3 - Select the Report
   Step 4 – Save Test Cases With all data collected, press the button to save tests to the test-cases folder. The generated CSV includes permutations of test cases (pages/tabs, bookmarks, and roles).

   Figure 4 - Save Test Cases

Sharing with the Team

Thanks to the feedback from this amazing community, I’m excited to share the latest version with instructions. This version is also compatible with Azure DevOps. Just follow these instructions to set up the pipeline.

If you’ll be at TechCon365 Dallas, please stop by my session on November 15th, where I go in-depth on broken visual testing and the setup.

As always, let me know your thoughts on LinkedIn or Twitter/X on the approach and ways to improve it and the documentation.