John Kerski’s Blog

Five Years of Bringing DataOps to Power BI and Fabric

2026-07-26T00:00:00+00:00

It has been five years since I started this blog, but somehow it feels like almost a decade has passed since 2021.

When I launched the blog, my goal was simple: bring DataOps principles to Power BI. Today, with Microsoft Fabric and AI reshaping the landscape, the application of those principles has expanded significantly.

Over the past month, I’ve been reflecting on how DataOps principles are showing up throughout Fabric and Power BI.

Improving Cycle Times

Whether it was intentional design or a happy accident, the introduction of Power BI Project files at the dawn of mainstream AI adoption dramatically accelerated how we build Power BI solutions.

Today, we’re not only developing semantic models and reports more efficiently, but we’re also building user interfaces with Fabric Apps and custom JavaScript applications layered on top of Power BI and Fabric.

AI has further accelerated solution development across the Fabric platform. It has enabled my teams and me to build projects such as Power Query Lint and Power Query Escape Room, both powered by Fabric Real-Time Intelligence and Fabric SQL.

Reuse

Outside of AI, features such as User-Defined Functions (UDFs), TMDL View, and DAX Query View are finally making it easier to reuse logic and share code across teams.

For years, reusability has been a challenge in Power BI development. These capabilities represent meaningful progress toward standardizing and sharing patterns, which is a core DataOps principle.

Quality Is Paramount

Testing is now becoming a first-class citizen within semantic models.

User-Defined Functions have opened the door for automated testing scenarios, and they enabled my team to develop PQL.Assert, which we shared with the community to help automate semantic model testing.

As DataOps practitioners know, quality cannot be an afterthought. The faster we can build, the more important automated validation becomes.

Room for Improvement

With all these advances, it’s easy to become complacent. Success can sometimes blind us to the opportunities that still exist.

However, complacency eventually leads to missed opportunities. While Microsoft has made tremendous progress, there are still several improvements that could further strengthen the application of DataOps principles within Fabric and Power BI.

Here is my wish list for 2026.

1. Azure Key Vault as a Fabric Artifact

This has been a frustration of mine for years, and it is clearly something many in the community have been asking for.

When Variables were introduced, I hoped they would become the foundation for securely storing secrets within Fabric tenants. Unfortunately, that hasn’t happened.

Today, obtaining and managing an Azure Key Vault can still be a bureaucratic process or a bridge too far for many organizations. As a result, some practitioners take the path of least resistance and store secrets directly in Power Query, Dataflows, or other places where they do not belong.

Yes, secrets can be passed from a pipeline, but that process is still cumbersome and ultimately depends on having a Key Vault.

In the age of AI, this becomes even more important.

Imagine if Fabric included a native Key Vault artifact. We could reference secrets securely within the Fabric ecosystem, and AI-powered code reviews could identify exposed credentials and automatically recommend moving them into a governed secret store. It would improve security while simplifying development.

2. Break Up expressions.tmdl

Let’s face it: tokens are the currency of AI.

When an agent has to read a 2,000-line expressions.tmdl file just to update a single function or table, we’re wasting tokens and increasing processing overhead.

The rapid growth of User Defined Functions has made this file even larger, which can create challenges in Visual Studio Code and increase merge conflicts during parallel development.

A more modular approach that stores tables, parameters, and functions in separate files would reduce merge conflicts, improve Git workflows, and make AI-assisted development significantly more efficient.

3. Native Regular Expression Support in Power Query

The community first requested regex support years ago, and it remains one of the most obvious gaps in the Power Query language.

There are workarounds involving HTML and JavaScript, but they do not work reliably in the service. Many of my teams have resorted to using Azure AI Foundry or external services simply to perform regex operations.

This feels unnecessary when .NET already provides robust regex functionality that could be surfaced directly through Power Query.

Regular expressions are a foundational capability for data transformation and should not require external dependencies.

4. OAuth Support for POST Requests in Web.Contents

If we cannot have a native Key Vault artifact and we still need external services for capabilities like regex, then we need better authentication support in Web.Contents.

Today, many APIs require POST requests that include payloads and OAuth authentication. In practice, developers often end up using anonymous connections alongside API keys and then face the challenge of securing those keys.

It is a cascading problem that creates unnecessary complexity and security concerns.

Supporting OAuth-authenticated POST requests natively would remove a significant friction point for Power BI and Fabric developers.

5. Skills as a Fabric Artifact or Environment Resource

Skills have quickly become an essential part of AI-enabled development.

However, they have also created a version-control challenge.

Today, skills files are often copied to individual developer machines. While convenient, this also means that different developers can experience different outcomes based on local configuration differences.

This is not an unsolved problem. The software industry has managed package dependencies for decades through ecosystems like npm and PyPI. In fact, Microsoft has produced the agent package manager to solve this problem for agents and skills.

I’d love to see skills, agents, and MCP configuration become first-class artifacts, or perhaps part of a Fabric Environment, so they can be versioned, managed, shared, and governed consistently across teams in the service and locally.

Even more importantly, I would love Fabric Copilot and related AI experiences to automatically leverage those skills so that AI interactions align with established standards and best practices.

6. GitHub App Connections for Fabric Git Integration

Git integration is critical for DataOps.

While Azure DevOps integration is excellent, GitHub integration still relies heavily on Personal Access Tokens (PATs).

I would love to see support for GitHub App connections that eliminate the need to manage, rotate, and secure PATs manually. It would simplify administration and align better with modern GitHub security practices.

7. Extend User Data Functions to Be More Like Azure Functions

I am a big fan of User Data Functions (not to be confused with User Defined Functions in DAX) and their writeback capabilities.

However, as Fabric increasingly becomes the backend platform for storing, processing, and orchestrating data for front-end applications, something Fabric Apps is making increasingly apparent, we need more of the capabilities offered by Azure Functions.

I would love to see support for additional runtimes such as Node.js, TypeScript, and PowerShell, along with features like deployment slots.

Azure Functions are incredibly powerful, but many organizations struggle with the overhead of provisioning Azure resources. Bringing more of those capabilities directly into Fabric would lower barriers to adoption and accelerate innovation.

8. Expand Fabric Apps Beyond Entra-only Internal Applications

Fabric Apps have introduced an exciting possibility: JavaScript applications backed entirely by Fabric.

With existing data layers, GraphQL support, and User Data Functions acting as an API layer, Microsoft has an opportunity to think beyond internal enterprise applications.

Fabric Apps already provide application hosting and static content hosting. The next step is enabling organizations to build truly customer-facing applications within Fabric.

Imagine being able to designate an application or workspace for public consumption while supporting authentication providers beyond Microsoft Entra ID. Organizations could build and publish customer-facing data applications directly from Fabric without introducing additional hosting platforms.

Today, solutions like Azure Static Web Apps and Vercel fill this gap. However, once a project extends beyond the Fabric boundary, adoption often slows as teams encounter additional security reviews, coordination efforts, and infrastructure costs.

Bringing these capabilities closer to Fabric could dramatically simplify the path from analytics solution to external-facing application.

Looking Back and Forward

Despite the opportunities I’ve outlined, I am genuinely pleased with the progress Microsoft has made over the past five years.

Many of the capabilities we have today, from Power BI Project Files to TMDL, DAX Query View, User-Defined Functions, and AI-assisted development, have fundamentally improved how we apply DataOps principles in Power BI and Fabric.

I also deeply appreciate the generosity of the data community. Whether through open-source projects, blog posts, conference sessions, or GitHub repositories, practitioners continue to share best practices and push the ecosystem forward.

The next five years will likely move even faster than the last five.

What would you like to see improved in Fabric and Power BI? Let me know on LinkedIn or Twitter/X.

Building an Apache Arrow Parser for Power BI’s ExecuteDaxQueries Endpoint

2026-07-17T00:00:00+00:00

One of the biggest advantages of Large Language Models is that, when used effectively, they can help carry an idea from planning to implementation, testing, and deployment.

Recently, Microsoft introduced the ExecuteDaxQueries endpoint for the Power BI REST API. This endpoint enables you to execute DAX queries programmatically against Power BI semantic models/datasets and retrieve results in a structured format, making it ideal for automation, testing, and integration scenarios. While many will ask, “Didn’t we already have ExecuteQueries?” the answer is yes, but the new endpoint adds some important capabilities, including support for INFO functions and preservation of native data types that were previously lost in the JSON output returned by ExecuteQueries.

For years, I’ve used ExecuteQueries in a custom Power BI REST API connector that I shared with the community. However, when it came to leveraging UDFs alongside INFO functions, we hit a wall. ExecuteDaxQueries opened the door, but there was a challenge: the response is returned in Apache Arrow IPC format, and Power Query has no native Arrow parser.

The idea of building an Arrow parser by hand was daunting. Instead, I took an AI-Driven Development (AIDD) approach using a template from Eric Elliott and guided the process with a test-first mindset. The goal was simple: build an Apache Arrow IPC parser for Power Query while continuously validating results against Python-based parsers and parity tests. After several rounds of planning, implementation, testing, and refinement, I believe we now have a working solution.

The result is a pre-release version of the connector that supports ExecuteDaxQueries, including Apache Arrow parsing, and it’s available now.

Resources

I’ll be sharing a video soon that walks through the connector and explains how it can help Power BI developers interact with APIs more effectively. For organizations in Commercial, GCC, and DoD tenants, this also creates new opportunities for testing, automation, and governance scenarios.

In the meantime, give it a try and let me know what you think.

Why User Defined Functions (UDFs) Changed Testing in Power BI

2026-06-20T00:00:00+00:00

User Defined Functions (UDFs) are, in my opinion, the biggest update to Power BI Desktop since PBIP.

That may sound dramatic, but if you care about DataOps, semantic model quality, and reusable development patterns, UDFs fundamentally change what is possible with DAX.

Reuse is one of the core principles of DataOps. For years we have been able to build reusable patterns in Power Query, PowerShell, Python, YAML, and infrastructure automation. But DAX was always missing a key capability: reusable logic that could live inside the semantic model itself.

Until now.

The Problem We’ve Had for Years

Testing semantic models has always been harder than it should be.

I’ve seen teams create entire ecosystems around testing:

Separate DAX query libraries
Python validation scripts
YAML configuration for test orchestration
PowerShell execution frameworks
Custom CI/CD runners in Azure Pipelines and GitHub Actions

All of that just to validate whether a semantic model was behaving correctly.

The bigger problem was consistency.

Every team built testing differently. Every organization defined “quality” differently. Some teams had robust automation. Others had manual spot checks. Some validated measures. Others validated refreshes. Some validated nothing at all.

Testing maturity became subjective.

And because these tests lived outside the model, they constantly drifted from the thing they were actually validating.

Why UDFs Matter

UDFs finally give us a native way to build reusable testing patterns directly into the semantic model.

That changes everything.

Now we can:

Standardize testing patterns
Reuse assertions across models
Keep tests versioned with the model
Eliminate orchestration complexity
Build portable testing libraries
Make testing part of the semantic model itself

That is a massive leap forward for DataOps in Power BI.

When UDFs became available, my team immediately saw the opportunity.

We wanted a comprehensive testing framework that lived inside the model instead of around it.

That’s how PQL.Assert was born.

Introducing PQL.Assert

PQL.Assert is a reusable testing library for Power BI semantic models built using DAX User Defined Functions.

The goal was simple:

Standardize semantic model testing
Make assertions reusable
Reduce pipeline complexity
Encourage automated testing adoption
Bring software engineering testing practices into BI

Instead of every team reinventing testing from scratch, PQL.Assert provides a common schema and reusable assertion framework directly within the model.

Since launching in February 2026, PQL.Assert has already surpassed 1,800 downloads, which tells me the Power BI community understands the value and has been waiting for something like this.

“But Why Would I Test?”

Because AI is accelerating semantic model development faster than ever.

Models are now being generated, modified, and optimized at a pace we simply have not seen before. Copilot experiences, Fabric workloads, agents, and automation are all increasing how quickly BI solutions are created.

But speed without validation is dangerous.

How do you know a generated measure is correct?

How do you know a schema change did not break downstream logic?

How do you prove a semantic model is trustworthy?

Testing.

Semantic Models Are the Foundation

Enterprise BI without testing practices cannot honestly claim that decisions are being made on a fully validated foundation.

Our reports rely on semantic models.

Our agents rely on semantic models.

Our Fabric implementations rely on semantic models.

Even AI experiences like Rayfin ultimately depend on trustworthy semantic models underneath the surface.

Everything stacks on top of that foundation.

Without testing and automated validation, organizations are effectively building analytical systems on assumptions instead of guarantees.

That becomes even more risky as AI-generated content increases.

Testing is no longer optional, it’s necessary.

Learn More

If you want to get started with semantic model testing using PQL.Assert, check out the setup documentation and examples below.

How DataOps principles help to reduce GenAI risk and improve data quality

2026-05-09T00:00:00+00:00

Over the past several articles, I’ve shared how my teams and I have navigated deployments and consulting projects with Fabric, Power BI, and Azure, where using AI has become essential for efficiency. The rapid adoption of tools like Claude for updating Power BI Semantic Models, the FabricNotebook agent for notebooks, and custom agents for enabling data chat has made it clear that GenAI is now a core part of the data landscape.

However, these advances also highlight the growing importance of DataOps principles. Without strong DataOps practices, AI can just as easily introduce new risks and undermine trust in your data solutions as it can accelerate productivity. I’ve compiled nine practical tips to help teams reduce GenAI risk and improve data quality. Read the full article on Simple Talk to learn more.

Share your thoughts on LinkedIn or Twitter/X.

5 Ways Microsoft Fabric Is Embracing Simplicity

2026-04-02T00:00:00+00:00

5 Ways Microsoft Fabric Is Embracing Simplicity

The last couple of weeks have been exciting to watch in the Microsoft Fabric space. These are a few of the announcements that I think help us move closer to a core DataOps principle of simplicity:

Simplicity:
We believe that continuous attention to technical excellence and good design enhances agility; likewise simplicity, the art of maximizing the amount of work not done, is essential.

— DataOps Manifesto

Mirroring for SharePoint Lists

In the past, I have written about pulling data from SharePoint, and this has the potential to eliminate the need to learn that pesky OData syntax or deal with the slow V1 or V2 SharePoint list connectors. The ability to surface SharePoint list data into Power BI just got simpler.

Excel Shortcuts

You cannot take the Finance team away from Excel, but you can now mirror the Excel data they manage. This should make handling version history and schema much easier, and it helps us avoid the slow SharePoint.Files connector.

Compare Code Changes

This is the question I kept getting in presentations on Git integration: ‘Can I see what changed in the service without going to Azure DevOps or GitHub?’ Now I can say yes, and life is a little simpler.

Fabric Notebook Auto Binding for Lakehouses

Ever move a notebook to another workspace and wonder why it is not behaving correctly, only to realize it is pointing to the wrong lakehouse? Yep, been there. This feature makes deployments across workspaces much simpler.

Workspace Tags

Workspace sprawl got you struggling to find where you need to go? Workspace tags help make managing and locating workspaces simpler. This is especially helpful when you have multiple workspaces for the same project, like dev, test, and prod.

I know I spend a lot of time writing about how Microsoft can make things harder for DataOps practitioners, but this time they are clearly moving in the right direction and making things simpler.

Which of these five Fabric features do you think will have the biggest impact on simplifying your data workflows? Are there other areas where you’d like to see Microsoft continue embracing the DataOps principle of simplicity? Share your thoughts on LinkedIn or Twitter/X.

UDFs in GCC and DoD: A big step for reusable, testable semantic models

2026-03-31T00:00:00+00:00

UDFs in GCC and DoD: A big step for reusable, testable semantic models

Thanks to the Microsoft team, we recently got User Defined Function (UDF) support in GCC and DoD sovereign tenants.

Why is that so important? If you have seen what DAXLib offers with SVG image and chart generation, or if you are trying to simplify and reuse measures, UDFs unlock a lot of value. Even better, many of these UDFs are open source under the MIT License, which makes it easier to adopt and adapt them for your own needs.

For my teams, this fits directly into how we approach testing. We have been using PQL.Assert along with Power Automate to validate semantic models and catch issues early. With continued government investment in G3 and G5 licenses, Power BI and Power Automate already give us a solid way to automate data refreshes. Now we can go a step further and validate that those refreshes did not introduce any issues into the semantic model.

It also changes how we think about production issues. When something breaks, we do not just fix it. We add a test for it. Over time, that builds a safety net that helps prevent the same issue from happening again.

If you want to learn more about testing semantic models, check out the first three parts of the series:

I cannot always say features work in GCC or DoD, especially when it comes to Fabric, but this is one you can try today to make your semantic models and reports better.

The AI-Enabled Power BI Tester

2026-02-22T00:00:00+00:00

The AI-Enabled Power BI Tester

If you’ve followed my blog over the years, you’ll notice I emphasize the importance of testing your semantic models. DataOps stresses the principle of “Quality is Paramount” so that when we change our Power Query, DAX, or refresh our data, we can automatically check for issues whether we are developing the solution or publishing to production. Too many times have I seen the “Publish and Pray” method of Power BI development where we hope nothing goes wrong… hope is not a strategy. Furthermore, how do we expect our younger or newer team members to contribute positively and quickly if they have no safety checks? When they make a change, it should not negatively impact the model or its customers.

From the customer perspective, people will often forgive you when you make a mistake. But when they experience the same mistake again in production just days or weeks later, they lose trust.

Admittedly, Microsoft has made testing difficult in the past. But when the DAX Query View was released, this led me to identify a DAX Query View Testing Pattern. While this was a step forward, executing the tests still required consistent syntax and setup. What if we could standardize some of the common tests and, better yet, leverage AI to build and run these tests? Enter User Defined Functions (UDFs), and the nuget of UDFs… DAX Lib.

Over the past few months, my team and I have built a set of functions to aid in using the DAX Query View Testing Pattern while promoting reuse and allowing AI to assist. This library is called PQL.Assert. Assert is a common term used in testing where you assert a certain behavior, like “I assert this column in this table is distinct.” And if that assertion fails, then the test result indicates as such.

We’ve published the open-source library (yes, it’s free under MIT License) at this link.

You can also follow our video series demonstrating PQL.Assert to see it in action.

Let me know your thoughts on LinkedIn or Twitter/X.

The AI-Enabled Power BI Developer

2026-01-29T00:00:00+00:00

The AI-Enabled Power BI Developer

Over the past year, the amount of time I’ve spent working with AI to build solutions has steadily increased. With low-code tools like Power BI, there appears to be a paradigm shift from the classic “clicky, clicky, draggy, droppy” to “chatty, chatty, checky, choosey.” In other words, we are spending less time using our mouse and instead using our mouth to explain to these AI tools what to create and what to modify.

Based on what my teams and I have learned so far, I wanted to share this approach using GitHub Copilot. Through my company’s Power Query Lint YouTube Channel, we started a series called “The AI-Enabled Power BI Developer”.

Our goal with this series is to show how to leverage AI as a Power BI Developer while having the guardrails of version control to ensure you can roll back or review any changes that AI may make.

Let me know your thoughts on LinkedIn or Twitter/X.

Bringing DataOps to Fabric’s Real-Time Capabilities

2025-12-07T00:00:00+00:00

Bringing DataOps to Fabric’s Real-Time Capabilities

Over the past year, I have been working with Eventhouses to build analytic solutions that react to real-time events occurring in Fabric as well as monitoring third-party applications that can ship results to Event Streams.

If you’ve been following my blog for a while, you know that I’ve emphasized version control, testing (many are probably sick of me talking about testing), and orchestration (automated deployments). With that, I want to talk about the state of Fabric Real-time supporting version control and testing, key tenets to practicing DataOps. For the uninitiated, this includes Eventhouse (aka Azure Data Explorer), Event Streams, and Real-time Dashboards in Fabric.

I will caveat that I am new to Fabric Real-time and was surprised during my research that the underlying technology for Eventhouses (Azure Data Explorer and Kusto) turned 10 years old recently (so Power BI and Eventhouse are nearly the same age). However, I am not new to version control, testing, and orchestration, so my perspectives come from through the DataOps lens. Here are my thoughts:

Version Control

Eventhouses support Git Integration, which is a great start. Figure 1 shows an example of the .kql script that gets pushed to Git for the Eventhouse I have to support the series I have on Making Your Power BI Teams More Analytic. Can you guess what’s wrong with this from a version control perspective?

Figure 1: Eventhouse .kql script pushed to Git

It’s one large file and version control fares much better with separate files. When you look at Power BI’s move to version control, the TMDL format separates tables, functions, relationships, etc. into their own files. Even the PBIR format for Power BI has navigated away from one large report.json file that represents the report to multiple json files to represent each visual and page. Eventhouse should follow this best practice. For example, if it saved each file like the structure shown in Figure 2, we could more easily integrate changes amongst teams and better track changes to the files.

Figure 2: Eventhouse File Structure in Fabric Service

Here is the Fabric Idea if you’d like to vote for it.

Testing

When I realized that Eventhouse’s technology was over 10 years old, I was like surely someone has written about Unit Tests and Integration Tests. I found these posts:

I also came across Kusto Explorer and a docker container where testing could occur.

It’s very important that with Event Streams being the source of data feeding these Eventhouses, we test the transformations that occur in update policies. There seems to be sizable effort to build tests and structure your KQL to support testing. Update Policies must use Functions instead of raw queries, forcing the concept of Separation of Concerns. For example, let’s say I need to add a column to a bronze table based on some logic before having it moved to silver. The update policy could look like this:

.alter table SilverTable policy update 
@'[{"Source": "BronzeTable", "Query": "BronzeTable | extend NewColumn = iff(Amount > 1000, \"High\", \"Low\")", "IsEnabled": true}]'

But to support testing it should look like this:

.create-or-alter function fnAddNewColumn() {
    BronzeTable 
    | extend NewColumn = iff(Amount > 1000, "High", "Low")
}

.alter table SilverTable policy update 
@'[{"Source": "BronzeTable", "Query": "fnAddNewColumn()", "IsEnabled": true}]'

And then the code could be tested like this:

////////////////////////////////////////////////////////////////////////////////
// Step 1: Clean up existing tables
.drop table BronzeTable ifexists 
.drop table TestResults ifexists

////////////////////////////////////////////////////////////////////////////////
// Step 2: Create the source table
.create table BronzeTable (Name:string, Amount:int)

////////////////////////////////////////////////////////////////////////////////
// Step 3: Load sample data into BronzeTable
.ingest inline into table BronzeTable <|
Alice,500
Bob,2000

////////////////////////////////////////////////////////////////////////////////
// Step 4: Create a “pure-ish” function that reads BronzeTable
.create-or-alter function fnAddNewColumn() {
    BronzeTable
    | extend NewColumn = iff(Amount > 1000, "High", "Low")
}

////////////////////////////////////////////////////////////////////////////////
// Step 5: Create the test results table
.create table TestResults (TestName:string, Name:string, Expected:string)

////////////////////////////////////////////////////////////////////////////////
// Step 6: Ingest test assertions
.ingest inline into table TestResults <|
"Alice Amount Check","Alice","Low"
"Bob Amount Check","Bob","High"

////////////////////////////////////////////////////////////////////////////////
// Step 7: Run the tests and output TestName, Actual, Expected, Result
fnAddNewColumn
| join kind=inner TestResults on Name
| project TestName, Actual=NewColumn, Expected, Result=iif(NewColumn==Expected,"Pass","Fail")

Therefore the output looks like Figure 3.

Figure 3: Test Results

But how do you run a bunch of these tests with a command line like “kql test” like “npm test”, like in the Javascript world? There is a dearth of libraries supporting KQL testing. I’ll be writing more about this, but if you know of a library, I am all ears.

Orchestration

I have been pleased with Event Streams and Eventhouse’s support with Deployment Pipelines, and I have been happy using Deployment Pipelines with the autobinding that occurs for deploying. Real-time Dashboards do need some work, however, with parameters. Autobinding does not always work and you’ll often get the issue below when deploying the dashboard, leading you to manually reset the parameter.

Figure 4: Common Deployment Error

That’s enough for now, and don’t think I am sour on Fabric Real-time. I find value in its capabilities, and I wouldn’t have come out with my series Making Your Power BI Teams More Analytic if I didn’t find value. I also see value in the Eventhouses built into the new monitoring feature in Fabric. If you’re not getting more familiar with real-time, I highly suggest doing so. I just want it to be better.

Did I miss something? Let me know your thoughts on LinkedIn or Twitter/X

Agents are the Pupils, We are the Teachers

2025-11-23T00:00:00+00:00

Agents are the Pupils, We are the Teachers

Agents are everywhere. If you haven’t been inundated at conferences, ads, videos, or social media posts… count yourself lucky. From the Microsoft perspective, there are a myriad of options. Fabric has Data Agents, SharePoint has SharePoint Agents, Copilot Studio lets you build more agents, and as I am writing, there is probably a new agent being announced. Under the hood, you can think of agents as consisting of three components:

A large language model – This is a model (e.g., GPT-4.0) and allows the agent to consume and respond to prompts.
A data source – This is what builds context to answer those questions. This can be a set of files, a database, or other agents.
Agent Instructions – This is a setting that empowers the builder of the agent to tailor how the agent should behave and provide context to help the agent answer questions.

After spending some years working with Large Language Models (LLMs) and recently agents, and seeing others work with them, there is an investment of time to get them to perform as expected.

This has created three critical things to consider when building an agent:

1) Return on Investment (ROI) – What is the ROI of creating an agent? You and/or your company are spending resources to build and maintain this agent, so will this agent result in cost savings, increased revenue, a redirection of resources, or some other altruistic outcome? If you can’t answer those questions definitively or have no way to track the metric(s) behind the ROI, you’re increasing the risk of failure.

2) Agents are Software – A wonderful book by Alan Cooper, Inmates are Running the Asylum, had this inquiry that stuck with me years ago: What do you get when you combine a computer and a car… a computer. Therefore, we can’t lose perspective that when we combine data and LLMs, you have a computer. We tend to, and I fall victim to this, anthropomorphize agents, thinking they are not software. LLMs were built with code and trained with code, and you’re adding context built by code (e.g., a SQL database built with T-SQL or that semantic model built in Power Query). Computers are built by imperfect beings and therefore can act imperfectly.

3) Testing Is Fundamental, and You Are the Teacher – You should think of an agent as a student in school. A student has the capacity to learn and understands (to an extent) the language for the teacher to impart lessons via instructions. But to make sure the student has grasped the lessons, what does the teacher do to measure how well the student understood the instructions? The teacher gives the student a test. And that test isn’t done just once; over the course of years, tests are applied to make sure the student is progressing.

This process of Educational Assessment means we are the teachers of agents, and we must test and continually test. As new data sources or updated data sources arrive for the agent, we should be testing. As new agent instructions are provided, we should be testing. For example, we should make sure an agent responds appropriately to a question like “show me Alice’s salary?” or “Tell me a joke” in a corporate setting. This is not only prudent but also mitigates the risk your agent does not use something like a curse word, cause data leakage, or provide the wrong answer.

With these three considerations in mind, I have noticed agents offered by Microsoft have opportunities and challenges to support these considerations. For example, let’s look at Fabric Agents:

Fabric Data Agents

ROI

Opportunities – Workspace Monitoring, recently introduced by Microsoft, is giving us diagnostic data like what DAX query was used by an agent when given a prompt. Chris Webb wrote an excellent blog on the subject. This allows us to get some telemetry to show if you have ROI.

Challenges – Fabric Data Agents lack a built-in qualitative feedback mechanism. We should easily be able to incorporate a Microsoft Form that saves feedback to a Fabric SQL database to understand if the end user is finding value. I will note Copilot Studio (another agent building tool) does have this feature.

Testing

Opportunities – The Fabric Data Agent SDK allows you to test in a Fabric Notebook and save results to the lakehouse.

Challenges – The Fabric Data Agent SDK only runs in a notebook, which leads to more capacity utilization. I would prefer to run these tests in a build agent (much cheaper). This leads us to another challenge I hope is rectified soon: service principal support. We should be able to automate testing with these agents using different service principals representing different personas.

Fabric Data Agents are not the exception; SharePoint Agents also lack an API to automate testing. I could go on, but this isn’t a blog to air my grievances with Microsoft’s current state of agent development. Rather, I want to emphasize that you need to consider these three concepts when evaluating agents, regardless of the company that provides them or whether the product is in preview or generally available.

And I know these aren’t the only considerations. Which ones should I include? Let me know your thoughts on LinkedIn or Twitter/X.