Application, Service principal and credentials usage reports in Azure AD

The Azure AD blade has long included some basic reporting capabilities, such as the Application activity report, which we covered back in 2019. Now, Microsoft is expanding the coverage of usage reports to service principals, introducing a new application credentials usage report and exposing all aforementioned reports under the Graph API’s /reports endpoint. Let’s take a look.

We start with the Application sign-in report, which has been available for a while as mentioned above. The report comes with two “views”: a “summary” one listing all Azure AD integrated applications (including Microsoft’s own apps, tenant app registrations and third-party apps) along with the number of successful and failed logins, as well as “details” view, which includes additional sign-in related details per application object. The “summary” view can be queried via the /reports/getAzureADApplicationSignInSummary endpoint, and exposes a weekly or monthly dataset (“period” should be set to either “D7” or “D30”):

GET https://graph.microsoft.com/beta/reports/getAzureADApplicationSignInSummary(period='D30')

To access the data exposed via the “summary” report view via the Graph SDK for PowerShell, you can leverage the Get-MgReportAzureAdApplicationSignInSummary cmdlet:

Nothing seems to have changed in this report since its introduction in 2019, apart from now being exposed via the Graph API. It’s mindboggling to see it still in “preview”, but that’s Microsoft for you 🙂

Next, we move on to the “detailed” view of the same report, accessible via the /reports/applicationSignInDetailedSummary endpoint. This view “enriches” the data provided for each Azure AD application, providing per-date aggregated statistics and focuses specifically on sign-in failures. Data is returned for the past 30 days, and for each day you will get the number of sign-ins (days with no sign-ins are omitted), and more importantly, the status code/details for any failed sign-ins. Here’s how an example looks like via the Graph explorer tool:

GET https://graph.microsoft.com/beta/reports/applicationSignInDetailedSummary

The output will include per-day data on each of the applications we examined via the “summary” view, for the last 30 days. As multiple entries will be retrieved per app, the output will have to be reworked to something more useful, which is where PowerShell comes in. To access the same report view via the Graph SDK for PowerShell, we can use the Get-MgReportApplicationSignInDetailedSummary cmdlet. We can store the output in a variable, then filter it based on the app(s) we want to examine. As usual, the Graph SDK for PowerShell output sucks, and you have to spend additional effort on it, but at the end you can obtain some useful data:

Get-MgReportApplicationSignInDetailedSummary | ? {$_.AppDisplayName -eq "Windows Sign In"} | sort AggregatedEventDateTime | select AggregatedEventDateTime,AppDisplayName,AppId,SignInCount,@{n="Status";e={$_.Status.ErrorCode}},@{n="Details";e={$_.Status.FailureReason}} | ft -AutoSize

Until now, we haven’t actually covered anything new – those report views have been available for years. In addition to bringing the existing report/views to the Graph API, Microsoft has introduced two new reports: the Service principal sign-in activity report, and the Application credential activity one. Let’s start with the former.

The Service principal sign-in activity report gives you the last time a given service principal was used in your tenant, broken down per delegate and application authentication flows. In addition, data is included for both scenarios where the app/service principal acts as a client, and as a resource. As not every service principal will have usage across all these scenarios, you can expect some (or most) of the fields to be empty. Overall, for each SP object within your tenant, you will get a set of four datetime fields, and an “aggregate” one, designating the latest of these dates.

To get the data from the Service principal sign-in activity report, you can use the /reports/servicePrincipalSignInActivities endpoint. In a welcome improvement over the Application sign-in report, this endpoint supports filtering (via $filter), so you can get only the data for specific application/service principal, if needed. Here’s an example on how a generic query looks like via the Graph explorer tool:

GET https://graph.microsoft.com/beta/reports/servicePrincipalSignInActivities

As mentioned above, it’s perfectly normal to have some of the fields return null values. In fact, you can expect even less data than shown on the above screenshot, as the entry shown therein corresponds to the SharePoint Online service principal, which is involved in myriad of auth scenarios within the service. In any case, some additional tinkering with the output might be needed, which is where PowerShell steps in. The cmdlet to access the Service principal sign-in activity report data is non-existent yet, so we can use Invoke-MgGraphRequest instead:

$report = Invoke-MgGraphRequest -Method GET -Uri "https://graph.microsoft.com/beta/reports/servicePrincipalSignInActivities"

The output will now be stored in a series of hashtables, which doesn’t exactly make it easy to work with, but it’s manageable. For example, to filter out the data based on a given app, and only surface the relevant “client” datetime fields, you can use:

$report["value"] | ? {$_["appId"] -eq "b22e0fdc-148a-4dbe-b596-da3ab8e46b00"} | select @{n="delegatedClientLastSignIn";e={$_["delegatedClientSignInActivity"]["lastSignInDateTime"]}},@{n="applicationClientLastSignIn";e={$_["applicationAuthenticationClientSignInActivity"]["lastSignInDateTime"]}}

delegatedClientLastSignIn applicationClientLastSignIn
------------------------- ---------------------------
16/05/23 06:28:30 18/05/23 11:10:50

While the robustness of the data returned by this endpoint is much appreciated, it’s a bit puzzling to see first-party Microsoft apps included in it, as we simply have no say in the process of configuring auth flows for them. I suppose knowing that a given first-party app uses delegate vs application auth flow has some value, and of course the overall “last used” datetime is certainly a useful one. Anyway, moving on.

The last report we will cover is the Application credential activity one. The idea behind this report is to give you the last usage timestamp for each of the credentials configured on your Azure AD registered applications. In addition, the expiration date for each credential is also surfaced, greatly increasing the value of the report. At least, in theory, as the current implementation is… funny to say the least.

In order to obtain the data for the Application credential activity report, one can issue a GET request against the /reports/appCredentialSignInActivities endpoint. Filtering is also supported, and you can use a filter against specific appId or keyId. Here’s an example via the Graph explorer tool:

GET https://graph.microsoft.com/beta/reports/appCredentialSignInActivities

The output includes details about the application, the type and identity of the credentials used, the resource accessed and of course, the timestamps. All great info to have, if it were correctly populated. If you take a closer look at the screenshot above, you will see some oddities, especially when it comes to the appId value. Similarly, the appObjectId and servicePrincipalObjectId values are erroneous. To top if off, the lastSignInDateTime value is also off. In other words, the report is currently useless.

To get an idea of what the report should look like and the value it can provide, for the time being best refer to the official documentation, where you can find some mockups. Hopefully, Microsoft will sort out the output issue in the coming weeks, as even for beta/preview, this is just unacceptable. I’ll make sure to update the article after the report starts working as expected. Until then, no point covering the non-existent PowerShell cmdlet either 🙂

In summary, we explored few Azure AD reports, all focused on application/service principal usage. The Application sign-in report has been ported to the Graph API, and the Service principal sign-in activity and Application credential activity reports keep him company. All the aforementioned reports are currently only available under the /beta endpoint, and all of them require AuditLog.Read.All permissions (both delegate an application permissions are supported) to access. While some oddities exist, the data presented in said reports can certainly be helpful in some scenarios, such as enriching the Azure AD  application/service principal permissions inventory reports with additional usage details.