Reporting on Entra ID integrated applications (service principals) and their permissions

Today, we’re going to be looking at reporting for Entra-integrated third-party applications (or their local representation, service principals). Since the last time we examined this, Microsoft has released some additional reporting and analytics, added support for Custom security attributes, thus allowing us to target “tagged” service principals within Conditional Access policies, and introduced the Microsoft 365 App certification program. We’ve also seen the increase in attacks that specifically target applications and service principals, such as the one outlined in this article.

As to why reporting on Entra-integrated applications is important, the gist of it is that those are applications that can potentially access or even manage data and directory objects within your Microsoft 365 organization. Understanding just what access such applications get and reviewing them periodically is becoming increasingly important, especially since bad actors started abusing the consent model few years back. And while Microsoft does provide some tools to report on this scenario, they are either put behind a paywall or severely outdated. So, think of this article/script as one of the tools to use in tackling the illicit consent grants issue.

With the above in mind, it was time to release an updated version of the service principal reporting script. We are actually releasing two versions of the script: one based on direct Graph API queries and leveraging application permissions, and one based on the Graph SDK for PowerShell and delegate permissions. As always, treat those script samples as “proof of concept” and make sure you update them to leverage your preferred method of authentication. And if you plan to run them against any production environment, make sure to add more robust error handling!

First things first, let’s talk authentication. In order to collect the set of service principal object and their properties, as well as any permissions granted, you will need the Directory.Read.All scope. While said scope is quite “wide”, on the positive side it gives us the means to resolve user, group and role identifiers to “human readable” values. Now, for the bad news. If you want to include any of the newly introduced properties, additional permissions will be needed, as follows:

• Directory.Read.All (hard-requirement for oauth2PermissionGrants, covers everything else needed)
• CustomSecAttributeAssignment.Read.All (optional, needed to retrieve custom security attributes)
• AuditLog.Read.All (optional, needed to retrieve Sign-in stats)
• Reports.Read.All (optional, needed to retrieve Sign-in summary stats)
• CrossTenantInformation.ReadBasic.All (optional, needed to retrieve owner organization info)

To avoid unnecessary 403 errors, the script will try and determine whether the necessary permissions have been granted, by either parsing the access token, or examining the scopes via the Get-MgContext cmdlet. Do not expect wonders from the error handling here, and make sure to manually confirm all the required permissions have been granted in case you are not getting the expected results.

Next, let’s talk about the script parameters. Most of them relate to the inclusion of new properties and the aforementioned permission requirements, with one exception: the switch to control the inclusion of built-in, or “system” service principals (i.e. those without the WindowsAzureActiveDirectoryIntegratedApp tag). Here’s the full set of parameters accepted by the script:

• IncludeBuiltin – whether to include built-in/system service principals in the output. Default value is $false.
• IncludeOwnerOrg – whether to “resolve” the appOwnerOrganizationId property to human-readable value, such as the tenant’s default domain. Using this switch requires the CrossTenantInformation.ReadBasic.All scope. Default value is $false.
• IncludeCSA – short for “include Custom security attributes”, does what it says. Comes with the requirement for the CustomSecAttributeAssignment.Read.All scope. Default value is $false.
• IncludeSignInStats – whether to include additional analytics about service principal usage and last sign-in timestamps. Requires both AuditLog.Read.All and Reports.Read.All scopes. Default value is $false.
• Verbose – use this switch to show additional information about the script’s processing. Default value is $false.

Before moving into some examples on how to run the script and the output it provides, a few words on its inner workings. After authenticating, the first step is to retrieve the set of Service principal objects within the tenant. Here, depending on the set of parameters used, additional properties and/or objects will be collected. If the –IncludeSignInStats parameter is used, two additional queries are made, as follows:

GET "https://graph.microsoft.com/beta/reports/servicePrincipalSignInActivities?$top=999"

GET "https://graph.microsoft.com/beta/reports/getAzureADApplicationSignInSummary(period='D30')"

Here’s probably a good place to mention that the /beta Graph API endpoints are used for most queries, the reason being that some properties are not yet exposed under /v1.0. For example, the publisherName one. Similarly, some methods/queries are only available under /beta or the corresponding Get-MgBeta* cmdlets. Feel free to switch to the GA version if you must, but don’t expect everything to work as is.

Next, the script will iterate over each service principal object, as additional calls are required to expose things like the set of permissions granted, the list of groups and directory roles the SP is assigned to and the list of owners. Technically, we can get the last one via an $expand query in the original LIST request, however that returns the full set of properties, which can be a mile long in some cases. If you feel strongly about this, feel free to update the script to use $expand instead 🙂

Lastly, the output is prepared. Here is where the most of the script’s helper functions are being utilized, as they are needed to parse the values to a more manageable format, replace GUIDs and such with human-readable values, and avoid performing the same calls multiple times. Do note that the script’s output will depend on the set of parameters used, as some properties are only surfaced when the corresponding switch is used.

Without further ado, let’s look into some examples on how to run the script. First, get your copy from my GitHub repo for either the Graph API version or the Graph SDK one. For the former one, make sure to edit the authentication bits before running the script (lines 186-210). By default, the script will only cover the WindowsAzureActiveDirectoryIntegratedApp-tagged objects. Use the set of the parameters as needed. For the Graph SDK version, replace the script name with .\app_Permissions_inventory_GraphSDK.ps1 in the examples below.

#Run the script with the default parameter values
.\app_Permissions_inventory_GraphAPI.ps1

#To include "system" service principals, use the -IncludeBuiltin switch
.\app_Permissions_inventory_GraphAPI.ps1 -IncludeBuiltin

#To include custom security attributes, use the switch -IncludeCSA
.\app_Permissions_inventory_GraphAPI.ps1 -IncludeCSA

#To enrich the report with all possible properties, use
.\app_Permissions_inventory_GraphAPI.ps1 -IncludeCSA -IncludeOwnerOrg -IncludeSignInStats

#You can also use -Verbose to output additional processing info and -OutVariable to store the output in a variable for reuse
.\app_Permissions_inventory_GraphAPI.ps1 -Verbose -OutVariable out

Output will be exported to a CSV file in the working directory. Some properties might contain multiple values, i.e. the ones detailing the permissions granted to the SP. The format used is as follows: the “resource” name is presented in square brackets “[]”, followed by a comma-separated list of permissions granted. If the service principal has permissions granted for multiple resources, they are stringed together in semicolon-delimited list. For example:

[Office 365 Management APIs]:ActivityReports.Read,ActivityReports.Read,ThreatIntelligence.Read,ActivityFeed.ReadDlp,ActivityFeed.Read,ServiceHealth.Read;[Microsoft Graph]:User.Read

And yes, there are duplicate values in the above. Don’t blame the script though, this is how Microsoft returns them, no idea why, and whether it is safe to just “merge” them together. So we’re playing it safe. Oh, and for delegate permissions, where we have user-specific consents, you can expect to also see the UPN of the user who has consented to a given permissions (no username means admin consent was granted).

[Microsoft Graph(vasil@michev.info)]:User.Read,profile,email,openid,offline_access

Similar logic/formatting is applied to the Custom Security Attributes property. If you don’t like the format used, feel free to replace it with your own. The relevant logic is part of the parse-AppPermissions, parse-DelegatePermissions and the parse-CustomSecurityAttributes helper functions.

Due to the large number of properties contained in the report, and the number of service principals you can expect to find in any tenant of size, it is near impossible to fit the output on a single screen. Instead, the screenshot below shows a curated view of the report, filtered by all third-party applications for which admin consent has been granted (delegate permissions only). The focus is on the newly introduced “statistics” columns, such as the last login timestamps.

Instead of additional screens, here’s a list of new or important properties you should focus on when reviewing the report:

• ApplicationId, aka the clientID of the parent application. When including built-in SPs, expect to see thing like the Microsoft Graph resource (00000003-0000-0000-c000-000000000000).
• IsBuiltIn, i.e. is the SP object tagged with WindowsAzureActiveDirectoryIntegratedApp.
• Owned by org has been updated to include the default domain of the owning organization, when you call the script with the –IncludeOwnerOrg parameter.
• Publisher and Verified give you information about the publisher verification process. Unfortunately, Microsoft is still not exposing the Microsoft 365 app certification status for SP objects.
• ObjectId is the GUID of the service principal.
• Created On gives you its creation timestamp, i.e. when was it provisioned in your tenant.
• Enabled gives you the status, if FALSE authentication via the SP is disabled.
• Owners is a newly introduced property, giving you the UPN(s) of any owner(s) assigned to the SP.
• Member of (groups) and Member of (roles) give you the list of groups and directory roles the given SP is a member of, respectively. You definitely want to review any SP with directory roles assigned, especially if it’s not a built-in one.
• PasswordCreds, KeyCreds and TokenKey are also newly introduced, and you should examine any SP with non-null values here. Refer to this article for more information and remediation steps.
• Permissions (application) gives you the list of Roles granted to the SP. Periodically review those!
• Authorized By (application) signals whether admin consent has been granted to the Roles from the field above, whereas Last modified (application) gives us the date and time consent was granted.
• Permissions (delegate) gives you the list of Scopes granted to the SP. Periodically review those, or at least the ones requiring admin consent.
• Authorized By (delegate) gives you the list of users that have consented to at least one delegate permission, and also signals whether admin consent has been granted on the SP.
• Valid until (delegate) is not used, but included for completeness.
• Last sign-in is newly added field, exposed when invoking the -IncludeSignInStats switch. It gives you the timestamp when the SP was last used, i.e. the date and time of last authentication. It is the “most recent” out of the other four newly added timestamps: Last delegate client sign-in, Last delegate resource sign-in, Last app client sign-in and Last app resource sign-in.
• Sign-in success count (30 days) and Sign-in failure count (30 days) are also newly introduced and only populated when using the -IncludeSignInStats switch. They give you the number of delegate login attempts for the past 30 days.
• CustomSecurityAttributes is the last new field, giving you the list of CSAs. Only included when running the script with the –IncludeCSA switch.

Out recommendation would be to filter the report based on few key properties, starting with permissions – review those periodically and disable (or outright remove) any SP object that has been granted permissions, unless you absolutely trust the Owner org. Pay special attention to any SP object with password or key credentials – there are only a handful of scenarios where such should exist. Similarly, pay special attention to any SP object with Directory role assignments, even the built-in ones! Review the usage stats and timestamps therein, and disable/remove any “unused” SP objects.

Before closing, few additional notes. With the newly gathered properties, the list of calls the script is making has increased, and so has its runtime. While some basic anti-throttling controls have been added, it’s possible that you run into throttling issues, so make any necessary adjustments. The Graph SDK version of the script completely relies on the its internal logic for that, so don’t blame the script. Speaking of which, the Graph SDK version has a number of oddities in its handling of data, so do not expect both versions of the script to produce the exact same output, even though the helper functions have been altered to account for such differences. For one, the Graph SDK defaults *some* timestamps to UTC, whereas others are presented “as is”.

As always, feel free to modify the scripts to better suit your own needs and environments, and don’t forget to send feedback. There are always more improvements to be made, and never enough time… so adding things like the set of Conditional access policies scoped to the SP will have to wait 🙂