As part of my investigation into the Microsoft 365 Group email address management story, I run a bunch of searches against Exchange Online’s Admin audit log as well as the Microsoft 365 Unified audit log. Some interesting observations arose from these, which hint that Microsoft might have changed the way some cmdlets work, or even the dual-write model itself. Read on for the details.
Setting the stage
First, a reminder that Microsoft is planning to remove access to Exchange Online’s Admin audit log by the end of this month (April 2024). In theory, not much should change for us customers, as events from the Admin audit already flow to the Unified audit log and can be interrogated therein. The obvious downside is that you will have to wait some time for the events to be ingested into the UAL, whereas on Exchange Online’s side of things they appear almost instantaneously in the Admin audit log.
The thing that prompted this investigation was the fact that changes I made to Microsoft 365 Group’s email addresses did not consistently generate audit log events. To be more specific, the change were all made via the Set-UnifiedGroup cmdlet, used with either the –PrimarySmtpAddress or –EmailAddresses parameter. Interestingly, none of these generated entries within Exchange Online’s Admin audit log. Yet, running the Set-UnifiedAuditLog cmdlet with other parameters did result in such entries. Moreover, the change in email addresses was reflected on both on Exchange Online and Entra ID’s side, so obviously the cmdlet succeeded. In effect, the need for further investigation arised!
Oddities in Microsoft 365 Group audit log events
The first set of tests were performed on a Microsoft 365 Group object, by means of running the Set-UnifiedGroup cmdlet. I made sure the tests include both properties that are specific to Exchange (thus should not generate a dual-write event or any other trace in the Unified audit log), as well as properties “authored” by Exchange Online, but available in other parts of the service. For the latter, one should expect to see an entry in Exchange Online’s admin audit log, followed by audit log events corresponding to the changes made in other directories, such as Entra ID or Teams. Here’s an example set of cmdlets I run:
Now let’s compare this to the set of audit log events generated for the Exchange Online Admin audit log, depicted on the screenshot below. Note the lack of any events corresponding to two of the Set-UnifiedGroup executions, namely the ones leveraging the -EmailAddresses and -DisplayName parameters. Yet, the changes did happen, as one could easily verify by running the Get-UnifiedGroup cmdlet. Both these properties are obviously synchronized to Entra ID as well and can be queried by the Graph API, which also showed the updated values. Yet, for some reason, no trace of said changes was generated in Exchange Online’s Admin audit log!?
On the other hand, two Update group. events were generated on Entra ID side of things. Interestingly, none of the events were generated in the context of the beloved Microsoft Substrate Management. In this case this is a good thing, as instead said events expose information about the actual user who executed the changes. In turn, said events are exposed within the Unified audit log, along with the events from the Exchange Admin audit log. So overall, we have a good record of the changes, be it a bit more convoluted than one would expect. With one minor detail – no IP information! Instead of the actor’s real IP address, as exposed in the other Set-UnifiedGroup events, we are only given an internal IPv6 address.
Things got a bit more interesting when repeating the test on a different tenant. The same set of examples were used against a Microsoft 365 Group within the new tenant, the only difference being using the –PrimarySmtpAddress parameter instead of the –EmailAddresses one. As before, only some of the Set-UnifiedGroup executions generated entries within the Exchange Online Admin audit log. The changes to the DisplayName and PrimarySmtpAddress were not visible therein, but did result in events generated within Entra ID Audit log. The combined set of events, as exposed by the Unified audit log is shown on the screenshot below:
Interestingly, this time around we get two events reflecting the change in email addresses. One of them is attributed to the actual user, whereas the other one is attributed to the Office 365 Exchange Online service principal. This is in contrast to the expected behavior, where the Microsoft Substrate Management service principal is pointed as the culprit. As usual, events associated with said principal only give us a clue as to what changed, but not who executed the change, so we’re lucky to have the “regular” Update group. event as well. Having two events is the “norm” when it comes to changes performed via the dual-write model (or so I thought), so in this tenant things behaved a bit better. I’m still puzzled by the lack of Set-UnifiedGroup events though. And we still lack the IP information.
Testing against other object types
Next, we turned to performing similar tests against a User object. The Set-User cmdlet was used to update some Exchange-specific properties and some that do replicate to Entra ID (i.e. City). The Set-Mailbox cmdlet was used to update the set of email addresses. Unlike the Microsoft 365 Group scenario, all instances of executing the Exchange cmdlets were properly logged within Exchange Online’s Admin audit log. When it comes to Entra ID side of things, Update user. entries were generated for updating properties such as City or Office, and interestingly, the Notes property (which afaik is NOT exposed within the Graph API for User objects). This time around, all Update user. events were generated in the context of Microsoft Substrate Management, so they’re not that useful. And, the change in email addresses was reflected in both logs, unlike the Microsoft 365 Groups scenario. Another difference is that the change in email addresses also resulted in event attributed to the Microsoft Substrate Management principal.
Lastly, we performed tests against “traditional” Distribution group objects. Similar to the above scenarios, we used the Set-DistributionGroup cmdlet to set some Exchange-specific properties (i.e. BccBlocked) as well as some that replicate to Entra ID, including a change in the set of email addresses. Similar to the User scenario above, all instances of executing the Set-DistributionGroup cmdlet were present in the Exchange Online Admin audit log. On Entra ID side of things, the change in email addresses was reflected in an Update group. event, again generated in the context of the Microsoft Substrate Management principal. In other words, the observed behavior matched that of the case of updates to user objects. And the other good news is that it is consistent across tenants!
Summarizing our findings
To summarize our little investigations, most of the time the Exchange cmdlets behave as expected – they generate proper audit log trail, which ends up exposed in the Unified Audit log, the soon-to-be singular source of information. Whenever a property also “consumed” by Entra ID/Graph API is changed, a corresponding event is generated within the Entra ID audit log as well, and again ingested into the Unified audit log. These Entra ID events however are of relatively low value, as while they do tell you which properties changed, they do not reveal details about the actor. Instead, you can expect the actor to be listed as the Microsoft Substrate Management principal. This underlines the importance of having the standalone Exchange Online events, or the “duplicate” Update group. event, as you need to correlate both to get the full picture.
Enter Microsoft 365 Groups. The most puzzling observation from the investigation is the fact that the Set-UnifiedGroup cmdlet did not generate any audit event when properties such as EmailAddresses, ProxyAddresses or DisplayName were modified. One explanation of this behavior is that Microsoft is calling another API on the backend, effectively using the Set-UnifiedGroup cmdlet as a proxy. Still, one would expect the corresponding event to be generated after calling the cmdlet. After all this is the whole idea behind having an audit trail.
Alas, this is not the case and the operation is effectively “transparent” to Exchange Online’s Admin audit log. Granted, there are ways we can confirm whether the execution was successful, but to do so after the fact, you must know which cmdlet was run in the first place. We know the changes took effect because a matching event can be found within the Entra ID audit log. We can also find “failed” execution events therein. So overall, the Unified Audit log will still contain the full audit trail.
The next interesting bit is that said Entra ID events are generated in the context of the user that performed the operation. This is a good thing, as if they were attributed to the “standard” Microsoft Substrate Management principal, we would be unable to obtain information about the actor. On the other hand, said events do not expose the IP of the actor, but use a generic (internal) IPv6 value instead, so we already have some information loss.
In addition, we also observed difference in the behavior across different tenants. In my personal tenant, updating the email addresses for a Microsoft 365 Group never resulted in generating an event attributed to either the Microsoft Substrate Management service principal or the Office 365 Exchange Online one. An event attributed to my own user object was generated, so we’re not completely left in the dark (if we discount the lack of IP information). Nevertheless, it is strange that the same operation would generate a single event in one tenant and “double” events in others.
Some other interesting observations to note: it looks like the set of CustomAttributeXX do not “qualify” for dual-write, thus updating them only results in event within the Exchange Online Admin audit log, not Entra ID Audit log. On the other side of the spectrum, updating the Notes property for a user in Exchange Online seems to generate Entra ID Audit log entry, even though said property is not exposed within Entra ID.
Conclusion
In summary, some changes are noticeable in the way Microsoft captures audit events corresponding to some operations against Microsoft 365 Group objects. Such events are no longer visible in the Exchange Online Admin audit log, and only generate an audit trail within Entra ID/Unified audit log. They also do not follow the “standard” pattern of having two events for the same action, one representing the real actor and another one attributed to the dual-write mechanism via the Microsoft Substrate Management service principal. Instead, a single event is generated, in the context of the real actor, at least in some tenants. Regardless of the number of events generated though, no proper IP address information is exposed.
While those changes do not seem to be directly related to the planned deprecation of Exchange Online Admin audit log cmdlets, they highlight the importance of paying attention to details. And doubly so when things as important as the audit trail are concerned. If the findings above turn out not to be just another fluke, this effectively means that some changes to Microsoft 365 Groups no longer generate the same level of detail as they did before. Which is never a good thing.
1 thought on “Changes in Set-UnifiedGroup result in lack of proper audit trail”