This is not the first post I’m writing on the subject of auditing in Microsoft 365 and how it can slowly cause your teeth to grind away, and I have no delusions it will be the last. But as auditing is (supposedly) an integral part of the suite, we have no choice than to deal with its oddities. Which is what I’ve been wasting time on for the most part of this week, so now it’s time for a little payback in the form of yet another rant post.
In this installment of the series of “why auditing in M365 continues to disappoint”, we will examine two scenarios. Two very simple scenarios if fact, and two cases where the audit trail fails to provide you with sufficient information as to who performed the changes or even what exactly was changed. Great situation to be in, especially considering Office 365 has been around for almost 11 years now. Sigh.
Let’s start with the first scenario, namely changing a user’s Company attribute via the Set-User Exchange Online cmdlet. The cmdlet execution is properly captured by the Exchange admin audit log and later ingested in the Unified audit log, so no issues here. You can see who executed the cmdlet, the timestamp, the IP the request come from, what changes were made, etc. But, if you try to see the same information from the Azure AD Audit log, things are rather disappointing.
The record is present, but the information contained within it leaves a lot to be desired. Putting aside the lackluster “Microsoft Substrate Management” designation for the actor (which those of you who have followed the developments in recent years will likely point out “is by design” and reflects the switch to dual-write model for Exchange Online operations), there’s still a lot wrong with the audit record. At the very least, it does NOT tell me what changed. Here’s what the record shows under Modified Properties:
In effect, we see some vague entry telling me that something might have changed, no specifics as to what exactly or who did the change. You’re expected to use the Unified audit log if you want to get the complete picture. We’ll see how well this goes in our next scenario. Still, having to redo the query in another system, so I can see what changed for an object Azure AD is authoritative for is not a good experience.
Sadly, this has been the behavior for almost three years now. Complaints from customers only resulted in Microsoft adding a documentation page, mentioning that you might run into this. No plans to address the impotence of the Azure AD record, by at least adding a list of properties that were actually modified, or some indication who the real actor is. No plans to add complimentary records either. Just go and run a query against the Unified audit log. Disregard the convenience of the available Graph API endpoints for Azure AD audit, use PowerShell or the neglected Management activities API. Another sigh.
And that was the good example I had. The other one revolves around changing proxy addresses for a Microsoft 365 Group, as detailed in this thread over at the MTC. Basically, the OP was complaining that an alias (proxy address) he removed from the group object was miraculously reappearing. So I went to test the behavior. First, I selected one of the Microsoft 365 Groups in my tenant, made sure it has at least one extra alias added. Then, I proceeded to remove one of the aliases:
So far so good. After checking few minutes later, the alias was still gone. I then proceeded to change the Primary SMTP address, with the intention of removing said alias later on. Both operations completed as expected. Yet, few refreshes later, something very strange happened. All the aliases I have previously removed have reappeared. At the same time, the change in Primary SMTP address persisted:
In effect, I seem to have reproduced the strange behavior the OP was describing in this MTC post, or something akin to it. As I couldn’t think of any functionality that would cause such “rollback” of the changes I made (Exchange Address Policies are not the explanation here!), my next step was to examine the audit trail. And here’s where things really started to fall apart.
It has been over three days since I made the changes showcased above. Yet, there are 0 (Z E R O) events in the Exchange admin audit log related to the use of the Set-UnifiedGroup cmdlet above. Or any other record for the changes I made to the group, at least according to the Exchange built-in audit capabilities. Explain that!
Turning to the Unified audit log, unsurprisingly we again don’t see any Set-UnifiedGroup events. What we do see is an “Update group.” event. A single one, corresponding to the change in PrimarySMTPaddress value. No events whatsoever for the other two changes made, each of which removed an alias from the group object. Zero information as to who performed the change, instead we have another service principal referenced. Another, as in it does not match the “Microsoft Substrate Management” ID we saw in the Set-User scenario. Most importantly, we see zero record of whatever process reverted those changes.
On the positive side of things, the event at least shows which properties were changed. If you take a closer look however, you will see that the “old” value for the ProxyAddresses attribute lists all four aliases, which is reflective of the “initial” state of the object, and completely disregards the changes made to remove an alias. One can then reason that the changes were never made in the first place, and point the lack of audit records as additional evidence to this hypothesis. Yet, the PowerShell output above clearly shows that at least on Exchange side, those changes were reflected, if only for a brief time.
Turning back to the audit trail, the Unified audit log record has a matching counterpart found in the Azure AD audit log. As with the Set-User example we examined above, the “actor” is not my admin user. Instead, the Exchange Online service principal, the one corresponding to appID value of 00000002-0000-0ff1-ce00-000000000000 is listed, as referenced by its objectID value (f5c70b13-9271-48f0-82cf-e6b2ca7c98a1). As before, we have zero information on who the actual actor was. And this time it’s actually worse, as the execution of the Set-UnifiedGroup cmdlet was not captured in either the Exchange admin audit log nor the Unified audit log, so you cannot obtain information about the actor therein. Zero accountability is the new moto 🙂
What’s also interesting is that the Unified audit log reveals few RecipientChange events, which seem to correspond to the changes made via the Set-UnifiedGroup cmdlet. Unfortunately, they are very light on detail, and do not feature any information as to who performed the change (“Admin” is shown as the UserId), nor what changed exactly. Correlating their timestamps and the RecipientId value gives sufficient evidence that they were triggered due to the changes made to the Group object. And two additional such events can be seen after the last change I made, indicating that indeed some internal process was at work. An example of one such event is shown below. Note the ResultIndex/ResultCount values, as well as the timestamp.
So, to summarize out findings for this scenario: a strange behavior with the process of changing proxy addresses for a Microsoft 365 Group resulted in another round of disturbing discoveries about the inadequacies of the audit story, this time in both Exchange Online and Azure AD. Even if we are to blame the dual-write model for the lack of detail on Azure AD side of things, nothing can justify the complete absence of audit trail on Exchange side. In effect, changes made in the service cannot be attributed to any given user or process, resulting in complete loss of accountability, and trust.
And while I am in a full bitch mode, can we talk about the Audit search experience in the Compliance center, which remains as crappy as ever. All the issues I listed an year ago are still present, usability remains as poor. In addition I’ve now noticed that scrolling down to load the next batch of results can cause additional entries to appear in the already loaded set, meaning that not all results are loaded chronologically. The first screenshot below shows the “initial” set of 150 results, the second one the set of results after fetching two more “pages” (450 in total). Note the highlighted in red entries, that are simply missing from the first set of results, even though they fall in the same time range supposedly covered by it (up to the red line).
So in effect, if you want to make sure you’re not missing any events in a specific timeframe, you have to scroll scroll scroll until all the results have been fetched. Surely a pleasant experience. At least the download functionality seems to export all the entries, I think…
UPDATED 19/04: Another day, another set of audit entries that fail to give you the complete picture. This time with regards to Teams admin operations, such as creating a new resource account, adding a call queue, assigning a number to it, etc. While we do get an audit record entry for such events, under the TeamsAdminAction operation type, the record itself is again light on details. Here’s for example all the info you get for the call queue event:
There is sufficient information to identify who performed the change, but not exactly what’s changed, as zero details are given with regards to the object created/modified. The same goes to other event types, such as assigning a number, etc. This behavior is puzzling, considering Teams is one of the few workloads that provide audit records even for Get-* entries. Oh well.