(Ab)using the REST API endpoints behind the new ExO cmdlets

Anyone that’s running admin tasks in Exchange Online is probably aware that a new, “V2” module has been released, featuring some faster and much more reliable cmdlets (in case you aren’t, watch this Ignite session). Those are backed by REST API endpoints, similar to the Graph API that covers most other workloads in Office 365. While Exchange Online is yet to support the Graph (or should it be the other way around?), the new REST endpoints can be exploited in a similar manner, as long as you know what you’re doing. Let’s see how.

First things first, you need to authenticate and in order to do that you need a token. To obtain a token, you must specify few bits of information, such as the clientID and resource, all of which you can obtain by running a Fiddler trace or in some cases by crawling the sign-in logs. Once you have all the details, you can either use the ADAL/MSAL binaries or directly issue a web request to obtain the token. I find it easier to use the former approach, so here’s a code snippet that loads the corresponding binaries and calls the AcquireTokenAsync method:

Add-Type -Path 'C:\Program Files\WindowsPowerShell\Modules\AzureAD\2.0.1.10\Microsoft.IdentityModel.Clients.ActiveDirectory.dll' 

$authContext3 = New-Object "Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext" -ArgumentList "https://login.windows.net/michev.onmicrosoft.com"

$plat = New-Object Microsoft.IdentityModel.Clients.ActiveDirectory.PlatformParameters -ArgumentList "Auto"

$authenticationResult = $authContext3.AcquireTokenAsync("https://outlook.office365.com", "fb78d390-0c51-40cd-8e17-fdbfab77341b", "urn:ietf:wg:oauth:2.0:oob",$plat);

Once you get the token, you can issue standard web requests against the corresponding endpoints. And in case you are wondering which endpoint you need and the exact syntax to use, you can easily obtain this information by running the corresponding new cmdlet with the -Debug switch. Here’s an example:

And here’s an example on how to get a list of all the mailboxes via the REST endpoint:

$authHeader = @{'Authorization'=$authenticationResult.Result.CreateAuthorizationHeader()}

$mailboxes = Invoke-RestMethod -Method Get -Uri "https://outlook.office.com/adminApi/beta/xxxxxxxx-352a-4eda-bece-09d0684d0cfb/Mailbox" -Headers $AuthHeader

$mailboxes.value

As another example, we can get the properties for a single mailbox:

Invoke-RestMethod -Method Get -Uri "https://outlook.office.com/adminApi/beta/xxxxxxxx-352a-4eda-bece-09d0684d0cfb/Mailbox('vasil@xxxxxx.info')" -Headers $AuthHeader 

@odata.context : https://outlook.office.com/adminApi/beta/xxxxxxxx-352a-4eda-bece-09d0684d0cfb/$metadata#Mailbox/$entity
@odata.id : https://outlook.office.com/adminApi/beta/xxxxxxxx-352a-4eda-bece-09d0684d0cfb/Mailbox(xxxxxxxx-888c-4b85-8871-c9766cb4791b')
@odata.editLink : https://outlook.office.com/adminApi/beta/xxxxxxxx-352a-4eda-bece-09d0684d0cfb/Mailbox(xxxxxxxx-888c-4b85-8871-c9766cb4791b'
ExternalDirectoryObjectId : xxxxxxxx-888c-4b85-8871-c9766cb4791b
UserPrincipalName : vasil@xxxxxx.info
Alias : vasil
DisplayName : Vasil Michev
EmailAddresses : {smtp:vasil@sb1.xxxxxx.info, SPO:SPO_xxxxxxxx-e473-451a-85e9-097d3c08307e@SPO_xxxxxxxx-352a-4eda-bece-09d0684d0cfb, SIP:vasil@xxxxxx.info, smtp:vasil@xxxxxx.onmicrosoft.com...}
PrimarySmtpAddress : vasil@xxxxxx.info
RecipientType : UserMailbox
RecipientTypeDetails : UserMailbox
Identity : vasil
Id : vasil
ExchangeVersion : 0.20 (15.0.0.0)
Name : vasil
DistinguishedName : CN=vasil,OU=michev.onmicrosoft.com,OU=Microsoft Exchange Hosted Organizations,DC=EURPR03A001,DC=prod,DC=outlook,DC=com
OrganizationId : EURPR03A001.prod.outlook.com/Microsoft Exchange Hosted Organizations/michev.onmicrosoft.com - EURPR03A001.prod.outlook.com/ConfigurationUnits/michev.onmicrosoft.com/Configuration
Guid : xxxxxxxx-4e45-4c28-a80f-b57b7441490f

And so on. Now, it goes without saying that this method is not supported in any way, so you should not use it in production. But, the samples above illustrate the inner workings of the new cmdlets, and they also hint what you can expect from the future Graph API endpoints for Exchange Online. To get more details on this as well as other tips on using the new cmdlets, join me and fellow MVP Ingo Gegenwarth on February 5th for the Working with the new Exchange Cmdlets webinar.

Posted in Exchange Online, Office 365 | Leave a comment

Expanding the functionality of the Start-HistoricalSearch cmdlet

Those of you accustomed to running message traces in Office 365 are probably aware of the Start-HistoricalSearch cmdlet, used to asynchronously request a trace for events past the 10-day horizon supported by Get-MessageTrace. In addition, the cmdlet is used to obtain “extended” trace reports, listing mail flow events. Now, it seems that Microsoft is positioning this cmdlet to provide other types of reports as well.

If you check the official documentation of the cmdlet, you might notice that the –ReportType parameter now lists quite few additional values, as follows:

  • ATPReport: Advanced Threat Protection File Types Report and Advanced Threat Protection Message Disposition Report
  • ATPV2: Exchange Online Protection and Advanced Threat Protection E-mail Malware Report.
  • ATPDocument: Advanced Threat Protection Content Malware Report for files in SharePoint, OneDrive and Microsoft Teams.
  • DLP: Data Loss Prevention Report.
  • Malware: Malware Detections Report.
  • MessageTrace: Message Trace Report.
  • MessageTraceDetail: Message Trace Details Report.
  • Phish: Exchange Online Protection and Advanced Threat Protection E-mail Phish Report.
  • SPAM: SPAM Detections Report.
  • Spoof: Spoof Mail Report.
  • TransportRule: Transport or Mail FLow Rules Report.
  • UnifiedDLP: Unified Data Loss Prevention Report.

To request each of these additional reports, you can use the following syntax:

Start-HistoricalSearch -ReportTitle testSpoof -ReportType Spoof -NotifyAddress vasil@domain.info -StartDate "14 Dec 2019" -EndDate "30 Dec 2019"

To get the status of any submitted reports, use the Get-HistoricalSearch cmdlet:

Get-HistoricalSearch

JobId                                SubmitDate          ReportTitle Status Rows ErrorCode ErrorDescription
-----                                ----------          ----------- ------ ---- --------- ----------------
e8fe474e-424c-4dc3-a5be-6f41515174f6 20/01/2020 08:39:07 testATP     Done   0
0aa02ccb-6a3a-4d68-961c-3c0c701e2eb5 20/01/2020 08:39:33 testDLP     Done   0
cdcad072-2873-49de-a2cd-278ceeb35ecd 20/01/2020 08:39:53 testPhish   Done   22
33e3f7f6-b13c-4ec8-9ff3-8cec042fea8e 20/01/2020 10:15:17 testSpam    Done   28
80b1d7ce-e8c7-4e13-8843-15c47f7f0cb2 20/01/2020 10:15:26 testSpoof   Done   4

As with the extended message trace reports, you will be submitting an async request, and after the corresponding report has been generated, you will receive an email notification with a link to download the report. Unfortunately, automatic this part is not possible.

The reports themselves seem to offer similar level of detail to the data obtained via the Get-MailDetail* cmdlets, where applicable, but also add some additional fields. For example, the Spam report will show the “standard” sender/recipient information, but also lists the original sender IP, connecting IP, hello string and SCL of the message:

In effect, the reports save you from running additional traces, as they readily provide those additional details. Here’s also an example Phish report generated for my tenant:

Since the documentation page is the only resource mentioning this new reports, it’s unclear at this point whether this is Microsoft’s planned replacement for the “old” reporting cmdlets, or just a new feature. I’ll update the article as I get new information about the reports.

Posted in Exchange Online, Office 365 | Leave a comment

Compliance center and Security center are rolling out to Office 365 customers

Since we seem to be on a compliance-related spree, here’s another article on the topic. Over this week, the specialized Compliance center and Security center have finally started rolling out to Office 365 customers. They were announced back at Ignite 2018, then released in preview in the beginning of 2019, but only for Microsoft 365 customers. Now, an year later, Office 365 customers can finally get their hands on the new portals.

I already covered those new portals back when they were initially made available in preview, so I will not go over them in detail here. You can find my original article over at Practical 365, rest assured not much has changed since then. Some new “cards” have been added, few things have been renamed (for example you can find Supervision policies under Communication compliance now, although the term itself is not mentioned anywhere else on the portal), and we have a customizable navigation pane now (or are supposed to have it, as it doesn’t seem to work for me just yet), akin to the Office 365 admin center.

In terms of new functionalities, you might find the Compliance score or the Data Classification Content explorer useful (both are in preview still). Third-party data connectors now can be configured via the UI. There are still a lot of features though that reuse the old SCC UI bits or simply redirect you to the SCC. Examples include the Edit functionality for DLP policies, Content searches and DSRs, labels and policies, and more. This should come at no surprise, as Microsoft simply continues to follow the “best practice” or releasing a MVP product and taking an year or two to actually bring parity with the functionality available in the old portal. And as discussed previously, the lack of attention to detail is alarming, to a point that it makes you wonder whether someone has actually looked at the live product. Just glance at the Information governance section for example…

The Security center seems to have gotten even less attention than the Compliance center over the last year. The only useful thing here is the new Secure score console. You will find the previous version of the Classification tab, listing the corresponding Retention and Sensitivity labels and policies. Every link under the Policies tab takes you to an external portal, and of course some engineer took the opportunity to shine here as well – the Antispam policy link takes you to the “dogfood” version of the SCC portal (https://df.protection.office.com/antispam).

And that’s pretty much all the Security center currently offers. Not even a hint of functionalities such as the Unified Audit log. Lots of “data isn’t available right now” cards, and eye candy such as the below:

Posted in Office 365 | 2 Comments

Issue with Office 365 compliance raises some serious questions

The year started with some compliance related news, namely the deprecation of eDiscovery/In-Place Holds in the EAC, as well as the beloved Search-Mailbox cmdlet. Some questionable decisions were made there IMO, as the suggested replacements either require you to spend additional $$$ on premium SKUs, or do things manually, but we’ve already discussed those in a previous article. Now, we continue the compliance-focused discussion with a recently surfaced issue that might have some severe consequences.

The issue can be summarized as follows: if you use the Outlook client in cached mode to remove an attachment from received message, the copy-on-write page protection mechanism will fail and the original item will not be saved. To give you some additional context, the COW page protection ensures that for mailboxes that are on hold (Litigation hold, In-place hold or retention policy created in the SCC), a copy of the original message is preserved when certain properties are modified. As the documentation explains, this includes messing with any of the attachments of a message or post. The copy of the original, unmodified item will be then saved in the /Versions subfolder of the Recoverable Items subtree and will still be available for eDiscovery investigations.

To my surprise though, a user posting on the Spiceworks forums reported that he was unable to find the original item after removing an attachment. Instead, only the current version of the message, with the attachment missing, was returned in a content search. Realizing the implications of such an issue, I was quick to perform few tests on my own, and after confirming I’m seeing the same behavior, asked for clarification on the Exchange MVP DL. Fellow MVPs Ingo Gegenwarth and Tony Redmond were able to independently confirm the issue, at which point it was clear that it’s not just a random fluke.

It took several days and a SevA case to get Microsoft’s attention on this, and almost a week after it was reported we still have no clear statement for the scope of the issue, its full impact and possible remediation. There is a SHD post, but afaik it is limited to just the tenants that have reported the issue, and the details presented therein leave a lot to be desired. In any case, as the issue is still under investigation, things will certainly change. I will make sure to update this article as we get additional information, but in the meanwhile, I feel obliged to issue another general rant. In case you only care about the technicalities, you can stop reading here and head out to Tony’s article at petri.com.

OK, rant time. First of all, this is a MAJOR issue, the type of which should result in updating few resumes. We are not only talking about potential data loss here, we are talking about being able to circumvent the hold functionality, which is the corner stone of Microsoft’s compliance solutions across most of Office 365. Being able to immutably protect any and all data stored inside user mailboxes is a requirement for a number of important regulations, such as the SEC 17a-4 regulation (other workloads also use the mailbox store to make sure compliance requirements are met). Now, it might turn out that Exchange Online has failed to comply with such regulations for an extended period of time. A point as to how exactly such compliance is validated by third-party auditors can also be extended here, and you might want to start independently verifying any and all claims in case your organization is in a highly-regulated industry.

Now, before I go full bitch mode, few mitigation factors should be mentioned. First of all, the COW page protection seems to only fail on processing attachment changes. Editing other properties will result in a copy of the original item correctly saved under the /Versions subfolder. In addition, the issue only seems to affect the Outlook client, more specifically Outlook running in cached mode. We’ve confirmed that several different builds of Outlook are affected, so the code issue has been around for at least an year. Luckily, most other email clients do not have the functionality to remove attachments or are unaffected. Even considering all these factors, this is still a huge issue, with potentially severe implications.

Which brings us to another rant point: why on Earth would Microsoft choose to code things in such a way that COW is enforced client-side? If I am to make a stupid analogy here – take any online game, if no server checks are performed one can easily modify practically every parameter/variable and enable “god mode”. But we’re not talking about games here, we’re talking about business that is bringing billions of revenue to Microsoft, based on the promises they’ve made with regards to protecting customers’ data and complying with regulations. And what’s even worse, this isn’t the first client-side bug that bypasses holds that we’ve uncovered. Back in 2014 I stumbled upon an issue with OWA’s move folders functionality that also resulted in data loss, even when the mailbox was put on hold. Another client-side issue that resulted in purging data from the mailbox store. One would think that Microsoft would learn their lesson and start using a server-side solution, but alas…

The next point, and one I’ve made repeatedly over the past few years, is the total neglect of QA in the service. Add to that the “standard” practice of releasing minimal viable products instead of complete, feature-proof solutions, and we get all the ingredients we need for a service plagued with issues. We’ve basically come to accept not only minor annoyances such as typos and UI glitches (which albeit minor for the most part, show an alarming lack of attention to detail), but even compromises with features that are required for the services to operate normally. Reporting such issues to Microsoft has mixed results, as some teams seem to be only interested in releasing the next roadmap feature and moving on to something else, instead of ensuring flaws are being disposed of in a timely manner. Even compliance issues are being ignored for months, such as the fact that any user can single-handedly prevent you from performing eDiscovery searches against his own ODFB site collection.

Since we know that this issue isn’t affecting on-premises Exchange installs (up to at least Exchange 2019 RTM), it is mind-boggling how a change to functionality as important as the COW page protection mechanism got released and pushed to the entire Exchange Online active base without conducting proper testing.  One would imagine that testing every aspect of compliance features is on the top of list for any code change. The issue itself is quite easy to reproduce, it takes a minute to test and it’s very hard to miss if you have even a basic understanding on how things are supposed to work. The obvious conclusion here is that this is simply another code change that got pushed directly into production, with minimum or no verification. Long live the DevOps model and its proper applications in practice…

The scope of the issue is another thing that should haunt Microsoft’s execs. Not only it seems to affect everyone in the service, it has been around for months, if not years. While we still have no official statement, nor a post-incident report giving us the gritty details, we know that a broad range of server and client versions are affected. In turn, this means that the impact on customers is immeasurable. What’s even worse, because of the lack of backups in the service, there is no way for Microsoft to actually restore the missing data, even if they knew all the individual tenants and items that have been affected by this. Which they don’t anyway. Even addressing the client-side aspect of this can be problematic, considering the Office update model and the support windows for the different channels.

Last but not least, Microsoft is yet again failing to communicate things to its customer base. Yes, there is a SHD posting on the issue, but you are probably not able to see it (you can find a screenshot of the latest update below). As customary, the SHD item description is understating the scope of the issue and its impact. The language used in these posts continues to be vague at best, even after years and years of feedback from MVPs and customers alike. I’m sure there are a lot of reasons and countless hours have been spent polishing the tone and language of these messages in the company of top lawyers, but the simple fact is that they fail to fulfil their primary purpose.

There is no doubt that Office 365 is highly successful, but it might be becoming a victim of its own success. From the position of a leader, it’s easy to cite statistics and telemetry to justify yet another questionable decision (*cough* self-service purchases *cough*) or claim that support is top notch or that forcing a shitty chat bot experience actually helps. The lack of competition allows Microsoft to get away with many frivolities, and such committed not just by their marketing machine. We get examples of the aforementioned on a monthly, if not weekly basis. But when functionalities such as holds/retention start failing, alarms should be sounding left and right. Lawyers will probably have the last say on this particular topic, as I imagine quite few organizations will have a lot to ask and demand about this issue.

Hopefully this time around Microsoft will actually learns a lesson or two, and something positive comes out of this fiasco.

Posted in Exchange Online, Office 365 | 1 Comment

Microsoft is deprecating Search-Mailbox and the legacy eDiscovery experience in July 2020

Happy new year 🙂

Some of you might already be aware of the plans to deprecate the old, Exchange-only eDiscovery experience that is still available as part of the Exchange Admin Center, as well as the underlying PowerShell cmdlets. In an effort that spans multiple years, Microsoft first announced their intentions to remove this functionality, then postponed the process as the suggested replacement (the eDiscovery functionality as part of the Security and Compliance Center) lacked certain functionality. Now, it seems like a final decision has been made and the deprecation dates are as follows:

  • On 01 April 2020 you will no longer be able to create new eDiscovery searches via the EAC or the corresponding New-MailboxSearch cmdlet.
  • On 01 July 2020, the rest of the cmdlets will be deprecated as well. This includes Set-MailboxSearch, Start-MailboxSearch, Stop-MailboxSearch.
  •  The Get-MailboxSearch and the Remove-MailboxSearch cmdlets will continue to be available in order to list/remove existing holds and searches.

What’s more important, this deprecation includes the invaluable Search-Mailbox cmdlet, running which will also give you the 01 April 2020 date as a warning. This might present few challenges to organizations that have created certain workflows around this very useful cmdlet, as to date Microsoft has not provided us with comparable functionality in the SCC eDiscovery stack. I will not go into listing all the differences here (you can refer to Tony’s article over at petri.com), but will just mention that we still don’t have any way to copy eDiscovery/Content Search results to a Discovery mailbox when using the SCC functionality. This in turn means that you need to either export the results to PST file and use the download tool to transfer them to a machine equipped with Outlook, or pay some extra $$$ and use the Advanced eDiscovery SKU and leverage the functionality available there.

At this time it’s not clear whether Microsoft actually plans to provide replacement for this and other missing functionalities, nor what the future holds for Discovery mailboxes for that matter. The only bit of documentation mentioning the deprecation can be found here, and is lacking any useful details currently, but will hopefully be updated soon.

 

P.S. The documentation is updated now, and you might also take a look at this article for advice on how to migrate your existing holds.

Posted in Exchange Online, Office 365, PowerShell | Leave a comment