Moving folders via OWA can bypass litigation hold

Since it has been officially confirmed now and there is already a KB article about it, I guess I can tell my story about it.

It was of course a random discovery. When I first noticed this, I was testing something completely different. See, for a while we’ve had an issue with one of our customers’ identity management process. Every time a person in the company changes his position, a new record was being generated by their HR system, and correspondingly a new user account was created in AD for that user. The users of course want to keep all the mail from their previous account, which provides some challenges with Exchange Online (on a side note, there is one really ugly hack which allows us to attach a mailbox to another user object in EO, I should write another post about this I guess).

It didn’t take long until I got sick of explaining to users how to move items between different mailboxes and of their (justified) complaints that it takes ages, so I decided to test whether I can find a better way for doing this. And by better I mean faster. Since OWA works ‘server-side’, it seemed like a good idea to do some tests with moving messages between different mailboxes there. Which indeed is quite possible and very easy to do. It does come with some limitations however, at least in the current version.

To actually prove that it does happen faster compared to moving items with Outlook (even in online mode), I did a simple test. I moved a folder with size of about 200MB between one shared mailbox and my own. I monitored the process with PowerShell and timed how long it took for the move. You can easily monitor this with the following PowerShell cmdlet:

Get-MailboxFolderStatistics vasil |? {$_.FolderSize -ne '0 B (0 bytes)'} | ft FolderPath,Folder*Size,Items*

The process didn’t took long to complete, and once it was over I was pretty happy with the results. It was way faster than anything Outlook had to offer, and most importantly it didn’t ‘lock’ the client. Just to make sure that I didn’t just imagine things, I decided to repeat the process from Outlook, so I moved the same folder back from my own mailbox (where it was currently residing) to the shared one. I was still using OWA to make the move. And just as I run the above cmdlet for the last time, making sure that all the 200MB have been moved out of my mailbox, something clicked inside my head. The results showed some items in the Purges folder of the Recoverable Items sub-tree. Nothing wrong there, it just reminded me that being the nice and trustworthy GA, I’ve put my mailbox on Litigation hold long time ago. The troubling part was that the size and the number of items in the whole Recoverable Items sub-tree didn’t add up. I’ve just moved 200MB worth of messages to and then out of this mailbox, which was under litigation hold, yet no trace of the messages was left!

My initial reaction was that I’m probably forgetting some detail, so I went to refresh my knowledge on the hold process. One very nice illustration on how the process works can be found in this article. So there I was, running the same cmdlet time after time, trying to figure out what I did wrong, while waiting for the results to finally update with the ‘correct’ info. I waited for 30 minutes, just to make sure it’s not some sort of crazy replication issue. Still nothing. I then decided to repeat the move process, effectively replicating the issue. Still nothing new in the Recoverable Items sub-tree. Decided to repeat the process in Outlook, using Online mode. Well it worked as expected there, there was a copy of all the moved messages preserved in the Recoverable Items sub-tree, thus no issue with the hold process.

I was then set to reproduce it in as many scenarios as possible. New set of mailboxes, freshly created mailboxes, new O365 tenant. The results were consistent, so I was convinced enough to raise a case with Microsoft about this, and also post it on TechNet and Yammer. Luckily, my Yammer post caught the attention of Microsoft MVP Tony Redmond. He was quick to reproduce it with on-prem built of Exchange 2013 CU6 and used his connections with the PG to greatly speed up the investigation process. Encouraged by his assistance, I dusted out my Exchange 2013 RTM virtual machine to check how far back we can repro this. Exchange 2013 RTM however lacks the functionality to open shared folders in OWA, thus it’s safe from this issue. Updating to CU1, which brought back the ‘open shared folders’ functionality, makes your setup vulnerable to this issue.

Some other scenarios have been tested by me, Tony and the Microsoft support engineer that handled the case. It all boils down to the following: moving folders between different mailboxes in OWA can and will bypass hold settings. You do not need to have a full access to another mailbox, only write access to some shared folder. Alternatively, you can simply grant delete permissions to a colleague on one of the folders you want to get rid of, and since the delete action in such scenario is in fact a move between different mailboxes, the issue will be reproduced. I’m sure that the PG engineers have tested even more scenarios, but so far the only way we can reproduce this is by performing a move action on a folder, between different mailboxes. Moving single items works as expected. Outlook works as expected.

Here’s the official KB article about this: KB 2996477. I’m positive that this issue will be resolved as soon as possible and I know that Microsoft are handling it with very high priority.

A big thank you goes to Tony Redmond, his involvement on this made sure that Microsoft took the issue seriously. It would’ve probably taken weeks to push this through the official support channels. Here’s a link to the blog post he made about this. Make sure to read the other stuff he posts as well, lots of good info on the site.

I will also use this as an example on how bringing together regular users, MVPs and Microsoft staff on a social media like Yammer can be rewarding for all involved parties. I’ll make sure to keep nagging the Yammer staff to push more and more Microsoft representatives in frequenting the groups, because sometimes it really pays off to pay attention to what that unknown guy has to say 🙂

UPDATE 05/09/2014: Tony Redmond has posted a nice article discussing the two workarounds suggested by Microsoft for this issue, you can find the article here: http://thoughtsofanidlemind.com/2014/09/05/reporting-delegate-access-to-exchange-mailboxes/

UPDATE 20/09/2014: Microsoft have crafted a fix for the issue and will push it as part of CU7. For EO customers, the deployment should already be happening. The KB article has been updated accordingly. As usual, thanks to Tony Redmond for keeping us informed.

This entry was posted in Exchange Online, Office 365, Outlook, OWA. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *