Office 365 Secure Score Analyzer

Security is one of the major investment points for Microsoft in their cloud journey. Office 365 is no exception, and while the occasional issue can popup from time to time, overall Microsoft is doing great in the area, as evident from numerous independent audits on the service. If anything, the pace at which they are releasing new security related features is accelerating. Just over the last few months we got the comprehensive Audit logs in the Security and Compliance Center as well as the Activity alerts, Owner audit for Exchange Online, Hybrid audit for SharePoint and SharePoint Online, Advanced eDiscovery, to name a few.

With the ever-increasing number of settings and features, keeping up with all the relevant controls can be a challenge. This is where different self-service audit tools can help you, and we will introduce you to some of them in the current article.

Short history of self-service auditing tools in Office 365

It all started with a webpage named Advanced Privacy Options for Administrators, which puts together couple of security and privacy related settings across Exchange, SharePoint and Lync Online. The structure of the page was nice however, offering a short description of the feature along with steps to turn it on or off, including screenshots to help the less experienced administrators. At the time, this was a comprehensive resource, but unfortunately become outdated and now it hardly contains any useful data.

Then came the Office 365 Customer Security Considerations Preview and Office 365 Customer Security Considerations Reference Guide, a set of documents published on the Office 365 Service Trust portal (part of the SCC now). They cover a lot more features, which hardly comes as a surprise given the pace of change. The controls are organized by Product and Category, a detailed description is provided as well as instructions on how to configure them via the Office 365 Portal or PowerShell where applicable.

Another similar tool is the Office 365 Secure Score, designed to help you analyze and act to improve your security risk in Office 365. Its initial version was a PowerShell script, which one downloads and runs locally on the machine. As such it relied on some prerequisites, such as the presence of the relevant PowerShell modules and/or the Sign-In Assistant. The tool has since moved to become an app integrated with Office 365, which is what we will review in this article.

Meet the Secure Score Analyzer app

The new version of the tool is delivered as Office 365 app so all you need to do in order to start using it is to navigate to and consent to using the app. The consent part is also my first remark about the (current version of the) app – for a tool designed to bring awareness to common and not so common privacy and security settings, it could handle the OAuth flows better. The publisher is not clearly marked as Microsoft and it wouldn’t hurt if the official documentation/announcement blog post had a clear explanation on why exactly the app needs the permissions listed below. This is purely cosmetic of course, but in my opinion should be addressed.

The app itself is quite nice on the eyes, with simplistic interface and few UI glitches as expected for a Preview. It has two main sections: the Dashboard and the Score Analyzer. On the top of the Dashboard page, a quick introduction to the app is shown in the form of “carousel slider” as depicted below:

A minor annoyance in the current version – if you dismiss the introduction, it will reappear the next time you navigate back to the Dashboard page.

Below the introduction, you will be presented with your current score. Do note the date listed, in my case it has not been updated in the last few days, even though I have changed some of the audited settings. More about this later, for now let’s focus on the looks:

Doesn’t look too good of a score, does it? If you compare it to the average score across all Office 365 tenants, which sits at the whooping 16, it doesn’t look that bad either. But the biggest benefit of using the tool is getting the insight of settings you might have overlooked, so right below the current score you are presented with the Improve your score section. You simply use the slider to setup a target score, which in turn will generate a list of actions for you to review and work with. I would fully expect to see some sort of Templates here in the future, for example selecting a “HIPAA compliance” scenario will set the score and action goal for you to work against. The current max score is 524 points, which in my case results in 55 additional actions.

The best part of the app is working with the Actions. They’re presented with a short description, Category, User Impact and points Score, and also categorized by Threat type. Based on your selection for the Score slider and the current configuration of your tenant, the list of Active and Completed actions will be adjusted accordingly. For example, in my case I have not configured the Outbound spam notification settings and the following action was recommended:

You might notice that the description is lacking proper Action Score numbers, which is just another thing that should be addressed for the GA version.

Each of the actions also features a Learn more button, pressing it will bring up a new pane on the right giving you additional information relevant to the action: What am I about to change and How will this affect my users. Both of these offer some short guidance, which can potentially be further enhanced by adding some links. Lastly, you have the Launch Now button with conveniently takes you to the correct place to configure the setting in question, something that I’m sure will be appreciated by admins with less experience. In this particular case, pressing the button will take you to the EAC (

If you are looking at one of the Completed actions (those are listed on the Score Analyzer page, but otherwise look quite similar to Incomplete actions), the Learn more button will be replaced with Undo button. The effect is pretty much the same – pressing it will bring the right pane with some additional information and link to the proper portal to revert the configuration changes. Changing a setting isn’t immediately reflected in the score, at least in the current version of the app. The whole “refresh” process is actually a bit unclear, as there is no way for the administrator to force a recheck on the settings. In my case, even three days after changing a setting the score remains the same, which hints that the refresh happens probably on a weekly basis. Perhaps a nice improvement would be to add some sort of Rescan button.

The current number of Actions available in the app is 27, with the goal of reaching 80 after the official launch, though the numbers don’t really match what you see in the app portal. Many actions are currently prefixed with [Not scored], as they are not contributing to the overall Security score. Over time, these should become functional and more actions should be added. Even without considering the missing ones, the number of actions is substantial, so the dropdown allowing you to quickly filter them by Category, Impact or Complexity, comes handy. You cannot however combine filters such as “account” actions with “low user impact”, but perhaps such functionality will come in the future. If you are browsing the Score Analyzer page, you can also filter them by Completed or Incomplete status.

Speaking of the Score Analyzer page, the first thing you will notice there is a nice historical graph representation of how your score has improved in the past few months, as well as how it relates to the average Office 365 score across tenants:

You can easily change the date range, but the coolest feature here is the Export button. It allows you to get a report of all actions and their relevant properties in a PDF or CSV file, with the later offering the obvious benefits of being able to filter, visualize or compare data as per your preferences.

Below the graph, you will also see a breakdown of your score across areas. Further down, you will find the list of Completed and Incomplete actions, which offers the same level of detail as what we already saw on the Dashboard page.

To finish up on the UI elements, we can also cover the top bar. While it does feature the familiar Office 365 view, none of the elements are currently working. I expect to see the full start menu and working Help/Profile links in the release version of the app.


This concludes our short review of the new Secure Score Analyzer app. It has been redesigned to integrate directly with the Office 365 portal and while the preview version still has some glitches, the overall impressions are positive. The app is pleasant on the eye, simple to use and offers a lot of context information to help Office 365 admins better understand each of the audited controls.

Currently, the list of monitored settings is not that impressive, and it focuses mostly on Exchange. By the time the app is officially launched, the list should grow to about 80 and will hopefully include actions across more workloads. For example, a notable omission in the current version is External sharing for SharePoint Online and OneDrive for Business. I would also love to see things such as Azure AD Privileged Identity Management or Activity alerts included, even as “informational” actions, or some of the “hidden” Azure AD settings and so on. Apart from the actions, some additional functionality might be handy, such as notifications for newly added actions or score changes, a Rescan button, some additional baselines and filters per industry or security standard, and so on.

In any case, the Security Score Analyzer is something you should keep an eye on!

This entry was posted in Office 365. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *