Seems that there is a known issue with the Term Store admins in SharePoint Online, namely when a user is added as Global Administrator (and as such added as SharePoint Admin and to the Term Store Admins) and later removed, the permissions they have on the term store are preserved (seems that they are stored in separate database). When you browse any page that incorporates Managed Metadata, SharePoint will enumerate all the Taxonomy Administrators, even the removed ones, and will try this lookup for a fixed period, which in turn will result in delay when getting the server response in the client application.
The solution to this is to remove the admin from the Term Store before deleting his account, or if already deleted, create account with the same UPN and add/remove it from the Term Store admins. Only the UPN is stored, so the fact that this is entirely new object with different *IDs should not play any role. Alternatively, you can raise a case with Microsoft support to have the orphaned entries removed.
Microsoft is aware of the issue and is working on resolving it. They seem to be keeping this one quiet though and there is no information regarding this issue available (took several months in our case to identify the cause).