Heading
I use the M365 Services Supplemental Monitoring management pack in my lab. I recently noticed that calls to:
https://graph.microsoft.com/v1.0/admin/serviceAnnouncement/healthOverviews
within my application have been failing with “403 Forbidden“. A few other folks that I know started seeing the same problem. Nothing had changed on our ends, we all checked absolutely everything. All other Graph calls continued to work correctly, only /serviceAnnouncement/… presented this problem.
Below is the output seen while troubleshooting in my lab.
Message: Failed attempt to retrieve Services object(s) at Graph URL: [https://graph.microsoft.com/v1.0/admin/serviceAnnouncement/healthOverviews].
See error data.
WorkflowName: M365SSVC.ServicesMon.PB.PA_(M365SSVC.GetServicesData.AgentTask)
ScriptName: M365SSVC.ServicesMon.ps1
Invocation/Function: LogIt
ThisScriptInstanceGUID: 45FB1F
ScriptLine: 181
Running As: contoso\gmsa-omaa$
ClientID: 8b2c08fc-faa6-xxxx-xxxx-xxxxxxxxxx
M365_AccountName: scomdemo1@monitoringguys.onmicrosoft.com
MgmtGroupRegKey:
TenantName: monitoringguys.onmicrosoft.com
ApiTokenScopeURL: https://graph.microsoft.com/.default
ApiTokenURL: https://login.microsoftonline.com
ApiURL: https://graph.microsoft.com
OnDemandDiscovery: N/A
TLSVersion: 1.2
PoshLibraryPath: C:\Program Files\Microsoft System Center\Operations Manager\Server\Health Service State\Resources\330\M365Library.ps1
ScriptOutputType: Serialized
EventIDFilter: 9990,9991,9992,9995,9996,9997,9998,9999
WriteToEventLog: True
Any Errors:
ERROR OBJECT:
PSMessageDetails :
Exception : System.Net.WebException: The remote server returned an
error: (403) Forbidden.
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.
GetResponse(WebRequest request)
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.
ProcessRecord()
TargetObject : System.Net.HttpWebRequest
CategoryInfo : InvalidOperation:
(System.Net.HttpWebRequest:HttpWebRequest)
[Invoke-WebRequest], WebException
FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Comm
ands.InvokeWebRequestCommand
ErrorDetails :
InvocationInfo : System.Management.Automation.InvocationInfo
ScriptStackTrace : at Script_42b8f6d4bc5d499581693ecbc1ad8377, <No file>:
line 172
PipelineIterationInfo : {}
-----------------------------------------------------------------
RESPONSE OBJECT:
IsMutuallyAuthenticated : False
Cookies : {}
Headers : {Transfer-Encoding, Strict-Transport-Security,
request-id, client-request-id...}
SupportsHeaders : True
ContentLength : -1
ContentEncoding :
ContentType : application/json
CharacterSet :
Server :
LastModified : 7/28/2023 1:21:30 PM
StatusCode : Forbidden
StatusDescription : Forbidden
ProtocolVersion : 1.1
ResponseUri : https://graph.microsoft.com/v1.0/admin/serviceAnnounc
ement/healthOverviews
Method : GET
IsFromCache : False
-----------------------------------------------------------------
RESPONSE HEADERS:
Transfer-Encoding:chunked
Strict-Transport-Security:max-age=31536000
request-id:c5b1dbea-0cca-4677-bcd2-xxxxxxxxxxxx
client-request-id:c5b1dbea-0cca-4677-bcd2-xxxxxxxxxxxx
x-ms-ags-diagnostic:{"ServerInfo":{"DataCenter":"Central US","Slice":"E","Ring":"2","ScaleUnit":"002","RoleInstance":"DS3PEPF0000943C"}}
X-Instance:southcentralus/_readvm_5
X-TimeTaken:0
Content-Type:application/json
Date:Fri, 28 Jul 2023 18:21:30 GMT
-----------------------------------------------------------------
For delegated permissions, the effective permissions of your app are the intersection of the delegated permissions the app has been granted (via consent) and the privileges of the currently signed-in user. Your app can never have more privileges than the signed-in user. Within organizations, the privileges of the signed-in user are determined by policy or by membership in one or more administrator roles.
-some nerd
App Registration
Below is what the app permissions should look like (at the time of this writing) for the aforementioned M365 Services Supplemental MP.
Nerdy Background Info
Skip to the bottom for the solution. Otherwise, here’s some extra detail about app registrations.
This management pack leverages PowerShell to perform synthetic transactions to test functionality and measure performance of various M365 services: OneDrive, Teams, Licensing, etc. The synthetic transactions are conducted through Microsoft Graph API, a RESTful web API that enables programmatic access to Microsoft Cloud service resources. This programmatic access requires a registered App in Azure Active Directory and that registration must be assigned specific permissions. Microsoft Graph exposes granular permissions that control the access that apps have to resources, like users, groups, and mail. Microsoft Graph has two types of permissions: “Application” and “Delegated”. It’s important to understand the differences.
Delegated permissions are used by apps that have a signed-in user present. For these apps, either the user or an administrator consents to the permissions that the app requests and the app can act as the signed-in user when making calls to Microsoft Graph. Some delegated permissions can be consented by non-administrative users, but some higher-privileged permissions require administrator consent.
Application permissions are used by apps that run without a signed-in user present. For example, apps that run as background services or daemons. Application permissions can only be consented by an administrator.
Effective permissions are the permissions that your app has when making requests to Microsoft Graph. It’s important to understand the difference between the delegated and application permissions that your app is granted, and its effective permissions when making calls to Microsoft Graph.
For delegated permissions, the effective permissions of your app are the intersection of the delegated permissions the app has been granted (via consent) and the privileges of the currently signed-in user. Your app can never have more privileges than the signed-in user. Within organizations, the privileges of the signed-in user are determined by policy or by membership in one or more administrator roles. For more information about administrator roles, see Assigning administrator roles in Azure Active Directory.
For example, assume your app has been granted the User.ReadWrite.All delegated permission. This permission nominally grants your app permission to read and update the profile of every user in an organization. If the signed-in user is a global administrator, your app can update the profile of every user in the organization. However, if the signed-in user is not in an administrator role, your app can update only the profile of the signed-in user. It won’t update the profiles of other users in the organization because the signed-in user does not have those privileges.
For application permissions, the effective permissions of your app will be the full level of privileges implied by the permission. For example, an app that has the Mail.Send application permission can send mail as every user in the organization.
To date, the M365 Supplemental management packs use only “Delegated” permissions.
Use of delegated permissions requires an M365 user account so that the monitoring workflows can simulate the user. Before the monitoring workflows can interact with Microsoft Graph API, they must authenticate to Azure and retrieve an access token for the user. The authentication request must include the account password and App secret/key.
During configuration of the Watcher node, both the App secret and account password get encrypted by the security context of the configuration task (usually the default RunAs account) and both are stored locally in the Watcher node registry as encrypted strings. The account context that performed the original encryption is the only account that may decrypt the values and only on that same computer. The strings cannot be decrypted by any other account, and they cannot be decrypted on any other computer. For this reason, do not provide credentials when executing the configuration tasks.
At this time, certificate authentication is not supported by the management pack; only application secret/key. Although some in the IT community may prefer certificates for app registrations, when all things above are considered, the perceived benefits don’t outweigh the administrative and configuration overhead.
If a specific RunAs account is desired, use the security profile: M365 Supplemental Library Default RunAs Profile. It is most common to leave this profile unused/vacant as it is usually unnecessary.
Solution
It turns out that the user account now requires the Message Center Reader role to view service status stuff, regardless of the app registration delegated permissions. Add this role to your monitoring account and your Services monitoring (and discovery) should return to normal. Changes to the role assignment might take a few minutes to become recognized.
Test
Be sure to run the test task.
