SCOM Enhanced Email Notification Script Version 2.X



In September of 2010 Tao Yang wrote a fantastic Powershell script (v2.0 here) for command channel notifications in System Center Operations Manager. Here is my adaptation of the script.

NOTE: This post was migrated from Technet and the formatting leaves a lot to be desired. I’ve tested this thoroughly in my lab and I suggest you do the same before using in any other environments. This script is meant to be an example and therefore use it at your own risk. Tested with SCOM 2012, 2012 SP1, 2012 R2, 2016, 2019.

SCOM Enhanced Email Notification Script
Version: 2.3.66


Version History

  • Update!   2024.10.22: v2.3.66 Fixed XSLT conversion for Knowledge article. Also fixed $tmpHTMLPath creation. Fixed email override feature within alert description. Removed unnecessary Cleanup function.
  • Update!   2023.02.07: v2.3.65 Added Culture and DateTimeFormat to Config. Improved knowledgearticle retrieval. Added $SubscriptionID to Logging.
  • 2021.04.28: v2.3.63 Improved logging and MAMLtoHTML conversion.
  • 2017.10.10: v2.3.62  Fixed resend loop on “Closed” alert resolution state.
  • 2017.9.28: v2.3.61 Added support for execution policy bypass from command line. This is an alternative to modifying the global execution policy. 
  • 2017.8.30: v2.3.6 Improved network device details for CustomField9.
  • 2016.5.17:  v2.3.5  Fixed muli-subscriber failure problem. Delayed alert CustomField updates until the very end of the script due to stale data issue (event ID 33333: “Request to update alert ignored due to invalid TimeModified”). Cleaned up a few unnecessary lines of code.
  • 2015.11.5: v2.3  Added configuration file per Tao’s suggestion. Thanks, Tao!
  • 2015.7.12: v2.2.5  Added mail send logging.
  • 2015.5.14: v2.2.4  Added LastModified field. The script adds the “LastModified” time to the email body. However, the script still takes a few more seconds to finish and right before it finishes, it updates the alert status to “Notified” thereby modifying the alert. If you look in the Console at the LastModified time, it won’t precisely match what is shown in the email notification; the Console will show a timestamp that is slightly more recent. 
  • 2015.5.14: v2.2.3 Removed a few unnecessary lines/commands that could potentially cause script errors and/or failure
  • 2015.5.3: v2.2.2 Removed older style SCOM cmdlets, only 2012R2+ supported now.
  • 2015.5.1: v2.2.1 Added support to include Parent name (IP address) and ‘sysName’ properties for network devices into CustomField9 of alert, also included in notification. Added Powershell code snippet to documentation for setting registry value to increase asynchronous process limit.
  • 2015.3.12: Changed how the PSModulePath gets set.
  • 2015.1.30: Added additional detail to instructions section.
  • 2015.1.8: Added Subscriber schedule validation AND Subscriber Address schedule validation logic. Cleaned up a few minor typos.
  • 2014.12.2: Fixed a typo in the error logging output message.
  • 2014.10.17: Updated so that email body would contain customer field 4 (CF4) before sending the email for “New” alerts.
  • 2014.9.16: Update the subject line format.
  • 2014.8.19: Updated documentation with notes on email subscriptions overrides.

Email Critical Alert Format Example:

From: SCOM-Alert <scomlab@monitoringguys.com>
Date: Wed, Apr 28, 2021 at 6:01 PM
Subject: Notified, SCOMDev: devdb01.CONTOSO.COM;monitoringguys.onmicrosoft.com \Microsoft 365 suite, M365 Services – Incident Message Alert Rule
To: <rickSanchez@gmail.com>

Severity: Critical Error

Severity:Critical Error
Alert Description:Workload: OSDPPlatform
IncidentID: MO251848
Title: Admins may notice that all usage reports data is delayed since April 18, 2021 within the Microsoft 365 admin center
Severity:
PublishedTime: 04/21/2021 23:31:21
MessageText: Title: Admins may notice that all usage reports data is delayed since April 18, 2021 within the Microsoft 365 admin center

User Impact: Admins may notice all usage reports data is delayed since April 18, 2021 within the Microsoft 365 admin center.

More info: Admins should be able to access the following reports: Browser usage, Exchange Online user activity, Forms, Yammer reports, and Apps Activation reports

Current status: We’ve successfully restored most of the affected reports and are continuing to restore the remaining portion to fully resolve this issue.

Scope of impact: This issue affects any of your admins attempting to view usage reports data after April 18, 2021.

Start time: Monday, April 19, 2021, at 12:00 AM UTC

Root cause: A portion of upstream data source service infrastructure was performing below acceptable thresholds, resulting for impact.

Next update by: Friday, April 23, 2021, at 5:30 AM UTC
MessageType: Incident
Classification: Advisory
Command Channel:SCOMEnhancedEmailNotIfication.PS1
Source:Microsoft 365 suite
Path:devdb01.CONTOSO.COM;monitoringguys.onmicrosoft.com
Principal Name:devdb01.CONTOSO.COM
Alert Name:M365 Services – Incident Message Alert Rule
Alert Resolution State:Notified (2)
Alert Rule Name:M365 Services – Incident Message Alert Rule
Alert Rule Description:This will raise an alert for new incident messages.
Time Raised:04/21/2021 23:34:53
Last Modified:04/28/2021 14:32:23
Alert ID:4446412e-b8f2-43fa-82e6-5243b1bc1a7b
SCOM Operations Console Info:Your Company Wiki
SCOM Web Console link:SCOM Web Console
Research It:Bing It!
ADMIN DIAGNOSTIC INFO——————–
**Use this command to view the full details of this alert in SCOM Powershell console:Get-SCOMalert -Id “4446412e-b8f2-43fa-82e6-5243b1bc1a7b” | format-list *
This email sent from:MS01.CONTOSO.COM
CF1: Alert.NetBIOSName:devdb01
CF2: Alert.NetBIOSDomain Name:CONTOSO
CF3: Alert.PrincipalName:devdb01.CONTOSO.COM
CF4: SubscriberList:Subscribers:rickSanchez@gmail.com
CF5: Management Pack Name:M365 Supplemental Services
CF6: Alert Class Name:M365SSVC.Services.Service
CF7: Alert.Category:Alert
CF8: Workflow Type:RULE
CF9:RESERVED
CF10: This Email Generated From Script:C:\SCOM_SCRIPTS\Notifications\SCOMEnhancedEmailNotification.ps1
Context:
Note: This context data is only relevant to the moment/time at which this alert was sent.
NameValue CategoryServiceComms ClassificationAdvisory IDMO251848 MessageTextTitle: Admins may notice that all usage reports data is delayed since April 18, 2021 within the Microsoft 365 admin center User Impact: Admins may notice all usage reports data is delayed since April 18, 2021 within the Microsoft 365 admin center. More info: Admins should be able to access the following reports: Browser usage, Exchange Online user activity, Forms, Yammer reports, and Apps Activation reports Current status: We’ve successfully restored most of the affected reports and are continuing to restore the remaining portion to fully resolve this issue. Scope of impact: This issue affects any of your admins attempting to view usage reports data after April 18, 2021. Start time: Monday, April 19, 2021, at 12:00 AM UTC Root cause: A portion of upstream data source service infrastructure was performing below acceptable thresholds, resulting for impact. Next update by: Friday, April 23, 2021, at 5:30 AM UTC MessageTypeIncident PublishedTime04/21/2021 23:31:21 Severity TitleAdmins may notice that all usage reports data is delayed since April 18, 2021 within the Microsoft 365 admin center WorkloadOSDPPlatform

Knowledge Article:


Summary

Will alert on new Service status messages. 

Resolutions

  • ?

Additional

The condition detection filters allow the user to determine which types of messages become collected/reported based on Classification and MessageType. The previously processed messages are cached in xml files that may be located in the following locations. Listed in order of liklihood:
C:\Windows\Temp\M365SSM\Services\ServiceMessages_<workloadName>.xml
C:\Users\<username>\AppData\Local\Temp\M365SSM\Services\ServiceMessages_<workloadName>.xml

If you wish to re-alert on all known/current messages, simply delete the related .xml file(s). 

External

https://MonitoringGuys.com

Overridable Parameters

NameDescriptionDefault Value
CD_ClassificationRegexRegex value for which a match will trigger the condition detection.
Options:Advisory, Incident
Incident|Advisory
CD_MessageTypeRegexRegex value for which a match will trigger the condition detection.
Options: Incident, MessageCenter
Incident
EventIDFilterThis can be used to filter which EventIDs get written by the workflow to the Operations Manager event log. This is only relevant when logging is enabled. See ‘WriteToEventLog’. Typically this is for customer support engineer use only.
PoshLibraryPathFor customer support engineer use only.
ProbeActionTimeoutSecondsIf the workflow module does not exit gracefully by this time limit, the module will be forced to terminate.180
SyncTimeThis can be set to force a workflow to synchronize its Interval to a specific start time. If no SyncTime value is provided to the workflow, the workflow will be initiated at the agent’s earliest opportunity after receiving a configuration change or restarting. Typically no SyncTime is preferred. Format is 00:00. Example for 5:36pm: 17:36. Example for 2:15am: 02:15.
WorkflowNameThis value gets passed into the datasource script and becomes noted in logged events (if logging is enabled). The value provided for this parameter gets appended to the ProbeAction/WriteAction name used for the datasource. If the datasource is used by more than one workflow an override value for this parameter could break cookdown. This could be used to differentiate/identify a specific instance of the scripted datasource. Typically this is for customer support engineer use only.
WriteToEventLogThis will enable/disable script logging to the Operations Manager event log.false

Note:

The context information provided in this notification is limited and is not always helpful. To see more detailed information about this alert, log into the appropriate SCOM console for the applicable datacenter (SCOM management group) and use the Health Explorer to find more details about the state change event for the object.




Closed Alert Example

From: SCOM-Alert [mailto:NOREPLY@contoso.com]
Sent: Thursday, October 8, 2015 3:32 PM
To: Tyson Paul <Tyson@contoso.com>
Subject: Closed, TEST:MS02\http://gunbot.net/ammo/rimfire/22short/, HTTP Request Error: Content Validation Error

Severity: Critical Error

Severity:Critical Error
Alert Description:Content Validation Error

 

Monitor Settings:
URL: http://gunbot.net/ammo/rimfire/22short/
ContentMatch String: Pickle Rick!
GroupID: URLGenie_Default
DNSResolutionTime: 0.1516305
Interval: 300
RetryCount: 1
Wiki: http://blogs.msdn.com/b/tysonpaul/archive/2015/05/04/urlgenie-management-pack-for-scom-an-easy-solution-for-bulk-website-monitoring.aspx
Description: Gunbot.net, Near realtime tracking of who has ammo, mags and reloading supplies in stock.

******* Request Headers *******
GET /ammo/rimfire/22short/ HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; Windows NT 6.1; en-US)
Content-Type: text/xml;charset=utf-8
Accept-Language: en-US
Accept-Charset: utf-8
From: SCOM@yourdomain.com
Connection: Keep-Alive

******* End Request Headers *******

******* Response Headers *******
ResponseHeaders: HTTP/1.1 200 OK
Connection: close
Date: Thu, 08 Oct 2015 17:46:12 GMT
Content-Length: 4842
Content-Type: text/html; charset=UTF-8
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.3.3

******* End Response Headers *******

Command Channel:MyDev
Source:http://gunbot.net/ammo/rimfire/22short/
Path:MS02.Contoso.com;\\Db01.contoso.com\scom_do_not_touch\URLGenie
Principal Name:MS02.Contoso.com
Alert Name:HTTP Request Error: Content Validation Error
Alert Resolution State:Closed (255)
Alert Monitor Name:URLGenie Content Monitor
Alert Monitor Description:None
Time Raised:10/08/2015 10:45:42
Last Modified:10/08/2015 15:30:32
Alert ID:b88f30ec-b008-4fc5-b785-a19cc3646238

SCOM Operations Console Info:

Operations Console Login Info

SCOM Web Console link:

Web Console

Research It:

Bing It!
 
ADMIN DIAGNOSTIC INFO——————–
**Use this command to view the full details of this alert in SCOM Powershell console:Get-SCOMalert -Id “b88f30ec-b008-4fc5-b785-a19cc3646238” | format-list *
This email sent from:MS01.Contoso.com
CF1: Alert.NetBIOSName:MS02
CF2: Alert.NetBIOSDomain Name:CONTOSO
CF3: Alert.PrincipalName:MS02.Contoso.com
CF4: SubscriberList:Subscribers:Subscriber1;
CF5: Management Pack Name:URLGenie Management Pack
CF6: Alert Class Name:URLGenie.HttpRequest
CF7: Alert.CategoryAvailabilityHealth
CF8: Workflow TypeMONITOR
CF9:RESERVED
CF10: This Email Generated FromC:\SCOM_SCRIPTS\Notifications\SCOMEnhancedEmailNotification_v2.2.4.ps1
Context: 
Note: This context data is only relevant to the moment/time at which this alert was sent.
type : Microsoft.SystemCenter.WebApplication.WebAp
plicationData
time : 2015-10-08T15:30:32.3385560-07:00
sourceHealthServiceId : 120875E8-C79E-91CA-E3AC-EA09A391F4BD
RequestResults : RequestResults
TransactionResponseTime : 0.1240154
TransactionResponseTimeEvalResult : 0
CollectPerformanceData : CollectPerformanceData

Knowledge Article:


Summary

The context information for these alerts is not always helpful. To see more detailed information about this alert log into the console for the management group.

Wiki


Note:

The context information provided in this notification is limited and is not always helpful. To see more detailed information about this alert, log into the appropriate SCOM console for the applicable SCOM management group and use the Health Explorer to find more details about the state change event(s) for the object.

Directions:

In the script there are instructions at the top that describe a few of the settings (lines of code) that must be configured before use. Simply edit the “Settings” area within the script. Use Notepad or your favorite text editor. I like Powershell ISE or Notepad++ .
Configuration is very simple and there are examples for each setting within the script comments at the top.

Here’s what I recommend:
PERFORM THESE TEST PROCEDURES IN A LAB ENVIRONMENT (NON PRODUCTION):
 

Remove all management servers from your Notifications Resource Pool except for one single server. This will help you troubleshoot during testing if needed.

Create this folder on your test mgmt server: “C:\SCOM_SCRIPTS\Notifications

Put the script into that new folder.

Open the Config.xml file and customize the necessary fields: smtp server, port, smtp sender, web console url, etc.  See the example files also in the folder. This Config.xml file must be located in the same directory as the .SCOMEnhancedEmailNotification.ps1 script. I suggest you put these files in this path on your management servers:   C:\SCOM_SCRIPTS\Notifications

Make sure that your default action account is a local administrator on your test mgmt server.

Make sure that your Powershell execution policy is not blocking the command channel execution.

  • View policy: Get-ExecutionPolicy 
  • Or just run the following to be sure the policy is “unrestricted”. In an elevated Powershell console on the mgmt server, run this command:
    Set-ExecutionPolicy Unrestricted
  • Note: a reboot may be necessary after modifying the execution policy.

Create the command channel:

Field ParameterNotes
Full path of the command file:%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exeStandard path to Powershell.exe (all versions)
Command line parameters:“C:\SCOM_SCRIPTS\Notifications\SCOMEnhancedEmailNotification.ps1” -Description ‘MyDev’ -AlertID ‘$Data/Context/DataItem/AlertId$’ -SubscriptionID ‘$MPElement$’* IF additional Ops Manager event log details are desired,  use the -Verbose switch in the command line parameters.
Startup folder for the command line:C:\SCOM_SCRIPTS\Notifications 

NOTE: If you don’t want to set the global execution policy to Unrestricted then consider adding the following to the beginning of your Command line parameters:
-ExecutionPolicy Bypass -File

Example:  

-ExecutionPolicy Bypass -File "C:\SCOM_SCRIPTS\Notifications\SCOMEnhancedEmailNotification.ps1" -Description 'MyDev' -AlertID '$Data/Context/DataItem/AlertId$' -SubscriptionID '$MPElement$'

Note: When using -ExecutionPolicy parameter notice the double quotes in the command above!  If you use single quotes around the file name instead of double quotes your command channel won’t trigger and you won’t get any logging or events in the event log and you will lose hours troubleshooting this (like I did). 

Subscribers:

  • Create a Test Subscriber with your email address. You can configure multiple subscribers, each with one or more recipients. The script will send to To, CC, and BCC. Notice the “TO” and “CC” preceding the addresses. If “TO”, “CC”, or “BCC” are omitted, the default becomes “TO”.
    Note: the email address is in the “Name” field. With other channels the “Name” field is used for a friendly/display name. With the Command channel you use the actual email address here.


 Subscriptions:

Create a test subscription, criteria: Any alerts with “synthetic transaction” in the name
The name match will be: %Synthetic Transaction%

  • Pick one or more “Subscribers”

New! Added support for schedules on Subscriber and individual Addresses.

Note: ANY Exclusions will take precedence over any Inclusions.

Testing:

  • Generate a test alert with this command from regular command prompt (not Powershell console), paste the following into command window:
    EventCreate /T ERROR /ID 101 /L APPLICATION /SO TEST /D “This is a synthetic transaction test only. Disregard this event. Note: This alert was generated by entering the below command into a command pompt on the server in question:> EventCreate /T ERROR /ID 101 /L APPLICATION /SO TEST /D
  • Watch Task Manager on the test mgmt server, look for a Powershell.exe process to spin up. Usually the process will take 10 – 20 seconds to send the mail.
  • You will likely want to adjust your max asynchronous process limit for Powershell as described in the ReadMe.txt included in the download.
  • You should receive the email quickly if you configured everything correctly.
  • Copy the script folder to your other management servers that will be in the Notifications Resource Pool. It must be in the same path on each mgmt. server.
  • Put each management server alone in the Notifications Resource Pool, one at a time,  and test each with the Synthetic Transaction procedure described above.

Troubleshooting:

If the script doesn’t appear to be running, open Task Manager, sort by Name, and watch for a new Powershell.exe process to spin up under the Management Server Action Account context/user . The script takes from 10-20 seconds to complete. If the new Powershell.exe process doesn’t open within 60 seconds or opens briefly and then disappears, then you have a problem.

Make sure the script itself is not being blocked from running. Right-click the script, select Properties. If you see a “Unblock” button, click it to make sure the script is not being blocked from execution.

The Powershell execution policy may be Restricted. Change this to Unrestricted. Set-ExecutionPolicy Unrestricted

There may be an error with the Channel configuration or path(s). Double check the path to the .ps1 script. Make sure the full file path is precise. I suggest navigating to the actual path where the script lives and copying the full path right from the File Explorer address/path field. This way there’s no chance of misspelling the path to the script. See command line parameter field example above.

Check the Notifications folder for any errors that might appear in the error log. Use the -WriteToEventLog command line parameter switches for additional Ops Manager event logging detail.

If needed, you can manually run the script. You must first set the required variables near the top of the script.

############### UNCOMMENT FOR MANUAL TESTING ############
 # $Description = "ManualTest"
 # $alertID = (Get-SCOMAlert)[0].ID.Guid

# Create a test subscription with “Synthetic” in the name
# $SubscriptionID = (Get-SCOMNotificationSubscription -DisplayName *Synthetic* ).ID.Guid

# $Verbose = $true
################## FOR TESTING ##########################

An additional method to gain visibility into any errors that might occur (before the script has a chance to write error output to the event log) is to append the line shown directly below at the end of the command line parameters:

 ; Write-Output "$($Error | Out-String)" |  Out-File -Filepath C:\SCOM_SCRIPTS\Notification_Errors.txt

Example:

"C:\SCOM_SCRIPTS\Notifications\SCOMEnhancedEmailNotification.ps1" -Description "MyDev" -AlertID '$Data/Context/DataItem/AlertId$' -SubscriptionID '$MPElement$' -Verbose -WriteToEventLog ; Write-Output "$($Error | Out-String)" |  Out-File -Filepath C:\SCOM_SCRIPTS\Notification_Errors.txt

This will output any errors that occur when the workflow attempts to launch the command channel.

Also, I’ve seen some weird things happen when copying/pasting from online sources into Powershell scripts. Make sure you are not copying some weirdly encoded characters. I’m pretty sure this blog doesn’t use any strange encoding but to keep your sanity, copy the code portions into Notepad++ and force the encoding to be UTF-8, then copy from there into the Console wizard.

*Please let me know if you have any suggestions to improve this article.

Page History:

  • 2023.02.02: updated a few tiny things to reflect current script version functionality
  • 2021.06.10: fixed broken images
  • 2021/04/28: Updated the email examples. Updated the configuration instructions.
  • 2015/11/5: Updated the email examples. Updated the configuration instructions.
  • 2015/5/14: Cleaned up some of the text. Added script version history near top.
  • 2014/9/16: Updated script. Improved notification subject line format.
  • 2014/10/15: Updated command channel arguments examples, removed some quotes
  • 2014/10/17: Updated troubleshooting section. Also updated the script to add the Subscribers to the CF4 field on “New” alerts. Previously subscribers would be empty for new alerts.
  • 2014/10/27: Updated troubleshooting section.
  • 2014/10/29: Updated some sloppy grammar.

Disclaimer: Sample scripts in this guide are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

4 Replies to “SCOM Enhanced Email Notification Script Version 2.X”

  1. Afternoon and thank you for this. Of course I have questions and this might be painful as I have not done a lot with notifications, just basic stuff. In the config.xml, line 14, do I remove the possible values I do not want to use.
    Also, when the notification is sent and the ResolutionState flips to 85, will it then get reset to 0 by the system or does it stay at 85?
    Thanks very much!
    TS

  2. Afternoon Paul, do you have any plans to update this, which by the way is awesome. I chose your method because it provides much more info (and colors) than the default method. Only issue I have is it is very resource intensive in busy environments. Since it is powershell, I keep coming up against Out of resources alerts and 100 messages limit limit thing. Also, is there a way to remove the “notified” portion so it only looks for new and then all else works? This would help to relieve some of the notifications count.
    Thanks!
    Tony

    1. @Tony,
      I don’t have any plans to update it but can you be a little more specific about your request?
      "...is there a way to remove the “notified” portion so it only looks for new and then all else works? This would help to relieve some of the notifications count."

      The script will only update an alert object if ResolutionState is currently 0/New. This includes setting the ResolutionState to whichever integer value you configured for “Notified”.
      The script (command channel) is triggered by the subscription criteria. If your subscription criteria does not specify “New” only, then upon the initial processing by the script the alert will get updated thereby causing the subscription to process the alert again which will trigger the script a second time. The alert object will not get updated a second time because the script only updates “New” alerts, however the email will get sent.
      If I understand your problem correctly, the solution is to configure the subscription criteria to catch “New” alerts only. “with a specific resolution state” = “New (0)”

Leave a Reply

Your email address will not be published. Required fields are marked *