File Transport for SCOM

Tyson Paul1 commentsAuthoringManagement PacksSCOM 2016SCOM 2019file transermanagement pack



Retrieve files and/or folders from any agent-managed computer with this nifty management pack.

Download “File Transport Management Pack” FileTransport_1.0.1.25.zip – Downloaded 1294 times – 26.85 KB


How does it work?

The simple answer:
Provide one or more paths in the agent task. The paths get zipped into a binary file which gets chopped up and sent to the mgmt server inside property bags. The mgmt server reassembles the pieces. That’s right, it uses the agent connection. No firewalls or ports need to be mentioned here.

The nerdy details:

  • The “Get Any File” agent task writes an event to the Operations Manager event log of the agent. This event contains information about which paths to “get”.
  • An event rule detects the special event and then passes the necessary information to a PowerShell workflow.
  • The script archives all of the requested paths into a single .zip archive.
  • IF the archive is bigger than the establish property bag size limit, then the archive is broken into individual bite-sized binary chunks (“bite” not “byte”) and written to temp files. Additionally, a corresponding “Get Any File” event is written to the event log for every individual file fragment. This recursively triggers the same event detection workflow to run again for each of the smaller file fragment this time, instead of the original path(s).
  • The target file (.zip archive or tmp file chunk/fragment) is a binary file. The file data gets converted into Base64 string data, then returned within a property bag along with some identifying information about the operation; FileHash, Archive Name, BatchGUID, Index number (for file fragments), etc.
  • The management server converts the property bag payload data Base64 (string) back to binary, then writes all of the file fragments to temp files based on their identifying “index” numbers.
  • Once all of the fragments have been written, the mgmt server combines all of the data into the original archive (.zip).
  • After verifying the SHA256 file hash is correct, the archive is written to the FileShare repository path.
  • Profits.



Show Me How

Overview:

  1. Identify a suitable repository path
  2. Enable the FileShare discovery with the repository path
  3. Run the task


1) Identify a Repository Path

In my lab I chose to use the standard admin share on the D drive of one of my mgmt servers, “MS01”. You can use any file share as long as the Management Server Action Account has read/write access.
Note: You can specify a local folder but you probably only want to discover a single instance of the FileShare in this case.

\\MS01.CONTOSO.COM\D$\FILETRANSPORT

All of my management servers use the Management Server Action Account which is a local admin on all mgmt servers.



2) Enable the FileShare Discovery

I enable the discovery for the class so that all management servers will discover the same shared folder path and should have full access to the share (read/write).

Tip: Immediately upon saving the override, the discovery should trigger on the affected agents. However, here’s a snippet that you can use to force discovery to run for a specific server.

#Collection server name. YOUR server name here.
$FileShareServer = "MS01.contoso.com"

$Task = Get-SCOMTask -name Microsoft.SystemCenter.TriggerOnDemandDiscovery
$TargetInstance = Get-SCOMClass -name 'microsoft.systemcenter.healthservice' | Get-SCOMClassInstance | ? Displayname -eq $FileShareServer
$DiscoID = (Get-SCOMDiscovery -Name FileTransport.FileShare.Discovery ).Id.Guid
$Override = @{
  TargetInstanceId = "$($TargetInstance.Id.Guid)"
  DiscoveryId = "$DiscoId"
}
$T = Start-SCOMTask -Instance $TargetInstance -Task $Task -Override $Override

# Show Task result
Get-SCOMTaskResult -BatchID $T.BatchId



3) Run the Task

Example 1

Collect event logs from multiple agents

Use a comma-separated list of event log names.

Zipped event logs appear within a few seconds.



Example 2

This is another way to get a file but shown with more detail.


First, I run a PowerShell command task to export the Operations Manager event log to local disk for multiple agents.
Optionally I can include MKDIR C:\Temp; to ensure the destination folder exists. See below

MKDIR C:\Temp; C:\Windows\System32\wevtutil.exe epl 'Operations Manager' C:\Temp\OpsMan.evtx

All of the agents have exported the same event log to the same location, with the same file name.
Example:

Next, I run the “Get Any File” task. I have provided a path, archive name, and batch name.
“Path” is the only required override parameter.

“GetFile” event is written on the agent:

I can see the file request data event has been collected. This is purely informational.

I can see the file receipt event has been collected. This is purely informational.


Example 3

I will collect multiple folders, a specific file, and a wildcard mask.

PathType
C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Management PacksDirectory
C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Connector Configuration CacheDirectory
C:\Program Files\Microsoft Monitoring Agent\Agent\*.configFile mask
(wildcard)
C:\Windows\System32\drivers\etc\hostsFile


I simply combine all the paths, comma-separated like this: 

C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Management Packs,C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Connector Configuration Cache,C:\Program Files\Microsoft Monitoring Agent\Agent\*.config,C:\Windows\System32\drivers\etc\hosts

Almost instantly the archive appears in the repository:



Example 4


In this example I specify the SCOM “Resource” folder at the path below. This will contain all resources from all active management packs and should be quite large so I also override the MaxTransferSizeMB to 300. 

C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Resources

I can see the temp binary files being written to the temp folder on the collection server:


285MB (193 property bags) took almost exactly 10 minutes on my lab machines.

The temp files get deleted on the agent and collection server after the file transfer is complete.



Can I integrate this ability into my own solution?

Yes.

Just write a basic GetFile event to the Operations Manager log and the workflows will do the rest to get the file(s).

EventID: 9994
Source: FileTransport
Level: Information (4)
The only required data is the Path as shown below.
GetFile:<paths>

The Data fields are as follows:

  1. Paths
  2. NewArchiveName
  3. BatchName
  4. NewArchiveNameIncludesAgentName
  5. NewArchiveNameIncludesTimestamp
  6. MaxTransferSizeMB
  7. N/A
  8. N/A
  9. N/A

Here’s a function you can use with an example of how to create the event:


################################################################
Function LogThis {
  [CmdletBinding(DefaultParameterSetName='Parameter Set 1', 
      SupportsShouldProcess=$false, 
      PositionalBinding=$false,
      HelpUri = 'http://www.microsoft.com/',
  ConfirmImpact='Medium')]
  [OutputType([Bool])]
  Param (
    [int]$EventID,
    [string]$EventLogName,
    [int]$Type=4, 
    [string[]]$Message="No message specified.", 
    [string] $Source
  ) 

  $Error.Clear()
  $objEvent = New-Object System.Diagnostics.EventLog
  $objEvent.Source = $Source
  $objEvent.Log = $EventLogName

  # EventID, CategoryID, EventLogEntryType 
  # EventLogEntryType: 1=Error, 16=FailureAudit, 4=Information, 8=SuccessAudit, 2=Warning
  $objEventID = New-Object System.Diagnostics.EventInstance($EventID,1,$Type)
  Try {
    $objEvent.WriteEvent($objEventID, @($Message))
  } Catch {
    Return $false
  }
  If ($Error){
    Return $false
  }
  Return $true
}
################################################################


$Source = "FileTransport"
$EventLogName = "Operations Manager"
$EventId = 9994
$Level = "info"
$EventType = @{
  'info' = 4
  'critical'= 1
  'warn' = 2
}

<# 
The logging function accepts an array to populate the numerous Data fields in the event.
The array order is as follows
Paths - multiple paths can be separated by comma
NewArchiveName
BatchName
NewArchiveNameIncludesAgentName
NewArchiveNameIncludesTimestamp
MaxTransferSizeMB
N/A
N/A
N/A

Example 1 - Build an array. The order must be as shown.
[System.Collections.ArrayList]$arrMessage = @()
$Paths = 'C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Resources'
$null = $arrMessage.Add("GetFile:$($Paths)") # Paths
$null = $arrMessage.Add("Resources.zip") # NewArchiveName - give the zip file a name
$null = $arrMessage.Add("DevAgentResourceFiles") # BatchName or job name
$null = $arrMessage.Add("true") # NewArchiveNameIncludesAgentName - include the agent name within the archive name
$null = $arrMessage.Add("false") # NewArchiveNameIncludesTimestamp - include a timestamp within the archive name
$null = $arrMessage.Add("300") # MaxTransferSizeMB - default is 15 but you can increase this as needed.


Example 2 - convert a single string into array. Separate fields with caret ^ symbol. 

$Message = 'GetFile:D:\Somefile.log,C:\webroot\Logs^MyLogFile.zip^ProdLogs^false^false^^^^'
[array]$arrMessage = @($Message.Split('^'))
#>


# Example
[System.Collections.ArrayList]$arrMessage = @()
$Paths = 'C:\Reports\*.html'
# These array elements will end up being the event log Data fields
$null = $arrMessage.Add("GetFile:$($Paths)") # Paths
$null = $arrMessage.Add("MyArchiveName.zip") # NewArchiveName - give the zip file a name
$null = $arrMessage.Add("MyBatchName") # BatchName or job name
$null = $arrMessage.Add("false") # NewArchiveNameIncludesAgentName - include the agent name within the archive name
$null = $arrMessage.Add("false") # NewArchiveNameIncludesTimestamp - include a timestamp within the archive name
$null = $arrMessage.Add("50") # MaxTransferSizeMB - default is 15 but you can increase this as needed.
$null = $arrMessage.Add('') # Must be empty. 
$null = $arrMessage.Add('') # Must be empty. 
$null = $arrMessage.Add('') # Must be empty. 



# Create event Source if it does not already exist.
If ($False -eq [System.Diagnostics.EventLog]::SourceExists($Source))
{
  [System.Diagnostics.EventLog]::CreateEventSource($Source, $EventLogName)
}


LogThis -EventID $EventId -Source $Source -Type $EventType[$Level] -EventLogName $EventLogName -Message $arrMessage



Integration Example

In my SCOMAgentHelper management pack there exists a task to initiate an agent trace file. I simply added the FileTransport code snippet to the tracing workflow script. Now the trace .zip file will automatically be transported to my mgmt server.

I include the function:

I add the snippet in which I specify the path to the target file. In this example script the path is $LocalZipFilePath .

Run the task

The agent trace archive will appear automatically in the FileShare folder.

Note: If you experience agent puke, consider increasing the MaximumQueueSizeKb . I set my lab agents to 76800 (75MB).

Default size: 15,360 (15MB)



Version History:
2024.04.03 – Updated the workflow targets to ‘Microsoft.SystemCenter.ManagementService’ to avoid effects of maintenance mode.
2022.04.27 – Fixed powershell probe type for TestPath task to allow serial output

← →

One Reply on “File Transport for SCOM”

Leave a Reply

Your email address will not be published. Required fields are marked *