Monitoring processes on remote Windows machines with Procmon

Process Monitor capture

I do a fair bit of work with VMs somedays, using them for a variety of purposes, including integration tests for Windows installers and building components for different platforms. Much of this work runs headless without a GUI and without manual intervention.

Sometimes things can go awry with a process, such as not shutting down as expected and keeping a handle open to a file that should be deleted, or spawning other processes and threads unexpectedly. When this happens on a Windows VM, it's useful to run Process Monitor a.k.a. Procmon on the remote VM to capture process activity for further investigation. This is the topic of today's post.

I've put together a small PowerShell script module to make the process of running Procmon on a remote Windows VM easier, and will walkthrough how to use it.

Assuming we're using Vagrant with the Virtualbox provider, we can create a minimal Vagrantfile with the adaptiveme/windows10 box from the gallery with

vagrant init adaptiveme/windows10

which will create a Vagrantfile in the current directory with the following contents

Vagrant.configure("2") do |config|
  config.vm.box = "adaptiveme/windows10"
end

Now to bring the box up

vagrant up

If this is the first time running the box, Vagrant will download it first

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'adaptiveme/windows10' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'adaptiveme/windows10'
    default: URL: https://vagrantcloud.com/adaptiveme/windows10
==> default: Adding box 'adaptiveme/windows10' (v1.0) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/adaptiveme/boxes/windows10/versions/1.0/providers/virtualbox.box
    default:
==> default: Successfully added box 'adaptiveme/windows10' (v1.0) for 'virtualbox'!
==> default: Importing base box 'adaptiveme/windows10'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'adaptiveme/windows10' is up to date...
==> default: Setting the name of the VM: vagrant_default_1541210094234_50798
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 3389 (guest) => 3389 (host) (adapter 1)
    default: 22 (guest) => 2222 (host) (adapter 1)
    default: 5985 (guest) => 55985 (host) (adapter 1)
    default: 5986 (guest) => 55986 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: WinRM address: 127.0.0.1:55985
    default: WinRM username: vagrant
    default: WinRM execution_time_limit: PT2H
    default: WinRM transport: negotiate
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /vagrant => C:/Users/russc/Desktop/vagrant

With the box now up and running, we can run PowerShell commands from a Windows host with the PowerShell CLI command

vagrant powershell -c "gci env:"

which will write the Environment variables to stdout.

Vagrant winrm

From a Mac or Linux host, there are numerous ways to run PowerShell scripts on a Windows vagrant box, including setting up an SSH server on the box and opening an SSH session with an SSH client on the host. The approach that I've had the most success with in the past however is to use the vagrant winrm plugin to executes scripts with Windows Remote Management (WinRM). Vagrant itself uses WinRM to execute provisioning steps for a Windows box, so it's a tried and trusted mechanism for executing commands remotely on Windows. The one piece of functionality that I've found to be missing is the ability to send input from stdin, which WinRM supports, but the plugin didn't the last time I checked. It's a small inconvenience, but not insurmountable.

To install the winrm plugin

vagrant plugin install vagrant-winrm

And to send the same command from a Mac or Linux host as before

vagrant winrm -c "gci env:"

For some commands, you may need to elevate privileges using the -e switch.

An aside: Elevated winrm commands

The implementation for elevating privileges through WinRM is interesting. Asking to elevate privileges creates a Windows scheduled task on the vagrant box, which is executed immediately and the output sent to a known file. The file is then read and returned as the result of the executed winrm CLI command. This little trick/hack is employed to circumvent "The Second/Double Hop" problem; a remote connection is made from the host to the Vagrant box, delegating to the vagrant credentials on the vagrant box. Asking to elevate on the Vagrant box requires delegating credentials again, which is disallowed for remote connections by default. Scheduled tasks however get around this because they originate from the vagrant box, with the full credential context.

Arguably, the correct way to allow double hops is to enable CredSSP with WSMan, but the Scheduled task solution works with less setup.

Running Procmon remotely

The PowerShell module can be installed on the remote machine from the gist, if it has access to the internet

vagrant winrm -c '. { iwr -useb https://gist.githubusercontent.com/russcam/3a44c8dad43cf5ccbda0b4dd0832e0b5/raw/810530b919466eb882fa7bb42914db9dc226c75c/Procmon.ps1 } | iex;'

This installs the module into this WinRM session only, so we can start using the module cmdlets to run Procmon by executing subsequent commands. the following will start Procmon, wait 2 seconds then stop Procmon. I'm going to use the shortened url https://git.io/fxhUD from this point onwards over the longer winded gist raw URI because it's a bit of a mouthful!

vagrant winrm -c '. { iwr -useb https://git.io/fxhUD } | iex; Start-Procmon; Start-Sleep 2; Stop-Procmon'

By default, Start-Procmon

  1. Downloads Procmon zip and unzips the contents to the $ProcmonDir, with the default location at $env:TEMP\ProcessMonitor\
  2. write process events to a backing file configurable by the $EventFile parameter, with the default location in $env:TEMP\ProcessMonitor\events.pml

Great! We've captured a firehose of events, but how do we get them back to the host machine in a format that will be easy to manipulate? If we're working on a host Windows machine, we might have Procmon available locally to be able to open the events.pml file. What would be better however is to convert the file to CSV and read that back. This is where the ConvertTo-ProcmonCsv cmdlet comes in, which will convert the PML file specified by $EventFile into a CSV file at $CsvFile. The cmdlet has default values that assume files are in $env:TEMP\ProcessMonitor

vagrant winrm -c '. { iwr -useb https://git.io/fxhUD } | iex; Start-Procmon; Start-Sleep 2; Stop-Procmon; ConvertTo-ProcmonCsv; Get-Content $env:TEMP\ProcessMonitor'

This will return all the events to stdout in CSV format. If the host machine is running PowerShell, we can capture the returned event CSV lines as objects and further filter and inspect

$events = $(vagrant winrm -c '. { iwr -useb https://git.io/fxhUD } | iex | Out-Null; Start-Procmon; Start-Sleep 2; Stop-Procmon; ConvertTo-ProcmonCsv; cat $env:TEMP\ProcessMonitor\events.csv' -e | ConvertFrom-Csv)

$events[0..9] | Format-Table

which produces something similar to

Time of Day      Process Name    PID  Operation                    Path                                            Result      Detail
-----------      ------------    ---  ---------                    ----                                            ------      ------
21:44:32.5296919 wsmprovhost.exe 3744 QueryOpen                    C:\Users\vagrant\AppData\Local\Temp\tmp2094.tmp SUCCESS     CreationTime: 02/11/2018 21:44:30, LastAccessT... 
21:44:32.5298646 wsmprovhost.exe 3744 QueryOpen                    C:\Users\vagrant\AppData\Local\Temp\tmp2094.tmp SUCCESS     CreationTime: 02/11/2018 21:44:30, LastAccessT... 
21:44:32.5315801 wsmprovhost.exe 3744 QueryOpen                    C:\Users\vagrant\AppData\Local\Temp\tmp2094.tmp SUCCESS     CreationTime: 02/11/2018 21:44:30, LastAccessT... 
21:44:32.5320442 wsmprovhost.exe 3744 QueryOpen                    C:\Users\vagrant\AppData\Local\Temp\tmp2094.tmp SUCCESS     CreationTime: 02/11/2018 21:44:30, LastAccessT... 
21:44:32.5322627 wsmprovhost.exe 3744 QueryOpen                    C:\Users\vagrant\AppData\Local\Temp\tmp2094.tmp SUCCESS     CreationTime: 02/11/2018 21:44:30, LastAccessT... 
21:44:32.5324631 wsmprovhost.exe 3744 CreateFile                   C:\Users\vagrant\AppData\Local\Temp\tmp2094.tmp SUCCESS     Desired Access: Generic Read, Disposition: Ope... 
21:44:32.5325423 wsmprovhost.exe 3744 QueryStandardInformationFile C:\Users\vagrant\AppData\Local\Temp\tmp2094.tmp SUCCESS     AllocationSize: 0, EndOfFile: 0, NumberOfLinks... 
21:44:32.5325726 wsmprovhost.exe 3744 ReadFile                     C:\Users\vagrant\AppData\Local\Temp\tmp2094.tmp END OF FILE Offset: 0, Length: 4,096, Priority: Normal
21:44:32.5326005 wsmprovhost.exe 3744 ReadFile                     C:\Users\vagrant\AppData\Local\Temp\tmp2094.tmp END OF FILE Offset: 0, Length: 4,096, Priority: Normal
21:44:32.5326419 wsmprovhost.exe 3744 CloseFile                    C:\Users\vagrant\AppData\Local\Temp\tmp2094.tmp SUCCESS

Pretty handy!

Filtering Procmon events

As I mentioned earlier, Procmon is a firehose of events and often we want to capture events specific only to one process. We can do this by specifying filters and passing them to Start-Procmon

$filters = @(New-ProcmonFilter -Column 'Process Name' -Relation is -Value chrome.exe -Action Include)
# add the default exclude filters
$filters += Get-DefaultProcmonFilters

$filters | Start-Procmon

This persists the given filters in the registry, which will be applied to Procmon on startup

Process Monitor capture of chrome.exe

When converting the events.pml to CSV, you can choose whether to apply this filter to the captured events with the $ApplySavedFilter switch, which defaults to $true.

All of the cmdlets available can be explored with Get-Help Procmon -Full

Name                              Category  Module   Synopsis
----                              --------  ------   --------
Clear-ProcmonFiltersRegistry      Function  Procmon  Clears the Process monitor filter bytes in the registry
ConvertTo-ProcmonCsv              Function  Procmon  Converts a Process monitor event file to CSV file
Download-Procmon                  Function  Procmon  Downloads Process monitor zip to the destination file
Get-DefaultProcmonFilters         Function  Procmon  Gets the default Process monitor filters
Get-ProcmonFiltersBytes           Function  Procmon  Gets the bytes for a collection of Process monitor filters
Invoke-Procmon                    Function  Procmon  Invokes Process monitor with given arguments
New-ProcmonFilter                 Function  Procmon  Creates a new Process monitor filter
Start-Procmon                     Function  Procmon  Starts Process monitor with given filters applied,...
Stop-Procmon                      Function  Procmon  Stops Process monitor
Unzip-Procmon                     Function  Procmon  Unzips Process monitor zip file to the destination directory
Write-ProcmonFiltersBytesToReg... Function  Procmon  Writes the Process monitor filter bytes to the registry
Write-ProcmonFiltersToRegistry    Function  Procmon  Writes the Process monitor filters to the registry

Let me know if you find these useful!

Comments

comments powered by Disqus