File Integrity Monitor with Powershell

A few years back I was doing some reading about file systems. I was interested in how ZFS and BTRFS hash every sector of the disk in order to detect data corruption. I think of it as a file integrity monitor but at a lower level in the system. Most of the time these filesystems are in a raid setup. So when they detect corruption they can repair it with data from another disk. If not then they can at least let you know before you overwrite a backup with bad data.

Then last September my old NAS box running ZFS died. I built a new NAS using Debian and EXT4 but I started thinking about data corruption. What would happen if some precious family photos or videos silently got corrupted? Then the corruption got stored in all my backups? What a nightmare!

Then I thought that there must already be some kind of file integrity software out there?

File Integrity Monitoring Options

I asked around and found out about some products like OSSEC, Wazuh, and AIDE. But this stuff was built with security and intrusion detection in mind. Overkill for what I wanted to do.

So I thought why don’t I make my own? How hard could it be? I could hash files and make sure the hash values do not change.

Checking File Integrity with Powershell

For reasons I can not remember I decided to go with Powershell for this task.

Make Baseline Hash List

First, we need to scan the folder or drive and make a baseline list of all the files and their hashes. I made a separate Powershell script for this job. You only need to run it one time to get everything setup.

#Make Baseline Hashes
$version = "20191007"
$scanPath = "D:\"
$baseStorePath = "D:\FileIntegrityCheck\baseline_hashes.csv"
$counter = 0

Write-Output "Setup - File Integrity Monitor (Sysjolt.com) version: $version"

cd $scanPath

Write-Output "Getting list of files"

$fileList = Get-ChildItem -Recurse
$fileNumber = $fileList.Count

$startime = $(Get-Date)
Write-Output "Scan started at $startime"

$Results = foreach ($FL_Item in $fileList)
{
	#Setting up progress bar
    $counter++
    Write-Progress -Activity "Hashing Files: $FL_Item " -Status "Processed $counter of $fileNumber" -PercentComplete (($counter / $fileList.Count) * 100)
	
	#Getting file hash. Using MD5 for speed
    $fileHash = Get-FileHash $FL_Item.FullName -Algorithm MD5

    if ($fileHash)
    {
        [PSCustomObject]@{
        Hash = $fileHash.Hash
        Path = $fileHash.Path
        }
    }
}

Write-Output "Scan started at $startime and ended at $(Get-Date)"


# on screen display
#$Results

# send to CSV file    
$Results | Sort Path | Export-Csv -Path $baseStorePath

Checking File Hashes For Changes

Now that we have the baseline hashes saved we can run this next Powershell script. I like to do this before a big backup to see if anything changed unexpectantly. If so it will show you what changed. If the changes are ok you can choose to update the baseline hashes.

I also added a progress bar in the code. Depending on the number of files you are scanning this can take a long time. It is nice to see some action on the screen to know it is working.

#Check File Hashes
$scanPath = "D:\"
$baseStorePath = "D:\FileIntegrityCheck\baseline_hashes.csv"
$currentStorePath = "D:\FileIntegrityCheck\current_hashes.csv"
$differencesStorePath = "D:\FileIntegrityCheck\differences_hashes.txt"

$version = "20191007"
$counter = 0

Write-Output "File Integrity Monitor (Sysjolt.com). version: $version"

cd $scanPath

Write-Output "Getting a list of files..."

$fileList = Get-ChildItem -Recurse
$fileNumber = $fileList.Count

$startime = $(Get-Date)
Write-Output "Scan started at $startime"

$Results = foreach ($FL_Item in $fileList)
{
    $counter++
    Write-Progress -Activity "Checking File Hashes: $FL_Item " -Status "Processed $counter of $fileNumber" -PercentComplete (($counter / $fileList.Count) * 100)
    $fileHash = Get-FileHash $FL_Item.FullName -Algorithm MD5

    if ($fileHash) #if here so we do not write blanks for folders
    {
        [PSCustomObject]@{
        Hash = $fileHash.Hash
        Path = $fileHash.Path
        }
    }
}

Write-Output "Scan started at $startime and ended at $(Get-Date)"

# Save to CSV file    
$Results | Sort Path | Export-Csv -Path $currentStorePath

# Write differences between old and new hashes to a text file
Compare-Object (Get-Content $baseStorePath)(Get-Content $currentStorePath) | Format-Table -Wrap | Out-File $differencesStorePath

Write-Output "Scan started at $startime and ended at $(Get-Date)"

# Read in text file
$changes = Get-Content -Path $differencesStorePath

# Display changes
$changes

$input = Read-Host -Prompt 'Would you like to update the base hashes? y/n'

if ($input -eq 'y')
{
    Remove-Item $baseStorePath
    $Results | Sort Path | Export-Csv -Path $baseStorePath
    Write-Output "Base hashes updated."
}

Write-Output "Done..."

How it works

To sum it up quickly I basically make a list of every file and its hash that you want to monitor. I store this into a CSV file and the next time the scan is running I compare the results with the old results in the CSV file.

Then I list any file that’s not matching up so you can see what has changed. In theory, if a file gets corrupted somehow the hash will change and no longer match what is stored. Then you will have an early warning that file integrity is bad before you back the data up.

Download

If you would like to download the files for these scripts you can do that here.

Download File Integrity Monitor

I hope you enjoy and please come back and let me know what you think!