Friday, November 13, 2015

Capturing a Memory Dump from a Hung Machine

We recently deployed McAfee's File and Removable Media Protection which seems to hang some Windows 7 machines. It is still early and it is possible that some of our other security products are interfering with McAfee's tool. The hard part is capturing some important diagnostic information, including a McAfee MER.

We are getting reports of the failures from computers more often from Europe. Those computers usually have just booted up within the first hour of work. When the computer hangs, the mouse is frozen and the computer will not respond to Ctrl-Alt-Del.

Here are some of the steps we needed to take to capture a full memory dump using a keyboard command. There are good articles on each step but I didn't find any articles that put them all together.

  1. Make sure the workstation is able to capture a full memory dump
    • https://msdn.microsoft.com/en-us/library/windows/hardware/ff542953(v=vs.85).aspx
    • Paging file needs to be bigger than the size of the RAM by at least 100 MB
    • Stop automatic reboots. This should ensure that the dump is written before rebooting.
    • Make sure to do a complete memory dump
  2. Set the computer to NMICrashDump to capture hardware failures in a BSOD
    • https://technet.microsoft.com/en-us/library/cc957353.aspx
    • HKLM\SYSTEM\CurrentControlSet\Control\CrashControl\NMICrashDump
  3. Set the computer to crash on keyboard command
    • https://msdn.microsoft.com/en-us/library/windows/hardware/ff545499(v=vs.85).aspx
    • There are two keys to worry about depending on USB keyboard or PS/2 keyboads
    • HKLM\SYSTEM\CurrentControlSet\Services\kbdhid\Parameters\CrashOnCtrlScroll
We have started playing with this McAfee KM article to see if it gives any relief. https://kc.mcafee.com/corporate/index?page=content&id=KB81384

One of the more distressing issues is that we cannot capture a memory dump, blue screen or otherwise, when McAfee FRP is fully installed. I am going to test if disabling the local driver will allow the capture shown above.

Monday, June 29, 2015

Alternative Data Streams

I've now run into this twice which means I need a way to remember this problem solution.

Windows NTFS has a method for recording Alternative Data Streams with files.  When you download files from the Internet IE may add an Alt. data stream that includes the Internet Zone the file was downloaded from.  There is a neat PowerShell command for viewing the stream called get-item filename.exe -stream *.

You can use SysInternals stream.exe to view and delete these alternative data streams.  I have not found a way to delete streams within PowerShell.

Here is the best article about the topic:

http://blogs.technet.com/b/askcore/archive/2013/03/24/alternate-data-streams-in-ntfs.aspx

Edit:
Found the PowerShell method

get-item <filename> -stream *
remove-item <filename> -stream zone.identifier

Tuesday, June 2, 2015

COM+ Permission Issue on Win7

Here is a subtle problem I started finding in my environment.  I've only seen it on 3-Win7 machines but now I'm worried there are more machines out there broken but without an easy way to identify the machines.

COM+ has a permission issue if you use USMT 4.0 to migrate from 32-bit to 64-bit machines (https://support.microsoft.com/en-us/kb/2481190).  We only noticed the problem because Outlook Add-ins started failing to load and seemed to crash.  Specifically, a user was not able to preview a PowerPoint presentation within the Outlook preview window unless they were running Outlook elevated as an administrator but a newer machine worked just fine running Outlook as a standard user.

We then noticed that Component Services showed "My Computer" with a red arrow and a message saying "You do not have permission to perform the requested action..."

We probably used USMT 4.0 for a year with who knows how many migrations from 32-bit to 64-bit.  I'm going to see if I can find a way to test a machine for this error and maybe create an SCCM compliance test.

Thursday, March 12, 2015

Determining Admin Rights

I'm now a global product lead for Avecto Privilege Guard (now called DefendPoint).  Privilege Guard is used to create an application execution policy to elevate processes, passively execute process, or block processes.  We are using the product to tighten security on desktops removing users from the administrator group but giving users the ability to continue to function.

One of our biggest issues was determining whether or not a user was running as an administrator.  On the outset, this seems like an easy question to answer.  However, with User Access Control (UAC) in place, it becomes more difficult. An administrator on Windows computers with UAC enabled will execute all scripts and applications with standard user rights until something triggers UAC and asks for elevation.  In a script, it is fairly easy to check for admin rights prior to execution by calling something that only an administrator would have access.  For example, you can call "net session" to determine if your shell is currently elevated.

To find out how many users are running with access to elevated rights is more difficult.  You could try to export the list of groups and users that are a member of the local administrator group but it quickly becomes difficult when you start dealing with nested domain groups.  Your account could receive administrator group membership by being part of a domain local group that is part of a domain global group that is part of another domain global group which is the actual group that has been added to the local administrators group.  Given enough time and recursive power, you can write a script that can give you the truth but it will not scale very well.

We found a great little utility on Windows 7 and greater computers that can quickly express every group membership your account is part connected.
whoami.exe /groups | find /i "administrators"
It "whoami.exe /groups" exports every group membership that is captured in the authentication token.  It is clean, it is locally cached, and doesn't require any processing or complicated recursive language that can easily have a loop error.

After testing and feeling comfortable we created an SCCM compliance powershell script that was very simple:
whoami.exe /groups | Select-String "s-1-5-32-544" -quiet
It seemed to work great with a slight problem, 10% of our inventory would generate an error instead of an answer of true or null for false.  I would connect to these computers and try to figure out what was happening and would find whoami.exe either fail or hang.  What I found in the event log was usually a couple group policy errors or warnings prior to a script failure.  It looked like we were picking up issues where group policy was temporarily failing because of one issue or another.  Either the secure channel was compromised waiting for a reset or something else was a problem.  I could easily fix the issue by rebooting the machine.

Now I think we found an even better solution.  I believe that whoami.exe is doing some type of secure channel look up on the domain which is not what we want to capture.  We want to capture whether a user is part of the builtin\administrator group.
[System.Security.Principal.WindowsIdentity]::GetCurrent().UserClaims.Value | Select-String "S-1-5-32-544" -quiet
We were able to test a second SCCM compliance baseline with both CI's attached to get side-by-side results.  One CI was set to query whoami and the other just used the powershell script shown above.  In our test group we went from 13% failures to 0.71% failures.

I did not find very good documentation on the difference of Claims or UserClaims so I picked the one that seemed most likely and started testing.  If someone could point me to more information, I would appreciate it.

We were looking to follow this great SCCM process from Wells Fargo to capture every computer's local group membership.  I really like their elegant method of capturing the data in WMI through a compliance script and then using SCCM to collect the information at a different time.  I'll have to implement this for another project.

http://mnscug.org/blogs/sherry-kissinger/244-all-members-of-all-local-groups-configmgr-2012