Published by OpenTask, Republic of Ireland
Copyright © 2009 by OpenTask
DLL Injection DebugWare: An Anatomy of Needle © Kapildev Ramlal
Modeling CPU Spikes © Konstantin Chebotarev and Dmitry Vostokov
All other articles in this issue © Dmitry Vostokov
All rights reserved.
Debugged! MZ/PE: MagaZine for/from Practicing Engineers
Volume 1, Issue 2, June 2009
Welcome to the second issue of Debugged! MZ/PE magazine! After publishing the first issue some engineers asked why we chose MZ/PE abbreviation. This is because ‘MZ’ is the first 2 bytes of every MS-DOS executable and ‘PE’ is the signature of the Windows program header that follows. Recall that we can launch every Windows program in MS-DOS and the output will be “This program cannot be run in DOS mode”. If we load any program into WinDbg as a crash dump file we can inspect its file contents or we can use any hex editor for this purpose:
Fault Injection: A New Perspective
Dmitry Vostokov, 15th
June
2009
http://www.dumpanalysis.org
Why should we be interested in constructing (or modeling) software defects? Besides fun and developing debugging exercises there is a scientific side to it: we can construct software bugs to check hypotheses when troubleshooting and debugging complex and hard to reproduce issues. The closest term is called “fault injection”. When working on this issue I accidentally discovered this now out-of print book published more than 10 years ago:
Software Fault Injection:
Inoculating Programs Against Errors
ISBN: 978-0471183815
Software defect construction is a more general term than fault injection. The latter is used for testing but we want to simulate bugs and abnormal system conditions to study debugging and memory dump analysis techniques or to build reproduction environments. In this issue, Kapildev Ramlal, a senior Citrix escalation engineer, recollects his experience with DLL injection tools he wrote to effectively and efficiently debug numerous problems in terminal services environments. Then Konstantin Chebotarev, a Citrix development analysis engineer, shows us how to construct a CPU spike and finally I discuss how to model a runaway exception handling.
There are many tools that inject faults, for example NotMyFault sysinternals tool[1]. It is very useful for studying crash dump analysis of kernel memory dumps and minidumps. Another use for these tools is to check certain debugging configurations like a postmortem debugger or WER settings are set up correctly. Typical example for user space is TestDefaultDebugger package[2]. Generalizing above, we can say that injecting bugs helps to test troubleshooting methods and tools. This is what I call metatroubleshooting, troubleshooting and debugging troubleshooting and debugging tools (DebugWare).
Creating software defects deliberately is entertaining and has great educational value for understanding internals and doing deep down debugging. I’m currently working on the following title:
Software Defect Construction:
Simulation and Modeling of Software Bugs ISBN: 978-1906717759
The book will cover all spectrum of software faults including kernel space bugs, implementation and design defects, provide debugging suggestions and compilations of resources.
DLL Injection DebugWare: An Anatomy of Needle
Kapildev Ramlal, 1st
June
2009
http://www.mindarray.org
When debugging software sometimes we’re faced with challenging and complex problems, forcing us to deviate from the traditional path and create new and clever ways to catch the bug. Some cases require intense research on the software component being debugged, including the lower level API’s that it utilizes. In Windows debugging, we are referring to the Windows API, however this can even go as far as an application specific API in some cases (although in most cases application symbols are not available).
So in considering the architecture of an application that is being debugged, one key point here is that the application will at some level utilize the Windows API. In C/C++ programming, these API’s are typically exposed by function calls exported from DLL’s.
In Windows, each process file starts with a PE format header which has several sections used for various purposes[3].
One of these sections is Import Address Table (IAT). This table contains the list of DLL modules that the executable links against and calls functions from. It also contains the function name and relative address of the function within the imported module. Leveraging the knowledge of how this works, the imported function address can be hooked to point to a different function – one that can be used for debugging, such as printing out the function parameters and return code. It’s a great concept and one that has been used for years within several versions of Windows. It’s not the only way to implement hooking in Windows, as there are also other mechanisms such as the SetWindowsHookEx API.
So we have a problematic EXE, and need to figure out what it is doing and if the function calls within it are working. To be more specific, we want to see if the Windows function calls are working. For example, we figure out that it is the EnumPrinterDrivers() function that is under suspicion, and we would like to dig deeper into its parameters and return code. Timing is critical for this problem, so any interception from a debugger such as WinDbg or NTSD causes the problem to subside…
We decide to hook the EnumPrinterDrivers() call to print out the debug information… How would we get our code loaded into the problem EXE to be hooked?
The most common method would be to create a DLL which can be loaded by, or injected into the process being debugged. It’s really not as difficult as one would imagine because a DLL is very similar to an EXE and we just need to supply it with the appropriate entry point. Here is a sample skeleton that can be used to flesh out a real working DLL:
#include <Windows.h>
HWND g_hMod = 0;
#ifdef __cplusplus // If used by C++ code,
extern "C" { // we need to export the C interface
#endif
// Entry point for DLL (main function for DLL)
BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved)
{
//Check load reason
switch(ul_reason_for_call)
{
case DLL_PROCESS_ATTACH:
{
// Eliminate un-necessary overhead...
DisableThreadLibraryCalls(hModule);
//Save the handle to the DLL for later use
g_hMod = (HWND)hModule;
break;
}
case DLL_PROCESS_DETACH:
{
break;
}
default:
break;
}
return TRUE;
}
#ifdef __cplusplus
}
#endif
Very simple. So we see that there’s a case statement when the DLL gets attached to the process and another for when it gets detached. That gives us a good opportunity to plant our hook, and another opportunity to unhook and cleanup our tracks. We can include our hook code in separate files in the DLL project, and then write our own hook functions that match the exact signature as the function being hooked. Keep in mind that Microsoft typedef’s a lot of their API’s, so don’t forget to check if an API really expands to an ANSI or UNICODE version ending with “A” or “W” (OutputDebugStringA vs. OutputDebug StringW). Our hook function signature needs to match exactly. Detailed instructions on creating hook functions are documented in Windows via C/C++ book by Jeffrey Richter and Christophe Nasarre (ISBN 978-0735624245).
So having a DLL, how do we invade the remote process being debugged and get our DebugWare DLL loaded? Well we have a few methods we can choose, and each comes with its own caveats.
The easiest method to inject a DLL is by using the registry. Simply edit the “AppInit_DLLs” value under HKLM\Software\Microsoft\WindowsNT\CurrentVersion\Windows and add the hook DLL to the list. The path to the DLL must not contain spaces and an accompanying value, LoadAppInit_DLLs of REG_DWORD type must also be created (under the same key) with a value set to 1. As long as the process being debugged is linked against user32.dll (like all GUI based apps) then this method should work. One major thing to consider is that all GUI apps will load the hook DLL.
Another method is by using SetWindowsHookEx method. This method uses the Windows API (generally for hooking window procedure message calls) called SetWindowsHookEx, by specifying a DLL handle as the 3rd parameter which causes the DLL to get loaded indirectly by the processes being hooked. This method is useful for GUI DebugWare applications.
We can also trick the process by thinking that it’s loading one of its own DLL’s (the Trojan DLL approach) or we can spawn the process being debugged as a child process and leverage the CreateProcess API to perform injection. These methods are discussed elsewhere (such as in Windows via C/C++ book), so we won’t go into more details here.
My favorite method which we discuss here in details is a method of DLL injection using remote threads. Yes it requires some understanding of several architectural concepts, but it is quite powerful. The beauty of this technique lies within the CreateRemoteThread API. We simply spawn a new thread in the process being debugged, and have that new thread to load the DLL being injected. In fact, we’ve once come across a problem so unique, that it has actually pushed me far enough to create my own DLL injection utility called Needle.
We’ll go ahead and share some of the core ingredients that make Needle pierce.
Get Debug Privileges
Needle doesn’t require debug privileges to inject all processes, but for some system processes it might be required. Here’s some code that shows how to do this:
bool NEEDLE::TweakPrivilege(TCHAR * pszPrivilege, BOOL bEnabled)
{
TOKEN_PRIVILEGES tp = {0};
LUID luid;
//Handle of token to adjust
HANDLE hToken = 0;
if (!LookupPrivilegeValue(NULL, pszPrivilege,
&luid))
{
this->DebugTrace(TEXT("NEEDLE: Failed to lookup privelege value. Error code: "), GetLastError());
return false;
}
//Retrieves token from current process
if(!OpenProcessToken(GetCurrentProcess(),
TOKEN_ADJUST_PRIVILEGES, &hToken))
{
this->DebugTrace(TEXT("NEEDLE: Failed to retrieve token handle. Error code: "), GetLastError());
return false;
}
tp.PrivilegeCount = 1;
tp.Privileges[0].Luid = luid;
if (bEnabled)
{
tp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
}
else
{
tp.Privileges[0].Attributes = 0;
// Enable the privilege or disable all privileges.
if ( !AdjustTokenPrivileges(
hToken,
FALSE,
&tp,
sizeof(TOKEN_PRIVILEGES),
(PTOKEN_PRIVILEGES) NULL,
(PDWORD) NULL) )
{
this->DebugTrace(TEXT("NEEDLE: Error
adjusting token privelege: "), GetLastError());
CloseHandle(hToken);
return false;
}
if (GetLastError() ==
ERROR_NOT_ALL_ASSIGNED)
{
this->DebugTrace(TEXT("NEEDLE: The token
does not have the requested privelege"), 0);
CloseHandle(hToken);
return false;
}
CloseHandle(hToken);
return true;
}
Grab the Process ID of the process being debugged
The Windows Terminal Services API exposes a nice function that allows easy process enumeration, where the results are stored in a neat array which can be easily looped to walk the results. It’s called WTSEnumerateProcesses(). After the PID is obtained, we can then proceed to get the proc address of the LoadLibrary function, and then open a handle to the remote process.
BOOL NEEDLE::GetPID(TCHAR * pProcName)
{
//First enumerate all processes on the system
if(!WTSEnumerateProcesses(
WTS_CURRENT_SERVER_HANDLE, 0, 1,
&this->m_pWtsPInfo, &this->m_dwCount))
{
this->DebugTrace(TEXT("NEEDLE: Enumerating
processes failed! Error code: "),
GetLastError());
return FALSE;
}
//Now parse processes and find Application.exe
for(unsigned int c=0; c<this->m_dwCount; c++)
{
//Find Application.exe
if(!lstrcmpi(
this->m_pWtsPInfo[c].pProcessName,
pProcName))
{
//Use PID to open handle to process
return this->m_pWtsPInfo[c].ProcessId;
}
}
this->DebugTrace(TEXT("NEEDLE: Failed to match
process name with enumerated processes!"),
GetLastError());
//If we arrived here, then we failed
return FALSE;
}
Get the LoadLibrary method
We need to extract the address of the LoadLibrary function that is specific to the LoadLibrary version we need to use. For example, LoadLibraryW vs.
LoadLibraryA.
Example:
this->m_hKern32 =
GetModuleHandle(TEXT("kernel32.dll"));
if(!this->m_hKern32) return false;
#ifdef _UNICODE
this->m_pLoadLib = (PVOID) GetProcAddress(
this->m_hKern32, "LoadLibraryW");
#else
this->m_pLoadLib = (PVOID) GetProcAddress(
this->m_hKern32, "LoadLibraryA");
#endif
if(!this->m_pLoadLib) return false;
Open a handle to the remote process
Since we’ve obtained the PID, we can open a handle to the remote process using OpenProcess. Instead of using PROCESS_ALL_ACCESS we can specify the granular flags such as
PROCESS_CREATE_THREAD| PROCESS_VM_OPERATION| PROCESS_VM_WRITE.
this->m_hProcess =
OpenProcess(PROCESS_CREATE_THREAD|
PROCESS_VM_OPERATION|PROCESS_VM_WRITE,
0, iPID);
The serum is getting loaded (Injection time)
After obtaining the remote process’ handle, we then proceed to allocate memory in the remote process to store the path to the injected DLL, and then write that path into the processes address space before creating the remote thread.
Example:
//First we need to allocate memory in the remote
process for our string
this->m_pBaseAddr =
VirtualAllocEx(this->m_hProcess,
0, this->m_dwModLen,
MEM_RESERVE|MEM_COMMIT,
PAGE_READWRITE);
if(!this->m_pBaseAddr)
{
this->DebugTrace(TEXT("NEEDLE: Failed to
allocate memory in remote process! Error code: "), GetLastError());
return 0;
}
if(!WriteProcessMemory(this->m_hProcess,
(LPVOID)this->m_pBaseAddr, (LPCVOID)pszModPath, this->m_dwModLen, 0))
{
this->DebugTrace(TEXT("NEEDLE:
WriteProcessMemory Failed!
Error code: "), GetLastError());
return 0;
}
// Now let us inject Application.exe
// with our module
if(!CreateRemoteThread(this->m_hProcess, 0, 0,
(LPTHREAD_START_ROUTINE)this->m_pLoadLib,
this->m_pBaseAddr, 0, 0))
{
this->DebugTrace(TEXT("NEEDLE: Failed to
create remote thread! Error code: "),
GetLastError());
return 0;
}
else
{
this->DebugTrace(TEXT("NEEDLE: Successfully
created remote thread!"),
GetLastError());
return true;
}
Bringing it all together
We’ve seen firsthand what makes Needle work. Now when cornered with a complex problem, we can consider writing a hook DLL to get more insights, and using DLL injection to get the hook DLL loaded into the remote process. We have seen how writing a DLL is easy, and implementing hooks within it really depends on our creativity and the situation at hand.
Developing a DLL injection utility like Needle is simple using the points outlined in this article, which involves using the following Windows API’s (OpenProcess,
VirtualAllocEx, WriteProcessMemory,
CreateRemoteThread).
Some gotcha’s that we encountered while developing Needle
It is helpful to call CreateProcess function and create the process being debugged suspended, giving a perfect opportunity to inject the debug DLL. Be sure to cleanup any memory allocated by the API’s used to inject the remote process.
It is tricky to inject Windows services. We believe this depends on the point in time when the injection is attempted, since the service process is managed by the Service Control Manager.
CreateRemoteThread fails when attempting to run against processes which exist in a separate session. There is a way around this using an undocumented Microsoft API, but we won’t go into the details of that here.
Modeling CPU Spikes
Konstantin Chebotarev and
Dmitry Vostokov, 25th
June
2009
http://support.citrix.com
Once we were discussing various ways to save memory dumps during high CPU consumption and both of us independently had an idea about writing such a small tool which allows to model different types of CPU load with different number of threads and different priorities. Because this tool can help people to understand CPU spike issues and play with different scenarios we decided to write an article for this magazine issue.
Cpuload tool was created usign MFC framework. Almost all our tools are created with the help of MFC instead of raw Win32 API to save developing time. First, we briefly discuss the code we added to MFC skeleton. Most of error handling is omitted for clarity there. Then we show the tool in action, and finally, show it internals from the debugger point of view (WinDbg).
The tool dialog resource is very simple. It consists from a slider to dynamically adjust CPU load and a static field to show CPU load percentage as a number:
The dialog initialization code sets realtime priority level for the current thread to boost GUI response. Because it is relative to the thread priority class (normal by default) this doesn’t freeze the system even on a uniprocessor machine. We need a good timer to model CPU balance so we are goimg to use the multimedia timer and set its resolution to 1 millisecond. Then we query the number of processors and call a function to create a CPU spiking thread for each processor.
BOOL CcpuloadDlg::OnInitDialog()
{
CDialog::OnInitDialog();
// …
SetThreadPriority(GetCurrentThread(),
THREAD_PRIORITY_TIME_CRITICAL);
MMRESULT res=timeBeginPeriod(1);
SYSTEM_INFO SystemInfo;
GetSystemInfo(&SystemInfo);
CPU_Num = SystemInfo.dwNumberOfProcessors;
for(int i=0;i<CPU_Num;++i)
{
CreateCPUSpiking(i);
}
return TRUE;
}
The following MFC code handles slider events and should be self-explanatory for MFC developers. If you don’t use MFC then after any slider move cpuval variable (int) is set to the value between 0 and 100:
CcpuloadDlg::CcpuloadDlg(CWnd* pParent /*=NULL*/)
: CDialog(CcpuloadDlg::IDD, pParent)
, m_CPU_val(0)
{
m_hIcon = AfxGetApp()->LoadIcon(IDR_MAINFRAME);
}
void CcpuloadDlg::DoDataExchange(CDataExchange* pDX)
{
CDialog::DoDataExchange(pDX);
DDX_Control(pDX, IDC_CPULOAD_SLIDER, mSlider);
DDX_Text(pDX, IDC_CPU_PERCENT, m_CPU_val);
}
BEGIN_MESSAGE_MAP(CcpuloadDlg, CDialog)
ON_WM_SYSCOMMAND()
ON_WM_PAINT()
ON_WM_QUERYDRAGICON()
//}}AFX_MSG_MAP
ON_WM_HSCROLL()
END_MESSAGE_MAP()
void CcpuloadDlg::OnHScroll(UINT nSBCode, UINT nPos,
CScrollBar* pScrollBar)
{
if(nSBCode==TB_THUMBTRACK)
{
m_CPU_val=cpuval=nPos;
UpdateData(FALSE);
CDialog::OnHScroll(nSBCode, nPos,
pScrollBar);
}
}
The function that creates spiking threads also sets their thread affinity (assigns a unique processor for each thread):
void CreateCPUSpiking(int CPU_idx)
{
DWORD ThreadID;
HANDLE hThread;
hThread=::CreateThread(NULL,0,SpikingThread,
NULL,0,&ThreadID);
DWORD_PTR mask=1<<CPU_idx;
SetThreadAffinityMask(hThread,mask);
}
Thread routine that runs thread code sleeps appropriate amount of milliseconds necessary to model CPU load:
DWORD WINAPI SpikingThread(LPVOID lpParameter)
{
double a=1.1;
DWORD time1=timeGetTime();
DWORD time2;
DWORD dt=100; // time to loop in milliseconds
DWORD ts; // time to sleep in milliseconds
do
{
a=a+0.1; // some dummy code to make CPU busy
a=a-0.1; //
time2=timeGetTime();
if((time2-time1) > dt) // if time to loop
// finished calculate new times
{
dt=cpuval*10; ts=1000-dt;
Sleep(ts);
time1=timeGetTime();
}
}
while(true);
return 0;
}
This code models CPU load almost exactly and its value corresponds to one that we see in Task Manager (all pictures were taken from 2 processor PC running x64 Windows Server 2008):
However 100% is not attainable when Task Manager is running because it eats approximately 1% of CPU time:
Other processes can eat CPU too, for example, this screenshot was taken while Windows Media Player was playing a CD (wmplayer.exe) with graphical animation turned on that ate csrss.exe time too:
In order to see what’s inside cpuload process we set CPU load to 100% and noninvasively attached WinDbg to it:
0:000> ~*k
. 0 Id: 710.ec4 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0017fba0 778a67f0 USER32!NtUserGetMessage+0x15
0017fbbc 00510211 USER32!GetMessageW+0x33
WARNING: Stack unwind information not available. Following frames may be wrong.
0017fbe8 0051173c cpuload+0x110211
0017fbf4 0051031f cpuload+0x11173c
0017fc00 00535ea5 cpuload+0x11031f
0017fc28 00515561 cpuload+0x135ea5
0017fc94 004f814d cpuload+0x115561
0017fe84 006a43b2 cpuload+0xf814d
0017fea8 006a42a8 cpuload+0x2a43b2
0017fec0 00626d0f cpuload+0x2a42a8
0017ff80 00626a4d cpuload+0x226d0f
0017ff88 775de4a5 cpuload+0x226a4d
0017ff94 77eecfed kernel32!BaseThreadInitThunk+0xe
0017ffd4 77eed1ff ntdll!__RtlUserThreadStart+0x23
0017ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
1 Id: 710.994 Suspend: 1 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
02e7ff88 775de4a5 WINMM!_alldiv+0x4d
02e7ff94 77eecfed kernel32!BaseThreadInitThunk+0xe
02e7ffd4 77eed1ff ntdll!__RtlUserThreadStart+0x23
02e7ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
2 Id: 710.d20 Suspend: 1 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
02f7ff88 775de4a5 cpuload+0xf87cf
02f7ff94 77eecfed kernel32!BaseThreadInitThunk+0xe
02f7ffd4 77eed1ff ntdll!__RtlUserThreadStart+0x23
02f7ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
We deliberately not applied symbol files for cpuload to show real life CPU spike scenario when symbol files are not available for 3rd-party software vendor modules. We see that threads 1 and 2 consumed plenty of CPU time in user mode relative to their life span:
0:000> !runaway f
User Mode Time
Thread Time
1:994 0 days 0:05:06.089
2:d20 0 days 0:04:45.076
0:ec4 0 days 0:00:00.156
Kernel Mode Time
Thread Time
0:ec4 0 days 0:00:00.452
2:d20 0 days 0:00:00.062
1:994 0 days 0:00:00.015
Elapsed Time
Thread Time
0:ec4 0 days 0:18:56.449
2:d20 0 days 0:18:56.143
1:994 0 days 0:18:56.143
So we have all information to diagnose Spiking Thread memory dump analysis pattern pointing to cpuload module[4]. Notice also that even noninvasive attach suspended threads and real CPU load becomes 0%. To resume the program we use ~m WinDbg command:
0:000> ~*m
Then we can randomly catch our thread stack traces:
0:000> ~*k
. 0 Id: 710.ec4 Suspend: 0 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0017fba0 778a67f0 USER32!NtUserGetMessage+0x15
0017fbbc 00510211 USER32!GetMessageW+0x33
WARNING: Stack unwind information not available. Following frames may be wrong.
0017fbe8 0051173c cpuload+0x110211
0017fbf4 0051031f cpuload+0x11173c
0017fc00 00535ea5 cpuload+0x11031f
0017fc28 00515561 cpuload+0x135ea5
0017fc94 004f814d cpuload+0x115561
0017fe84 006a43b2 cpuload+0xf814d
0017fea8 006a42a8 cpuload+0x2a43b2
0017fec0 00626d0f cpuload+0x2a42a8
0017ff80 00626a4d cpuload+0x226d0f
0017ff88 775de4a5 cpuload+0x226a4d
0017ff94 77eecfed kernel32!BaseThreadInitThunk+0xe
0017ffd4 77eed1ff ntdll!__RtlUserThreadStart+0x23
0017ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
1 Id: 710.994 Suspend: 0 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
02e7ff88 775de4a5 cpuload+0xf87cf
02e7ff94 77eecfed kernel32!BaseThreadInitThunk+0xe
02e7ffd4 77eed1ff ntdll!__RtlUserThreadStart+0x23
02e7ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
2 Id: 710.d20 Suspend: 0 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
02f7ff88 775de4a5 cpuload+0xf87cf
02f7ff94 77eecfed kernel32!BaseThreadInitThunk+0xe
02f7ffd4 77eed1ff ntdll!__RtlUserThreadStart+0x23
02f7ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
If we apply symbols we get nicer stack traces:
0:000> ~*kL
. 0 Id: 710.ec4 Suspend: 0 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0017fba0 778a67f0 USER32!NtUserGetMessage+0x15
0017fbbc 00510211 USER32!GetMessageW+0x33
0017fbe8 0051173c cpuload!AfxInternalPumpMessage+0x21
0017fbf4 0051031f cpuload!CWinThread::PumpMessage+0xc
0017fc00 00535ea5 cpuload!AfxPumpMessage+0x1f
0017fc28 00515561 cpuload!CWnd::RunModalLoop+0x1d5
0017fc94 004f814d cpuload!CDialog::DoModal+0x1e1
0017fe84 006a43b2 cpuload!CcpuloadApp::InitInstance+0xad
0017fea8 006a42a8 cpuload!AfxWinMain+0x82
0017fec0 00626d0f cpuload!wWinMain+0x18
0017ff80 00626a4d cpuload!__tmainCRTStartup+0x2af
0017ff88 775de4a5 cpuload!wWinMainCRTStartup+0xd
0017ff94 77eecfed kernel32!BaseThreadInitThunk+0xe
0017ffd4 77eed1ff ntdll!__RtlUserThreadStart+0x23
0017ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
1 Id: 710.994 Suspend: 0 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
02e7fe74 004f87e7 WINMM!timeGetTime+0x14
02e7ff88 775de4a5 cpuload!SpikingThread+0x67
02e7ff94 77eecfed kernel32!BaseThreadInitThunk+0xe
02e7ffd4 77eed1ff ntdll!__RtlUserThreadStart+0x23
02e7ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
2 Id: 710.d20 Suspend: 0 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr
02f7ff88 775de4a5 cpuload!SpikingThread+0x58
02f7ff94 77eecfed kernel32!BaseThreadInitThunk+0xe
02f7ffd4 77eed1ff ntdll!__RtlUserThreadStart+0x23
02f7ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
We have highlighted in red the currently executing module and function.
The cpuload application and its symbols can be downloaded from:
http://www.dumpanalysis.org/Debugged/Jun09/ download/cpuload.zip
Modeling Exception Handling
Dmitry Vostokov, 15th
June
2009
http://www.dumpanalysis.org
Once we got a process memory dump with a stack overflow pattern[5]. There were hundreds of repeated frames involving exception dispatch:
(d70.111c): Stack overflow - code c00000fd (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00113354 ebx=00113068 ecx=0011307c edx=7c9485ec esi=001133c8 edi=00000000
eip=7c95153e esp=00112ff4 ebp=00113050 iopl=0 nv up ei pl nz ac po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00210212
ntdll!RtlDispatchException+0x8:
7c95153e 56 push esi
0:000> k 1000
ChildEBP RetAddr
00113050 7c94855e ntdll!RtlDispatchException+0x8
00113050 7c978002 ntdll!KiUserExceptionDispatcher+0xe
001133b0 7c94855e ntdll!RtlDispatchException+0xf7
001133b0 7c978002 ntdll!KiUserExceptionDispatcher+0xe
00113710 7c94855e ntdll!RtlDispatchException+0xf7
00113710 7c978002 ntdll!KiUserExceptionDispatcher+0xe
00113a70 7c94855e ntdll!RtlDispatchException+0xf7
00113a70 7c978002 ntdll!KiUserExceptionDispatcher+0xe
00113dd0 7c94855e ntdll!RtlDispatchException+0xf7
00113dd0 7c978002 ntdll!KiUserExceptionDispatcher+0xe
00114130 7c94855e ntdll!RtlDispatchException+0xf7
00114130 7c978002 ntdll!KiUserExceptionDispatcher+0xe
00114490 7c94855e ntdll!RtlDispatchException+0xf7
[...]
0020f830 7c978002 ntdll!KiUserExceptionDispatcher+0xe
0020fb90 7c94863c ntdll!RtlDispatchException+0xf7
0020fe70 7c80bee7 ntdll!RtlRaiseException+0x3d
0020fed0 00058dc9 kernel32!RaiseException+0x53
WARNING: Stack unwind information not available. Following frames may be wrong.
0020ff34 000a7d5a Application+0x48dc9
0020ff54 000a7bf2 Application+0x97d5a
0020ff78 000a7de2 Application+0x97bf2
0020ffc0 7c82f23b Application+0x97de2
0020fff0 00000000 kernel32!BaseProcessStart+0x23
If we look at raw stack we could see repeated hidden exception pattern[6]:
0:000> dds esp
00112ff4 00000000
00112ff8 00000000
00112ffc 00000000
00113000 00000000
00113004 00000000
00113008 00000000
0011300c 00000000
00113010 00000000
00113014 00000000
00113018 00000000
0011301c 00000000
00113020 00000000
00113024 00000000
00113028 00000000
0011302c 00000000
00113030 00000000
00113034 00000000
00113038 00000000
0011303c 00000000
00113040 00000000
00113044 00000000
00113048 00000000
0011304c 00000000
00113050 001133b0
00113054 7c94855e ntdll!KiUserExceptionDispatcher+0xe
00113058 00113068
0011305c 0011307c ; .cxr
00113060 00113068 ; .exr
00113064 0011307c
00113068 c0000025
0011306c 00000001
00113070 001133c8
0:000> .exr 00113068
ExceptionAddress: 7c978002 (ntdll!RtlDispatchException+0x000000f7)
ExceptionCode: c0000025
ExceptionFlags: 00000001
NumberParameters: 0
0:000> !error c0000025
Error code: (NTSTATUS) 0xc0000025 (3221225509) - {EXCEPTION} Cannot Continue Windows cannot continue from this exception.
Looking closer at RaiseException call we noticed that its second parameter is 1:
0:000> kv 100
ChildEBP RetAddr Args to Child
[...]
0020fed0 00058dc9 0eedfade 00000001 00000007 kernel32!RaiseException+0x53
[...]
This function has the following prototype[7]:
void RaiseException(
DWORD dwExceptionCode,
DWORD dwExceptionFlags,
DWORD nNumberOfArguments,
const DWORD* lpArguments
);
What caught our attention was the fact that dwExceptionFlags paramater can have the value EXCEPTION_NONCONTINUABLE and if we continue to execute the code afterwards another exception is raised:
EXCEPTION_NONCONTINUABLE_EXCEPTION
Therefore, to test our hypothesis that the code of Application module attempted to continue code excution after raising non-continuable exception we created a small C++ application:
#include "stdafx.h"
#include <excpt.h>
#include <windows.h>
int _tmain(int argc, _TCHAR* argv[])
{
__try
{
RaiseException (0xdeaddead,
EXCEPTION_NONCONTINUABLE, NULL, NULL);
}
__except (EXCEPTION_CONTINUE_EXECUTION)
{
}
return 0;
}
The application raises a non-continuable exception but SEH exception filter attempts to continue execution. If the second parameter is 0 then the program quietly exits as expected. However if this parameter is set to 1 (EXCEPTION_NONCONTINUABLE) then the program crashes:
We then attached WinDbg non-invasively to SEHTest.exe:
0:000> k
ChildEBP RetAddr
00081128 00000000 ntdll!RtlRaiseException+0xa
0:000> r
eax=00081140 ebx=0017ff34 ecx=00000000 edx=ffffffd8 esi=000811bc edi=00000000
eip=77052fb2 esp=00080e58 ebp=00081128 iopl=0 nv up ei pl nz na po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
ntdll!RtlRaiseException+0xa:
77052fb2 54 push esp
Unfortunately WinDbg was unable to reconstruct stack trace so we did that manually[8]:
0:000> dds ebp
00081128 000811a4
0008112c 77096fb7 ntdll!RtlDispatchException+0x18a
00081130 00081140
00081134 00000000
00081138 000811bc
0008113c 00081560
00081140 c0000025
00081144 00000001
00081148 000811bc
0008114c 00000000
00081150 00000000
00081154 00000000
00081158 00000000
0008115c 00000000
00081160 00000000
00081164 00000000
00081168 00000000
0008116c 00000000
00081170 00000000
00081174 00000000
00081178 00000000
0008117c 00000000
00081180 00000000
00081184 00000000
00081188 00000000
0008118c 00000000
00081190 00000000
00081194 00000000
00081198 00180000
0008119c 00081000
000811a0 00000000
000811a4 00081548
0:000> k L=00081128 00081128 77052fb2 1000
ChildEBP RetAddr
00081128 77096fb7 ntdll!RtlRaiseException+0xa
000811a4 77052eff ntdll!RtlDispatchException+0x18a
000811a4 77096fb7 ntdll!KiUserExceptionDispatcher+0xf
00081548 77052eff ntdll!RtlDispatchException+0x18a
00081548 77096fb7 ntdll!KiUserExceptionDispatcher+0xf
000818ec 77052eff ntdll!RtlDispatchException+0x18a
000818ec 77096fb7 ntdll!KiUserExceptionDispatcher+0xf
00081c90 77052eff ntdll!RtlDispatchException+0x18a
00081c90 77096fb7 ntdll!KiUserExceptionDispatcher+0xf
00082034 77052eff ntdll!RtlDispatchException+0x18a
00082034 77096fb7 ntdll!KiUserExceptionDispatcher+0xf
000823d8 77052eff ntdll!RtlDispatchException+0x18a
000823d8 77096fb7 ntdll!KiUserExceptionDispatcher+0xf
0008277c 77052eff ntdll!RtlDispatchException+0x18a
0008277c 77096fb7 ntdll!KiUserExceptionDispatcher+0xf
00082b20 77052eff ntdll!RtlDispatchException+0x18a
00082b20 77096fb7 ntdll!KiUserExceptionDispatcher+0xf
00082ec4 77052eff ntdll!RtlDispatchException+0x18a
00082ec4 77096fb7 ntdll!KiUserExceptionDispatcher+0xf
00083268 77052eff ntdll!RtlDispatchException+0x18a
[...]
0017f7dc 77096fb7 ntdll!KiUserExceptionDispatcher+0xf
0017fb80 77052eff ntdll!RtlDispatchException+0x18a
0017fb80 75b7f328 ntdll!KiUserExceptionDispatcher+0xf
0017ff04 0040104b kernel32!RaiseException+0x58
0017ff44 004011d8 SEHTest!wmain+0x4b
0017ff88 75bde4a5 SEHTest!__tmainCRTStartup+0x10f
0017ff94 770acfed kernel32!BaseThreadInitThunk+0xe
0017ffd4 770ad1ff ntdll!__RtlUserThreadStart+0x23
0017ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
0:000> kv L=00081128 00081128 77052fb2 1000
ChildEBP RetAddr Args to Child
[...]
0017ff04 0040104b deaddead 00000001 00000000 kernel32!RaiseException+0x58 (FPO: [4,20,0])
[...]
We can also see the same hidden exception on raw stack: c0000025. Therefore we conclude that we sucessfully modeled the problem.
The SEHTest application and its symbols can be downloaded from:
http://www.dumpanalysis.org/Debugged/Jun09/ download/SEHTest.zip
Memory Dump Analysis Certification Voucher can be found in the printed version.
Memory Dump Analysis Certification Resources (Part 1)
|
Windows Debugging: Practical Foundations ISBN: 978-1-906717-10-0 (Paperback) ISBN: 978-1-906717-67-4 (Hardcover) |
Written by the founder of DumpAnalysis.org this book is not about bugs or debugging techniques but about background knowledge everyone needs to start experimenting with WinDbg, learn from practical experience and read other advanced debugging books. Solid understanding of fundamentals like pointers is needed to analyze stack traces beyond !analyze -v and lmv WinDbg commands. This is the book to help technical support and escalation engineers and Windows software testers without the knowledge of assembly language to master necessary prerequisites to understand and start debugging and crash dump analysis on Windows platforms. It doesn't require any specific knowledge, fills the gap and lowers the learning curve. The book is also useful for software engineers coming from managed code or Java background, engineers coming from non-Wintel environments, Windows C/C++ software engineers without assembly language background, security researchers and beginners learning Windows software disassembling and reverse engineering techniques. This book can also be used as Intel assembly language and Windows debugging supplement for relevant undergraduate level courses. |
|
|
Memory Dump Analysis Anthology, Volume 1 ISBN: 978-0-9558328-0-2 (Paperback) ISBN: 978-0-9558328-1-9 (Hardcover) This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in 2006 - 2007 for software engineers developing and maintaining products on Windows platforms, technical support and escalation engineers dealing with complex software issues and general Windows users. |
|
|
Memory Dump Analysis Anthology, Volume 2 ISBN: 978-0-9558328-7-1 (Paperback) ISBN: 978-1-906717-22-3 (Hardcover) This is a revised, edited, cross-referenced and thematically organized volume of selected DumpAnalysis.org blog posts about crash dump analysis and debugging written in January - September 2008 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software on Windows platforms and technical support and escalation engineers dealing with complex software issues. The second volume features: - 45 new crash dump analysis patterns - Pattern interaction and case studies - Updated checklist - Fully cross-referenced with Volume 1 - New appendixes |
