Recently, KoreLogic examined a number of malware downloaders that use API callback functions to redirect the flow of execution and make malware analysis more difficult. While this is not a new technique our research did not find many public resources discussing this topic. The purpose of this blog post is to describe KoreLogic's analysis on what callback functions are, how malware uses them, and how this technique can be detected and analyzed.
Callback functions are routines that are passed to Windows API functions as a parameter, and are called later by the API to perform some type of functionality, such as processing messages or handling events. Callbacks are used in a number of places within the Windows API, and a quick MSDN search finds hundreds of Windows functions that use them.
An example Windows API that uses a callback function is RegisterClass, which defines the characteristics of a GUI window within a program. When called, RegisterClass is passed a structure named WNDCLASS. Within the WNDCLASS structure is a pointer to a callback function WindowProc. This function is a user-defined procedure that processes messages sent to the window by the operating system. A benign example WindowProc can be found here.
Malware will often set up their own malicious callback functions and pass them to APIs. When the API function is called, the callback function is executed and the malicious code is run. Malicious callback functions have been seen to, amongst other things, unpack and/or decrypt code, redirect execution, deobfuscate strings, or perform anti-debugging techniques.
There are generally two ways to detect malicious callback functions in malware:
Trojan DownloaderThe first example is from a downloader that came in through malicious spam (MD5: 28d4a53893aba210fed7600cc759cbcd). After initializing memory, the malware calls DialogBoxParamW, which creates a dialog box. One of the parameters, lpDialogFunc, is a DialogProc callback function that is used to process messages sent to the dialog box. Messages let the dialog box know when something happens to it, such as as when a button is pressed or when the box is closed.
As soon as DialogBoxParamW executes, DialogProc begins receiving messages to process. This malicious callback function is fairly simplistic. It compares the uMsg parameter, which contains the type of message being processed, against a list of known values. If the message is WM_COMMAND (0x111), the malware will load an address into a global variable (renamed mal_code in IDA Pro). This address contains additional code that continues execution of the malware and is called after the return from DialogBoxParamW.
Due to the function of the callback function - to set up the address to jump to later - there is no need to put a breakpoint on the callback function entry point. The call to DialogBoxParamW can be stepped over and a breakpoint can be set on the call to mal_code set, or on the first instruction in mal_code itself. Analysis can then continue on the rest of the malware execution.
Upatre DownloaderThe second example is from an Upatre downloader (MD5: e49e7b907499c8b4e31447eaffd112b1) that was also distributed through malicious spam and utilizes callback functions. In this case, the malware uses CreateWindowEx to create an invisible window. (The window is invisible because ShowWindow is never called, and WS_VISIBLE is not given to the function as a window style.)
One of the parameters to CreateWindowEx is lpClassName, which is the name of a Window class used to define the window's features. For this sample, the class name is "cligas", and was previously registered in a call to RegisterClassEx. RegisterClassEx receives a pointer to a WNDCLASSEX structure, which among other things, contains a callback function that handles messages sent to the window. The callback function implemented by this malware is a bit more complicated than the previous one examined, so we'll look at it in pieces.
When the callback function is called, it examines the uMsg parameter which contains the message sent to the window, and will perform do different things depending on the message received.
Note that once the malware begins to process the WM_PARENTNOTIFY messages and calls the decoding function, it never returns. Therefore, in order to debug the malware, a breakpoint has to be set on the callback function. If the call to CreateWindowEx were stepped over, the malware would execute without intervention.
SummaryCallback functions normally used by the Windows API to handle program operations are used by malware to redirect the flow of execution and increase analysis difficulty. The two examples examined in this post only represent a small percentage of what and how malware uses malicious callback functions. However, by understanding the purpose of callback functions, and how to find and analyze them, analysts will be better prepared when they come across malware samples that use them.
Comments are closed for this story.
|Please contact us if you would like more information about our services, tools, or careers with us.|