Recently, KoreLogic examined a number of malware downloaders that use API callback functions to redirect the flow of execution and make malware analysis more difficult. While this is not a new technique our research did not find many public resources discussing this topic. The purpose of this blog post is to describe KoreLogic's analysis on what callback functions are, how malware uses them, and how this technique can be detected and analyzed.

Callback functions are routines that are passed to Windows API functions as a parameter, and are called later by the API to perform some type of functionality, such as processing messages or handling events. Callbacks are used in a number of places within the Windows API, and a quick MSDN search finds hundreds of Windows functions that use them.

An example Windows API that uses a callback function is RegisterClass, which defines the characteristics of a GUI window within a program. When called, RegisterClass is passed a structure named WNDCLASS. Within the WNDCLASS structure is a pointer to a callback function WindowProc. This function is a user-defined procedure that processes messages sent to the window by the operating system. A benign example WindowProc can be found here.

Malware will often set up their own malicious callback functions and pass them to APIs. When the API function is called, the callback function is executed and the malicious code is run. Malicious callback functions have been seen to, amongst other things, unpack and/or decrypt code, redirect execution, deobfuscate strings, or perform anti-debugging techniques.

There are generally two ways to detect malicious callback functions in malware:

• Watch for calls by the malware to known Windows APIs that utilize callbacks. These include RegisterClass, DialogBox, waveOutOpen, and others. Remember that the actual callback function address may be buried in an object or set up prior to the API call, as with WNDCLASS and RegisterClass.
• Look for functions being passed as parameters to API calls. Depending on the disassembler or debugger you are using and the amount of obfuscation the malware has gone through, these may appear as names or addresses.

Keep in mind that when debugging malware that use malicious callback functions, a breakpoint will have to be placed on the callback function itself, and not the Windows API that uses it (unless you want to waste time stepping into the API). Stepping over the API call may result in the malicious callback function being executed without the analyst being able to debug the malicious code.

Trojan Downloader

The first example is from a downloader that came in through malicious spam (MD5: 28d4a53893aba210fed7600cc759cbcd). After initializing memory, the malware calls DialogBoxParamW, which creates a dialog box. One of the parameters, lpDialogFunc, is a DialogProc callback function that is used to process messages sent to the dialog box. Messages let the dialog box know when something happens to it, such as as when a button is pressed or when the box is closed.

As soon as DialogBoxParamW executes, DialogProc begins receiving messages to process. This malicious callback function is fairly simplistic. It compares the uMsg parameter, which contains the type of message being processed, against a list of known values. If the message is WM_COMMAND (0x111), the malware will load an address into a global variable (renamed mal_code in IDA Pro). This address contains additional code that continues execution of the malware and is called after the return from DialogBoxParamW.


Due to the function of the callback function - to set up the address to jump to later - there is no need to put a breakpoint on the callback function entry point. The call to DialogBoxParamW can be stepped over and a breakpoint can be set on the call to mal_code set, or on the first instruction in mal_code itself. Analysis can then continue on the rest of the malware execution.

Upatre Downloader

The second example is from an Upatre downloader (MD5: e49e7b907499c8b4e31447eaffd112b1) that was also distributed through malicious spam and utilizes callback functions. In this case, the malware uses CreateWindowEx to create an invisible window. (The window is invisible because ShowWindow is never called, and WS_VISIBLE is not given to the function as a window style.)

One of the parameters to CreateWindowEx is lpClassName, which is the name of a Window class used to define the window's features. For this sample, the class name is "cligas", and was previously registered in a call to RegisterClassEx. RegisterClassEx receives a pointer to a WNDCLASSEX structure, which among other things, contains a callback function that handles messages sent to the window. The callback function implemented by this malware is a bit more complicated than the previous one examined, so we'll look at it in pieces.

When the callback function is called, it examines the uMsg parameter which contains the message sent to the window, and will perform do different things depending on the message received.

• WM_CREATE (0x1): The window receives this after it is created but is not yet visible. In this case, the callback function will create three hidden child windows. The first and third windows are edit controls, but the second window is a RichEdit control that contains lines of text used by the malware later. The creation of these windows also generate WM_PARENTNOTIFY messages to itself.
• WM_DESTROY (0x2): This is received when the window is being destroyed. When this is received, the function exits gracefully.
• WM_PARENTNOTIFY (0x210): The WM_PARENTNOTIFY message is received when a window's child windows are created or destroyed. In our sample, this is where the malware begins the majority of its work.
• Any other message the window receives calls DefWindowProc, the API that handles all default processing.

When the WM_PARENTNOTIFY message is received, the callback function checks to see if all three child windows have been created by decrementing a global value. Once all three have been created, the callback function calls another function, which begins the decoding process for the malware.


Note that once the malware begins to process the WM_PARENTNOTIFY messages and calls the decoding function, it never returns. Therefore, in order to debug the malware, a breakpoint has to be set on the callback function. If the call to CreateWindowEx were stepped over, the malware would execute without intervention.

Summary

Callback functions normally used by the Windows API to handle program operations are used by malware to redirect the flow of execution and increase analysis difficulty. The two examples examined in this post only represent a small percentage of what and how malware uses malicious callback functions. However, by understanding the purpose of callback functions, and how to find and analyze them, analysts will be better prepared when they come across malware samples that use them.