Monitoring ActiveX execution

Intro

My project this year involved hooking the load of ActiveX modules, and the next step is to actually monitor its execution and check for exploits. Let's take a step back first and have a recap of the ways in which a piece of software can be exploited.

Exploits

To exploit an application, there are two steps - injecting code, and altering the flow of execution. There are many ways of injecting code, and these are often hard to distinguish from legitimate data provisioning, so we will look at ways in which the flow of data can be altered. They are as follows:

1. Overwriting a segment of code. This is the hammer in our knife set, as not only is it almost impossible (without having first exploited the application anyway), it destroys the underlying data.
2. Overwriting a pointer which is used by a JMP or CALL command. This is slightly more precise, but requires somehow overwriting a specific part of memory, which is also quite difficult, but is much more feasible than #1
3. Overwriting the pointer to the previous stack frame. This is traditionally done through an overflow of a stack-based buffer, but can also be done by a heap overflow exploit. This used to be quite common, but there is a fourth option that has come into play recently
4. Overwriting the pointer to an exception handler. This requires that an exception be somehow triggered, and that the handler is in a module that doesn't have safeSEH activated. It's a bit of a stretch, but it is one of the ways that DEP is bypassed in modern exploits.

Now I'm by no means an expert here, but I think it's safe to rule out #1 due to code sections generally not having write privileges, and tentatively rule out #2, although a clever heap overflow exploit could take advantage of something like this (although the reason I can think of is in a non-DEP environment while trying to evade detection techniques such as the one described below, since in a non-DEP environment, #3 is much easier).

This leaves us with #3 and #4, each of which requires very different approaches. #3 should theoretically be disabled by DEP, however applications which have memory spaces that are both writeable and executable are still vulnerable, since an overflow would load the following addresses in the stack frame (memcpy, [padding DWORDs for each argument of the original function], address of w/x memory, address of w/x memory, address of shellcode, pointer to length) which would make the return instruction jump to the memcpy function, copy the shellcode to the w/x buffer, and then on return execute that memory. Given also that not every computer has DEP enabled (even windows 7 it is a per-application opt-in by default), and old hardware may not be DEP capable, this is still an area where protection is required.

An approach for protecting against attacks in category #4 requires that either modifications to the SEH chain are monitored, or that each SEH handler is tested when an exception is raised. The first option means breakpointing every SEH record in the chain (easy, but requires lots of breakpoints). The advantage of this is that the instruction pointer can be checked for every modification to the SEH chain, and suspicious modifications can be flagged (eg writes from within HeapFree and the like are probably a result of a heap overflow). The second option is a little easier but possibly not as effective, however the advantage lies in how non-invasive it is - the KiDispatchUserException function is patched to add extra checks, and these checks can include checking the page that the handler is executing from - if it's writeable (which is normally not the case for executable memory), then mark as suspicious.

Category #3 attacks are still important to block, even on modern systems. As mentioned earlier, even windows 7 uses the optin DEP rather than optout scheme for most apps, and apps that do take advantage of this can be circumvented by a return-to-libc attack, although with aslr this becomes more difficult. Since these (and most of category #2 except JMP[] ) all alter the stack, it is sufficient to breakpoint stack memory, and then either keep a separate collection of stack frames to compare after each RET instruction, or just check the permissions of the memory that the instruction pointer references.

ActiveX

Applying these to ActiveX modules is fairly straightforward. For performance reaspns it's best not to trace through the whole execution, but rather activate monitors when they're needed. Fortunately, the ActiveX interface makes this easy for us.

Each ActiveX object is instantiated by the CoGetClassObject() function, which returns an IClassFactory object, which exports a CreateInstance function. The first step then, is to hook CoGetClassObject() and then the CreateInstance function of every IClassFactory which is returned.

After this, we can detect the instantiation of every object, each of which contains a reference to its vtable, which is a structure containing references to functions and data objects. The function references point to functions in code memory, while the object references point to data memory, so these are easy to distinguish. A memory breakpoint across the whole structure allows us to hook every function that is called, and every object that is referenced.

Cunning plan

1. Hook the CoGetClassObject function
2. Hook the IClassFactory::CreateInstance function on every IClassFactory returned
3. Breakpoint the vtable of every IUnknown returned
4. Implement stack breakpoints whenever ActiveX functions are called

Once this is set up, every RET instruction will invoke a check of the stack to see if the destination memory is writeable and/or in data memory (covering #3), and at some stage this same function needs to be patched into KiDispatchUserException (covering #4) giving us fairly high confidence that any low level exploits will be detected.