Explicit buffer overflow detection

I've spent the bulk of this year working on my Honours project of explicitly detecting the loading of ActiveX modules in Internet Explorer, and the obvious follow-on is explicit detection of the the exploit within the module.

That would take quite a bit of work however, but I had another idea after seeing Microsoft's recent out-of-band patch release. One of the exploits was for the native font handler in Windows, and looking at the code I could see stack cookies within each function, but these appeared to be hardcoded (and thus predictable from an attacker's point of view). This got me thinking - why not take advantage of exception handlers and breakpoints to create a better overflow detection and add that to the security cookie stuff that's already present in modern compilers?

For those of you that spend more time finding bugs in your own code rather than searching through somebody else's work of art for something exploitable, here's a quick intro to buffer overflows.

void vulnerable(char* in){
    char test[10];
    strcpy(test, in);
    printf("argv[1] = %s\n", test);
}

What's wrong with this picture? If you've written something like this before, you've probably been presented with a "Segmentation Fault" error and an abrupt crash. The bug here is fairly straight-forward - the buffer is only 10 bytes long (which will hold 9 chars + terminating null char), so passing an argument with more than 9 characters will start overwriting other memory.... but what exactly?

When you call a function, the return address (next instruction after your function call) is saved onto the stack, and the stack pointer is decreased (stack memory grows downwards). Declaring local variables like this makes the compiler reserve space on the stack, giving you a buffer just below your return address in memory. See where this is going? If you overflow your buffer (string copies go from low to high in memory) you'll start overwriting your other stack variables until you eventually overwrite the return address, and when the function ends, it attempts to relocate to where it was called from, but actually ends up somewhere completely different. If you do it accidentally, you'll probably end up in unexecutable memory, triggering a segfault. If you craft your input well, you can execute anything you like!

Compiler designers got around this by setting stack cookies, making the compiled code look more like this:

void vulnerable(char* in){

    char test[10];

    int cookie=34531;

    strcpy(test, in);

    printf("argv[1] = %s\n", test);

    if(cookie!=34531) ThrowException();

}

The cookie sits just below the return address, so will be overwritten before the return address is. Then when the function is about to return, the cookie is checked, and if it isn't the expected value, then we know an overflow has occurred, and can crash the program.

There are two problems with this though:

1. This only detects the overflow just before the function exits
2. Hardcoded stack cookies can be accounted for by hackers, so really only detect unintentional overflows

We can fix #2 by randomly generating these at runtime, but we're still being quite passive about it. This is where the exception handlers and breakpoints come in. Since we're already wrapping our functions (or at least our compilers are doing it for us), it's quite reasonable to want to extend these.

Our new framework has the following things then:

1. A preamble, which installs an exception handler and breakpoints the return address
2. An exception handler, which explicitly detects modifications to the return address
3. A postamble, which removes the breakpoints before leaving.

I've said enough, it's time for the code.

// oflow.cpp : Defines the entry point for the console application.
// thanks to http://www.yashks.com/breakpoints.html for an excellent reference
// heavily based on SimpleSEH http://www.oldschoolhack.de/tutorials/seh.pdf

#define WIN32_LEAN_AND_MEAN        // Exclude rarely-used stuff from Windows headers
#include <stdio.h>
#include <tchar.h>
#include <windows.h>

// dirty global var
DWORD retval=0;
DWORD byteswritten=0;

LONG WINAPI TopExceptionFilter(EXCEPTION_POINTERS* ep){
    if(ep->ExceptionRecord->ExceptionCode==EXCEPTION_SINGLE_STEP){
        if(ep->ContextRecord->Dr6 & 1){
            // overflow
            printf("Overflow detected... ");
            // overwrite
            if(WriteProcessMemory(GetCurrentProcess(), (LPVOID) (ep->ContextRecord->Dr0), &retval, 4, &byteswritten))
                printf("Return address patched\n");
            else
                printf("Patch failed\n");
            return EXCEPTION_CONTINUE_EXECUTION;
        }   
    }
    return EXCEPTION_CONTINUE_SEARCH;
}

void Preamble(PDWORD d){
    // install handler
    DWORD prev;
    printf("Return address of last function: %X\n", *d);
    retval=*d;
    SetUnhandledExceptionFilter(TopExceptionFilter);

    CONTEXT ContextRecord={CONTEXT_DEBUG_REGISTERS};
    printf("Add h/w breakpoint at %X\n", d);
    // add hw breakpoint
    ContextRecord.Dr0=(DWORD)d;
    ContextRecord.Dr6=0;
    ContextRecord.Dr7 |= 0xD0001; // 4 bytes wide, break on writes only
    printf("DR0 set to %X\nDR7 set to %X\n", ContextRecord.Dr0, ContextRecord.Dr7);
   
    if(!SetThreadContext(GetCurrentThread(), &ContextRecord)){
        printf("Couldn't write context\n");
    }
}

void Postamble(){
    // remove hook
    CONTEXT ContextRecord={CONTEXT_DEBUG_REGISTERS};
    printf("Leaving, remove breakpoint\n");
    // add hw breakpoint
    ContextRecord.Dr0=0;
    ContextRecord.Dr6=0;
    ContextRecord.Dr7=0; // 4 bytes wide
    printf("DR0 set to %X\nDR7 set to %X\n", ContextRecord.Dr0, ContextRecord.Dr7);
   
    if(!SetThreadContext(GetCurrentThread(), &ContextRecord)){
        printf("Couldn't write context\n");
    }
}

_declspec(noinline) void vulnerable(char* in){
    char test[10];

    PDWORD stack;
    _asm{
        mov stack,ebp
    }
    stack++;
    Preamble(stack);
    printf("Doing strcpy\n");
    strcpy(test, in);
    printf("argv[1] = %s\n", test);
    Postamble();
}

int _tmain(int argc, _TCHAR* argv[])
{
    if(argc>1)
        vulnerable(argv[1]);
    else
        printf("Takes an argument\n");
    printf("done\n");
    return 0;
}


That didn't hurt now, did it? Here's the output

C:\Documents and Settings\Driverz\My Documents\Visual Studio 2005\Projects\oflow
\release>oflow garygarygarygarygarygary
Return address of last function: 40121B
Add h/w breakpoint at 12FF74
DR0 set to 12FF74
DR7 set to D0001
Doing strcpy
Overflow detected... Return address patched
Overflow detected... Return address patched
Overflow detected... Return address patched
Overflow detected... Return address patched
argv[1] = garygarygarygarygary←↕@
Leaving, remove breakpoint
DR0 set to 0
DR7 set to 0
done

And we're done. You'll see that each time the return address was overwritten, it was immediately returned to its original value, so the function could return normally. In a lot of circumstances this isn't going to be enough - so much of the stack is going to be totally mangled - but you could get a just-in-time debugger to be invoked here to patch up the memory properly, rather than when the security cookie kicks in, or when a poorly coded exploit fails and crashes your application further down the line.

There are a few other issues to consider here, so we'll look at them one by one

1. How do I debug it? Debuggers use the hardware breakpoints themselves, so the detection breaks (making it fun trying to iron out the bugs). The simplest way is to formally prove that it does what it needs to, and leave it out until you've finished developing your code
2. There are only 4 hardware breakpoints, what if I have more than 4 nested functions? This shouldn't be an issue - each invocation of the Preamble() function can store the previous hook in a Stack-type collection, since each function's call stack is going to be below its parent in memory, so overwriting of the current return address will be detected before the stacks of other functions are mangled.
3. Will this fix all exploits? No. It only detects stack overflows, not heap overflows (which are another story and are probably too extensive to protect with hardware breakpoints, but memory breakpoints are a possibility).
4. Can this be realistically applied to compiled binary code? Not easily. This is the principle I would like to use, but it would have to be done alongside a debugger which traces through each instruction, and manually invoking preamble and postamble before each CALL and RET, which is going to be really slow, but should be okay for research projects.

All done! Big thanks to:

http://www.yashks.com/breakpoints.html - good breakpoint reference

http://www.oldschoolhack.de/tutorials/seh.pdf - where the bulk of my code was stolen from

PS: Make sure you build the code with /GS- in compiler commandline and /SAFESEH:NO in linker commandline if building with visual studio