GCC and pointer overflows

Posted Apr 16, 2008 19:50 UTC (Wed) by iabervon (subscriber, #722)
Parent article: GCC and pointer overflows

Note that the wrap-around check isn't necessarily sufficient if sizeof(*buffer) > 1, as in:

    int buffer[BUFLEN];
    int *buffer_end = buffer + BUFLEN;

    /* ... */
    unsigned int len;
    if (buffer + len >= buffer_end || buffer + len < buffer)
	loud_screaming_panic("len is out of range\n");

At a bit more than 1/(sizeof int) of MAX_INT, the pointer sum will probably go back through the buffer. For ideal security on this sort of stuff, GCC would generate code where a pointer addition that overflows compares greater than any valid pointer value when compared in the same expression. It should be more efficient than the second check anyway, since it's only checking processor flags on calculations it's doing anyway in a correctly-predicted not-taken branch.

to post comments

GCC and pointer overflows

Posted Apr 16, 2008 20:58 UTC (Wed) by JoeBuck (guest, #2330) [Link] (1 responses)

The compiler then would have to insert such a check into every array reference in a loop, and your code would run about 3x slower. After all, they could wrap around the end, right?

But in a properly structured program, such checks are not needed, as pointers are not random, but rather refer to objects that have definite sizes and that do not wrap around the end of memory. If you think that you need such a check, there's something wrong with your program structure, some check that was omitted elsewhere. What matters is the size of the buffer, and you can go beyond its end without wrapping around the end of the address space.

GCC and pointer overflows

Posted Apr 17, 2008 14:06 UTC (Thu) by iabervon (subscriber, #722) [Link]

How so? I was only suggesting that the expression "pointer1 + offset >= pointer2" be treated
differently (that is, the magic does away if you store the LHS in a variable and use the
variable). (And, of course, the change would only apply to cases where the LHS overflows the
address space, which is clearly undefined, so having odd things matter isn't a problem.)

If you do something with "pointer + offset" other than comparing it with something else, I
wouldn't have anything change (what would it do, anyway, without a test and branch in the
code?); and in the case where you're doing the comparison, processors generally have a status
bit that gets updated in the multiply/shift and add and is pretty much free to have an
unlikely jump based on. Except on architectures where the instruction decode significantly
limits the pipeline, it should be at most one cycle to test so long as you do each test right
after the ALU operation.

C, in general, doesn't make good use of the fact that processors make it trivial to test for
overflow, but that doesn't mean that C compilers can't do extra-clever things with this
ability in cases where there's undefined behavior associated with it.