Abstract: As modern return-oriented programming (ROP) attacks have become more sophisticated, preventing or detecting these attacks is essential for real-world exploit defense. As an alternative to many defense techniques of ROP attacks that require software modification and hardware assistance, researchers have proposed ROP defense techniques using hardware performance counters (HPCs) to mitigate concerns about additional cost and compatibility issues. However the existing HPC data-based ROP detection techniques typically suffer from low detection performance mainly because of the non-deterministic nature of HPCs. To address these issues, we propose DROPSYS, an enhanced detection of ROP attacks using system information. DROPSYS is based on the detection of the abnormal change of system information that takes place during ROP attacks. Differing from the existing techniques, DROPSYS harnesses not only HPC data, but also system utilization data to mitigate the non-deterministic nature of HPCs. Using both HPCs of processors and system utilization of operating systems makes transparent operation without requiring any modifications to the protected programs. DROPSYS uses a long short-term memory-based variational autoencoder to effectively analyze the multivariate time-series HPC data and system utilization data for better detection performance. DROPSYS also performs feature selection for low computational overhead while maintaining the attack detection performance. In our experiments with real-world ROP exploits, DROPSYS successfully detected ROP code execution in all tested programs. Evaluation results show that DROPSYS effectively captures the behaviors and effects of ROP attacks and can detect the attacks with a 0.028% false positive rate. The accuracy of DROPSYS is 95.3%, and its F1 score is 94.9%---a figure much higher than those of existing techniques that utilize only HPC data.

Abstract: Malicious attackers exploit memory vulnerabilities to carry out control-flow attacks, such as a ROP attack, negatively impacting the system. They execute malicious code with adverse effects on the system through these control flow attacks. While techniques to safeguard programs from ROP attacks have primarily focused on preserving the integrity of return addresses and function pointers, there is insufficient research on protection techniques for frequently used data pointers residing in memory. This is due to the more frequent declaration and use of data pointers compared to other pointer objects, resulting in a higher cost of integrity verification. In this paper, we propose a technique that utilizes the pointer authentication function provided by the ARM architecture for efficient integrity verification of pointers. The hardware support functions help significantly mitigate negative performance impacts while ensuring high-security levels for data pointer protection. The proposed technique operates through hardware-based keyed MAC generation and verification instructions. It analyzes the declaration and usage of data pointers during the compilation process, inserts ARM PA instructions, and generates MAC values with appropriate context values.

Abstract: Code obfuscation serves as a method to safeguard a program's intellectual property rights against malicious software analysis through reverse engineering. Within code obfuscation, control flow obfuscation makes it challenging to analyze the program by intricately altering its control flow. However, in general, the control flow of a program with obfuscated control flow is statically determined, rendering it susceptible to dynamic analysis. Recently, a technique was proposed to address this issue, utilizing dynamic opaque predicate. However, the technique has a limitation—once the control flow is determined through the first opaque predicate, the subsequent control flow becomes fixed. This paper introduces a novel dynamic control flow obfuscation technique designed to address these limitations. The strength of the proposed technique lies in the fact that its deobfuscation not only necessitates additional technology, leading to a high cost for malicious software analysis, but it can also be seamlessly integrated with existing control flow obfuscation techniques.

Abstract: Recent studies on control-flow attack defense techniques aim to mitigate the analysis cost when applied to real applications. Nevertheless, their defense strategies, incorporating both hardware implementations and algorithms, still prove less effective in tackling the challenges of attack analysis and cost issues in real-world industries. This paper introduces a lightweight detection technique that utilizes system side-channel information, specifically the system utilization generated by running systems and processes, to address the limitations of existing defense techniques. The newly implemented detection technique dynamically profiles the program's process I/O, memory, and context switch information through threading during the execution of the analyzed program. In the experiment, we analyzed specific fluctuation sections and detected control-flow attacks through pattern comparison using actual system side-channel information. Specifically, through approximately 1,600 profiling experiments conducted 1,000 times, we confirmed that system side-channel information exhibits a normal pattern in the control flow of a typical program but displays a distinct change pattern when a control-flow attack occurs.

Abstract: Code pointer manipulation attacks, commonly used to hijack the control flow of an application, enable attackers to execute code with a detrimental impact on the system. In particular, backward-edge exploits are frequently employed in real-world system hacking through ROP attacks. Backward-edge protection techniques are primarily implemented through code instrumentation in the prologue and epilogue of a function block. Instrumented code includes instructions to either encrypt/decrypt the return address of the function or store it in a separate memory space. However, when a library function, used to handle a program's exception, is called, a mismatch between the function prologue and epilogue occurs. The instrumented code fails to adjust for this mismatch, resulting in a program crash. In this paper, we propose an enhanced exception handler that takes into account security, compatibility, and efficiency in the exception handling process for programs utilizing the backward-edge CFI technique. In detail, we implemented a practical and efficient C/C++ exception handler compatible with backward-edge CFI. We assessed compatibility and performance by applying the proposed prototype to the SPEC CPU2017 benchmark program. The results of the measurements confirmed no compatibility issues and revealed a performance overhead of less than 1%.