How ROP (Return Oriented Programming) works


Gear up for a fascinating deep dive into exploit development using return-oriented programming. This blog post covers a concept used widely in exploit development when bypassing restrictions when attempting memory corruption hacking. We will look at how the stack works, vanilla buffer overflow attacks, progress to conditions to prevent buffer overflows, introduce ROP attacks, and how they are used to gain Remote Code Execution on target machines.


To gain the most out of this walk-through it is essential to have some background knowledge on the following:

  • Linux fundamentals
  • Basic understanding of assembly
  • Ability to read and interpret code
  • Vanilla buffer overflows


By the end of this walk-through, readers should be able to:

  • Perform a basic ROP attack
  • Use python to automate ROP attacks
  • Build an understanding of how the stack works

The stack and memory layout

We begin by reviewing how the stack works. The stack is a region in memory that temporarily stores data related to the current function/thread being executed. It behaves in a LIFO(last in first out) manner when handling data. Data pushed last into the stack is popped out first – just like a stack of plates.

A typical memory layout of a C program consists of the following sections:

  • Text segment – contains executable instructions
  • Initialized data segment – contains global and static variables
  • Uninitialized data segment (bss) – contains statically declared variables that haven’t been assigned a value yet
  • Stack – stores data temporarily
  • Heap – dynamic memory allocation takes place here

How data is handled in the stack

In standard computer architecture, the stack grows backwards into memory (opposite direction). A stack pointer (ESP) is a register that keeps track of the top of the stack, while a base pointer (EBP) keeps track of the bottom of the stack.

Each time data is pushed onto the stack; the stack pointer is adjusted. The set of data-pushed for one function is called a stack frame. The function should know where to return from the stack once it finishes executing, so an address is pushed onto the stack known as the return address. It shows where we return to in memory

buffer is a region in the stack that holds data. Keep this in mind as it will come into play later in the blog-post

Stack-based buffer overflows

A buffer overflow is a vulnerability that arises when the amount of data being supplied to the program exceeds the storage capacity of the buffer. As a result, the program attempting to write the data to the buffer overwrites adjacent memory locations.

Buffer Overflow

The first buffer overflow attack occurred in 1988. Being an old and well-understood vulnerability, they are still a significant security problem to date, occurring in way more complicated scenarios.

Example of vulnerable C code

In a basic example such as the one below, we will see how the vulnerability arises.

This simple C program takes in one argument and passes it to vulnerable(), which creates a buffer variable and copies the argument into it. It finishes off by printing “exiting”, then quits.

It is not a very useful program, but it sure is vulnerable. Line 7 gives a call to strcpy() a well-known weak method in C that has no bounds checking for the size of input it copies. So, we could supply an input larger than the buffer cause a buffer overflow in this case.

Exploiting a vanilla buffer overflow

We start by doing a practical example on an intentionally vulnerable program. Our benchmark steps for performing the exploitation will be as follows

  1. Triggering the vulnerability
  2. Finding the offset
  3. Controlling the return address
  4. Complete the exploit

Our practical example will involve a CTF challenge. CTFs are security/hacking competitions that involve challenges meant to teach hacking in a fun way.

CTFlearn is a good platform with some resources which we will utilize for this example. Let us ssh into the following server to access the challenge we will be solving involving a buffer overflow.

ssh color@ -p 1001 (password: guest)

Viewing the files in the current directory, we have a C file, its compiled binary, and a flag.txt file which our goal should be to read it. If you try to read it, you get a permission error.

Source code analysis

Our first step involves reviewing the code in the C file and finding the vulnerable part we will be taking advantage of.

I like to start from the main function and viewing the functions it calls. setresuid and setresgid allow the user to run the file with temporarily elevated privileges. setbuf specifies the buffer to be used by the stream for I/O operations. These three lines are standard and not our focus for now.

We have a conditional that checks if the vuln function has been called and gives us a shell via the system(“/bin/sh”) command. Since we will be running the program with elevated privileges, access to the shell will give us temporary privileges to read the flag.

The vuln function creates a buffer variable that holds 32 characters. It asks for input from the user and stores it in the buffer variable. The main point to note is that a call to gets is made. gets is a very dangerous method in C and should never be used to get input from the user as it performs no checks and causes buffer overflows.

1. Triggering the vulnerability

We begin by interacting with the program.

In my defence, brown is a decent colour 🙂

Let us supply considerable input and try to crash the program. Segmentation fault is a standard error resulting from a buffer overflow. Perfect. We can begin the exploitation process.

2. Finding the offset

We need to find the exact number of bytes that causes the application to crash. This will allow us to know the location of the instruction pointer in the program. Because if we overwrite the return pointer, the program will not know where to return control, hence causing a crash.

We will use gdb to find the location of the instruction pointer. gdb is a free debugger that allows us to see what is happening inside a program while executing.

We will Metasploit pattern_create module to generate a cyclic pattern of bytes to help us find the location of the EIP. The command below generates bytes of length 100.

/usr/bin/msf-pattern_create -l 100

The address displayed as 0x62413762 is a byte representation of some of the ASCII characters that we supplied to the program. This means that the 100-byte ASCII text overwrote the instruction pointer. Hence the program did not know where to return. Let us find the location of that offset.

/usr/bin/msf-pattern_offset -l 100 -q 0x62413762

This means that after exactly 52 bytes, we have the EIP (instruction pointer). Since we know the location of the EIP, we can use it to point to an area of memory we want, e.g., somewhere we have inserted malicious code or, in this case, to where system(‘/bin/sh’) is in memory and gain a shell!

3. Controlling the return address

During exploit development, I like to confirm I have control over the instruction pointer by supplying four more bytes after 52 characters and viewing the output in gdb.

You can see the instruction pointer being overwritten with 424242 which is hex representation for ‘BBBB’. This confirms control of the instruction pointer.

4. Completing the exploit

We finalize our exploit by writing a python script that will overwrite the instruction pointer with the memory address of/bin/sh location in memory. This will enable us to jump to where /bin/sh is and get a shell.

We begin by finding the memory address of system(‘/bin/sh’). Back to our debugger. Typing in info functions, we get a list of the functions used in the program.

As we saw in the source code, the main function had a call to system(‘/bin/sh’), if we decompile that function, we can get the memory address of the call to the system.

disass main decompiles the main function for us. Here you can see the highlighted line that contains the call to the system. When attempting this challenge, I realized that picking the system’s exact address will not work because the addresses shift slightly once the program runs.

We can therefore pick an address that is slightly below the system one. This would mean we will hit the system along the way and get a shell. I picked 0x08048677

Let us build our exploit with python. I will utilize the pwntools module to interact with the challenge remotely.

  • Line 1 imports the module so we can use it
  • From lines 3 to 7, we create an ssh session with the server
  • In line 10, we interact with the binary file
  • Line 13 fills up our buffer till the EIP with 52 characters, p32(address) packs the address in 32-bit byte format
  • Line 14 allows us to receive data from the binary, e.g. the first line it prints out
  • We send the payload to the binary at line 15
  • At 16 finally opens up a shell which we capture from the server

We get a shell and proceed to print the flag!

ROP (Return Oriented Programming)

As we saw in the practical bit of exploiting a buffer overflow, controlling the stack can be very dangerous. It allows us to overwrite the instruction pointer, giving us control over what the program does next.

Realistically, programs will not have a system(‘/bin/sh’) function that magically pops us a shell, so we need to conveniently invoke system or exec to give us a shell for arbitrary code execution.

Return Oriented Programming (ROP) is the idea of chaining together small snippets of assembly code with control over the stack to cause the program to do more complex things.

Data Execution Prevention (DEP)

ROP allows us to bypass security mechanisms like Data Execution Prevention (DEP). In a standard buffer overflow, the attacker would write the malicious payload onto the stack and then overwrite the return address with the location of these newly written instructions.

Eventually, Operating Systems began to combat exploitation of these bugs by marking the memory where data is written as non-executable. With this enabled, the machine would refuse to execute any code located in user-writable areas of memory. This prevents the attacker from placing a payload on the stack and jumping to it via a return address overwrite.

To defeat this, ROP came into play. This attack does not inject malicious code but instead uses instructions already present, “ROP gadgets“, by manipulating the return address. This attack does not use malicious code but rather combines assembly instructions within the program.

What are ROP Gadgets?

ROP gadgets are small instruction sequences ending with a ret instruction. Combining different ROP gadgets enables you to perform complex instructions within the program.

An example of ROP gadgets:

What can we do with ROP gadgets

With an ROP gadget, we can execute any kind of instruction in memory provided the right instruction sequence is found. I will focus on loading a constant into a register.

Loading a constant into a register

Loading a constant into a register saves a value on the stack to a register using the pop instruction for later use.

In the instruction pop eax; ret; we pop the value on the stack to eax register and then return to the address on top of the stack. The stack alignment will look as follows:

  • At the top of the stack we have the address of the gadget which overwrites the instruction pointer.
  • 0x123456 is the value that will be popped to eax
  • Finally we have the address of the next gadget, when ret is called , we return to it.

Let us write our first ROP exploit!

Where’s the fun if we don’t put our newly gained knowledge into practice. We will be solving a challenge from ROPemporium, one of the best platforms for learning ROP.

Split x86_64 Ropemporium

We will be solving the split challenge. Download the x86_64 version. I will showcase ROP using a 64-bit architecture.

Our goal for the challenge

As with other ropemporium challenges, we need to read flag.txt by exploiting the binary file. Our objective is to use an ROP gadget to execute system with ‘/bin/cat flag.txt’ string as the argument.

To call asystem with an argument, we need to utilize a pop rdi gadget

| syscall | arg0 | arg1 | arg2 | arg3 | arg4 | arg5 |
| %rax | %rdi | %rsi | %rdx | %r10 | %r8 | %r9 |

our syscall is system , and arg0 is ‘/bin/cat flag.txt’, and pop rdi; ret is our gadget to add the arg0 into rdi register from the stack.


Since we are dealing with a buffer overflow vuln, let us begin by finding the RIP offset (64-bit instruction pointer). Recreating steps from our buffer overflow example above, we get an offset of 40 bytes. In the screenshot below, RBP is at an offset of 32 bytes. Which is 8 bytes away from the RIP, so 32+8 = 40

I will reduce the technicality of this walkthrough and skip the detail on reverse-engineering the binary. Since our instructions say we need to use a system call, let us go ahead and search for the system call in the binary.

We have the call to system at memory address 0x0000000000400560

Next, we find the string /bin/cat flag.txt address using the command ‘find “/bin/cat”‘

The /bin/cat flag.txt address is located at 0x601060.

Our last step is finding a suitable rop gadget to load the /bin/cat flag.txt into system. I will use ropper, a tool used to find rop gadgets.

Luckily, we have a pop rdi; ret at address 0x00000000004007c3.

Our final payload will look as follows:

offset_padding + pop_rdi_gadget + "/bin/cat flag.txt"address + system_addr 

Our exploit will overwrite the instruction pointer with pop rdi; ret. This instruction will pop the next instruction on the stack (that of bin/cat flag.txt) into the rdi register. The ret instruction pops another address off the stack, in this case, the system. Which executes usually giving as the flag contents. 

Mitigations against buffer overflows

1. Secure coding

Writing secure code is the best way to prevent buffer overflow vulnerabilities. Developers who write in languages like C, more susceptible to buffer overflow vulnerabilities, should be more aware of the dangerous functions and methods. Methods like gets, which we have seen in the first example, should be avoided at all times.

SDLC cycles can also incorporate secure coding practices and periodic source code reviews to reduce the attack surface of organizations.

2. Non-executable stacks (NX)

Buffer overflow exploits often rely on putting malicious code into the stack and then changing the program’s control flow to jump to the malicious code and execute it. Such an attack will be preventable if all the writable addresses in memory are non-executable. An NX bit is enabled to make the stack non-executable.

3. Address Space Layout Randomization

This is a technique that relies on randomly positioning addresses of various program components in the memory space. This makes it harder for an attacker to determine a specific block of code in the target program, thus reducing the likelihood of a successful attack.

4. Stack Canary

This is a value placed on the stack so that it allows detection when overwritten by a stack buffer overflow. Before the function returns (which is when the attacker attempts to gain control over the instruction pointer), the integrity of the canary is verified.

5. Data Execution Prevention (DEP)

This security feature monitors and protects certain regions of memory from executing (usually malicious) code. We are enabling DEP to mark all data regions in memory as non-executable by default.


Despite being a technical and lengthy article, we have gone through buffer overflows and bypass security mitigations such as DEP using return-oriented programming. This blog post is an excellent starter to exploit development and understanding what goes on under the hood when programs run.


If you enjoyed this article, I would recommend my previous blog post on sql injection attacks

You May Also Like