SLAE 32 – Assignment 4

Assignment 4 – Custom Shellcode encoder

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID: SLAE-1112

This is the fourth from seven assignments in order to complete the SLAE (32bit) certification.

This time, the objective is to create a custom Shellcode encoder.


An encoder is an algorithm to retain payload functionality, but alter the byte sequence.

Encoding a Shellcode is a common practice in order to avoid bad characters or to evade some (bad) AVs, making them believe that the Shellcode is not malicious. The encoded Shellcode needs to be prefixed by a “decoding stub”, used for decoding the shellcode. The Stub then takes the payload and transforms it back to its original form and then jumps right to it:


Custom Encoder

The Shellcode to be encoded will be a simple 19 byte execve(“/bin//sh”, NULL,NULL).

The custom encoding scheme that I developed consists in a mixed usage of ROT 13, XOR and Right Shift. The bytes will be XORed against an iterator that starts with the value 19 (size of shellcode) and will decrement after each interation.

The encoder script will iterate over each byte and perform the three operations:


Just like in class, the output is presented in two formats:



Now, we need to revert the encoder process in Assembly in order to build the stub. To do this, the Jmp-Call-Pop technique will be used.

This technique consists in performing a jmp short to a label and immediately after that calling another label. As always, when a call is made the address of the next instruction is pushed into the stack, thus, when we reach the called label and execute a pop instruction we will obtain the address of the Shellcode.

After we obtain the address of the encoded Shellcode, we can iterate over that address and offsets and start transforming the encoded Shellcode back to its original form.

Since the encoder performed a ROT 13, XOR and Right Shift over each byte, we need to do the reverse process: Left Shift, XOR and undo the ROT 13.

esi will be used as a pointer to the current Shellcode byte that is being decoded. After that byte have been decoded, esi is incremented, pointing to the next byte.

After our counter cl hits 0 the Shellcode is back to its original form and a jmp short is performed to the decoded Shellcode:


After extracting the Shellcode from the decoder binary file using objdump and using it on the PoC C code, our execve(“/bin//sh”, NULL,NULL) works perfectly:


As we can see our Shellcode grew from 19 bytes to 44 bytes. The size of the Stub that decodes the Shellcode is 44-19= 25 bytes.

Decoding process with gdb

It is possible to follow the code execution with gdb and see the Shellcode decoding process on the fly with gdb. In the following picture we can observe the memory layout at the beginning of the Stub (left) and after the decode has been completed (right):


The address 0x0804a059 (marked in yellow) is where the Shellcode starts. Before the Stub execution this address holds the instruction dec ebx (0x4b)  and after being decoded holds the instruction xor eax, eax (0x31). In the following image it is possible to watch the full Shellcode before and after being encoded:


The full code can be found in my SLAE Github repository. That is all for this assignment, see you on the next one.