This was an interesting challenge and I really enjoyed solving it, so here’s a writeup! :)
First of all, we are provided with a binary and a PCAP file. Let’s start reversing the binary, here are some findings:
- The binary starts listening on port 4141 and every incoming connection is passed to the “handle” method
- From a high-level perspective we can infer that the binary is implementing a VM. When connecting it reads 2 lengths, 2 “magic words”, first message (bytecode for the VM) and second message (payload). More in detail, it reads messages in the following format:
- 1 word: length of bytecode (in number of instructions)
- 1 word: length of payload (in bytes)
- 1 word: first magic word
- 1 word: second magic word
- Bytecode
- Payload
- We can provide code for this VM and it will get executed for every 2 words of the payload.
- The VM has 13 opcodes and 16 32-bit registers. Registers are setup as following:
- r0: first magic word
- r1: second magic word
- r2: Nth word of payload
- r3: (N+1)th word of payload
- r4: first word of master key
- r5: second word of master key
- r6: third word of master key
- r7: forth word of master key
- The binary embeds a master key that was redacted in the version we have.
- After every execution of the VM r0 and r1 are sent back to the client.
Now we can move to the PCAP: we can notice that there is a communication of a client with the server, the bytecode is still there but the payload was redacted. The output from the server however does not seem helpful for now. We can then write a disassembler for the bytecode in order to understand what it is doing. This script implements it, and the following is its output:
xor reg[0], reg[0], reg[2]
xor reg[1], reg[1], reg[3]
mov reg[15], 0
mov reg[14], 2654435769
mov reg[13], 0
lshift reg[2], reg[1], 4
rshift reg[3], reg[1], 5
xor reg[2], reg[2], reg[3]
add reg[2], reg[2], reg[1]
mov reg[12], 3
and reg[11], reg[13], reg[12]
reg[10] = reg[4 + reg[11]]
add reg[10], reg[10], reg[13]
xor reg[2], reg[2], reg[10]
add reg[0], reg[0], reg[2]
add reg[13], reg[13], reg[14]
lshift reg[2], reg[0], 4
rshift reg[3], reg[0], 5
xor reg[2], reg[2], reg[3]
add reg[2], reg[2], reg[0]
mov reg[12], 3
rshift reg[9], reg[13], 11
and reg[11], reg[9], reg[12]
reg[10] = reg[4 + reg[11]]
add reg[10], reg[10], reg[13]
xor reg[2], reg[2], reg[10]
add reg[1], reg[1], reg[2]
mov reg[8], 1
add reg[15], reg[15], reg[8]
mov reg[8], 64
if reg[15] < reg[8]: goto 5
We can see that the code is doing some different operations involving our payload, the master key and the magic words. It is interesting to notice that there is some initial setup code (in particular, magic words are XORed with the payload), then lines after 5 are repeated 64 times. 2654435769 is an interesting constant so I tried to Google it, it give interesting results! Apparently it’s a constant used in the Tiny Encryption Algorithm. After finding this piece of information it was easy to find similarities between the XTEA algorithm and the code, the only difference was that the number of rounds is increased from 32 to 64.
At this point the goal of the challenge became clearer: a client connected to the server sending the bytecode of the almost-XTEA algorithm and a secret message (redacted from the PCAP), the server encrypted it and sent it back to the client. The master key is the XTEA key, the algorithm is called in CBC mode and the magic words are the IV. Some tests with the redacted binary confirmed these hypothesis.
The problem now is how to get the master key of the server, as we need it to decrypt the ciphertext from the PCAP. We can recall that the VM executes any bytecode we feed into it, the master key is initially in r4-r7 and r0-r1 are outputted to the client. We can just write some bytecode that does a “mov r0, r4; mov r1, r5” and get 2 words of the master key, then repeat the same operation for r6 and r7. This example code does the job.
Once we retrieve the master key from the server we just need to decrypt the ciphertext from the PCAP file. I used a Python implementation of the XTEA algorithm and I patched it to use 64 rounds instead of 32. Then we need to be careful with endianness.
This script performs the decryption…and ta-daaah, here’s the flag!
~~~~~~~~~~~~~~~~~~~~ RAD B) ~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~ Your flage is: ~~~~~~~~~~~~~~~~~~~~
~~~~~~~~ flag{g0t_a_pr0bl3m??_put_a_VM_0n_it!} ~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~