Hack The Box: Reverse Engineering Basics

Overview

Reverse engineering challenges appear frequently in CTFs and require understanding how compiled programs work. This writeup covers fundamental techniques I’ve used on Hack The Box and other CTF platforms.

Static vs Dynamic Analysis

Approach	Tools	When to Use
Static	Ghidra, IDA, strings	Understanding program structure
Dynamic	GDB, ltrace, strace	Runtime behavior, bypassing checks
Hybrid	Both	Most effective approach

Initial Analysis

Basic File Information

# File type
file challenge

# Linked libraries
ldd challenge

# Strings (ASCII)
strings challenge | less

# Strings (unicode)
strings -e l challenge

# Section headers
readelf -S challenge

# Symbols
nm challenge

Security Protections

# Check protections with checksec
checksec challenge

RELRO           STACK CANARY      NX            PIE
Partial RELRO   Canary found      NX enabled    No PIE

Protection	Description	Bypass
RELRO	Relocations read-only	GOT overwrite harder
Stack Canary	Stack buffer overflow protection	Leak canary
NX	Non-executable stack	ROP chains
PIE	Position Independent Executable	Leak base address

Static Analysis with Ghidra

Setup

1. Create new project
2. Import binary (File → Import File)
3. Analyze with default settings
4. Navigate to main function

Reading Decompiled Code

Example: Simple password check

undefined8 main(void) {
    int iVar1;
    char local_28[32];
    
    printf("Enter password: ");
    fgets(local_28, 0x20, stdin);
    iVar1 = strcmp(local_28, "s3cr3t_p4ss!");
    if (iVar1 == 0) {
        printf("Access granted!\n");
        print_flag();
    }
    else {
        printf("Wrong password!\n");
    }
    return 0;
}

Password found: s3cr3t_p4ss!

Common Patterns

XOR encoded strings:

char encoded[] = {0x48, 0x65, 0x6c, 0x6c, 0x6f};
for (int i = 0; i < 5; i++) {
    encoded[i] ^= 0x20;  // XOR key
}

Decode manually:

encoded = [0x48, 0x65, 0x6c, 0x6c, 0x6f]
decoded = ''.join(chr(b ^ 0x20) for b in encoded)
print(decoded)  # "hello"

Multi-stage validation:

if (strlen(input) == 16) {           // Length check
    if (input[0] == 'H') {            // First char
        if (input[1] == 'T') {        // Second char
            if (check_format(input)) { // Format check
                // ...
            }
        }
    }
}

Build the flag piece by piece from the checks.

Dynamic Analysis with GDB

Basic Commands

# Start GDB
gdb ./challenge

# Set Intel syntax
set disassembly-flavor intel

# Run program
run

# Run with arguments
run arg1 arg2

# Set breakpoint
break main
break *0x401234

# Continue execution
continue

# Step one instruction
si

# Step one line (over calls)
ni

# Print registers
info registers
print $rax

# Examine memory
x/20x $rsp          # 20 hex words at stack pointer
x/s 0x402000        # String at address
x/10i $rip          # 10 instructions at instruction pointer

Bypassing Checks

Example: Patching a jump

Original:

0x401234: cmp eax, 0
0x401237: jne 0x401250   ; Jump if not equal (wrong password)
0x401239: call print_flag

In GDB, skip the check:

break *0x401237
run
# Enter wrong password
set $rip = 0x401239
continue

Finding Hidden Functionality

# List all functions
info functions

# Disassemble specific function
disassemble print_flag

# Find cross-references in Ghidra
# Right-click function → References → Find References to

Common Challenge Types

1. Password/Key Validation

# Run ltrace to see library calls
ltrace ./challenge
# strcmp("input", "password123")

2. Serial Key Generator

Reverse the validation algorithm:

int validate(char* serial) {
    int sum = 0;
    for (int i = 0; i < strlen(serial); i++) {
        sum += serial[i] * (i + 1);
    }
    return sum == 0x1337;
}

Write a keygen:

def generate_key():
    target = 0x1337
    key = ""
    running_sum = 0
    
    for i in range(10):
        # Calculate required character
        remaining = target - running_sum
        positions_left = 10 - i
        char_val = remaining // (i + 1)
        
        if 32 <= char_val <= 126:  # Printable ASCII
            key += chr(char_val)
            running_sum += char_val * (i + 1)
    
    return key

3. Packed/Obfuscated Binaries

UPX packed:

# Detect
file challenge
# challenge: ELF 64-bit ... UPX compressed

# Unpack
upx -d challenge

Custom packer:

1. Set breakpoint at entry
2. Run until unpacking completes
3. Dump memory to file
4. Analyze dumped binary

4. Anti-Debugging

Common techniques:

// ptrace check
if (ptrace(PTRACE_TRACEME, 0, 0, 0) == -1) {
    exit(1);  // Debugger detected
}

// /proc/self/status check
FILE* f = fopen("/proc/self/status", "r");
// Look for TracerPid: 0

Bypasses:

# LD_PRELOAD to hook ptrace
# Compile: gcc -shared -fPIC -o bypass.so bypass.c
cat > bypass.c << EOF
long ptrace(int request, int pid, void *addr, void *data) {
    return 0;
}
EOF

LD_PRELOAD=./bypass.so ./challenge

Or patch the binary to NOP the check.

Python Scripting for RE

Pwntools for Automation

from pwn import *

# Connect to process
p = process('./challenge')

# Or remote
# p = remote('target.htb', 1337)

# Receive until prompt
p.recvuntil(b'Password: ')

# Send data
p.sendline(b's3cr3t_p4ss!')

# Get response
response = p.recvline()
print(response)

p.interactive()

Z3 for Constraint Solving

When validation is complex:

from z3 import *

# Create symbolic variables
flag = [BitVec(f'f{i}', 8) for i in range(16)]

s = Solver()

# Add constraints from reverse engineering
s.add(flag[0] == ord('H'))
s.add(flag[1] == ord('T'))
s.add(flag[2] == ord('B'))
s.add(flag[3] == ord('{'))
s.add(flag[15] == ord('}'))

# Example: characters must sum to specific value
s.add(sum(flag[4:15]) == 1000)

# Printable characters
for c in flag:
    s.add(c >= 32)
    s.add(c <= 126)

if s.check() == sat:
    m = s.model()
    result = ''.join(chr(m[c].as_long()) for c in flag)
    print(f"Flag: {result}")

Practical Workflow

# 1. Initial analysis
file challenge && checksec challenge && strings challenge | grep -i flag

# 2. Run and observe
./challenge

# 3. Trace library calls
ltrace ./challenge 2>&1 | tee ltrace.log

# 4. Open in Ghidra, find main
# 5. Identify validation logic
# 6. Dynamic analysis with GDB if needed
# 7. Write solve script

Cheat Sheet

Task	Command/Tool
File type	file binary
Strings	strings binary
Shared libs	ldd binary
Protections	checksec binary
Decompile	Ghidra, IDA
Debug	gdb binary
Trace libs	ltrace binary
Trace syscalls	strace binary
Unpack UPX	upx -d binary

Key Takeaways

1. Start with static analysis — Understand the program structure first
2. Use strings generously — Flags and passwords are often in plaintext
3. ltrace catches strcmp — Easy wins for password checks
4. GDB for runtime patching — Skip checks by modifying $rip
5. Z3 for complex constraints — Automate what you can’t solve manually

Reverse engineering is pattern recognition — the more challenges you solve, the faster you recognize common techniques.