security

Art of Exploitation

Wednesday, May 21, 2025 · 11 min read

Page 1 of 16

GNU Development Tools

objdump -M intel -D <a.out>
- Used for disassembling executables.
x86 processors → 80386, 80186, 80286, 80486

Using gdb

gdb -q ./a.out
- Add breakpoint → break main
- Run program → run
- Get register info → info registers

Registers

General Purpose

Accumulator:
- 16-bit: ax
- 32-bit: eax
Base:
- 16-bit: bx
- 32-bit: ebx
Counter:
- 16-bit: cx
- 32-bit: ecx
Data:
- 16-bit: dx
- 32-bit: edx

General Purpose (Extended)

Stack Pointer:
- 16-bit: sp
- 32-bit: esp
Base Pointer:
- 16-bit: bp
- 32-bit: ebp
Source Index:
- 16-bit: si
- 32-bit: esi
Destination Index:
- 16-bit: di
- 32-bit: edi

Instruction Pointer Register

eip

For 64-bit Registers

r8 - r15:
- Helps in efficiency.

Flags

e-flags → Combination of memory segmentation and flags.

Segment in Memory

cs → Thread local storage
ds, es, fs, gs → File system & general storage

Page 2 of 16

Intel x86-64 Assembly Syntax

operation <dest>, <src>

gdb "examine" Command

Used to examine memory and look for certain addresses.

Shorthand Syntax

x → Examine memory

Format Options:

x/o → Show in octal
x/x → Hexadecimal
x/u → Unsigned base 10
x/t → Binary
x/i → Instruction (show at the address)
x/s → Get string at the address
x/x b → Half-word (2 bytes, little endian)
- Little-endian: Left-to-right is opposite.
x/x l → 4 bytes
x/x g → Giant (8 bytes)

Shared Library Code

Shared libraries' code comes from libraries like:

strcpy() → from string.h

Commands

run
bt
continue / cont

Breakpoints in Shared Libraries

If we set breakpoints for shared library code, they may get reduced after the program runs.

Page 3 of 16

Memory Segmentation

Compiled Program's Memory

Text → Also called code segment.
Data
bss
Heap
Stack

Segments Overview

a) Text Segment

Execution Flow:
1. EIP reads the instruction.
2. Add byte length of instruction to EIP.
3. Execute the instruction that was read in Step 1.
4. Loop.
Write Permission:
- Disabled in text segment as it is not used for storing variables, only code.
- Memory size is fixed.

b) Data & bss Segment

Initialized Global Variables
Uninitialized Global Variables
Static Variables

c) Heap Segment

Segment of memory a programmer can control directly.
- Growth:
  - Grows downward → Larger address.
  - Grows upward → Higher address.

Page 4 of 16

d) Stack Segment

Variable Size:
- Stores local function variables and context during function calls.
- Growth:
  - Grows downward → Lower address.
  - Grows upward → Higher address.

graph TD
    A[Stack Frame]
    B[Lower Address]
    C[Higher Address]
    A --> B
    A --> C

File Access

Two Ways to Access Files in C:

File Descriptors → Low-level I/O functions.
File Streams → High-level buffered I/O.

File Descriptor

A number used to reference open files.

Common Functions

open(), close(), read(), write()

Page 5 of 16

Exploitations

Exploitations are usually done with buffer overflows or format string methods.

i) Overwrite variable value with overflow

Example:

strcpy(password_buffer, password);
char[16]

If we pass more than 16 characters, it will start to overwrite memory as there is no check on the compiler to restrict access.

ii) Overwrite the entire stack memory

When a function is called, a structure called stack frame is pushed onto the stack, and the EIP register jumps to the first instruction of the function.

Stack frame: Has local variables and return addresses so EIP can be restored.

Example:

int addr;
call func;
inst1 addr2;

<addr> is the return address. So we can check where exactly it is stored. If we modify the return address, execution can be controlled.

(INSERT IMAGE)

Page 6 of 16

iii) Heap Overflow

Heap overflow is similar to stack overflow and can overwrite memory.

malloc(): Allocates heap memory.
free(): Releases heap memory.

There might be checks to verify if heap headers are correct.

iv) Format String Exploits

Format string exploits are not very common anymore.

Common Format Specifiers:

%d: Decimal
%u: Unsigned integer
%x: Hexadecimal
%s: String
%n: Writes the number of bytes written so far.

This %n pushes the value into a pointer.

Example:

printf("Hello %d, %08x, %n", A, &A, B);

Stack layout:

Address of format string
Value of A
Address of A
Value of B
Stack bottom

If only two arguments are passed to printf, the value of %n pushed in the stack will point to the stack bottom instead.

Page 7 of 16

Networking

OSI Model Layers

Application Layer
Presentation Layer
Session Layer
Transport Layer
Network Layer
Data Link Layer
Physical Layer

Wrapped information is passed down the layers.

i) Socket

Sockets help in network communication through the OS.

Types of Sockets:

Stream Socket (TCP)
Datagram Socket (UDP)

Syntax:

socket(int domain, int type, int protocol);

domain: Protocol family (e.g., AF_INET).
type: Stream or datagram.
protocol: Protocol of the family (usually set to 0).

Common Socket Functions:

connect(fd, sockaddr, sockaddr_length)
bind(fd, addr, addr_length)
listen(fd, queue_size)
accept(fd, addr, addr_length)
send(fd, buffer, length, flags)
recv(fd, buffer, length, flags)

ii) sockaddr Structure

The sockaddr structure is different for each protocol family. It contains:

Address family
Address details

Page 8 of 16

ii) Network Sniffing

Types of Networks:

Switched: Packets pass through every device.
Unswitched: Devices look only at packets sent to destination addresses.

For unswitched networks, devices can be enabled in promiscuous mode, allowing them to look at all packets.

Example tools:

tcpdump
dsniff

These convert the device interface into promiscuous mode. This is called sniffing.

Example Command:

sudo ifconfig eth0 promisc

Packet Sniffing

Most programs such as telnet, FTP, and POP3 are connected to tcpdump. All these present nicely packed TCP/IP packets.

Accessing Lower Layer Packets

We can use raw sockets instead of SOCK_STREAM or UDP.

Example:

socket(PF_INET, SOCK_RAW, IPPROTO_TCP);

Here:

PF_INET: Protocol family (Internet)
SOCK_RAW: Raw socket type
IPPROTO_TCP: TCP protocol

Page 9 of 16

Introduction

These notes delve into the technical details of packet sniffing, network protocol headers, and potential exploits related to the TCP/IP stack. The text covers the structure of Ethernet, IP, and TCP headers, methods of packet manipulation, and common network attacks.

Packet Sniffing with `libpcap`

A library libpcap is used behind tcpdump and sniff.

It is cross-platform and has multiple features allowing sniffing on networks using raw sockets.
This will help decode headers and packets for all layers.

Ethernet Header (Defined in `if_ether.h`)

struct ethhdr {
    u_char h_dest[ETH_ALEN];
    u_char h_source[ETH_ALEN];
    __be16 h_proto; // Octets in Ethernet address = 6
};

Packet Type ID Field: Specifies the type of packet.

IP Header (Defined in `ip.h`)

struct iphdr {
    u_int8_t version;
    u_int8_t ihl;
    u_int8_t tos; // Type of service
    u_int16_t tot_len;
    u_int16_t id;
    u_int16_t frag_off;
    u_int8_t ttl;
    u_int8_t protocol;
    u_int16_t check; // Checksum
    u_int32_t saddr; // Source address
    u_int32_t daddr; // Destination address
};

Page 10 of 16

TCP Header (Defined in `tcp.h`)

struct tcphdr {
    u_short tcp_src_port;
    u_short tcp_dest_port;
    u_int tcp_seq;
    u_int tcp_ack;
    u_short reserved:4;
    u_short tcp_offset:4;
    u_short tcp_flags;
    u_short tcp_window;
    u_short tcp_checksum;
    u_short tcp_urgent;
};

Packets: Sent to the port they are destined for.
Clever workarounds are often needed for sending/receiving packets.

Methods:

Spoofing:
- Forging a source address in the packet.
ARP Reply:
- Requests MAC addresses for which a response is received on ZP.
ARP Traffic:
- Requires additional memory and would complicate a protocol.

Page 11 of 16

Denial of Service (DoS) Attacks

Program-Based Exploits:
- Crash services.
Network-Based Exploits:
- Flood services.

(a) SYN Flooding

Exhausts states in the TCP/IP stack.
Limited number of connections that the server can track.

sequenceDiagram
    participant Attacker
    participant Victim's Service
    Attacker->>Victim's Service: SYN
    Victim's Service->>Attacker: SYN+ACK
    Attacker->>Victim's Service: (No ACK)

Since ACK is never sent from the spoofed address, the victim's service will wait for a timeout, resulting in long delays.

(b) Ping of Death

ICMP echo messages can only have (2^{16}) bytes of data.
Some systems fail to handle packets larger than (2^{16}), leading to crashes.

Page 12 of 16

Other Network Attacks

(c) Teardrop

Related to IP fragmentation assembly.
When a packet is fragmented, offsets stored in the headers are used to reconstruct the original packet.
Teardrop Attack: Sends packets with overlapping offsets, causing implementation errors and crashes.

(d) Ping Flooding

Floods the victim with ping packets.

(e) Amplification Attacks

sequenceDiagram
    participant Attacker
    participant Broadcast Address
    participant Victim
    Attacker->>Broadcast Address: Large Packet
    Broadcast Address->>Victim: Amplified Packet

(f) Distributed DoS Flooding (DDoS)

Multiple sources attack the victim simultaneously.

(iv) TCP/IP Hijacking

Spoofed packets are used to take over a connection between a host and a machine.

Page 13 of 16

TCP Packet Sniffing and Spoofing

TCP Sequence Diagram:

sequenceDiagram
    participant Client
    participant Server
    Client->>Server: Seq# = 1, Len = 669
    Server->>Client: TCP ACK# = 670
    Client->>Server: Seq# = 670, Len = 1460
    Server->>Client: ACK# = 2130
    Client->>Server: Seq# = 2130, Len = 1460
    Server->>Client: ACK# = 3570

Key Observations:

If we can sniff a TCP packet from the receiver and send a spoofed packet with the correct sequence number, the server will believe the attacker is the client.
Note: This only works if the attacker is already on the network.

Steps:

Sniff already established connections (if ACK flag is set).
Filter only target IP packets.
Send a RST (reset) packet to the target with the correct sequence number.

Result:

Once the connection between the client and the server is hijacked, the server will remain in a desynchronized state.

Page 14 of 16

Port Scanning

Purpose:

To figure out which ports are listening and accepting connections (nmap tool).

Techniques:

(a) Stealth SYN Scan / Half-open Scan

Procedure:
- Send a SYN packet.
- Check for SYN/ACK from server.
- Send RST connection (do not SYN flood).
Command:
```
nmap -sS
```
Stealth Scan

(b) FIN, X-mas, and Null Scans

Purpose:
- Detect stealth SYN scans.
- Sends nonsensical packets to every port.
- If the port is listening, it gets ignored, otherwise RST packet is sent.
Limitations:
- Not reliable; some implementations might choose not to send RST packets.
Commands:
```
nmap -sF
nmap -sX
nmap -sN
```

(c) Spoofing Decoys

Procedure:
- Send decoy IP addresses.
- Keep the real IP hidden to avoid SYN flood.

Page 15 of 16

Advanced Techniques:

(d) Idle Scanning

Purpose:
- Scan a target using spoofed packets from an idle host by observing changes in the host.
Mechanism:
- Works on machines that increment the IPID packet numbers. (Does not work for randomized IPID.)

sequenceDiagram
    participant Attacker
    participant Zombie
    participant Target
    Attacker->>Zombie: IPID probe (IPID = 314)
    Zombie-->>Attacker: IPID = 315
    Attacker->>Target: Spoofed SYN (with zombie IP)
    Target-->>Zombie: SYN/ACK packet (if port open) or RST (if port closed)

Observations:
- If IPID increases by 1, the port is closed.
- If IPID increases by 2, the port is open.
Command:
```
nmap -sI <zombie> <target>
```

Page 16 of 16

Proactive Defense (Shroud or Shield)

Purpose:

Prevent sending RST packets on closed ports.

Procedure:

Use firewall rules:

iptables -h -A 20 "void" -j "send-reset"

Send SYN/ACK on all ports without opening all ports.
Filter SYN packets on closed ports.
Send back/spoof back SYN/ACK.

References & Related Topics

Network Security: Concepts and Techniques
Buffer Overflow Exploits
Socket Programming Essentials
Raw Sockets in Networking
RFC 791: Internet Protocol (IP) Specification
RFC 793: Transmission Control Protocol (TCP) Specification
ARP Spoofing: Techniques and countermeasures
DDoS Mitigation Strategies

Share via EmailShare on XShare on LinkedInShare on Reddit

GNU Development Tools

Using gdb

Registers

General Purpose

General Purpose (Extended)

Instruction Pointer Register

For 64-bit Registers

Flags

Segment in Memory

Intel x86-64 Assembly Syntax

gdb "examine" Command

Shorthand Syntax

Format Options:

Shared Library Code

Commands

Breakpoints in Shared Libraries

Memory Segmentation

Compiled Program's Memory

Segments Overview

a) Text Segment

b) Data & bss Segment

c) Heap Segment

d) Stack Segment

File Access

Two Ways to Access Files in C:

File Descriptor

Common Functions

Exploitations

i) Overwrite variable value with overflow

ii) Overwrite the entire stack memory

iii) Heap Overflow

iv) Format String Exploits

Common Format Specifiers:

Networking

OSI Model Layers

i) Socket

Types of Sockets:

Syntax:

Common Socket Functions:

ii) sockaddr Structure

ii) Network Sniffing

Types of Networks:

Example Command:

Packet Sniffing

Accessing Lower Layer Packets

Introduction

Packet Sniffing with libpcap

Ethernet Header (Defined in if_ether.h)

IP Header (Defined in ip.h)

TCP Header (Defined in tcp.h)

Methods:

Denial of Service (DoS) Attacks

(a) SYN Flooding

(b) Ping of Death

Other Network Attacks

(c) Teardrop

(d) Ping Flooding

(e) Amplification Attacks

(f) Distributed DoS Flooding (DDoS)

(iv) TCP/IP Hijacking

TCP Packet Sniffing and Spoofing

TCP Sequence Diagram:

Key Observations:

Steps:

Result:

Port Scanning

Purpose:

Techniques:

(a) Stealth SYN Scan / Half-open Scan

(b) FIN, X-mas, and Null Scans

(c) Spoofing Decoys

Advanced Techniques:

(d) Idle Scanning

Proactive Defense (Shroud or Shield)

Purpose:

Procedure:

References & Related Topics

Share this

Packet Sniffing with `libpcap`

Ethernet Header (Defined in `if_ether.h`)

IP Header (Defined in `ip.h`)

TCP Header (Defined in `tcp.h`)