Making System Calls in x86-64 Assembly

Introduction

In the previous article, we learned to use gdb and used it to debug our crashing program. Eventually, we discovered that after executing the last instruction, the CPU didn’t know the program had ended. It continued reading and executing past the end of the .text section in memory, causing a crash. So, we need some way to make our process exit or stop the execution before that happens. How can we do that?

When we ran our program using the shell command ./false, it was the shell that invoked the fork and execve system calls. These created a new process, loaded our program into memory, and scheduled it for execution on the CPU. Similarly, to terminate our program gracefully, we need to invoke another system call that tells the kernel our process is done.

This system call to exit a process is called “exit”. When we write code in high-level languages, the runtime automatically invokes it after the main function returns. However, when writing freestanding assembly, we need to do it ourselves. For that, we need to learn how to call syscalls from assembly.

In this part, we will:

Understand what system calls are
Learn how to invoke them in assembly
Fix our crashing program step-by-step
Write a second assembly program using getpid
Hands-on exercise: a limited version of the kill command

Recap: If you haven’t seen the previous articles in the series, here’s what you have missed:

This article is part of a paid subscriber series.
If you’re enjoying the content, please consider upgrading to a paid plan to unlock the rest of this series. Paid subscribers also get discounted access to courses and books, and the rest of the archive.

Alternatively, you can purchase an ebook version of this series. (If you’re already a paid subscriber, email me for a discounted link.)

I Want the PDF

Understanding System Calls

Before we learn how to invoke system calls, let’s first understand why they exist.

Modern operating systems serve two roles: they manage the execution of programs on the CPU and provide safe, unified access to hardware resources like files, memory, and networks. But, application code cannot directly access these hardware features. Why not?

There are three main reasons:

Hardware abstraction: Devices vary widely in design and interface. The OS hides this complexity by exposing a uniform way to access them. Whether you’re reading from an SSD or a magnetic disk, you use the same system call (read), and the OS handles the details.
Portability: Most modern OSes follow the POSIX standard, which defines a consistent set of system calls. If your application uses only POSIX-compliant syscalls, it can compile and run on any compliant OS with minimal changes.
Security: If user programs could directly access memory or I/O devices, they could corrupt system state or access other processes’ data. System calls act as a controlled gateway; only kernel code (running in a higher privilege level) is allowed to interact directly with hardware.

This separation of privilege is enforced by the CPU. On x86, the kernel runs in ring 0 (full privilege), while user programs run in ring 3 (restricted mode). All system calls are implemented inside the kernel at ring 0. To invoke them from ring 3, user space programs need a way to trigger a transition into kernel mode using a mechanism provided by the CPU.