Diving through the shell: the command interpreter 🤿

How does the shell work internally? What happens when we enter a command? What happens when we enter ls -l? What happens if we use an expansion like *.c?

Authors: Jean Pierre, Stiven Gonzalez Mahecha, and Diana Henao.

Photo by Joan Gamell on Unsplash

The shell is a program that allows interaction between the user and the operating system. This interaction occurs thanks to the shell interpreting the commands entered and executing the tasks assigned by those commands. But shell there is not only one. Over the years, several shells have been developed and these different shells can be used simultaneously on the same computer.

The first shell was the Bourne shell (sh) and this shell handles the main features of all shells (see shell types figure). From this, features such as aliases, history… have been added. Currently, the most used shell in Linux is bash.

Four of the main shells exist or have existed. All the shells perform similar tasks, but the csh, ksh, and bash provided additional interactive features (Kerrisk, M. 2010).

As we said above, the way to interact with the operating system is through this type of command interpreter. Internally, the shell communicates with the operating system Kernel.

The Kernel is in charge of managing and allocating the computer’s resources.

The Kernel takes care of functions such as: planning the execution of programs; the management of memory equitably and efficiently between the processes; provide a file system that allows among others the creation, deletion, updating of files; the creation and completion of processes under available computational resources.

Additionally, the kernel provides a system call application programming interface (API). This allows you to execute tasks with the use of system calls.

The system call is the fundamental interface between an application and the Linux kernel (Linux manual page — syscalls(2)).

System calls allow requesting services from the operating system kernel. These are the only entry points to the kernel system. The system calls can control processes, manage files and devices, obtain information, and communicate with the Kernel. Some of these calls can be seen in the following table:

Some frequently used system calls. Modified from GeeksforGeeks — system calls (https://www.geeksforgeeks.org/introduction-of-system-call/) and Pediaa (https://pediaa.com/what-is-the-difference-between-api-and-system-call/)

When executing a program, the computer operates in user mode. When the program requires computational resources, the program sends a request to the kernel and switches from operating in User mode to operating in Kernel mode. It is the system calls that allow these requests.

At this point, many of you will wonder: why if we were talking about the shell, we started talking about Kernel and system calls?

Because the shell works using system calls, which allows the entered commands to communicate and execute the tasks.

Initially, the shell upon receiving an input (command(s) and/or argument(s)) can work in an interactive or non-interactive mode. In interactive mode, the terminal input processes the input line by line and allows you to continue editing lines. In this mode, the lines are infinite until the “EOF” or “exit” instruction. additionally, the prompt (PS1) is shown in the terminal, and after executing the entered instruction, the prompt is shown again on the screen, waiting for a new instruction.
When the terminal works in a non-interactive mode, the shell takes care of reading the characters individually because they are not grouped in lines.
Unlike the interactive mode, it is necessary to press the enter key to indicate the completion of the instruction.

The command and arguments, which are entered as a line to the shell, are fragmented into arguments separated by a specific character (something called tokens). In the case of the example command “ls -l *.c” the separating character is the space “ ”. Each of these fragments is stored, as a token. This separation allows the shell to verify what the command is:

1. An executable program, available in some of the $PATH variable directories;
2. A command built into the shell itself, known as shell builtins (cd, env, exit, history, among others);
3. A shell function built into the environment;
4. An alias, built from other commands.

Depending on which of the previous options is the command received, the shell can: execute the function directly, search in the $PATH directories, or search in the current directory for an executable file with the function, command or program to be executed.

Upon finding it, the shell is responsible for creating a child process from the parent process that is calling it. This step allows the program to continue running in interactive mode without ending the main program that contains it (known as the parent process). Once the child process is created, the instruction is executed using a system call known as execve(). When the function is executed and finished, the parent process is in charge of cleaning the memory, printing a line break, and waiting for a new command. This is how the shell works generally, and yes! the processes are carried out by different systems call.

A possible general scheme of how a simple shell works. As always in programming, there is more than one solution!

Now we are going to see an example with “ls -l *.c”

First, for the specific case of “ls -l *.c” we have to talk about expansions as well. Expansions are a process that takes place before the order that has just been submitted is executed. With the expansion, some of the entered parameters are expanded by a bit more before the shell carries out the entire process seen above.

This is what our example looks like in the terminal :)

To use an expansion it is necessary to use wildcards (characters that allow one or more characters to be represented). Each wildcard has its definition. For the specific case of the character “*.c”, the wildcard “*” followed by the .c allows searching in the current directory for all filenames that have a .c extension, regardless of the name.

Initially, when using the wildcard an expansion is generated. Doing a pathname expansion or filename expansion matches every filename “.c” in the current directory. So what the command receives is not “*”, it is the complete list of filenames that meet the “.c” criterion.

When the enter key is pressed, the shell can expand automatically, so the entered command “ls” never sees the “*”, it only sees the result of the expansion. After this pathname expansion, the process of reading and executing the commands follows the process seen above.

So what happens when we run ls -l alone? All files, not hidden, available in the current folder are displayed as a list:

What happens when you type ls -l?

And what happens if we include the wildcard with a specific .c extension? Only those files that comply with what is established when using the wildcard are listed. In this case, we want to obtain a list of files with a .c extension contained in the current folder:

And now? What happens if you use wildcard “*” additionally to ls -l?

In summary: when entering a command in the shell, a series of system calls are generated that allow execution in kernel mode. In this mode, it is possible to allocate the necessary computational resources. In the case of using wildcards, an expansion is executed first.

References:

Shotts Jr, WE. 2013. The Linux command line: A complete introduction. No starch press. Second internet Edition.

Kerrisk, M. 2010. The Linux programming interface: A Linux and UNIX System programming handbook. No starch press.