分类 Linux 相关 下的文章

Linux 文件系统学习摘要

这是公司内学习 Linux Kernel 的第13章, 关于文件系统的部份.

inode

In Linux and other Unix-like operating systems, an inode (index node) is a data structure that stores information about a file or directory except its name and its actual data. Each file or directory has an inode that contains important metadata about the file or directory.

The information stored in an inode includes:

  1. File size
  2. Device ID
  3. User ID (owner)
  4. Group ID
  5. File permissions
  6. File creation, modification, and access times
  7. Number of links (how many file names point to this inode)
  8. Pointers to the disk blocks that store the file's data

The inode number is a unique identifier for the inode within the filesystem. You can view the inode information of a file or directory using the ls -i or stat command in the terminal.

dentry

In the Linux kernel, a dentry (directory entry) is a data structure that represents a specific inode in the cache. It's a key component of the Virtual File System (VFS) layer, which provides a common interface for all file systems.

The dentry object contains information about the inode, the file name, and pointers to the parent and child dentries, forming a dentry tree that represents the directory hierarchy. This structure allows the kernel to quickly look up files and directories, improving the efficiency of file system operations.

Dentries are stored in a dentry cache (dcache), which keeps track of recently accessed dentries to speed up subsequent file and directory lookups. When a file is accessed, the kernel first checks the dcache. If the dentry is found, the kernel can access the file directly without having to traverse the entire file system, which can significantly improve performance.

stat 命令

stat 命令显示文件, 路径, 文件系统信息. 具体看下面例子.

# 显示 /tmp 目录所在的文件系统信息
$ stat -f /tmp/
File: "/tmp/"
    ID: 7ae4c0a51947813 Namelen: 255     Type: ext2/ext3
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 122221576  Free: 73122076   Available: 66895184
Inodes: Total: 31121408   Free: 28709496

例子

当我们新建一个空文件和一个空目录的时候, 可以看到如下:

supra@suprabox:~/Downloads$ touch x
supra@suprabox:~/Downloads$ stat x
  File: x
  Size: 0             Blocks: 0          IO Block: 4096   regular empty file
Device: fd01h/64769d    Inode: 10748017    Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/   supra)   Gid: ( 1000/   supra)
Access: 2023-12-01 21:05:46.177415933 -0800
Modify: 2023-12-01 21:05:46.177415933 -0800
Change: 2023-12-01 21:05:46.177415933 -0800
 Birth: 2023-12-01 21:05:46.177415933 -0800
supra@suprabox:~/Downloads$ mkdir y
supra@suprabox:~/Downloads$ stat y
  File: y
  Size: 4096          Blocks: 8          IO Block: 4096   directory
Device: fd01h/64769d    Inode: 11153180    Links: 2
Access: (0775/drwxrwxr-x)  Uid: ( 1000/   supra)   Gid: ( 1000/   supra)
Access: 2023-12-01 21:06:28.061716185 -0800
Modify: 2023-12-01 21:06:28.061716185 -0800
Change: 2023-12-01 21:06:28.061716185 -0800
 Birth: 2023-12-01 21:06:28.061716185 -0800

下面是对x这个空文件输出的详细解释:

  1. File: x:文件名是x。
  2. Size: 0:文件大小是0字节,因为你刚创建了这个文件但还没有写入任何内容。
  3. Blocks: 0:文件占用的块数是0,这与文件大小为0是一致的。
  4. IO Block: 4096:文件系统的I/O块大小是4096字节。
  5. regular empty file:这是一个常规的空文件。
  6. Device: fd01h/64769d:文件所在的设备的设备号。
  7. Inode: 10748017:文件的inode号是10748017。
  8. Links: 1:硬链接数是1,表示只有一个文件名指向这个inode。
  9. Access: (0664/-rw-rw-r--):文件的权限是0664,也就是用户(owner)和组(group)有读写权限,其他人(other)只有读权限。
  10. Uid: ( 1000/ supra):文件的所有者的用户ID是1000,用户名是supra。
  11. Gid: ( 1000/ supra):文件的所有者的组ID是1000,组名是supra。
  12. Access: 2023-12-01 21:05:46.177415933 -0800:文件最后一次被访问的时间。
  13. Modify: 2023-12-01 21:05:46.177415933 -0800:文件最后一次被修改的时间。
  14. Change: 2023-12-01 21:05:46.177415933 -0800:文件状态最后一次被改变的时间。
  15. Birth: 2023-12-01 21:05:46.177415933 -0800:文件的创建时间。

Linux cgroup freezer subsystem 与 SIGSTOP/SIGCONT

在 Linux 上, 使用 cgroup freezer subsystemSIGSTOP/SIGCONT 都能暂停和继续进程的执行. 那么它的区别与联系是什么呢?

为什么需要 SIGSTOP/SIGCONT

暂停与继续进程.
SIGSTOP tells LINUX to pause a process to be resumed later.
SIGCONT tells LINUX to resume the processed paused earlier.

什么是 cgroup freezer subsystem

https://www.kernel.org/doc/Documentation/cgroup-v1/freezer-subsystem.txt
首先它是 cgroup 控制的一个子系统, 所以它有cgroup的层级结构. 它能把一个或者多个进程冻住, 然后就获得了该进程的一个snapshot(checkpoint), docker 的 pause 子命令就是使用 cgroup freezer subsystem 实现的. 当冻住该进程之后, 你能check 找个进程的 /proc 的状态, 内存的状态, 去收集它的运行时信息. 之后解冻之后, 就可以继续运行.

相比来说, 被 SIGSTOP/SIGCONT 的进程自己或者父进程能感知到自己收到了这些 signal.

cgroup freezer subsystem 的一个简单例子

先创建 cgroup freezer 目录, 然后挂载cgroup 文件系统, 然后创建一个文件夹, 然后把进程扔进去, 然后冻结, 解冻.
三种状态:

  1. FROZEN — tasks in the cgroup are suspended. 冻结.
  2. FREEZING — the system is in the process of suspending tasks in the cgroup. 过度, 瞬间状态.
  3. THAWED — tasks in the cgroup have resumed. 解冻.

    # mkdir /sys/fs/cgroup/freezer
    # mount -t cgroup -ofreezer freezer /sys/fs/cgroup/freezer
    # mkdir /sys/fs/cgroup/freezer/0
    # echo $some_pid > /sys/fs/cgroup/freezer/0/tasks
    # echo FROZEN > /sys/fs/cgroup/freezer/0/freezer.state
    # ps aux | grep $some_pid
    # echo THAWED > /sys/fs/cgroup/freezer/0/freezer.state

    为什么需要 cgroup freezer subsystem

进程的跨服务器迁移

整个服务器有可能, 也有可用的商业方案.
单进程, 不可行.

  1. /proc, debugFS 这些伪文件系统不可复制.
  2. 进程号冲突.
  3. 在用的文件描述符.
  4. IP 地址.
  5. socket buff 未读/未发的数据.

D - Uninterruptible Sleep Process

认识 Linux ps 命令的状态列:

$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0 169156 12824 ?        Ss   Nov03   6:05 /lib/systemd/systemd 
root           2  0.0  0.0      0     0 ?        S    Nov03   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   Nov03   0:00 [rcu_gp]
root          14  0.0  0.0      0     0 ?        I    Nov03  10:34 [rcu_sched]
root          68  0.0  0.0      0     0 ?        SN   Nov03   0:00 [ksmd]
root         868  0.0  0.0   2812  1048 ?        Ss   Nov03   0:00 /usr/sbin/acpid
root         875  0.0  0.0 484180 15888 ?        Ssl  Nov03   5:39 /usr/sbin/NetworkManager
root         894  0.0  0.1 917300 28164 pts/0    Ssl+ Nov03   1:52 /usr/local/qualys/cloud-agent 
rtkit       1415  0.0  0.0 155920  2584 ?        SNsl Nov03   0:08 /usr/libexec/rtkit-daemon
supra       2445  0.0  0.0  41472  3876 ?        S<sl Nov03   0:00 /usr/bin/pipewire
supra    2176300  0.0  0.0  15472  4596 pts/2    R+   16:58   0:00 ps aux

从上面的输出可以看到 STAT (state) 列有各种字母组合的状态, 分别代表什么意思呢?

State 含义

https://man7.org/linux/man-pages/man1/ps.1.html

    the state of a process:
               D    uninterruptible sleep (usually IO)
               I    Idle kernel thread
               R    running or runnable (on run queue)
               S    interruptible sleep (waiting for an event to
                    complete)
               T    stopped by job control signal
               t    stopped by debugger during the tracing
               W    paging (not valid since the 2.6.xx kernel)
               X    dead (should never be seen)
               Z    defunct ("zombie") process, terminated but not
                    reaped by its parent

       For BSD formats and when the stat keyword is used, additional
       characters may be displayed:

               <    high-priority (not nice to other users)
               N    low-priority (nice to other users)
               L    has pages locked into memory (for real-time and
                    custom IO)
               s    is a session leader
               l    is multi-threaded (using CLONE_THREAD, like NPTL
                    pthreads do)
               +    is in the foreground process group

虽然上面的各种状态比较详细, 简化一下, 其实主要有: R, S, D, T, Z.
LinuxProcessStateChange.png

关于 T & t Stopped 的状态

从运行或可运行状态,我们可以使用SIGSTOP或SIGTSTP信号将进程置于停止状态(T)。两种信号之间的区别在于,我们使用编程方式发送SIGSTOP,例如运行 kill -STOP {pid} 命令。进程不能忽略此信号,并将进入停止状态。当我们使用键盘CTRL + Z发送SIGTSTP信号, 也可以把它置于T状态。与SIGSTOP不同,进程可以选择忽略此信号并在接收SIGTSTP后继续执行。

在此状态下,我们可以通过发送SIGCONT信号将进程带回运行或可运行状态。

关于 D Uninterruptible 的状态

An uninterruptible sleep state is a sleep state that will not handle a signal right away. It will wake only as a result of a waited-upon resource becoming available or after a time-out occurs during that wait (if specified when put to sleep). It is mostly used by device drivers waiting for disk or network IO (input/output).

When the process is sleeping uninterruptibly, signals accumulated during the sleep will be noticed when the process returns from the system call or trap.

You cannot kill "D" state processes, even with SIGKILL or kill -9. As the name implies, they are uninterruptible. You can only clear them by rebooting the server or waiting for the I/O to respond.

为什么需要 Uninterruptible 状态

对于某些必须在系统可能出现故障或意外断电的情况下确保数据完整性和一致性的系统进程。 例如,当文件系统正在将数据写入磁盘时,如果系统接收到信号或其他事件,则它不能简单地在操作中途停止。 数据必须完整写入,以确保不丢失或损坏。

可以使用不间断睡眠的进程的其他示例包括一些设备驱动程序、备份进程和虚拟内存操作。

简而言之,不间断睡眠用于确保关键系统进程能够完成其任务,而不会出现数据损坏或丢失的风险,即使在发生断电或系统崩溃等意外事件时也是如此。

TASK_KILLABLE state.

这是可中断睡眠进程和不可中断睡眠进程之间的折衷方案。 处于 TASK_KILLABLE 状态的进程仍然不能被通常意义上的中断(即不能强制系统调用返回 EINTR); 然而,处于这种状态的进程可以被终止。 这意味着,例如,通过 NFS 执行 I/O 的进程如果进入楔入状态,则可能会被终止。 并非所有系统调用都实现这种状态,因此某些系统调用仍然有可能卡住不可杀死的进程,但这肯定比以前的情况有所改进。

Cgroup freezer subsystem -> D

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/sec-freezer
this subsystem suspends or resumes tasks in a cgroup.
什么情况下需要 cgroup freezer 子系统

  1. 暂停出问题的进程/线程, 查看它内存状态
  2. 搬家 checkpoint
  3. 暂停消耗资源(CPU/内存/网络)的进程/线程

使用cgroup 挂起进程的例子: https://www.kernel.org/doc/Documentation/cgroup-v1/freezer-subsystem.txt
与 SIGSTOP/SIGCONT 的区别:

  1. 进入的状态不同, SIGSTOP/SIGCONT 进入 stopped (T)状态, freezer 进入 Unineteruptiable sleep D.
  2. 被 SIGSTOP/SIGCONT 的进程能感应到信号, 被 freezer 的进程感应不到自己收到这个信号.

cgroup freezer 子系统和 SIGSTOP/SIGCONT 的区别?

进程的生命周期(from wikipidia):

pLifeCycle.webp

eBPF - 例子 进程在 CPU 的运行时间

probe 点: tracepoint:sched:sched_switch
查看参数:

supra@suprabox:~/work/ebpf/bpftrace$ sudo bpftrace -lv tracepoint:sched:sched_switch
tracepoint:sched:sched_switch
    char prev_comm[16]
    pid_t prev_pid
    int prev_prio
    long prev_state
    char next_comm[16]
    pid_t next_pid
    int next_prio

代码如下:

BEGIN
{
    printf("     usecs         count |      distribution                                  |");
}

tracepoint:sched:sched_switch { 
    $prev = args.prev_pid;
    if (@pid[$prev]) {
        @ns = hist((nsecs - @pid[$prev])/1000);
        delete(@pid[$prev]);
    }
    @pid[args.next_pid] = nsecs;
}

END
{
    clear(@pid);
}

BPF 例子 - 观测某个 tcp 连接的状态变化

给2个参数, 分别是 IP 和端口, 观察符合找个条件的tcp 连接的状态变化.

这个例子中本想同时对比 IP 和 port, 但是对于IP 遇到一个问题.

常见的 IPv4 是这么写的(字符串): 145.23.45.23
$sk->__sk_common.skc_daddr 拿到的地址 (unsigned int32): 0X91172d17.
使用 ntop 函数转换后是一个 inet_t, 虽然文档死硬说是 Sting 表示形式: inet_t = “145.23.45.23”

那么为了比较2个 IP 地址, 你想把 0X91172d17 转成 inet_t = “145.23.45.23”, 但是还不能比, 不能用 String 和 inet_t 比较.
要么把 145.23.45.23 通过 pton 转成 uint8[], 然后再通过强转 (uint32)pton("145.23.45.23")) 再和 0X91172d17 比较.

关键的关键是 pton 的参数一定要是 字符串常量, 变量不行, 因为它要在编译时知道类型. TMD.

#!/usr/bin/env bpftrace

#ifndef BPFTRACE_HAVE_BTF
#include <linux/socket.h>
#include <net/sock.h>
#else
#include <sys/socket.h>
#endif

BEGIN
{
    @host = "127.0.0.1";
    @port = (uint16)80;
    if ("" != str($1)) {
        @host = str($1);
    }
    if ("" != str($2)) {
        @port = (uint16)$2;
    }

    printf("looking for tcp connection related to : %s:%d\n", @host, @port);
    printf("%39s:%-6s %39s:%-6s %-10s -> %-10s\n", "src IP", "src port", "dest ip", "dest port", "old state", "new state");
    @states[1] = "ESTABLISHED";
    @states[2] = "SYN_SENT";
    @states[3] = "SYN_RECV";
    @states[4] = "FIN_WAIT1";
    @states[5] = "FIN_WAIT2";
    @states[6] = "TIME_WAIT";
    @states[7] = "CLOSE";
    @states[8] = "CLOSE_WAIT";
    @states[9] = "LAST_ACK";
    @states[10] = "LISTEN";
    @states[11] = "CLOSING";
    @states[12] = "NEW_SYN_RECV";
}

kfunc:vmlinux:tcp_set_state {
    $sk = ((struct sock *) args->sk);
    $inet_family = $sk->__sk_common.skc_family;
    if ($inet_family == AF_INET) {
      $daddr = ntop($sk->__sk_common.skc_daddr);
      $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
    } else {
      $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
      $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
    }
    $lport = $sk->__sk_common.skc_num;
    $dport = $sk->__sk_common.skc_dport;

    $dport = bswap($dport);

    if (($dport == @port) || ( $lport == @port)) {
        $curState = @states[args.state];
        $key = str($dport);
        $oldState = @keyMap[$daddr, $key];
        if ($oldState == "") {
            $oldState = "NONE";
        }
        @keyMap[$daddr, $key] = $curState;

        printf("%39s:%-6d %39s:%-6d %-10s -> %-10s\n", $saddr, $lport, $daddr, $dport, $oldState, $curState);
    }
}

END {
    clear(@states);
    clear(@keyMap);
    clear(@host);
    clear(@port);
}