Linux Memory Deep Dive - Part 1: How Much Memory Do You Really Have?

Series: Linux Memory Deep Dive, from free -h to kernel internals, built entirely on hands-on experiments.
Goal: cover everything about Linux memory that can come up in a FAANG/MAANG-level systems or SRE interview.

Series Roadmap

Lab Environment

All experiments in this part were run on a small test VM:

A small VM is intentional: later in the series we will deliberately run it out of memory.

Table of Contents

1. First look: free
2. Where free gets its numbers: /proc/meminfo
3. Where did 405 MiB go? Boot-time memory accounting
4. Proof that cache is not "used" memory: drop_caches
5. Page cache in action: Direct I/O, cold reads, warm reads, zero-copy
6. Memory over time: vmstat
7. Interview questions
8. Key takeaways

1. First look: free

free -h

               total        used        free      shared  buff/cache   available
Mem:           1.6Gi       354Mi       1.0Gi       4.8Mi       361Mi       1.3Gi
Swap:          2.0Gi          0B       2.0Gi

free does not compute anything itself. It simply reads /proc/meminfo, a file the kernel generates on the fly. The same is true for top, htop, vmstat and most other memory tools: they are wrappers around /proc.

What each column means

"Linux ate my RAM"

Linux deliberately fills idle RAM with cache, because reading from RAM is orders of magnitude faster than reading from disk. When processes need memory, the cache is dropped.

So a low free value is normal. The real warning signs are a low available value, growing swap activity, major page faults and OOM kills.

2. Where free gets its numbers: /proc/meminfo

cat /proc/meminfo

MemTotal:        1682240 kB
MemFree:          860248 kB
MemAvailable:    1328216 kB
Buffers:           20992 kB
Cached:           577952 kB
SwapCached:            0 kB
Active:           342256 kB
Inactive:         322972 kB
Active(anon):      78660 kB
Inactive(anon):        0 kB
Active(file):     263596 kB
Inactive(file):   322972 kB
Unevictable:       13612 kB
Mlocked:            9612 kB
SwapTotal:       2097148 kB
SwapFree:        2097148 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         79908 kB
Mapped:            80916 kB
Shmem:              4928 kB
KReclaimable:      23840 kB
Slab:              65112 kB
SReclaimable:      23840 kB
SUnreclaim:        41272 kB
KernelStack:        1768 kB
PageTables:         2536 kB
...
CommitLimit:     2938268 kB
Committed_AS:     283948 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       14868 kB
...
Hugepagesize:       2048 kB
DirectMap4k:       63336 kB
DirectMap2M:     2033664 kB

(Taken about two hours after section 1, so buff/cache has grown. Memory is alive.)

2.1 Mapping free columns to /proc/meminfo

2.2 How much RAM does the VM actually have?

DirectMap4k:       63336 kB
DirectMap2M:     2033664 kB

The sum is 2097000 kB ≈ 2 GiB: all physical memory the kernel has mapped into its own address space. So the VM really has 2 GiB, but MemTotal shows only 1.6 GiB. We find out why in section 3.

(Side note for Parts 3 and 5: almost all of it is mapped with 2 MiB pages, not 4 KiB, to save TLB entries.)

2.3 Anonymous vs file-backed memory

Active(anon):      78660 kB     Active(file):     263596 kB
Inactive(anon):        0 kB     Inactive(file):   322972 kB
AnonPages:         79908 kB     Mapped:            80916 kB

• Anonymous memory has no file behind it: heap (malloc), stack. The kernel cannot simply drop it, because the data would be lost. The only way to evict it is to write it to swap.
• File-backed memory is a copy of a file: page cache, binaries, shared libraries.
• A clean page can be dropped immediately and re-read from disk later.
• A dirty (modified) page must be written back to disk first.

AnonPages is the anonymous memory of all processes. Mapped is file pages currently mapped into processes with mmap (binaries, .so libraries).

Buffers is the page cache of raw block devices: filesystem metadata (superblocks, bitmaps, the ext4 journal) and anything that reads /dev/sdX directly. Cached is the page cache of regular files. On a modern system Buffers is small; file data always lands in Cached.

Shmem is a special case: tmpfs and shared memory pages are counted in Cached, but the kernel keeps them on the anon LRU lists, because like anonymous memory they have no file on disk and can only be evicted to swap.

If you add it up, Active(anon) + Inactive(anon) is 78660 kB, but AnonPages + Shmem is 84836 kB. The ~6 MiB gap is anon and shmem pages that are not on the anon LRU at all. They sit on the Unevictable list (13612 kB here, most of it Mlocked, 9612 kB): locked pages cannot be evicted, so the kernel does not waste time scanning them. A few pages can also sit briefly in per-CPU batches before they are added to an LRU list.

Active / Inactive are the kernel's LRU lists: recently used ("hot") pages vs pages nobody has touched for a while ("cold"). Under pressure the kernel reclaims inactive pages first. Here Inactive(file) is 322 MiB, so the cheapest option, dropping cold file cache, is readily available. (Covered in depth in Part 7.)

2.4 Why MemAvailable ≠ MemFree + Cached

MemAvailable is an estimate, not a sum. The kernel computes it in si_mem_available() (mm/show_mem.c):

available = MemFree − totalreserve_pages
          + pagecache   − min(pagecache / 2,   low watermark)
          + reclaimable − min(reclaimable / 2, low watermark)

pagecache   = Active(file) + Inactive(file)
reclaimable = KReclaimable (SReclaimable + other reclaimable kernel memory)

• totalreserve_pages is memory the kernel keeps for itself (high watermarks plus per-zone reserves), so it is never "available".
• It assumes up to half of the file cache and half of the reclaimable slab might be in active use and should not count.

The classic trap: Shmem (tmpfs, /dev/shm) is counted inside Cached, but it has no copy on disk, so it cannot be dropped. It can only be swapped. MemAvailable handles this correctly (it uses the file LRU lists, not Cached), but a script that computes MemFree + Cached does not. A host can show gigabytes of "cache" that is really pinned tmpfs data.

2.5 Kernel memory

Slab:              65112 kB   = SReclaimable 23840 + SUnreclaim 41272
KernelStack:        1768 kB
PageTables:         2536 kB
VmallocUsed:       14868 kB

• Slab: caches of kernel objects (inodes, dentries, network buffers). SReclaimable can be freed; SUnreclaim cannot. Steadily growing SUnreclaim is a classic sign of a kernel memory leak (Part 9).
• KernelStack: one kernel stack per thread (typically 16 KiB each).
• PageTables: page tables of all processes (Part 3).
• VmallocUsed: memory allocated with vmalloc.

VmallocTotal: 34359738367 kB (32 TiB) is not memory. It is the size of the virtual address range reserved for vmalloc.

2.6 Fields we will come back to

Summary of this VM: ~80 MiB is used by processes (anon), ~82 MiB by the kernel itself (slab 64 + vmalloc 14.5 + page tables 2.5 + stacks 1.7), and ~600 MiB is reclaimable cache.

3. Where did 405 MiB go? Boot-time memory accounting

dmesg | grep -iE 'Memory:|crashkernel'

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-7.0.0-30-generic root=UUID=... ro crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
[    0.008867] crashkernel reserved: 0x000000006b000000 - 0x000000007f000000 (320 MB)
[    0.011055] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.011057] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]
[    0.099092] Freeing SMP alternatives memory: 56K
[    0.207957] Memory: 1632348K/2096608K available (23799K kernel code, 4952K rwdata, 16764K rodata, 5292K init, 5856K bss, 457952K reserved, 0K cma-reserved)
[    0.627303] Freeing initrd memory: 33820K
[    0.688959] Freeing unused decrypted memory: 2028K
[    0.690161] Freeing unused kernel image (initmem) memory: 5292K
[    0.692246] Freeing unused kernel image (text/rodata gap) memory: 776K
[    0.693310] Freeing unused kernel image (rodata/data gap) memory: 1668K

3.1 The kernel command line

crashkernel is specified as a scale: with 2–4 GiB of RAM reserve 320 MiB, with 4–32 GiB reserve 512 MiB, and so on. Our VM has 2 GiB, so 320 MiB is reserved, at physical addresses 0x6b000000–0x7f000000 (0x14000000 bytes = exactly 320 MiB).

A subtle detail: the kernel sees only 2096608K, which is less than 2 GiB, yet the 2G-4G range still matched. That is because parse_crashkernel_mem() rounds system RAM up to 128 MiB before comparing, exactly to absorb firmware holes like ours.

The current command line is always available in /proc/cmdline.

3.2 The accounting

How to read the Memory: line: 1632348K/2096608K available (... 457952K reserved ...).

• 2096608K: all RAM pages the kernel found (physpages).
• 457952K reserved: pages taken before the page allocator started (physpages − totalram).
• 1632348K available: pages free at that instant. Early boot code had already allocated 6308K, which still belongs to MemTotal, so this is not the starting point for MemTotal.

MemTotal is totalram: physpages − reserved, plus what is given back later.

The 457952K reserved:

(Freeing SMP alternatives memory: 56K happened before the Memory: line, so it is already included in totalram and is not added again.)

3.3 What these items are

• crashkernel / kdump. If the main kernel panics, a second small kernel boots inside this reserved memory and saves a dump (vmcore) of the crashed kernel for post-mortem analysis. The memory must be reserved up front, and it sits idle while everything works. On this VM it is 16% of all RAM. Small VMs without kdump often drop it; large servers usually keep it.
• Kernel image sections: kernel code (text), rodata (read-only data), rwdata (writable globals), bss (zero-initialized globals) and init (code needed only during boot, freed afterwards).
• initrd / initramfs: a small temporary in-RAM filesystem used to load drivers and mount the real root filesystem. Freed once the system is up.
• struct page: for every 4 KiB physical page the kernel keeps a ~64-byte descriptor (owner, reference count, LRU list, flags…). That is ~1.56% of RAM: ~32 MiB on this VM, ~16 GiB on a 1 TiB server.

Bottom line: of 2 GiB of physical memory, ~405 MiB never shows up in MemTotal: 320 MiB for kdump, ~46 MiB for the kernel image (after init code is freed), ~32 MiB for page descriptors, ~7 MiB for other early allocations. That is why MemTotal is always smaller than the RAM you paid for.

4. Proof that cache is not "used" memory: drop_caches

sync; echo 3 > /proc/sys/vm/drop_caches; free -h

• sync flushes dirty pages to disk, because dirty data cannot be dropped.
• drop_caches: 1 = page cache, 2 = reclaimable slab (dentries, inodes), 3 = both.

               total        used        free      shared  buff/cache   available
Mem:           1.6Gi       311Mi       1.3Gi       4.8Mi       104Mi       1.3Gi
Swap:          2.0Gi          0B       2.0Gi

Before vs after

What this proves

1. Cache moved into free. ~500 MiB of cache became free memory. Nothing was "eaten".
2. available did not change. The kernel already counted that cache as available. drop_caches gave applications zero additional usable memory. It only moved numbers between columns. This is why monitoring must use available, not free.
3. used dropped slightly because MemAvailable is conservative: it does not count all reclaimable cache. Once the memory is truly free, it is counted in full.
4. buff/cache did not reach zero. drop_caches only drops clean, unmapped pages. What remains: mapped binaries and libraries of running processes (~80 MiB), Shmem (4.8 MiB), in-use slab objects, some buffers.

Do not do this in production. It does not "free memory". It throws away useful cache and immediately causes extra disk I/O as everything is re-read. drop_caches is a tool for benchmarks (cold-cache tests) and debugging.

5. Page cache in action

5.1 Writing a file around the cache: Direct I/O

dd if=/dev/zero of=/root/testfile bs=1M count=300 oflag=direct; free -h

300+0 records in
300+0 records out
314572800 bytes (315 MB, 300 MiB) copied, 0.0846984 s, 3.7 GB/s
               total        used        free      shared  buff/cache   available
Mem:           1.6Gi       354Mi       784Mi       4.8Mi       652Mi       1.3Gi
Swap:          2.0Gi          0B       2.0Gi

oflag=direct opens the file with O_DIRECT: data goes straight to the disk, bypassing the page cache.

Two surprises:

1. buff/cache grew by ~547 MiB, but the file is only 300 MiB. The numbers don't match, so "the file landed in cache" was not the only explanation.
2. 3.7 GB/s is suspiciously fast for this VM's virtual disk.

5.2 Verify, don't guess: fincore

fincore reports how many pages of a specific file are in the page cache. On Ubuntu/Debian it lives in the util-linux-extra package.

fincore /root/testfile; df -T /root

RES PAGES  SIZE FILE
 0B     0  300M /root/testfile
Filesystem     Type 1K-blocks    Used Available Use% Mounted on
/dev/sda2      ext4  32844560 5738676  25411932  19% /

Zero pages of the file are cached. O_DIRECT worked correctly on ext4; the cache growth came from unrelated background activity between measurements. Lesson: when measuring a live system, check the specific object, not the global totals.

Modern fincore uses the cachestat(2) syscall (Linux 6.5+), falling back to mincore(2). cachestat is cheaper and also reports dirty, writeback and recently evicted pages.

Why 3.7 GB/s? O_DIRECT bypasses only the guest's page cache. Below it are more layers:

dd → guest page cache (bypassed) → virtio disk → QEMU → host (cache mode, ZFS/LVM) → physical disk

The host may cache the writes, and we wrote zeros, which compression or thin provisioning can store almost for free.

Buffered vs Direct I/O

Databases use O_DIRECT to avoid double caching: the same data in both their buffer pool and the kernel page cache. PostgreSQL is the well-known counterexample: it relies on the page cache.

Note: O_DIRECT is not O_SYNC. It skips the page cache, but the data can still sit in the device's volatile write cache, and file metadata (size, block allocation) may not be on disk yet. Durability still needs fsync()/fdatasync() or O_DSYNC.

5.3 Cold read: filling the cache

{ time cat /root/testfile > /dev/null; } 2>&1; fincore /root/testfile; free -h

real	0m0.058s
user	0m0.000s
sys	0m0.054s
  RES PAGES  SIZE FILE
 300M 76800  300M /root/testfile
               total        used        free      shared  buff/cache   available
Mem:           1.6Gi       347Mi       427Mi       4.8Mi       1.0Gi       1.3Gi
Swap:          2.0Gi          0B       2.0Gi

• The whole file is now cached: 76800 pages = 300 × 1024 × 1024 / 4096, confirming the 4 KiB page size.
• free dropped, buff/cache grew, and available again did not change.
• real − (user + sys) = time spent waiting. Here 0.058 − 0.054 ≈ 4 ms: the process barely waited for the disk (the host cache made this "cold" read fast). On a slow disk you would see real = 3 s with sys = 0.05 s, which means the process was sleeping in D state (uninterruptible I/O wait).

When a file is read sequentially, the kernel also prefetches upcoming blocks. This is called read-ahead (Part 6).

5.4 Warm read: served from RAM

{ time cat /root/testfile > /dev/null; } 2>&1

real	0m0.008s
user	0m0.002s
sys	0m0.004s

7× faster, with no disk access at all. But 300 MB in 4 ms of system time is ~75 GB/s, faster than a plain memory copy. Something else is going on.

5.5 The zero-copy surprise

cat --version | head -1; dd if=/root/testfile of=/dev/null bs=1M

cat (uutils coreutils) 0.8.0
300+0 records in
300+0 records out
314572800 bytes (315 MB, 300 MiB) copied, 0.0256231 s, 13 GB/s

Ubuntu 25.10+ ships the Rust rewrite of coreutils (uutils), whose cat uses splice() on Linux instead of read() + write(). splice() moves data between file descriptors inside the kernel, without copying it into the process. When the destination is /dev/null, the data is simply discarded in the kernel. dd, using plain read(), really copies the data into user space, and lands at 13 GB/s, a typical memory-copy speed.

(dd here is the uutils build too; we will confirm the exact syscalls of both with strace -c in Part 12.)

Three reads of the same file

Compare only rows that differ in one thing:

• A vs B: same tool, same syscall, cold vs warm: ~7×. This is the value of the page cache.
• C vs B: same warm file, read() vs splice(): ~3×. This is the cost of copying into user space, which zero-copy avoids.

On bare metal with an HDD (~150 MB/s), the cold read would take ~2 s, and the A vs B gap would be ~250×, not 7×.

Lesson: before explaining a number, make sure you know what you actually measured. Without checking cat --version, this post would claim that the page cache delivers 75 GB/s.

6. Memory over time: vmstat

Everything so far was a snapshot. vmstat shows memory over time.

vmstat -w 1 5

--procs-- -----------------------memory---------------------- ---swap-- -----io---- -system-- ----------cpu----------
   r    b         swpd         free         buff        cache   si   so    bi    bo   in   cs  us  sy  id  wa  st  gu
   1    0            0       437932        15412      1027904    0    0    97   123  127    0   1   0  99   0   0   0
   0    0            0       437932        15412      1027912    0    0     0     0  261  153   0   0 100   0   0   0
   0    0            0       437932        15412      1027912    0    0     0     0  270  156   0   1  99   0   0   0
   0    0            0       437932        15412      1027912    0    0     0     0  199  121   0   0 100   0   0   0
   0    0            0       437932        15412      1027912    0    0     0     0  111   53   0   0 100   0   0   0

Note: in the first line, the rate columns (si, so, bi, bo, in, cs, CPU %) are averages since boot. The memory columns are a current snapshot. Read rates from line 2 onward.

Our VM: no swap activity, no I/O, CPU 99–100% idle. A good baseline for the experiments ahead.

Signs of memory pressure in vmstat

1. si / so consistently non-zero → active swapping (thrashing if severe);
2. free low and cache keeps shrinking → the kernel is evicting cache to make room;
3. b > 0 and wa rising → processes are waiting for I/O, including swap-ins;
4. bi rising under a steady workload → evicted files are being re-read from disk.

A large swpd on its own is not a problem: if si/so are near zero, the swapped-out pages are simply cold, and the freed RAM is doing useful work as cache.

7. Interview Questions

Q: free shows 100 MiB free on a 64 GiB server. Is that a problem? Not necessarily. Look at available, not free. If available is high, the memory is used by reclaimable cache, which is good. A problem starts when available is low, swap activity is high, major page faults grow, and OOM kills appear in the logs.

Q: What is the difference between MemFree and MemAvailable? MemFree is completely unused memory. MemAvailable is the kernel's estimate of memory obtainable without swapping, including reclaimable cache and slab, minus reserves. Monitoring and alerting should use MemAvailable.

Q: How does eviction differ for anonymous and file-backed pages? Clean file pages are simply dropped; dirty ones are written back first. Anonymous pages can only be written to swap. Without swap, anonymous memory cannot be evicted at all, which brings OOM closer.

Q: Why can't you treat all of Cached as free memory? It includes Shmem (tmpfs, /dev/shm), which has no backing file and cannot be dropped, and some cache is in active use.

Q: How does free compute used? It reads /proc/meminfo. In recent procps-ng, used = MemTotal − MemAvailable; older versions used total − free − buffers − cache. That's why the columns don't always add up.

Q: The server has 64 GiB, but MemTotal shows 62 GiB. Where is the rest? Reserved before the kernel starts handing out memory: firmware/ACPI regions, the crashkernel reservation for kdump, the kernel image, and the struct page array (~1.5% of RAM). In a VM, a balloon driver can take more. Check with dmesg | grep Memory:, /proc/cmdline, /proc/iomem.

Q: A colleague wants a cron job running echo 3 > /proc/sys/vm/drop_caches every hour. Thoughts? Bad idea. It does not increase available (we demonstrated it), it hurts performance by forcing re-reads from disk, and if the system really lacks memory it treats the symptom instead of finding the actual consumer.

Q: Why didn't buff/cache drop to zero after drop_caches? Only clean, unmapped pages are dropped. Mapped binaries and libraries of running processes, tmpfs/shmem, dirty pages and in-use slab objects remain.

Q: You benchmark a disk in a VM with dd ... oflag=direct and get 3 GB/s. Do you trust it? No. O_DIRECT only bypasses the guest page cache; the hypervisor cache, controller cache, compression and thin provisioning sit below. /dev/zero is also unrealistic data. Use fio with random data, a working set larger than all caches, and an understanding of the full storage stack.

Q: Why do databases often use O_DIRECT? What are the downsides? Pros: the database controls caching itself, no double caching, more predictable latency. Cons: it must implement its own caching and read-ahead, buffers must be aligned, and every uncached read hits the disk.

Q: How can you tell from time whether a program is CPU-bound or I/O-bound? user + sys ≈ real → CPU-bound. real ≫ user + sys → mostly waiting (disk, network, locks). High sys → lots of time in the kernel (syscalls, page faults, copying).

Q: Why are nginx and Kafka so fast at serving files? Page cache plus zero-copy: sendfile() moves data from the page cache straight to the socket, with no copy into user space and fewer context switches. Kafka intentionally relies on the page cache instead of its own cache.

Q: How do you detect memory pressure with vmstat? Watch the trend, not absolute values: sustained non-zero si/so, shrinking cache, rising b and wa. Ignore the rates in the first line, because they are averages since boot.

Q: What is the difference between Buffers and Cached? Buffers is the page cache of raw block devices: filesystem metadata and direct reads of /dev/sdX. Cached is the page cache of regular files, and it also includes Shmem. Both are reclaimable, except the Shmem part.

Q: How does the kernel compute MemAvailable? MemFree minus the kernel's reserves (totalreserve_pages), plus the file LRU pages and reclaimable kernel memory (KReclaimable), each minus whichever is smaller: half of it or the low watermark. It is a heuristic, and it deliberately does not count all cache as reclaimable.

Q: Does O_DIRECT guarantee the data is on disk when write() returns? No. It bypasses the page cache, not the device's write cache, and it does not persist file metadata. You still need fsync()/fdatasync() or O_DSYNC.

Q: Committed_AS is larger than CommitLimit. Is the system about to fail allocations? Only if vm.overcommit_memory=2. With the default 0 (heuristic overcommit), CommitLimit is not enforced, and Committed_AS just shows how much memory has been promised. The risk then shows up later, as OOM kills when the promises are actually used.

Q: What is steal time? Time when a vCPU was ready to run but the hypervisor gave the physical CPU to another VM. High st means an overloaded host or noisy neighbours, and you cannot fix it from inside the guest.

8. Key Takeaways

• free, top and vmstat are views over /proc/meminfo. Know the fields behind each column.
• Monitor available, not free. Low free is normal; Linux uses idle RAM as cache.
• MemTotal < physical RAM: on our 2 GiB VM the kernel reserved ~405 MiB at boot (320 MiB for kdump alone).
• Memory is either anonymous (evictable only to swap) or file-backed (droppable or written back).
• drop_caches moves cache into free but does not increase available. It is a benchmarking tool, not a fix.
• Same cat, cold vs warm: the page cache gave ~7× even in a VM. Same warm file, read() vs splice(): zero-copy gave another ~3×.
• Always verify what you measured: fincore, df -T, cat --version turned two wrong assumptions into correct conclusions.
• For memory pressure, watch trends: si/so, shrinking cache, b, wa.

Next, Part 2: Virtual memory. What does a process actually see? We'll open /proc/<pid>/maps, explore code, heap, stack and mmap regions, ASLR, and learn the difference between VSZ, RSS, PSS and USS, and which one to trust.