netatalk
4.5.0
Free and Open Source Apple Filing Protocol (AFP) Server
The Hidden Cost of Validation: How Netatalk's Directory Cache Optimization Reduces Storage Stack Pressure
When you access a file on a network share, you're triggering a cascade of operations through multiple cache layers. Each layer tries to optimize performance, but they can work against each other in surprising ways. This article explores how Netatalk's probabilistic directory cache validation reduces I/O by working with the storage page cache.
Modern file servers orchestrate multiple cache layers, each with its own purpose and behavior:
graph TB
subgraph "Application Layer"
AFP[AFP Client Request]
NT[Netatalk Server]
DC[Directory Cache
LRU]
CNID[CNID Database]
end
subgraph "Kernel Layer"
PC[Page Cache
LRU]
VFS[VFS Layer]
end
subgraph "Storage Layer"
FS[Filesystem
ZFS/ext4/XFS]
ARC[ZFS ARC Cache
Optional]
DISK[Physical Storage]
end
AFP --> NT
NT --> DC
DC --> CNID
NT --> VFS
VFS --> PC
PC --> FS
FS --> ARC
ARC --> DISK
style PC fill:#ffcccc
style DC fill:#ccffcc
style ARC fill:#ffcccc
Each cache layer uses an LRU (Least Recently Used) algorithm to decide what to keep in memory (ZFS uses ARC instead of LRU). This works well when access patterns match LRU assumptions, but breaks down with certain workloads (especially scan workloads)—and that's where our story begins.
The Linux page cache is incredibly fast and nearly invisible. It automatically caches file data and metadata in RAM, but here's the catch: you can't directly control what stays in cache.
LRU Performance Issues:
Degeneration Under Common Patterns: LRU can perform poorly with certain access patterns. For example, if an application loops over an array larger than the number of pages in the cache, it will cause a page fault for every access, leading to inefficiency (the cache churns and does not hold data long enough to achieve a good hit rate).
Cache Pollution: LRU may evict pages that are frequently accessed but not necessarily the most relevant, which can lead to less optimal cache performance. For example when enumerating paths to move around a directory tree, Netatalk historically (and still does today by default, until you apply the new afp.conf tuning options shown below) performed an IO stat operation every time it reads a dircache entry to validate the file/directory still exists. These IO stat validations allow Netatalk to detect external filesystem changes, but keep unneeded data warm in the page cache, pushing other more important objects off the page cache and increasing disk IO - even though Netatalk has the dircache containing everything needed.
When a client browses a Netatalk network folder, here's the actual sequence:
sequenceDiagram
participant Client as AFP Client
participant Netatalk
participant Disk as Filesystem
participant PageCache as Page Cache
participant DirCache as Directory Cache
Client->>Netatalk: List folder contents (FPEnumerate)
Netatalk->>Disk: readdir() - get entry names
Disk->>Netatalk: Return entry names
loop For EACH entry
Netatalk->>PageCache: stat() each entry
alt Cache miss
PageCache->>Disk: Read inode metadata
Disk->>PageCache: Load metadata
Note over PageCache: Evicts LRU data
end
PageCache->>Netatalk: Return stat data
alt Entry is directory
Netatalk->>DirCache: Check if cached
alt Not in cache
Netatalk->>DirCache: Add directory to cache
end
else Entry is file
Netatalk->>DirCache: Check if cached (if file caching enabled)
alt Not in cache
Netatalk->>DirCache: Add file to cache (if enabled)
end
end
end
Netatalk->>Client: Return file list with metadata
Key insights from enumerate.c:
Current enumeration inefficiency:
Future optimization opportunity - Cache-First Enumeration:
Optimal approach:
Would potentially transform repeated directory browsing from O(n) stat calls to O(n/100) stat calls, eliminating most filesystem I/O.
LRU caches have a fundamental weakness called the scanning problem:
When Netatalk validates thousands of entries, each stat() call:
Parent folder recursion:
Each directory entry, stores its parent ID, and each parent ID has its own directory cache entry, providing a recursive chain (path) to root. When a new directory is accessed/added, dirlookup() recursively calls itself until it finds a cached ancestor or reaches volume root. Every recursive lookup also results in many more stat calls. So even opening a small folder directly, still requires stat'ing every level of the whole path to be pushed into the page cache.
If the dircache max size is small (by default just 8192 entries), as you move around your file share, old entries are pushed off (evicted) as new ones are added. This high entry rotation is known as "scan eviction" and means by the time you want to go back to a previous directory and read a cached entry, it has likely already been evicted which can cause a cascade effect of recursive lookups and stats calls to restore the broken cached paths if parent entries are evicted. So unless your whole file server has less than 8192 file and directories, it is recommended to increase the dircache size value in afp.conf.
Future releases will increase the maximum size of the dircache once existing performance issues are addressed. We are also considering retaining directories over files during eviction to reduce recursive path discoveries after parent folders are evicted.
The page cache operates at memory speed when it hits, making overhead hard to measure:
The real cost isn't the individual stat() calls—it's the cascade effect:
However, there is only a single Page Cache per kernel. So this new dircache change in Netatalk to validate dircache periodically means less interference with other services on the same instance.
Before understanding the probabilistic solution, let's see how Netatalk finds files and the precedence between cache layers:
flowchart TD
Request[Client requests file/folder]
Request --> DC_Check{In Directory
Cache?}
DC_Check -->|Cache Hit| DC_Valid{Validate?}
DC_Check -->|Cache Miss| CNID_Query[Query CNID Database]
DC_Valid -->|Every Nth request| Validate[stat filesystem]
DC_Valid -->|Other requests| Use_Cached[Use cached entry]
Validate -->|Still valid| Use_Cached
Validate -->|Invalid| Remove_Entry[Remove from cache]
Remove_Entry --> CNID_Query
CNID_Query -->|Found| FS_Check[stat filesystem]
CNID_Query -->|Not found| Create_New[Create new CNID]
FS_Check -->|Exists| Add_Cache[Add to dircache]
FS_Check -->|Missing| Error[Return error]
Create_New --> Add_Cache
Add_Cache --> Return[Return to client]
Use_Cached --> Return
style DC_Check fill:#e6f3ff
style CNID_Query fill:#ffe6e6
style FS_Check fill:#ffffe6
graph TB
subgraph "Lookup Precedence"
L1[Directory Cache - Fastest - Memory]
L2[CNID Database - Fast - BerkeleyDB]
L3[Filesystem - Slow - Disk I/O]
L1 -->|Miss or Invalid| L2
L2 -->|Verify exists| L3
end
subgraph "Data Location"
D1[Dircache - Paths and Metadata]
D2[CNID - ID to Path mappings]
D3[Page Cache - File data and inodes]
D4[Disk - Authoritative source]
end
L1 -.-> D1
L2 -.-> D2
L3 -.-> D3
D3 -.-> D4
With the lookup hierarchy understood, here's how probabilistic validation optimizes the flow:
stateDiagram-v2
[*] --> CacheLookup: Directory Request
state ValidationDecision {
[*] --> CheckCounter
CheckCounter --> IncrementCounter
IncrementCounter --> Modulo: counter % freq
Modulo --> DoValidate: == 0 (validate)
Modulo --> SkipValidate: != 0 (skip)
}
CacheLookup --> Found: Cache Hit
CacheLookup --> NotFound: Cache Miss
Found --> ValidationDecision
DoValidate --> StatCall: Filesystem stat
SkipValidate --> TrustCache: Use cached data
StatCall --> Valid: Still exists
StatCall --> Invalid: Gone/moved
Valid --> UpdateMetadata: Refresh metadata
Invalid --> RemoveEntry: Evict from cache
RemoveEntry --> QueryCNID: Rebuild from CNID
UpdateMetadata --> ServeClient
TrustCache --> ServeClient
NotFound --> QueryCNID
QueryCNID --> ServeClient
ServeClient --> [*]: Return to client
note right of SkipValidate: 99% take this path
No filesystem I/O
The key insight: We don't skip the cache hierarchy, we skip the validation step. Files are still properly discovered through CNID when not cached, but we avoid repeatedly verifying cached entries still exist.
With this setting:
Different cache layers have different retention strategies:
graph TB
subgraph "Netatalk Dircache"
DC1[Entry Added]
DC1 --> DC2[LRU Queue]
DC2 --> DC3[Validated 1/N times]
DC3 --> DC4[LRU Evicted when full]
end
graph TB
subgraph "Page Cache"
PC1[Page Loaded]
PC1 --> PC2[Active List]
PC2 --> PC3[Inactive List]
PC3 --> PC4[LRU Reclaimed under pressure]
end
graph TB
subgraph "ZFS ARC"
ARC1[Block Cached]
ARC1 --> ARC2[MRU List]
ARC2 --> ARC3[MFU List]
ARC3 --> ARC4[Ghost Lists]
ARC4 --> ARC5[ARC Evicted by ARC sizing]
end
By trusting and reducing validation calls at the fastest layer (dircache), we reduce pressure on the subsequent cache layers:
You can observe the benefits of dircache validation optimization using these metrics:
When an afpd child process exits gracefully (user disconnects) or receives SIGUSR1, it logs detailed cache statistics. The metrics differ between LRU (Least Recently Used) and ARC (Adaptive Replacement Cache) modes.
These metrics are reported for both cache modes:
LRU mode provides straightforward hit/miss metrics:
Example LRU log output:
ARC typically outperforms LRU by maintaining both recent and frequent access history. However, it may underperform in edge cases such as rapidly changing access patterns (ARC relies on historical data) or workloads that fit entirely within cache (LRU's simpler design has less overhead).
Netatalk's ARC Implementation: Netatalk's ARC retains all data in ghost entries rather than metadata-only. This eliminates disk lookups on ghost hits while preserving ARC's learning benefits. The total cache footprint is cache + ghosts (2× dircache size), but performance is improved across virtually all workload patterns.
Example ARC log output:
High invalid_on_use counter: Low numbers are normal and expected. High values indicate possible external file changes or high multi-user interactions. Consider lowering dircache validation freq.
Low invalid_on_use counter: Consider increasing dircache validation freq for performance gains.
Low hit rate (< 20%): May indicate cache size is too small or workload doesn't benefit from caching. Consider increasing dircache size.
High eviction rate: Cache is undersized for working set. Increase dircache size.
Key metrics to monitor:
Netatalk includes an idle worker thread that performs non-critical dircache housekeeping during poll() idle periods, when the main AFP command thread is blocked waiting for client data.
The idle worker uses temporal separation rather than locks: the worker thread and main thread never access shared dircache data concurrently.
If the worker thread fails to start (e.g., pthread_create() failure or platforms without lock-free atomics), all operations fall back to synchronous behavior transparently. The idle_worker_is_active() guard ensures the dir_free_invalid_q() call is preserved in the DSIFUNC_CMD handler when the worker is not running.
afp_dsi_close() can be called from signal handler context. It uses idle_worker_stop_signal_safe() which performs only a single atomic store (no spin, no mutex). Since all afp_dsi_close() paths end in exit(), the kernel terminates the worker thread.
Worker statistics are logged at session close:
The real benefit of this optimization isn't just the eliminated stat() calls— it's the compound effect across the entire storage stack:
By understanding how cache layers interact and compete, we can make intelligent optimizations that benefit the entire system.
By default nothing changes, as the default value for dircache validation freq = 1. If you have Samba or other processes operating on the volume this must remain set to 1.
Directory cache optimizations available in Netatalk 4.4+ with configurable validation frequency via the dircache validation freq parameter.
Developed and Authored by Andy Lemin, with Contributions from the Netatalk team.
1.17.0