Build1 publisher3 min readPublished
AWS's FSx for ONTAP NVMe/TCP multipath check fails on three Amazon Linux 2023 kernel series
AWS's multipath check for FSx for ONTAP over NVMe/TCP fails on three Amazon Linux 2023 kernel series, a dev.to author found. Without kernel multipath support the host sees one namespace as two separate devices, a problem the author says one command can catch before FSx billing starts.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- The author traced the gap to a kernel built without Asymmetric Namespace Access, the feature that tells the host which path to a namespace is shortest.
- After moving to a kernel with multipathing, the author found nvme list-subsys shows that two paths exist, not that both are in use.
- Testing used one EC2 client against one Single-AZ file system; Multi-AZ is unverified and the post makes no performance comparison.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure A team that skips the check gets one ONTAP namespace as two disks, with nothing in the kernel marking which one is the standby path to the non-optimized controller.
- contradiction AWS's page says only package installs differ on other AMIs; the author's two counterexamples, neither fixed by a package, put these AL2023 kernels outside that statement.
- constraint The findings cover three kernel series on Single-AZ only, so teams on other kernels or Multi-AZ have to run the check themselves before trusting either the AWS page or the post.
The path in that step is a parameter of the nvme_core kernel module, and the procedure treats a Y in it as proof that multipath came up [1]. On the Amazon Linux 2023 AMIs the author tested, there was no file to read [2]. A step whose only success criterion is Y has no instruction for a missing file. The author traced the cause to a kernel build setting [4]. The kernel ships without Asymmetric Namespace Access (ANA), the feature that tells a host which of several paths to one namespace is the shortest [3].
ANA matters because of how FSx for ONTAP presents storage. In the author's setup, a second-generation SINGLE_AZ_2 file system, the HA pair's two controllers each hold a path to the same namespace. One path goes to the optimized controller, the other to the non-optimized one [11]. With ANA, the host can tell the second path is a standby. Without it, the author wrote, the paths "appear as two separate devices pointing at the same data" [12].
AWS scoped the procedure to one distribution. Its "Before you begin" section says "Create an EC2 instance running Red Hat Enterprise Linux (RHEL) 9.3" [7]. The same section says that, apart from installing packages, the commands are valid on other EC2 Linux AMIs [8]. The author reports two counterexamples to that sentence and says package installation explains neither [9]. The author also limits the charge, saying the post is not a claim that the documentation is wrong [10].
Sequence is where this costs money. The documented check is item four under "To discover the target NVMe nodes" [1]. Discovery needs a target, so the check runs only after the FSx file system exists [1]. The author says the kernel can be checked with one command before any billing starts [13]. The file depends on the client kernel, not on FSx [4]. I'd run that check when choosing the AMI, before creating anything on the storage side.
Fixing the kernel clears the first problem and exposes a second. After moving to a kernel with multipathing, the author found that the optimized and non-optimized lines in nvme list-subsys say there are two paths. They do not say both are in use [14]. Use has to be counted separately, and the author warns that choosing which table to count is itself a trap [14].
The scoping is careful work. The check covered the three kernel series returned by al2023-ami-kernel-default-x86_64 [5]. The author wrote that they cannot claim AL2023 always behaves this way [6]. All figures come from a Single-AZ configuration, and Multi-AZ is unverified [15]. The post makes no performance comparison and says its figures cannot be used for sizing [15]. It belongs to a dev.to series titled "FSx for ONTAP Block Protocols, Measured" [16].
What to watch
- Whether AWS revises the 'Before you begin' statement on the Provisioning NVMe/TCP for Linux page or adds an Amazon Linux 2023 note.
- Whether a later al2023-ami-kernel-default kernel series ships with Asymmetric Namespace Access, which would let the documented Y check pass.
- Results from a Multi-AZ FSx for ONTAP file system, which the author has not verified.