The NFS share that was secretly SMB: debugging an Azure Files CSI snapshot restore

---

We run a Sonatype Nexus IQ server on AKS. Its work directory, around 115 GiB of small report files, lives on an NFS 4.1 Azure file share in a Premium storage account that only accepts traffic from the cluster's subnet. As part of a backup and disaster recovery project, I added hourly snapshots of that share with Gemini, using the Azure Files CSI driver's VolumeSnapshot support.

Taking snapshots worked on the first try. Restoring one did not, and finding out why took me into the CSI driver, AzCopy and a misleading error message. This post walks through it, because the symptoms point you somewhere other than the cause.

The setup

The Azure Files CSI driver docs recommend useDataPlaneAPI: "oauth" for private storage accounts. With it, the driver talks to the storage data plane using an OAuth token from the AKS control plane identity, and Azure lets that through the account firewall as a trusted service. So the snapshot class and the StorageClass for restores both used it:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-azurefile-vsc-oauth
driver: file.csi.azure.com
deletionPolicy: Delete
parameters:
  useDataPlaneAPI: "oauth"
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azurefile-csi-nfs-restore
provisioner: file.csi.azure.com
parameters:
  protocol: nfs
  skuName: Premium_LRS
  storageAccount: <account>
  resourceGroup: <rg>
  useDataPlaneAPI: "oauth"
reclaimPolicy: Delete

A restore is just a PVC with a dataSource pointing at a VolumeSnapshot. The driver handles it in two steps:

1. Create a new, empty file share for the PVC.
2. Run azcopy copy from the snapshot into the new share. For NFS volumes it adds --from-to=FileNFSFileNFS.

The symptom

The PVC stayed Pending, and the events repeated this:

Cannot perform copy due to error: the destination share has NFS protocol enabled.
To copy to a NFS share, use the appropriate --from-to flag value

That reads like "the driver forgot to tell AzCopy this is an NFS copy". A missing --from-to flag was my first theory, and it was the one I put in the first version of the upstream issue.

Narrowing it down

Three observations changed the picture.

The new share was not NFS. The driver had left the half-created destination share behind, so I asked Azure what it was:

$ az storage share-rm show ... -n pvcn-<uuid> --query enabledProtocols
"SMB"

An NFS StorageClass had produced an SMB share.

The same restore without OAuth worked. I repeated it with an identical StorageClass that only lacked useDataPlaneAPI. The share came out as NFS, and about 50 minutes later all 114.7 GiB were restored and matched the live share file for file. So the difference was the OAuth, data plane path.

The driver does pass the flag. In copyFileShareByAzcopy:

if shareOptions.Protocol == armstorage.EnabledProtocolsNFS {
    azcopyCopyOptions = append(azcopyCopyOptions, "--from-to=FileNFSFileNFS")
}

So --from-to was fine. The problem was one step earlier.

Root cause 1: the data plane client dropped the protocol

The driver has two ways to create a share, and useDataPlaneAPI picks one:

	Management plane (ARM)	Data plane (Files REST API)
Used when	default	useDataPlaneAPI: "true" or "oauth", or provisioner secrets
Client	azureFileMgmtClient	azureFileDataplaneClient
Sets NFS protocol	yes	no

The management plane client always set the protocol:

if shareOptions.Protocol == armstorage.EnabledProtocolsNFS {
    shareOps.FileShareProperties.EnabledProtocols = to.Ptr(shareOptions.Protocol)
}

The data plane client only passed the quota:

_, err := f.Client.NewShareClient(shareOptions.Name).Create(ctx, &share.CreateOptions{
    Quota: to.Ptr(int32(shareOptions.RequestGiB)),
})

Everything else the StorageClass asked for was silently dropped: the protocol, access tier, root squash, metadata and provisioned IOPS and bandwidth. A share's protocol is fixed when it is created, and Azure's default is SMB. So every NFS volume created through the data plane path became an SMB share. Snapshot restore and clone fail loudly because AzCopy checks the protocols. A plain new NFS PVC would presumably only fail later, when a node tries to mount an SMB share over NFS.

Root cause 2: the error message described the wrong thing

AzCopy's check, validateShareProtocolCompatibility, compares two inputs for each side of the copy:

• shareProtocol: what the share actually is, read from Azure.
• fromTo: what you claimed on the command line.

if shareProtocol == common.ELocation.File() {            // the share is SMB
    if isSource && fromTo.From() != common.ELocation.File() {
        return errors.New("the source share has SMB protocol enabled. " +
            "To copy from a SMB share, use the appropriate --from-to flag value")
    }
    if !isSource && fromTo.To() != common.ELocation.File() {
        return errors.New("the destination share has NFS protocol enabled. " +
            "To copy to a NFS share, use the appropriate --from-to flag value")
    }
}

Three of the four branches describe the share's real protocol. The SMB-destination branch carries the NFS-destination text word for word. This is the one we hit: the share was SMB and the flag said NFS. The message said the opposite of what was true, and its advice ("use the right --from-to") pointed at the one thing that was already correct.

The history explains it. AzCopy used to have a single message filled in with the share's protocol, which would have said "The destination share has SMB protocol enabled". A later refactor split it into four hard-coded strings, and one of them was pasted from its neighbour.

It took me a while to see this myself. The condition fromTo.To() != File looks like "the flag is not SMB, so say NFS". The trick is that the message's first sentence is about the share, which the outer if has already established is SMB.

Why hadn't anyone hit this?

It is a real bug, but few people take that path:

• useDataPlaneAPI is opt-in. The default ARM path has always been correct.
• For SMB volumes, the most common kind, a dropped protocol gives Azure's default, which is what the user wanted anyway.
• The combination is recent. NFS snapshot restore arrived in late 2025, OAuth data plane support in 2025, and the docs started recommending OAuth for private accounts in mid-2026.
• The end-to-end tests that use useDataPlaneAPI only cover SMB, and the NFS tests only use the default path.
• When it does happen, the symptom looks like a network or flag problem. Anyone can drop useDataPlaneAPI, see it work, and move on.

The fixes

CSI driver (kubernetes-sigs/azurefile-csi-driver#3511, PR #3512): map every share option for the data plane create, like the management plane client does, in one small function with a unit test.

func newShareCreateOptions(o *ShareOptions) *share.CreateOptions {
    opts := &share.CreateOptions{Quota: to.Ptr(int32(o.RequestGiB))}
    if o.Protocol == armstorage.EnabledProtocolsNFS {
        opts.EnabledProtocols = to.Ptr(string(armstorage.EnabledProtocolsNFS))
    }
    // access tier, root squash, metadata, provisioned IOPS and bandwidth likewise
    return opts
}

Why not route OAuth requests through the ARM path instead? Because the data plane path is a feature people choose on purpose, to avoid ARM throttling or to run with data-plane-only permissions. Three different code paths also share this client, so fixing the client fixes all of them.

AzCopy: return "the destination share has SMB protocol enabled. To copy to a SMB share, ..." in that branch, and move the comparison into a small function with a table test covering all eight share and flag combinations.

A bonus: snapshots that never got deleted

Two days later I noticed the snapshot count was still climbing. Gemini keeps 24 hourly and 7 daily snapshots and deletes the rest, but none had ever been deleted. When the VolumeSnapshotClass has no snapshotter secrets, the driver deletes a share snapshot through ARM (FileShares.Delete with x-ms-snapshot), whatever useDataPlaneAPI says. That needs Microsoft.Storage/storageAccounts/fileServices/shares/delete, which our control plane identity deliberately did not have. Azure allows 200 snapshots per share, so at 24 a day new snapshots would have started failing within a week.

We kept the subscription-wide role without delete, and added a custom role containing only that action, assigned on the one storage account. The stuck deletions cleared within minutes.

Takeaways

• Check what Azure actually created. One az ... show of the resource the error was about turned a confusing message into an obvious mismatch.
• Change one variable. Repeating the restore with a single StorageClass parameter removed told me more than reading code did.
• Read error messages as claims to verify. This one was a copy-paste of its neighbour.
• Test restores, not just backups. Snapshots were "working" for days while both restore and cleanup were broken.
• Watch the cleanup side of retention. A backup that cannot delete its old copies fails slowly and quietly.

Until the driver fix reaches AKS, our restores use a StorageClass without useDataPlaneAPI. It is a one-line workaround, now that we know which line.