#Installation

#Prerequisites

#Common requirements

1. ACP version: v4.0 or later.
2. Cluster administrator access to the target ACP cluster.
3. Supported NPU hardware. NPU worker nodes must carry one of:
   • Ascend 910B
   • Ascend 310P
4. Alauda Build of Node Feature Discovery cluster plugin must be installed. The operator reads NPU presence and kernel/OS labels from NFD to decide which driver image to pull on each node.

#MindIO SDK (optional)

If you plan to enable MindIO TFT or MindIO ACP, separately stage the matching MindIO SDK package on each NPU node under /opt/openFuyao/mindio/. Skip this step otherwise.

#Procedure

#Step 1: Sync driver images and configure ImageWhiteList

kubectl -n npu-operator rollout restart ds/ascend-runtime-containerd

#1.1 Pull driver images from Docker Hub

The driver image is shipped independently on Docker Hub at docker.io/alaudadockerhub/ascend-driver — not as part of the operator bundle — because the driver .ko binaries are kernel-specific and the kernel list grows over time. Each tag follows the pattern <HDK>-<chip>-<kernel>-<os-stem>, for example 25.5.0-910b-6.6.0-145.0.4.135-oe2403sp3. Pick the tag matching your nodes' uname -r and chip.

List available tags:

curl -s 'https://hub.docker.com/v2/repositories/alaudadockerhub/ascend-driver/tags/?page_size=100' \
  | jq -r '.results[].name'

On a machine with internet access, mirror each selected tag into your cluster registry:

TAG=25.5.0-<chip>-<kernel>-<os-stem>

skopeo copy --all \
  docker://docker.io/alaudadockerhub/ascend-driver:$TAG \
  docker://<your-cluster-registry>/mlops/ascend-driver:$TAG

The operator configuration defaults spec.driver.image.repository to mlops/ascend-driver; override it in the deployment form if your registry uses a different namespace.

#1.2 Allow the driver image in ImageWhiteList

ACP gates which images pods are allowed to pull. Every driver-image tag the DaemonSet may pull must be listed explicitly in an ImageWhiteList.

Create one (or extend the existing ascend-driver ImageWhiteList in cpaas-system):

apiVersion: app.alauda.io/v1alpha1
kind: ImageWhiteList
metadata:
  name: ascend-driver
  namespace: cpaas-system
spec:
  repoList:
  # one entry per tag you mirrored in step 1.1 — full image reference, not bare repo path
  - <your-cluster-registry>/mlops/ascend-driver:25.5.0-910b-<kernel>-<os-stem>
  - <your-cluster-registry>/mlops/ascend-driver:25.5.0-310p-<kernel>-<os-stem>

Add one repoList entry per <chip, kernel> tag you mirrored. Each entry is a full image reference including the tag (the API does not accept bare repository paths). If you override spec.driver.image.repository later, list the new path instead.

#1.3 Verify

On each NPU node, the kubelet should be able to pull a driver image (use the tag matching the node's kernel):

crictl pull <your-cluster-registry>/mlops/ascend-driver:$TAG

If this succeeds, Step 1 is complete. A later ImagePullBackOff on the driver DaemonSet usually means a missing tag in the registry, a registry credential issue, or a missing repoList entry in the ImageWhiteList.

#Step 2: Download packages

#Step 3: Upload packages

The platform provides the violet command-line tool for uploading both operator packages and cluster plugin packages downloaded from the Customer Portal Marketplace.

For details, see Upload Packages.

#Step 4: Install the Node Feature Discovery cluster plugin

Alauda Build of Node Feature Discovery is a cluster plugin, not an operator. Install it first because the NPU Operator depends on its node labelling.

1. Navigate to Administrator > Marketplace > Cluster Plugins.
2. Switch to the target cluster.
3. Locate Alauda Build of Node Feature Discovery and click Install.

#Step 5: Install the Alauda Build of NPU Operator

Alauda Build of NPU Operator is delivered as an operator (OLM bundle). Installation has two distinct sub-steps on the platform UI:

1. Install the operator — the OperatorHub flow only brings up the operator's controller pods (npu-operator-controller-manager + npu-operator). It does not deploy any driver, device plugin, or other NPU components.
2. Create an NPUOperatorCtl instance — only at this step do you fill in the deployment form, and only after the instance is created do the controller pods start reconciling and rolling out the NPU components onto the nodes.

#5.1 Label nodes

Apply the label masterselector=dls-master-node to all master nodes and the label workerselector=dls-worker-node to the worker nodes that should host NPU components:

kubectl label nodes <master-node-id> masterselector=dls-master-node
kubectl label nodes <worker-node-id> workerselector=dls-worker-node

#5.2 Install the operator

1. Navigate to Administrator > Marketplace > OperatorHub, switch to the target cluster, and locate the Alauda Build of NPU Operator entry. If the status is Absent, confirm the operator package was uploaded with violet in Step 3.

2. Click the operator to open its details page, then click Install.

3. On the install page, leave Channel unchanged, confirm Version, leave Installation Location as npu-operator (the default; all NPU components created in the next sub-step land here), and select Manual for Upgrade Strategy. Click Install.

4. Wait for the subscription to reach Succeeded. The Alauda Build of NPU Operator tile should transition from Installing to Installed, and kubectl -n npu-operator get pod will show the two controller pods (npu-operator and npu-operator-controller-manager) Running.

#5.3 Create the NPUOperatorCtl instance

The deployment form opens when you create the instance, not when you install the operator above.

1. On the Installed Operators page, click the Alauda Build of NPU Operator tile, then click Create Instance (or open the NPUOperatorCtl tab and click Create NPUOperatorCtl).

2. Fill in the form (see the table below) and click Create.

3. The operator immediately reconciles: driver / device plugin / runtime sidecar / exporter / rebooter DaemonSets land on every NPU node, and the controller updates the NPUOperatorCtl status.conditions to Deployed=True / UpgradeSuccessful once everything is up.

Deployment form parameter description:

#Verification

1. Confirm the NPUOperatorCtl instance is reconciling cleanly:

   kubectl -n npu-operator get npuoperatorctl

   The Deployed condition should be True with reason UpgradeSuccessful. (Replace npu-operator below and in subsequent commands with the namespace you chose at install time if it differs.)

2. Wait for the npu-driver pod to become Running. First-time install takes a few minutes for the driver image to pull and the modules to be inserted into the host kernel:

   kubectl -n npu-operator get pod -w | grep npu-driver

3. Check that the NPU node is now reporting allocatable Ascend devices:

   kubectl get node ${nodeName} -o jsonpath='{.status.allocatable}'
   # Example output includes:
   #   "huawei.com/Ascend910":"8"   (910B nodes; specific value depends on card count)
   #   "huawei.com/Ascend310P":"1"  (310P nodes)

4. (Optional) Run npu-smi info on the host. The operator does not symlink npu-smi into the host PATH (/usr is read-only on KubeOS), so call the binary directly with its libraries loaded:

   LD_LIBRARY_PATH=/var/lib/Ascend/driver/lib64/driver:/var/lib/Ascend/driver/lib64/common \
     /var/lib/Ascend/driver/tools/npu-smi info

   Each card should report Health: OK and a non-zero Bus-Id.

5. Validate end-to-end with a sample NPU workload. v1.2.4 no longer requires runtimeClassName: ascend — the resource request alone triggers CDI device injection. In air-gapped or image-whitelist-enforced clusters, mirror the sample image to your cluster registry first, or replace it with an equivalent internal test image that includes npu-smi.

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: npu-smoke
   spec:
     restartPolicy: Never
     containers:
     - name: probe
       image: ascendai/pytorch:ubuntu-python3.8-cann8.0.rc1.beta1-pytorch2.1.0
       command: ["bash", "-c"]
       args:
         - |
           ls /dev/davinci*
           npu-smi info
           sleep 3600
       resources:
         limits:
           huawei.com/Ascend910: 1   # Change to huawei.com/Ascend310P on a 310P node
   EOF
   
   kubectl logs npu-smoke

   The pod should reach Running. ls /dev/davinci* should show /dev/davinci_manager plus one per-card device node (e.g. /dev/davinci0), and npu-smi info should print the card's status. Both confirm that CDI injected the device into the container.

#Step 6: Verify monitoring

If the NPU Exporter component was deployed when installing the Alauda Build of NPU Operator, the operator automatically deploys a ServiceMonitor named npu-exporter-servicemonitor in the operator namespace, wired up to the npu-exporter Service. No manual ServiceMonitor creation is required. You can verify it with:

kubectl -n npu-operator get servicemonitor npu-exporter-servicemonitor

To get a Grafana dashboard, import the JSON file by following Import Dashboard.

The JSON file is available in ascend-npu-dashboard.

{
  "tags": [
    "ascend",
    "昇腾"
  ]
}

{
  "tags": [
    "ascend"
  ]
}

#What's next

• Driver upgrade and self-healing — how to roll the driver version forward and how the chip self-healing path works.

#FAQ

#Where is npu-smi installed on the host?

In v1.2.4 the driver pod stages the Huawei tools tree to /var/lib/Ascend/driver/, so the binary is at /var/lib/Ascend/driver/tools/npu-smi. No host PATH symlink is created (KubeOS keeps /usr read-only). Call it with the matching LD_LIBRARY_PATH:

LD_LIBRARY_PATH=/var/lib/Ascend/driver/lib64/driver:/var/lib/Ascend/driver/lib64/common \
  /var/lib/Ascend/driver/tools/npu-smi info

If you prefer a PATH-resolvable command, write a small wrapper into a writable location (e.g. /opt/bin/npu-smi) that exports LD_LIBRARY_PATH and execs the real binary.

#Do I still need runtimeClassName: ascend on workload pods?

No. v1.2.4 uses CDI for device injection: requesting huawei.com/Ascend910 (or Ascend310P) is enough. Existing manifests that still set runtimeClassName: ascend continue to work — the RuntimeClass is kept for backwards compatibility — but no new manifests need it.

#What should I pay attention to when uninstalling Alauda Build of NPU Operator?

Uninstalling the operator removes the driver DaemonSet, but the driver modules already loaded into the host kernel stay loaded — rmmod would risk leaving the chip in an unrecoverable state. To fully remove the driver from a host, reboot the node after the operator is uninstalled; the modules will not auto-reload because the DaemonSet is gone.

Files staged to the host can be cleaned up manually if needed:

rm -rf /var/lib/Ascend /var/lib/ascend /home/bios/driver /etc/ascend_install.info /run/ascend

Run these only after the operator and its driver pod have been removed, and after the node has been rebooted (or before, if you intend to reboot anyway).