11.9. Binding memory

On a machine with more than one kind of memory, e.g DDR next to HBM, the NUMA node on which a process is cpu-bound to is not always the node with the best memory for it. Example is Sapphire Rapids (SPR) with DDR+CPU nodes and HBM-only nodes. On SPR, the local DDR+CPU node is also the default and closest node for memory allocation. HBM nodes hold no cpu cores, so are silently ignored by automatic page allocation and migration. Using HBM is usually done through custom allocator libraries (e.g. memkind), by explicit C API and system calls (hwloc, mbind(), set_mempolicy()) or with command-line binding tools like numactl.

PRRTE can memory-bind each process to the NUMA node(s) best suited to it by a chosen measure — capacity, locality, bandwidth or latency — among the nodes local to the cores the process is bound to. This is separate from, and complementary to, cpu-binding a process to cores (--bind-to). The cpu-binding defines where the process runs, while the memory-binding decides where its memory is allocated.

Note

PRRTE currently implements these directives with hwloc’s memory attributes, which need hwloc 2.3 or later. PRRTE itself builds with hwloc 2.1 or later, but with an older hwloc this feature is unavailable: asking for a memory binding is reported as a memory-binding failure (see Failure handling).

11.9.1. Usage

Per job or per app, with the --membind option of mpirun, prterun and prun:

mpirun --membind bandwidth -n 4 ./a.out
mpirun --map-by numa --membind bandwidth -n 4 ./a.out
mpirun --bind-to core --membind bandwidth -n 4 ./a.out

All three bind memory: the first relies on PRRTE’s default cpu binding, the second on the binding --map-by derives, the third on an explicit one. What matters is that each process ends up bound to cpus: see The process must be bound to cpus.

Currently supported directives are: bandwidth, latency, locality and capacity (and none, the default).

As the default for every job a DVM runs, use the hwloc_default_membind MCA parameter, which takes the same values. It is a PRRTE parameter, so it is given with --prtemca or in the environment as PRTE_MCA_hwloc_default_membind — PRTE_MCA_, not OMPI_MCA_:

prte --prtemca hwloc_default_membind bandwidth ...

A process is bound by its app’s --membind if it has one, else by its job’s, else by the MCA parameter.

11.9.1.1. The --membind option

Bind the memory of each process to the NUMA node(s) best suited to it by the chosen measure, among the nodes local to the CPUs the process is bound to. This is separate from --bindto, which decides where a process runs: --membind decides where its memory is allocated. It needs the process to be bound to CPUs (see below).

Supported directives:

  • NONE binds no memory. This is the default.

  • CAPACITY: the local node with the most memory

  • LOCALITY: every node local to the process’ CPUs

  • BANDWIDTH: the local node with the highest bandwidth from the process’ CPUs. This may be a node with no CPUs of its own — on a processor with on-package high-bandwidth memory (HBM), the HBM nodes hold no cores, so a process bound to a core sits on a DDR node and BANDWIDTH binds its memory to the HBM node next to it.

  • LATENCY: the local node with the lowest latency from the process’ CPUs

Nodes tied for the best value are all used.

Directives are case-insensitive and can be shortened to the minimum number of characters to uniquely identify them. Thus, bandwidth, BANDWIDTH and b are the same request. A shortening that fits more than one of them is refused rather than guessed at — L could be LOCALITY or LATENCY.

Memory is only bound for processes that are bound to CPUs, whether by --bindto, by the binding --mapby derives, or by PRRTE’s default. --bindto none leaves processes unbound, and so does an oversubscribed node, on which PRRTE drops its default binding: PRRTE then warns once per app and runs the job with that memory unbound. A --membind takes precedence over the hwloc_default_mem_alloc_policy MCA parameter, and a failure to bind memory is handled according to hwloc_default_mem_bind_failure_action. The directives are currently implemented with hwloc’s memory attributes, which requires hwloc 2.3 or later, and BANDWIDTH and LATENCY are only as good as the values hwloc has for the platform.

The hwloc_default_membind MCA parameter takes the same values, and supplies the memory binding for a job or app that gives no --membind.

Per-app-context memory binding (MPMD jobs)

The rule is the one described for --bindto: a --membind written on the first app segment and nowhere else describes the whole job; otherwise each app that carries one is bound by its own, and apps that carry none take the default — they do not inherit another app’s directive.

Example:

prun --bindto core --membind bandwidth -n 4 app1 : --bindto core --membind capacity -n 2 app2

Here the memory of app1 processes is bound to the node with the highest bandwidth from their core, and that of app2 processes to the node with the most memory, all within the same job.

The same directives can be supplied via the PMIx_Spawn API with the PMIX_MEMBIND key: in the job-level info[] array for the whole job, or in the info[] array of a pmix_app_t for that app alone. On that path there is no “first app” rule: the array a key was written in says what it describes.

11.9.1.2. Where the memory goes

The candidates are the NUMA nodes local to the process’ cpuset: those whose locality contains it (the DDR node holding a core, and an HBM node sharing that locality) and those inside it (each node of a package the process is bound to). The choice is never made across the whole machine, so a process is not sent to a node on another socket because it is larger.

Candidates the PRRTE daemon may not itself use (for example because of a cgroup) are dropped before the choice is made, since the process inherits that restriction: if the best node is forbidden, the best permitted one is chosen. Nodes tied for the best value are all kept. If no candidate is left, or none has a value for the chosen measure, the process’ memory is not bound, which is handled as a failure (see Failure handling).

Note

BANDWIDTH and LATENCY are only as good as the values hwloc has. On platforms that do not expose the HMAT information, or inside a container that hides it, hwloc has nothing to compare and the directive cannot be resolved - which is handled as a failure below.

11.9.2. Interaction with other parameters

11.9.2.1. The process must be bound to cpus

The memory binding is computed from the cpuset a process is bound to, so it is applied to every process the mapper binds to cpus — whether the binding policy was given with --bind-to, derived from --map-by (--map-by numa binds to the NUMA domain, for example), or PRRTE’s default — and to no other. A process is left unbound by --bind-to none, and when PRRTE drops its default binding because a node is oversubscribed.

When a memory binding is requested for processes that end up unbound, PRRTE prints one warning per app when it maps the job, naming how many of its processes are affected, and runs the job with their memory left unbound:

$ mpirun --bind-to none --membind bandwidth -n 4 ./a.out
A memory binding was requested for processes that are not bound to
cpus:
...

A process whose default cpu binding is refused by the operating system is tolerated as unbound (the default is a preference, not a requirement); its memory is then not bound either, without a warning of its own.

mem_alloc_policy. hwloc_default_mem_alloc_policy is PRRTE’s older, coarser control: it either does nothing (none) or restricts a process to its local NUMA domain (local_only). A memory binding requested by --membind or hwloc_default_membind takes precedence, for every process it applies to.

Topology files. A topology read from a file with hwloc_use_topo_file describes a machine other than the one the process runs on, so no memory binding is applied.

11.9.3. Failure handling

Failing to bind is governed by hwloc_default_mem_bind_failure_action, the same parameter that governs local_only:

  • silent - proceed without comment.

  • warn (default) - warn the first time, then let the job continue, possibly with degraded performance.

  • error - abort the job.

This covers the directive being unavailable (an hwloc older than 2.3, or no data for it), no usable local node, as well as the operating system refusing the request. Many container runtimes deny the system call that sets a memory policy; inside one, use silent or warn unless the job must not run without it.

11.9.4. Seeing what was chosen

Raise the hwloc verbosity to see the node set PRRTE selected for each process:

prterun --prtemca hwloc_base_verbose 40 \
        --bind-to core --membind bandwidth -n 2 ./a.out

and inspect the running process on its node: the mappings in /proc/<pid>/numa_maps carry a bind: entry naming the node(s), and numastat -p <pid> shows where its pages actually landed.