Hardware Locality (hwloc)  1.11.3
Frequently Asked Questions

Should I use logical or physical/OS indexes? and how?

One of the original reasons why hwloc was created is that physical/OS indexes (obj->os_index) are often crazy and unpredictable: logical processors numbers are usually non-contiguous (processors 0 and 1 are not physically close), they vary from one machine to another, and may even change after a BIOS or system update. This numbers make task placement hardly portable. Moreover some objects have no physical/OS numbers (caches), and some objects have non-unique numbers (core numbers are only unique within a socket). Physical/OS indexes are only guaranteed to exist and be unique for PU and NUMA nodes.

hwloc therefore introduces logical indexes (obj->logical_index) which are portable, contiguous and logically ordered (based on the resource organization in the locality tree). In general, one should only use logical indexes and just let hwloc do the internal conversion when really needed (when talking to the OS and hardware).

hwloc developers recommends that users do not use physical/OS indexes unless they really know what they are doing. The main reason for still using physical/OS indexes is when interacting with non-hwloc tools such as numactl or taskset, or when reading hardware information from raw sources such as /proc/cpuinfo.

lstopo options -l and -p may be used to switch between logical indexes (prefixed with L#) and physical/OS indexes (P#). Converting one into the other may also be achieved with hwloc-calc which may manipulate either logical or physical indexes as input or output. See also hwloc-calc.

# Convert PU with physical number 3 into logical number
$ hwloc-calc -I pu --physical-input --logical-output pu:3
5

# Convert a set of NUMA nodes from logical to physical
# (beware that the output order may not match the input order)
$ hwloc-calc -I numa --logical-input --physical-output numa:2-3 numa:7
0,2,5

I do not want hwloc to rediscover my enormous machine topology every time I rerun a process

Although the topology discovery is not expensive on common machines, its overhead may become significant when multiple processes repeat the discovery on large machines (for instance when starting one process per core in a parallel application). The machine topology usually does not vary much, except if some cores are stopped/restarted or if the administrator restrictions are modified. Thus rediscovering the whole topology again and again may look useless.

For this purpose, hwloc offers XML import/export features. It lets you save the discovered topology to a file (for instance with the lstopo program) and reload it later by setting the HWLOC_XMLFILE environment variable. The HWLOC_THISSYSTEM environment variable should also be set to 1 to assert that loaded file is really the underlying system.

Loading a XML topology is usually much faster than querying multiple files or calling multiple functions of the operating system. It is also possible to manipulate such XML files with the C programming interface, and the import/export may also be directed to memory buffer (that may for instance be transmitted between applications through a package). See also Importing and exporting topologies from/to XML files.

How many topologies may I use in my program?

hwloc lets you manipulate multiple topologies at the same time. However these topologies consume memory and system resources (for instance file descriptors) until they are destroyed. It is therefore discouraged to open the same topology multiple times.

Sharing a single topology between threads is easy (see Thread Safety) since the vast majority of accesses are read-only.

If multiple topologies of different (but similar) nodes are needed in your program, have a look at How to avoid memory waste when manipulating multiple similar topologies?.

How to avoid memory waste when manipulating multiple similar topologies?

hwloc does not share information between topologies. If multiple similar topologies are loaded in memory, for instance the topologies of different identical nodes of a cluster, lots of information will be duplicated.

hwloc/diff.h (see also Topology differences) offers the ability to compute topology differences, apply or unapply them, or export/import to/from XML. However this feature is limited to basic differences such as attribute changes. It does not support complex modifications such as adding or removing some objects.

Why is lstopo slow?

lstopo enables most hwloc discovery flags by default so that the output topology is as precise as possible (while hwloc disables many of them by default). This includes I/O device discovery through PCI libraries as well as external libraries such as NVML. To speed up lstopo, you may disable such features with command-line options such as --no-io.

When NVIDIA GPU probing is enabled with CUDA or NVML, one should make sure that the Persistent mode is enabled (with nvidia-smi -pm 1) to avoid significant GPU initialization overhead.

When AMD GPU discovery is enabled with OpenCL and hwloc is used remotely over ssh, some spurious round-trips on the network may significantly increase the discovery time. Forcing the DISPLAY environment variable to the remote X server display (usually :0) instead of only setting the COMPUTE variable may avoid this.

Also remember that these components may be disabled at build-time with configure flags such as --disable-opencl, --disable-cuda or --disable-nvml, and at runtime with the environment variable HWLOC_COMPONENTS=-opencl,cuda,nvml.

If loading topologies is slow because the machine contains tons of processors, one should also consider using XML (see I do not want hwloc to rediscover my enormous machine topology every time I rerun a process).

What should I do when hwloc reports "operating system" warnings?

When the operating system reports invalid locality information (because of either software or hardware bugs), hwloc may fail to insert some objects in the topology because they cannot fit in the already built tree of resources. If so, hwloc will report a warning like the following. The object causing this error is ignored, the discovery continues but the resulting topology will miss some objects and may be asymmetric (see also What happens if my topology is asymmetric?).

****************************************************************************
* hwloc has encountered what looks like an error from the operating system.
*
* L3 (cpuset 0x000003f0) intersects with NUMANode (P#0 cpuset 0x0000003f) without inclusion!
* Error occurred in topology.c line 940
*
* Please report this error message to the hwloc user's mailing list,
* along with the output from the hwloc-gather-topology script.
****************************************************************************

These errors are common on large AMD platforms because of BIOS and/or Linux kernel bugs causing invalid L3 cache information. In the above example, the hardware reports a L3 cache that is shared by 2 cores in the first NUMA node and 4 cores in the second NUMA node. That's wrong, it should actually be shared by all 6 cores in a single NUMA node. The resulting topology will miss some L3 caches.

If your application not care about cache sharing, or if you do not plan to request cache-aware binding in your process launcher, you may likely ignore this error (and hide it by setting HWLOC_HIDE_ERRORS=1 in your environment).

Some platforms report similar warnings about conflicting Packages and NUMANodes. Upgrading the BIOS and/or the operating system may help. Otherwise, as explained in the message, reporting this issue to the hwloc developers (by sending the tarball that is generated by the hwloc-gather-topology script on this platform) is a good way to make sure that this is a software (operating system) or hardware bug (BIOS, etc).

Does hwloc require privileged access?

hwloc discovers the topology by querying the operating system. Some minor features may require privileged access to the operation system. For instance memory module and PCI link speed discovery on Linux is reserved to root, and the entire PCI discovery on Solaris and BSDs requires access to some special files that are usually restricted to root (/dev/pci* or /devices/pci*).

To workaround this limitation, it is recommended to export the topology as a XML file generated by the administrator (with the lstopo program) and make it available to all users (see Importing and exporting topologies from/to XML files). It will offer all discovery information to any application without requiring any privileged access anymore. Only the necessary hardware characteristics will be exported, no sensitive information will be disclosed through this XML export.

This XML-based model also has the advantage of speeding up the discovery because reading a XML topology is usually much faster than querying the operating system again.

The utility hwloc-dump-hwdata is also involved in gathering privileged information at boot time and making it available to non-privileged users. However it only applies to Intel Knights Landing Xeon Phi for now (see Why do I need hwloc-dump-hwdata for caches on Intel Knights Landing Xeon Phi?). See also HWLOC_DUMPED_HWDATA_DIR in Environment Variables for details about the location of dumped files.

hwloc only has a one-dimensional view of the architecture, it ignores distances

hwloc places all objects in a tree. Each level is a one-dimensional view of a set of similar objects. All children of the same object (siblings) are assumed to be equally interconnected (same distance between any of them), while the distance between children of different objects (cousins) is supposed to be larger.

Modern machines exhibit complex hardware interconnects, so this tree may miss some information about the actual physical distances between objects. The hwloc topology may therefore be annotated with distance information that may be used to build a more realistic representation (multi-dimensional) of each level. For instance, the root object may contain a distance matrix that represents the latencies between any pairs of NUMA nodes if the BIOS and/or operating system reports them.

What happens to my topology if I disable symmetric multithreading, hyper-threading, etc. ?

hwloc creates one PU (processing unit) object per hardware thread. If your machine supports symmetric multithreading, for instance Hyper-Threading, each Core object may contain multiple PU objects:

$ lstopo -
...
  Core L#0
    PU L#0 (P#0)
    PU L#1 (P#2)
  Core L#1
    PU L#2 (P#1)
    PU L#3 (P#3)

x86 machines usually offer the ability to disable hyper-threading in the BIOS. Or it can be disabled on the Linux kernel command-line at boot time, or later by writing in sysfs virtual files.

If you do so, the hwloc topology structure does not significantly change, but some PU objects will not appear anymore. No level will disappear, you will see the same number of Core objects, but each of them will contain a single PU now. The PU level does not disappear either (remember that hwloc topologies always contain a PU level at the bottom of the topology) even if there is a single PU object per Core parent.

$ lstopo -
...
  Core L#0
    PU L#0 (P#0)
  Core L#1
    PU L#1 (P#1)

How may I ignore symmetric multithreading, hyper-threading, etc. ?

First, see What happens to my topology if I disable symmetric multithreading, hyper-threading, etc. ? for more information about multithreading.

If you need to ignore symmetric multithreading in software, you should likely manipulate hwloc Core objects directly:

/* get the number of cores */
unsigned nbcores = hwloc_get_nbobjs_by_type(topology, HWLOC_OBJ_CORE);
...
/* get the third core below the first package */
hwloc_obj_t package, core;
package = hwloc_get_obj_by_type(topology, HWLOC_OBJ_PACKAGE, 0);
core = hwloc_get_obj_inside_cpuset_by_type(topology, package->cpuset,
                                           HWLOC_OBJ_CORE, 2);

Whenever you want to bind a process or thread to a core, make sure you singlify its cpuset first, so that the task is actually bound to a single thread within this core (to avoid useless migrations).

/* bind on the second core */
hwloc_obj_t core = hwloc_get_obj_by_type(topology, HWLOC_OBJ_CORE, 1);
hwloc_cpuset_t set = hwloc_bitmap_dup(core->cpuset);
hwloc_bitmap_singlify(set);
hwloc_set_cpubind(topology, set, 0);
hwloc_bitmap_free(set);

With hwloc-calc or hwloc-bind command-line tools, you may specify that you only want a single-thread within each core by asking for their first PU object:

$ hwloc-calc core:4-7
0x0000ff00
$ hwloc-calc core:4-7.pu:0
0x00005500

When binding a process on the command-line, you may either specify the exact thread that you want to use, or ask hwloc-bind to singlify the cpuset before binding

$ hwloc-bind core:3.pu:0 -- echo "hello from first thread on core #3"
hello from first thread on core #3
...
$ hwloc-bind core:3 --single -- echo "hello from a single thread on core #3"
hello from a single thread on core #3

What are these Group objects in my topology?

hwloc comes with a set of predefined object types (Core, Package, NUMA node, Caches) that match the vast majority of hardware platforms. The HWLOC_OBJ_GROUP type was designed for cases where this set is not sufficient. Groups may be used anywhere to add more structure information to the topology, for instance to show that 2 out of 4 NUMA nodes are actually closer than the others. When applicable, the Type info attribute describes why a Group was actually added (see also Custom string infos).

hwloc currently uses Groups for the following reasons:

What happens if my topology is asymmetric?

hwloc supports asymmetric topologies even if most platforms are usually symmetric. For example, there could be different types of processors in a single machine, each with different numbers of cores, symmetric multithreading, or levels of caches.

In practice, asymmetric topologies mostly appear when intermediate groups are added for I/O affinity: on a 4-package machine, an I/O bus may be connected to 2 packages. These packages are below an additional Group object, while the other packages are not (see also What are these Group objects in my topology?).

Before hwloc v2.0, hwloc_topology_ignore_type_keep_structure() and hwloc_topology_ignore_all_keep_structure() may also make topologies assymetric by removing parts of levels, especially when part of the machine is disallowed by administrator restrictions (e.g. Linux cgroups).

To understand how hwloc manages such cases, one should first remember the meaning of levels and cousin objects. All objects of the same type are gathered as horizontal levels with a given depth. They are also connected through the cousin pointers of the hwloc_obj structure. Some types, such as Caches or Groups, are annotated with a depth or level attribute (for instance L2 cache or Group1). Moreover caches have a type attribute (for instance L1i or L1d). Such attributes are also taken in account when gathering objects as horizontal levels. To be clear: there will be one level for L1i caches, another level for L1d caches, another one for L2, etc.

If the topology is asymmetric (e.g., if a group is missing above some processors), a given horizontal level will still exist if there exist any objects of that type. However, some branches of the overall tree may not have an object located in that horizontal level. Note that this specific hole within one horizontal level does not imply anything for other levels. All objects of the same type are gathered in horizontal levels even if their parents or children have different depths and types.

See the diagram in Terms and Definitions for a graphical representation of such topologies.

Moreover, it is important to understand that a same parent object may have children of different types (and therefore, different depths). These children are therefore siblings (because they have the same parent), but they are not cousins (because they do not belong to the same horizontal level).

How do I annotate the topology with private notes?

Each hwloc object contains a userdata field that may be used by applications to store private pointers. This field is only valid during the lifetime of these container object and topology. It becomes invalid as soon the topology is destroyed, or as soon as the object disappears, for instance when restricting the topology. The userdata field is not exported/imported to/from XML by default since hwloc does not know what it contains. This behavior may be changed by specifying application-specific callbacks with hwloc_topology_set_userdata_export_callback() and hwloc_topology_set_userdata_import_callback().

Each object may also contain some info attributes (key name and value) that are setup by hwloc during discovery and that may be extended by the user with hwloc_obj_add_info() (see also Object attributes). Contrary to the userdata field which is unique, multiple info attributes may exist for each object, even with the same name. These attributes are always exported to XML. However only character strings may be used as key names and values.

It is also possible to insert Misc objects with a custom name anywhere as a leaf of the topology (see Miscellaneous objects). And Misc objects may have their own userdata and info attributes just like any other object.

The hwloc-annotate command-line tool may be used for adding Misc objects and info attributes.

There is also a topology-specific userdata pointer that can be used to recognize different topologies by storing a custom pointer. It may be manipulated with hwloc_topology_set_userdata() and hwloc_topology_get_userdata().

Why does Valgrind complain about hwloc memory leaks?

If you are debugging your application with Valgrind, you want to avoid memory leak reports that are caused by hwloc and not by your program.

hwloc itself is often checked with Valgrind to make sure it does not leak memory. However some global variables in hwloc dependencies are never freed. For instance libz allocates its global state once at startup and never frees it so that it may be reused later. Some libxml2 global state is also never freed because hwloc does not know whether it can safely ask libxml2 to free it (the application may also be using libxml2 outside of hwloc).

These unfreed variables cause leak reports in Valgrind. hwloc installs a Valgrind suppressions file to hide them. You should pass the following command-line option to Valgrind to use it:

  --suppressions=/path/to/hwloc-valgrind.supp

How do I handle ABI breaks and API upgrades?

The hwloc interface is extended with every new major release. Any application using the hwloc API should be prepared to check at compile-time whether some features are available in the currently installed hwloc distribution.

For instance, to check whether the hwloc version is at least 1.10, you should use:

#include <hwloc.h>
#if HWLOC_API_VERSION >= 0x00010a00
...
#endif

The hwloc interface will be deeply modified in release 2.0 to fix several issues of the 1.x interface. The ABI will be broken, which means applications must be recompiled against the new 2.0 interface.

To check that you are not mixing old/recent headers with a recent/old runtime library, check the major revision number in the API version:

#include <hwloc.h>
  unsigned version = hwloc_get_api_version();
  if ((version >> 16) != (HWLOC_API_VERSION >> 16)) {
    fprintf(stderr,
           "%s compiled for hwloc API 0x%x but running on library API 0x%x.\n"
           "You may need to point LD_LIBRARY_PATH to the right hwloc library.\n"
           "Aborting since the new ABI is not backward compatible.\n",
           callname, HWLOC_API_VERSION, version);
    exit(EXIT_FAILURE);
  }

To specifically detect v2.0 issues:

#include <hwloc.h>
#if HWLOC_API_VERSION >= 0x00020000
  /* headers are recent */
  if (hwloc_get_api_version() < 0x20000)
    ... error out, the hwloc runtime library is older than 2.0 ...
#else
  /* headers are pre-2.0 */
  if (hwloc_get_api_version() >= 0x20000)
    ... error out, the hwloc runtime library is more recent than 2.0 ...
#endif

You should not try to remain compatible with very old releases such as 1.1.x or earlier because HWLOC_API_VERSION was added in 1.0.0 and hwloc_get_api_version() came only in 1.1.1. Also do not use the old cpuset API since it was deprecated and superseded by the bitmap API in 1.1, and later removed in 1.5.

How do I build hwloc for BlueGene/Q?

IBM BlueGene/Q machines run a standard Linux on the I/O node and a custom CNK (Compute Node Kernel) on the compute nodes. To run on the compute node, hwloc must be cross-compiled from the I/O node with the following configuration line:

./configure --host=powerpc64-bgq-linux --disable-shared --enable-static \
  CPPFLAGS='-I/bgsys/drivers/ppcfloor -I/bgsys/drivers/ppcfloor/spi/include/kernel/cnk/'

CPPFLAGS may have to be updated if your platform headers are installed in a different directory.

How to get useful topology information on NetBSD?

The NetBSD (and FreeBSD) backend uses x86-specific topology discovery (through the x86 component). This implementation requires CPU binding so as to query topology information from each individual logical processor. This means that hwloc cannot find any useful topology information unless user-level process binding is allowed by the NetBSD kernel. The security.models.extensions.user_set_cpu_affinity sysctl variable must be set to 1 to do so. Otherwise, only the number of logical processors will be detected.

How do I find the local MCDRAM NUMA node on Intel Knights Landing Xeon Phi?

Intel Knights Landing Xeon Phi processors introduce a new memory architecture by possibly having two distinct local memories: some normal memory (DDR) and some high-bandwidth on-package memory (MCDRAM). Processors can be configured in various clustering modes to have up to 4 Clusters. Moreover, each Cluster (quarter, half or whole processor) of the processor may have its own local parts of the DDR and of the MCDRAM.

The upcoming hwloc 2.0 will address this new architecture by presenting memory in an improved way. For now, starting with 1.11.2, hwloc releases use the following approximate representation:

If a cluster only contains DDR or MCDRAM but not both, that memory is available as a local NUMA node above cores as usual.

If a cluster contains both, two distinct NUMA nodes appear. They are sibling children of a Group object of type Cluster (or sibling children of the Package object for non-clustered processors).

The DDR memory is the local NUMA node above cores as usual. Allocating memory from one core to its local NUMA node will therefore actually allocate it on the normal memory by default.

The local high-bandwidth MCDRAM is the second NUMA node (without any Core or PU below it). It is the next sibling of the local DDR NUMA node below the same parent object. To allocate on the faster MCDRAM, one should first find the local NUMA node (the DDR memory, by looking up parent objects), and then take the next sibling to reach the local MCDRAM NUMA node (if any).

The MCDRAM NUMA nodes may also be identified thanks to the Type info attribute which is set to MCDRAM.

Why do I need hwloc-dump-hwdata for caches on Intel Knights Landing Xeon Phi?

Intel Knights Landing Xeon Phi processors may use the on-package memory (MCDRAM) as either memory or a memory-side cache (currently reported as a L3 cache by hwloc). Details about this cache (especially its size) are currently only available to privileged users.

The hwloc-dump-hwdata utility may be used to dump this privileged binary information into human-readable and world-accessible files that the hwloc library will later load. The utility should usually run as root once during boot, in order to update dumped information (stored under /var/run/hwloc by default) in case the MCDRAM configuration changed between reboots.

hwloc-dump-hwdata requires dmi-sysfs kernel module loaded.

The utility is currently unneeded on non-KNL platforms.

See HWLOC_DUMPED_HWDATA_DIR in Environment Variables for details about the location of dumped files.

How do I build for Intel Xeon Phi coprocessor?

Note
This section does not apply to standalone Intel Knights Landing Xeon Phi.

Intel Knights Corner Xeon Phi coprocessors usually runs a Linux environment but cross-compiling from the host is required. hwloc uses standard autotools options for cross-compiling. For instance, to build for a Knights Corner (KNC) coprocessor:

If building with icc:

./configure CC="icc -mmic" --host=x86_64-k1om-linux --build=x86_64-unknown-linux-gnu

If building with the Xeon Phi-specific GCC that comes with the MPSS environment, for instance /usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc:

export PATH=$PATH:/usr/linux-k1om-4.7/bin/
./configure --host=x86_64-k1om-linux --build=x86_64-unknown-linux-gnu