cpuset - confine processes to processor and memory node subsets


   The  cpuset  filesystem  is a pseudo-filesystem interface to the kernel
   cpuset mechanism, which is used to control the processor placement  and
   memory placement of processes.  It is commonly mounted at /dev/cpuset.

   On systems with kernels compiled with built in support for cpusets, all
   processes are attached to a cpuset, and cpusets are always present.  If
   a  system supports cpusets, then it will have the entry nodev cpuset in
   the file /proc/filesystems.  By mounting the cpuset filesystem (see the
   EXAMPLE  section below), the administrator can configure the cpusets on
   a system to control the processor and memory placement of processes  on
   that  system.   By  default, if the cpuset configuration on a system is
   not modified or if the cpuset filesystem is not even mounted, then  the
   cpuset  mechanism,  though  present,  has  no  effect  on  the system's

   A cpuset defines a list of CPUs and memory nodes.

   The CPUs of a system include all the logical processing units on  which
   a  process can execute, including, if present, multiple processor cores
   within a package and Hyper-Threads within  a  processor  core.   Memory
   nodes  include all distinct banks of main memory; small and SMP systems
   typically have just one memory node that contains all the system's main
   memory,  while  NUMA  (non-uniform memory access) systems have multiple
   memory nodes.

   Cpusets are  represented  as  directories  in  a  hierarchical  pseudo-
   filesystem,  where  the  top  directory  in the hierarchy (/dev/cpuset)
   represents the entire system (all online CPUs and memory nodes) and any
   cpuset that is the child (descendant) of another parent cpuset contains
   a subset of that parent's CPUs and memory nodes.  The  directories  and
   files representing cpusets have normal filesystem permissions.

   Every  process  in the system belongs to exactly one cpuset.  A process
   is confined to run only on the CPUs in the cpuset it belongs to, and to
   allocate  memory  only  on  the  memory  nodes  in that cpuset.  When a
   process fork(2)s, the child process is placed in the same cpuset as its
   parent.   With  sufficient  privilege,  a process may be moved from one
   cpuset to another and the allowed CPUs and memory nodes of an  existing
   cpuset may be changed.

   When  the  system  begins  booting,  a  single  cpuset  is defined that
   includes all CPUs and memory nodes on the system, and all processes are
   in that cpuset.  During the boot process, or later during normal system
   operation, other cpusets may be created, as subdirectories of this  top
   cpuset,  under  the  control of the system administrator, and processes
   may be placed in these other cpusets.

   Cpusets  are  integrated  with  the   sched_setaffinity(2)   scheduling
   affinity  mechanism  and  the  mbind(2)  and  set_mempolicy(2)  memory-
   placement mechanisms in the kernel.  Neither of these mechanisms let  a
   process  make  use  of a CPU or memory node that is not allowed by that
   process's cpuset.  If changes to a process's cpuset placement  conflict
   with  these other mechanisms, then cpuset placement is enforced even if
   it means overriding these other mechanisms.   The  kernel  accomplishes
   this  overriding  by  silently  restricting  the  CPUs and memory nodes
   requested by these other mechanisms to those allowed  by  the  invoking
   process's  cpuset.   This  can result in these other calls returning an
   error, if for example, such a call ends up requesting an empty  set  of
   CPUs  or memory nodes, after that request is restricted to the invoking
   process's cpuset.

   Typically,  a  cpuset  is  used  to  manage  the  CPU  and  memory-node
   confinement  for  a  set  of  cooperating  processes  such  as  a batch
   scheduler job, and these  other  mechanisms  are  used  to  manage  the
   placement  of individual processes or memory regions within that set or


   Each directory below /dev/cpuset represents a  cpuset  and  contains  a
   fixed set of pseudo-files describing the state of that cpuset.

   New  cpusets are created using the mkdir(2) system call or the mkdir(1)
   command.  The properties of a cpuset, such as its flags,  allowed  CPUs
   and  memory  nodes, and attached processes, are queried and modified by
   reading or writing to the appropriate file in that cpuset's  directory,
   as listed below.

   The  pseudo-files  in  each  cpuset directory are automatically created
   when the cpuset is created, as a result of the mkdir(2) invocation.  It
   is not possible to directly add or remove these pseudo-files.

   A  cpuset  directory that contains no child cpuset directories, and has
   no attached processes, can be removed using rmdir(2) or  rmdir(1).   It
   is  not  necessary,  or possible, to remove the pseudo-files inside the
   directory before removing it.

   The pseudo-files in each cpuset directory are small text files that may
   be  read  and written using traditional shell utilities such as cat(1),
   and echo(1), or from a program by using file I/O library  functions  or
   system calls, such as open(2), read(2), write(2), and close(2).

   The  pseudo-files in a cpuset directory represent internal kernel state
   and do not have any persistent image on disk.  Each of these per-cpuset
   files is listed and described below.

   tasks  List  of the process IDs (PIDs) of the processes in that cpuset.
          The list is formatted as a series of ASCII decimal numbers, each
          followed  by  a  newline.   A  process  may be added to a cpuset
          (automatically removing  it  from  the  cpuset  that  previously
          contained  it)  by  writing  its PID to that cpuset's tasks file
          (with or without a trailing newline).

          Warning: only one PID may be written to  the  tasks  file  at  a
          time.   If  a string is written that contains more than one PID,
          only the first one will be used.

          Flag (0 or 1).  If set (1), that  cpuset  will  receive  special
          handling  after  it  is  released,  that is, after all processes
          cease using it (i.e., terminate or  are  moved  to  a  different
          cpuset) and all child cpuset directories have been removed.  See
          the Notify On Release section, below.

          List of the physical numbers of the CPUs on which  processes  in
          that cpuset are allowed to execute.  See List Format below for a
          description of the format of cpus.

          The CPUs allowed to a cpuset may be changed  by  writing  a  new
          list to its cpus file.

          Flag  (0 or 1).  If set (1), the cpuset has exclusive use of its
          CPUs (no  sibling  or  cousin  cpuset  may  overlap  CPUs).   By
          default,  this is off (0).  Newly created cpusets also initially
          default this to off (0).

          Two cpusets are sibling cpusets if they share  the  same  parent
          cpuset  in  the  /dev/cpuset  hierarchy.  Two cpusets are cousin
          cpusets if neither is the ancestor of the other.  Regardless  of
          the  cpu_exclusive  setting,  if  one  cpuset is the ancestor of
          another, and if both of these cpusets have nonempty  cpus,  then
          their  cpus  must  overlap,  because  the cpus of any cpuset are
          always a subset of the cpus of its parent cpuset.

          List of memory nodes on  which  processes  in  this  cpuset  are
          allowed  to  allocate  memory.   See  List  Format  below  for a
          description of the format of mems.

          Flag (0 or 1).  If set (1), the cpuset has exclusive use of  its
          memory  nodes  (no  sibling or cousin may overlap).  Also if set
          (1), the cpuset is a Hardwall cpuset (see below).   By  default,
          this  is  off (0).  Newly created cpusets also initially default
          this to off (0).

          Regardless of the mem_exclusive setting, if one  cpuset  is  the
          ancestor  of  another,  then  their  memory  nodes must overlap,
          because the memory nodes of any cpuset are always  a  subset  of
          the memory nodes of that cpuset's parent cpuset.

   cpuset.mem_hardwall (since Linux 2.6.26)
          Flag (0 or 1).  If set (1), the cpuset is a Hardwall cpuset (see
          below).  Unlike mem_exclusive, there is no constraint on whether
          cpusets  marked  mem_hardwall  may have overlapping memory nodes
          with sibling or cousin cpusets.  By default, this  is  off  (0).
          Newly created cpusets also initially default this to off (0).

   cpuset.memory_migrate (since Linux 2.6.16)
          Flag  (0  or  1).  If set (1), then memory migration is enabled.
          By default, this is off (0).  See the Memory Migration  section,

   cpuset.memory_pressure (since Linux 2.6.16)
          A  measure  of  how  much  memory pressure the processes in this
          cpuset are causing.  See the  Memory  Pressure  section,  below.
          Unless memory_pressure_enabled is enabled, always has value zero
          (0).  This file is read-only.  See the WARNINGS section, below.

   cpuset.memory_pressure_enabled (since Linux 2.6.16)
          Flag (0 or 1).  This file is present only in  the  root  cpuset,
          normally   /dev/cpuset.    If   set   (1),  the  memory_pressure
          calculations are enabled for all  cpusets  in  the  system.   By
          default,  this  is  off  (0).   See the Memory Pressure section,

   cpuset.memory_spread_page (since Linux 2.6.17)
          Flag (0 or 1).  If set (1),  pages  in  the  kernel  page  cache
          (filesystem buffers) are uniformly spread across the cpuset.  By
          default, this is off (0) in the top cpuset, and  inherited  from
          the  parent  cpuset  in  newly  created cpusets.  See the Memory
          Spread section, below.

   cpuset.memory_spread_slab (since Linux 2.6.17)
          Flag (0 or 1).  If set (1), the kernel slab caches for file  I/O
          (directory and inode structures) are uniformly spread across the
          cpuset.  By defaultBy default, is off (0) in the top cpuset, and
          inherited  from the parent cpuset in newly created cpusets.  See
          the Memory Spread section, below.

   cpuset.sched_load_balance (since Linux 2.6.24)
          Flag (0 or  1).   If  set  (1,  the  default)  the  kernel  will
          automatically  load  balance  processes  in that cpuset over the
          allowed CPUs in that cpuset.  If cleared  (0)  the  kernel  will
          avoid load balancing processes in this cpuset, unless some other
          cpuset with overlapping CPUs  has  its  sched_load_balance  flag
          set.  See Scheduler Load Balancing, below, for further details.

   cpuset.sched_relax_domain_level (since Linux 2.6.26)
          Integer,   between   -1   and   a  small  positive  value.   The
          sched_relax_domain_level controls the width of the range of CPUs
          over  which  the kernel scheduler performs immediate rebalancing
          of  runnable  tasks  across  CPUs.   If  sched_load_balance   is
          disabled,  then the setting of sched_relax_domain_level does not
          matter,   as   no   such   load   balancing   is    done.     If
          sched_load_balance  is enabled, then the higher the value of the
          sched_relax_domain_level, the wider the range of CPUs over which
          immediate  load  balancing  is  attempted.   See Scheduler Relax
          Domain Level, below, for further details.

   In  addition  to  the  above  pseudo-files  in  each  directory   below
   /dev/cpuset,  each  process has a pseudo-file, /proc/<pid>/cpuset, that
   displays the path of the process's cpuset  directory  relative  to  the
   root of the cpuset filesystem.

   Also the /proc/<pid>/status file for each process has four added lines,
   displaying  the  process's  Cpus_allowed  (on  which  CPUs  it  may  be
   scheduled)  and  Mems_allowed  (on  which  memory  nodes  it may obtain
   memory), in the two formats Mask Format and List Format (see below)  as
   shown in the following example:

          Cpus_allowed:   ffffffff,ffffffff,ffffffff,ffffffff
          Cpus_allowed_list:     0-127
          Mems_allowed:   ffffffff,ffffffff
          Mems_allowed_list:     0-63

   The  "allowed"  fields  were  added in Linux 2.6.24; the "allowed_list"
   fields were added in Linux 2.6.26.


   In addition to controlling which cpus and mems a process is allowed  to
   use, cpusets provide the following extended capabilities.

   Exclusive cpusets
   If  a cpuset is marked cpu_exclusive or mem_exclusive, no other cpuset,
   other than a direct ancestor or descendant, may share any of  the  same
   CPUs or memory nodes.

   A  cpuset that is mem_exclusive restricts kernel allocations for buffer
   cache pages and other internal kernel data pages commonly shared by the
   kernel  across  multiple  users.  All cpusets, whether mem_exclusive or
   not, restrict allocations of  memory  for  user  space.   This  enables
   configuring  a system so that several independent jobs can share common
   kernel data, while isolating each job's  user  allocation  in  its  own
   cpuset.  To do this, construct a large mem_exclusive cpuset to hold all
   the jobs, and  construct  child,  non-mem_exclusive  cpusets  for  each
   individual job.  Only a small amount of kernel memory, such as requests
   from interrupt handlers, is  allowed  to  be  placed  on  memory  nodes
   outside even a mem_exclusive cpuset.

   A  cpuset  that  has  mem_exclusive  or  mem_hardwall set is a hardwall
   cpuset.  A hardwall  cpuset  restricts  kernel  allocations  for  page,
   buffer,  and  other  data commonly shared by the kernel across multiple
   users.  All cpusets, whether hardwall or not, restrict  allocations  of
   memory for user space.

   This  enables configuring a system so that several independent jobs can
   share common kernel data, such as  filesystem  pages,  while  isolating
   each  job's user allocation in its own cpuset.  To do this, construct a
   large hardwall cpuset to hold all the jobs, and construct child cpusets
   for each individual job which are not hardwall cpusets.

   Only  a  small amount of kernel memory, such as requests from interrupt
   handlers, is allowed to be taken outside even a hardwall cpuset.

   Notify on release
   If the notify_on_release flag is enabled (1) in a cpuset, then whenever
   the  last process in the cpuset leaves (exits or attaches to some other
   cpuset) and the last child cpuset of that cpuset is removed, the kernel
   will run the command /sbin/cpuset_release_agent, supplying the pathname
   (relative to the mount point of the cpuset filesystem) of the abandoned
   cpuset.  This enables automatic removal of abandoned cpusets.

   The  default  value  of  notify_on_release in the root cpuset at system
   boot is disabled (0).  The default value of other cpusets  at  creation
   is the current value of their parent's notify_on_release setting.

   The  command  /sbin/cpuset_release_agent  is  invoked,  with  the  name
   (/dev/cpuset relative path) of the to-be-released cpuset in argv[1].

   The usual contents of the command /sbin/cpuset_release_agent is  simply
   the shell script:

       rmdir /dev/cpuset/$1

   As with other flag values below, this flag can be changed by writing an
   ASCII number 0 or 1 (with optional trailing newline) into the file,  to
   clear or set the flag, respectively.

   Memory pressure
   The  memory_pressure  of  a cpuset provides a simple per-cpuset running
   average of the rate that the processes in a cpuset  are  attempting  to
   free  up in-use memory on the nodes of the cpuset to satisfy additional
   memory requests.

   This enables  batch  managers  that  are  monitoring  jobs  running  in
   dedicated  cpusets  to efficiently detect what level of memory pressure
   that job is causing.

   This is useful both on tightly managed systems running a  wide  mix  of
   submitted jobs, which may choose to terminate or reprioritize jobs that
   are trying to use more memory than allowed on the nodes assigned  them,
   and  with  tightly coupled, long-running, massively parallel scientific
   computing jobs that will dramatically fail to meet required performance
   goals if they start to use more memory than allowed to them.

   This  mechanism provides a very economical way for the batch manager to
   monitor a cpuset for signs of memory pressure.  It's up  to  the  batch
   manager  or other user code to decide what action to take if it detects
   signs of memory pressure.

   Unless memory pressure calculation is enabled by  setting  the  pseudo-
   file /dev/cpuset/cpuset.memory_pressure_enabled, it is not computed for
   any cpuset, and reads from any memory_pressure always return  zero,  as
   represented  by  the  ASCII  string  "0\n".   See the WARNINGS section,

   A per-cpuset, running average is employed for the following reasons:

   *  Because this meter is per-cpuset  rather  than  per-process  or  per
      virtual  memory region, the system load imposed by a batch scheduler
      monitoring this metric is sharply reduced on large systems,  because
      a scan of the tasklist can be avoided on each set of queries.

   *  Because  this meter is a running average rather than an accumulating
      counter, a batch scheduler can detect memory pressure with a  single
      read,  instead of having to read and accumulate results for a period
      of time.

   *  Because this meter is per-cpuset rather than per-process, the  batch
      scheduler  can  obtain  the  key  information---memory  pressure  in a
      cpuset---with  a  single  read,  rather  than  having  to  query   and
      accumulate  results  over  all  the  (dynamically  changing)  set of
      processes in the cpuset.

   The memory_pressure of a cpuset is calculated using a per-cpuset simple
   digital  filter  that is kept within the kernel.  For each cpuset, this
   filter tracks the recent rate  at  which  processes  attached  to  that
   cpuset enter the kernel direct reclaim code.

   The  kernel  direct  reclaim  code is entered whenever a process has to
   satisfy a memory page request by  first  finding  some  other  page  to
   repurpose,  due  to  lack  of any readily available already free pages.
   Dirty filesystem pages are repurposed by first writing  them  to  disk.
   Unmodified  filesystem  buffer  pages are repurposed by simply dropping
   them, though if that page is needed again, it will have  to  be  reread
   from disk.

   The cpuset.memory_pressure file provides an integer number representing
   the recent (half-life of 10 seconds) rate  of  entries  to  the  direct
   reclaim  code caused by any process in the cpuset, in units of reclaims
   attempted per second, times 1000.

   Memory spread
   There are two Boolean flag files per  cpuset  that  control  where  the
   kernel allocates pages for the filesystem buffers and related in-kernel
   data  structures.   They  are  called   cpuset.memory_spread_page   and

   If  the  per-cpuset Boolean flag file cpuset.memory_spread_page is set,
   then the kernel will spread the filesystem buffers (page cache)  evenly
   over all the nodes that the faulting process is allowed to use, instead
   of preferring to put those pages on  the  node  where  the  process  is

   If  the  per-cpuset Boolean flag file cpuset.memory_spread_slab is set,
   then the kernel will spread some filesystem-related slab  caches,  such
   as  those  for  inodes and directory entries, evenly over all the nodes
   that the faulting process is allowed to use, instead of  preferring  to
   put those pages on the node where the process is running.

   The  setting  of  these  flags  does  not  affect the data segment (see
   brk(2)) or stack segment pages of a process.

   By default, both kinds of memory  spreading  are  off  and  the  kernel
   prefers  to  allocate  memory  pages  on  the  node  local to where the
   requesting process is running.  If that node  is  not  allowed  by  the
   process's  NUMA  memory  policy or cpuset configuration or if there are
   insufficient free memory pages on that node, then the kernel looks  for
   the nearest node that is allowed and has sufficient free memory.

   When  new  cpusets are created, they inherit the memory spread settings
   of their parent.

   Setting memory spreading causes allocations for the  affected  page  or
   slab  caches  to  ignore the process's NUMA memory policy and be spread
   instead.  However, the effect of  these  changes  in  memory  placement
   caused by cpuset-specified memory spreading is hidden from the mbind(2)
   or set_mempolicy(2) calls.  These two NUMA memory policy  calls  always
   appear  to  behave  as  if  no  cpuset-specified memory spreading is in
   effect, even if it is.  If  cpuset  memory  spreading  is  subsequently
   turned  off,  the  NUMA  memory policy most recently specified by these
   calls is automatically reapplied.

   Both  cpuset.memory_spread_page   and   cpuset.memory_spread_slab   are
   Boolean  flag  files.   By  default, they contain "0", meaning that the
   feature is off for that cpuset.  If a "1" is written to that file, that
   turns the named feature on.

   Cpuset-specified  memory  spreading  behaves similarly to what is known
   (in other contexts) as round-robin or interleave memory placement.

   Cpuset-specified memory spreading can provide  substantial  performance
   improvements for jobs that:

   a) need  to  place  thread-local data on memory nodes close to the CPUs
      which are running the threads that most frequently access that data;
      but also

   b) need  to  access  large  filesystem data sets that must to be spread
      across the several nodes in the job's cpuset in order to fit.

   Without this policy, the memory allocation  across  the  nodes  in  the
   job's  cpuset  can  become  very uneven, especially for jobs that might
   have just a single thread initializing or reading in the data set.

   Memory migration
   Normally,    under    the     default     setting     (disabled)     of
   cpuset.memory_migrate,  once a page is allocated (given a physical page
   of main  memory),  then  that  page  stays  on  whatever  node  it  was
   allocated,  so  long  as  it  remains  allocated,  even if the cpuset's
   memory-placement policy mems subsequently changes.

   When memory migration is enabled in a cpuset, if the  mems  setting  of
   the  cpuset  is  changed, then any memory page in use by any process in
   the cpuset that is on a memory node that is no longer allowed  will  be
   migrated to a memory node that is allowed.

   Furthermore,  if  a  process is moved into a cpuset with memory_migrate
   enabled, any memory pages it uses that were on memory nodes allowed  in
   its  previous cpuset, but which are not allowed in its new cpuset, will
   be migrated to a memory node allowed in the new cpuset.

   The relative  placement  of  a  migrated  page  within  the  cpuset  is
   preserved  during these migration operations if possible.  For example,
   if the page was on the second valid node of the prior cpuset, then  the
   page  will  be  placed  on  the second valid node of the new cpuset, if

   Scheduler load balancing
   The kernel scheduler automatically load balances processes.  If one CPU
   is  underutilized,  the  kernel  will  look for processes on other more
   overloaded CPUs and move those  processes  to  the  underutilized  CPU,
   within  the  constraints  of  such  placement mechanisms as cpusets and

   The algorithmic cost of load balancing and its  impact  on  key  shared
   kernel  data  structures  such  as the process list increases more than
   linearly with the number of CPUs being balanced.  For example, it costs
   more  to  load  balance  across  one  large set of CPUs than it does to
   balance across two smaller sets of CPUs, each of half the size  of  the
   larger set.  (The precise relationship between the number of CPUs being
   balanced and the cost  of  load  balancing  depends  on  implementation
   details  of  the  kernel  process scheduler, which is subject to change
   over time, as improved kernel scheduler algorithms are implemented.)

   The per-cpuset flag sched_load_balance provides a mechanism to suppress
   this automatic scheduler load balancing in cases where it is not needed
   and suppressing it would have worthwhile performance benefits.

   By default, load balancing is done across all CPUs, except those marked
   isolated  using  the  kernel  boot  time  "isolcpus="  argument.   (See
   Scheduler Relax Domain Level, below, to change this default.)

   This default load balancing across all CPUs is not well suited  to  the
   following two situations:

   *  On  large systems, load balancing across many CPUs is expensive.  If
      the system is managed using cpusets to  place  independent  jobs  on
      separate sets of CPUs, full load balancing is unnecessary.

   *  Systems  supporting  real-time  on some CPUs need to minimize system
      overhead on those CPUs, including avoiding process load balancing if
      that is not needed.

   When  the  per-cpuset  flag  sched_load_balance is enabled (the default
   setting), it requests load  balancing  across  all  the  CPUs  in  that
   cpuset's  allowed CPUs, ensuring that load balancing can move a process
   (not otherwise pinned, as by sched_setaffinity(2)) from any CPU in that
   cpuset to any other.

   When  the  per-cpuset  flag  sched_load_balance  is  disabled, then the
   scheduler will avoid load balancing across the  CPUs  in  that  cpuset,
   except  in  so  far as is necessary because some overlapping cpuset has
   sched_load_balance enabled.

   So, for example, if the top  cpuset  has  the  flag  sched_load_balance
   enabled,  then the scheduler will load balance across all CPUs, and the
   setting of the sched_load_balance flag in other cpusets has no  effect,
   as we're already fully load balancing.

   Therefore  in  the  above  two  situations, the flag sched_load_balance
   should be disabled in the top cpuset, and only  some  of  the  smaller,
   child cpusets would have this flag enabled.

   When doing this, you don't usually want to leave any unpinned processes
   in the top cpuset that might use nontrivial amounts  of  CPU,  as  such
   processes  may  be  artificially  constrained  to  some subset of CPUs,
   depending on  the  particulars  of  this  flag  setting  in  descendant
   cpusets.   Even  if  such  a process could use spare CPU cycles in some
   other CPUs, the kernel scheduler might not consider the possibility  of
   load balancing that process to the underused CPU.

   Of course, processes pinned to a particular CPU can be left in a cpuset
   that  disables  sched_load_balance  as  those  processes  aren't  going
   anywhere else anyway.

   Scheduler relax domain level
   The  kernel  scheduler performs immediate load balancing whenever a CPU
   becomes free or another task becomes  runnable.   This  load  balancing
   works  to  ensure  that  as many CPUs as possible are usefully employed
   running tasks.  The kernel also performs periodic  load  balancing  off
   the   software   clock   described   in   time(7).    The   setting  of
   sched_relax_domain_level applies  only  to  immediate  load  balancing.
   Regardless  of  the  sched_relax_domain_level  setting,  periodic  load
   balancing is attempted over all CPUs (unless disabled  by  turning  off
   sched_load_balance.)   In  any case, of course, tasks will be scheduled
   to  run  only  on  CPUs  allowed  by  their  cpuset,  as  modified   by
   sched_setaffinity(2) system calls.

   On  small  systems,  such as those with just a few CPUs, immediate load
   balancing is useful to improve system  interactivity  and  to  minimize
   wasteful  idle  CPU cycles.  But on large systems, attempting immediate
   load balancing across a large number of CPUs can be more costly than it
   is  worth,  depending  on the particular performance characteristics of
   the job mix and the hardware.

   The   exact    meaning    of    the    small    integer    values    of
   sched_relax_domain_level will depend on internal implementation details
   of the kernel scheduler code and on the non-uniform architecture of the
   hardware.   Both  of  these  will  evolve  over time and vary by system
   architecture and kernel version.

   As of this writing,  when  this  capability  was  introduced  in  Linux
   2.6.26,  on  certain  popular  architectures,  the  positive  values of
   sched_relax_domain_level have the following meanings.

   (1) Perform immediate load balancing across  Hyper-Thread  siblings  on
       the same core.
   (2) Perform  immediate  load  balancing  across other cores in the same
   (3) Perform immediate load balancing across other CPUs on the same node
       or blade.
   (4) Perform    immediate    load    balancing   across   over   several
       (implementation detail) nodes [On NUMA systems].
   (5) Perform immediate load balancing across over all CPUs in system [On
       NUMA systems].

   The  sched_relax_domain_level  value  of  zero  (0)  always means don't
   perform immediate load balancing, hence that  load  balancing  is  done
   only  periodically,  not  immediately  when  a CPU becomes available or
   another task becomes runnable.

   The sched_relax_domain_level value of minus one (-1) always  means  use
   the  system  default  value.   The  system  default  value  can vary by
   architecture and kernel version.  This  system  default  value  can  be
   changed by kernel boot-time "relax_domain_level=" argument.

   In  the  case  of  multiple  overlapping cpusets which have conflicting
   sched_relax_domain_level values, then the highest such value applies to
   all  CPUs  in any of the overlapping cpusets.  In such cases, the value
   minus one (-1) is the lowest value, overridden by any other value,  and
   the value zero (0) is the next lowest value.


   The  following  formats  are  used to represent sets of CPUs and memory

   Mask format
   The Mask Format is used to represent CPU and memory-node bit  masks  in
   the /proc/<pid>/status file.

   This  format  displays  each  32-bit  word  in hexadecimal (using ASCII
   characters "0" - "9" and "a" - "f");  words  are  filled  with  leading
   zeros,  if required.  For masks longer than one word, a comma separator
   is used between words.  Words are displayed in big-endian order,  which
   has  the  most significant bit first.  The hex digits within a word are
   also in big-endian order.

   The number of 32-bit words displayed is the minimum  number  needed  to
   display all bits of the bit mask, based on the size of the bit mask.

   Examples of the Mask Format:

          00000001                        # just bit 0 set
          40000000,00000000,00000000      # just bit 94 set
          00000001,00000000,00000000      # just bit 64 set
          000000ff,00000000               # bits 32-39 set
          00000000,000e3862               # 1,5,6,11-13,17-19 set

   A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:


   The  first  "1" is for bit 64, the second for bit 32, the third for bit
   16, the fourth for bit 8, the fifth for bit 4, and the "7" is for  bits
   2, 1, and 0.

   List format
   The  List  Format for cpus and mems is a comma-separated list of CPU or
   memory-node numbers and ranges of numbers, in ASCII decimal.

   Examples of the List Format:

          0-4,9           # bits 0, 1, 2, 3, 4, and 9 set
          0-2,7,12-14     # bits 0, 1, 2, 7, 12, 13, and 14 set


   The following rules apply to each cpuset:

   *  Its CPUs and memory nodes must be a (possibly equal) subset  of  its

   *  It can be marked cpu_exclusive only if its parent is.

   *  It can be marked mem_exclusive only if its parent is.

   *  If it is cpu_exclusive, its CPUs may not overlap any sibling.

   *  If  it  is  memory_exclusive,  its  memory nodes may not overlap any


   The permissions of a cpuset are determined by the  permissions  of  the
   directories and pseudo-files in the cpuset filesystem, normally mounted
   at /dev/cpuset.

   For instance, a process can put itself in some other cpuset  (than  its
   current  one)  if  it  can  write the tasks file for that cpuset.  This
   requires execute permission on the encompassing directories  and  write
   permission on the tasks file.

   An  additional  constraint  is  applied to requests to place some other
   process in a cpuset.  One process may not attach another  to  a  cpuset
   unless  it  would  have  permission  to send that process a signal (see

   A process may create a child cpuset if it  can  access  and  write  the
   parent  cpuset  directory.  It can modify the CPUs or memory nodes in a
   cpuset if it can access that cpuset's directory (execute permissions on
   the each of the parent directories) and write the corresponding cpus or
   mems file.

   There is one  minor  difference  between  the  manner  in  which  these
   permissions  are  evaluated  and  the manner in which normal filesystem
   operation permissions are evaluated.  The  kernel  interprets  relative
   pathnames  starting  at a process's current working directory.  Even if
   one is operating on a cpuset file, relative pathnames  are  interpreted
   relative  to  the  process's current working directory, not relative to
   the process's current cpuset.  The only ways that cpuset paths relative
   to  a  process's current cpuset can be used are if either the process's
   current working directory is its cpuset (it first did a cd or  chdir(2)
   to its cpuset directory beneath /dev/cpuset, which is a bit unusual) or
   if some  user  code  converts  the  relative  cpuset  path  to  a  full
   filesystem path.

   In  theory,  this  means  that  user  code should specify cpusets using
   absolute pathnames, which requires  knowing  the  mount  point  of  the
   cpuset  filesystem  (usually,  but  not  necessarily, /dev/cpuset).  In
   practice, all user level code that  this  author  is  aware  of  simply
   assumes that if the cpuset filesystem is mounted, then it is mounted at
   /dev/cpuset.  Furthermore, it is common practice for carefully  written
   user  code  to verify the presence of the pseudo-file /dev/cpuset/tasks
   in order to verify  that  the  cpuset  pseudo-filesystem  is  currently


   Enabling memory_pressure
   By  default, the per-cpuset file cpuset.memory_pressure always contains
   zero (0).  Unless this feature is enabled by writing "1" to the pseudo-
   file  /dev/cpuset/cpuset.memory_pressure_enabled,  the  kernel does not
   compute per-cpuset memory_pressure.

   Using the echo command
   When using the echo command at the shell prompt to change the values of
   cpuset files, beware that the built-in echo command in some shells does
   not display an error message if the write(2) system  call  fails.   For
   example, if the command:

       echo 19 > cpuset.mems

   failed  because  memory  node  19  was not allowed (perhaps the current
   system does not have a memory node 19), then the echo command might not
   display  any error.  It is better to use the /bin/echo external command
   to change cpuset file settings, as this command will  display  write(2)
   errors, as in the example:

       /bin/echo 19 > cpuset.mems
       /bin/echo: write error: Invalid argument


   Memory placement
   Not  all  allocations  of system memory are constrained by cpusets, for
   the following reasons.

   If hot-plug functionality is used to  remove  all  the  CPUs  that  are
   currently  assigned  to  a  cpuset,  then the kernel will automatically
   update the cpus_allowed of all  processes  attached  to  CPUs  in  that
   cpuset  to  allow  all  CPUs.   When  memory hot-plug functionality for
   removing memory nodes is available, a similar exception is expected  to
   apply  there as well.  In general, the kernel prefers to violate cpuset
   placement, rather than starving a process that has had all its  allowed
   CPUs  or  memory  nodes  taken  offline.   User code should reconfigure
   cpusets to refer only to online CPUs and memory nodes when  using  hot-
   plug to add or remove such resources.

   A  few  kernel-critical,  internal  memory-allocation  requests, marked
   GFP_ATOMIC, must be satisfied immediately.  The kernel  may  drop  some
   request  or  malfunction  if  one of these allocations fail.  If such a
   request cannot be satisfied within the current process's  cpuset,  then
   we relax the cpuset, and look for memory anywhere we can find it.  It's
   better to violate the cpuset than stress the kernel.

   Allocations of memory requested by kernel drivers while  processing  an
   interrupt  lack  any  relevant process context, and are not confined by

   Renaming cpusets
   You can use the rename(2) system call to rename cpusets.   Only  simple
   renaming is supported; that is, changing the name of a cpuset directory
   is permitted, but moving a directory into a different directory is  not


   The  Linux  kernel  implementation of cpusets sets errno to specify the
   reason for a failed system call affecting cpusets.

   The possible errno settings and their meaning  when  set  on  a  failed
   cpuset call are as listed below.

   E2BIG  Attempted  a  write(2)  on  a  special cpuset file with a length
          larger than some kernel-determined upper limit on the length  of
          such writes.

   EACCES Attempted  to  write(2)  the  process ID (PID) of a process to a
          cpuset tasks  file  when  one  lacks  permission  to  move  that

   EACCES Attempted  to  add,  using  write(2),  a CPU or memory node to a
          cpuset, when that CPU or memory node  was  not  already  in  its

   EACCES Attempted   to  set,  using  write(2),  cpuset.cpu_exclusive  or
          cpuset.mem_exclusive on a cpuset whose  parent  lacks  the  same

   EACCES Attempted to write(2) a cpuset.memory_pressure file.

   EACCES Attempted to create a file in a cpuset directory.

   EBUSY  Attempted  to  remove,  using  rmdir(2),  a cpuset with attached

   EBUSY  Attempted  to  remove,  using  rmdir(2),  a  cpuset  with  child

   EBUSY  Attempted  to  remove a CPU or memory node from a cpuset that is
          also in a child of that cpuset.

   EEXIST Attempted to create,  using  mkdir(2),  a  cpuset  that  already

   EEXIST Attempted to rename(2) a cpuset to a name that already exists.

   EFAULT Attempted  to  read(2)  or write(2) a cpuset file using a buffer
          that is outside the writing processes accessible address space.

   EINVAL Attempted to change a cpuset, using  write(2),  in  a  way  that
          would violate a cpu_exclusive or mem_exclusive attribute of that
          cpuset or any of its siblings.

   EINVAL Attempted to write(2) an empty cpuset.cpus or  cpuset.mems  list
          to a cpuset which has attached processes or child cpusets.

   EINVAL Attempted  to  write(2)  a cpuset.cpus or cpuset.mems list which
          included a range with the second number smaller than  the  first

   EINVAL Attempted  to  write(2)  a cpuset.cpus or cpuset.mems list which
          included an invalid character in the string.

   EINVAL Attempted to write(2) a list to a cpuset.cpus file that did  not
          include any online CPUs.

   EINVAL Attempted  to write(2) a list to a cpuset.mems file that did not
          include any online memory nodes.

   EINVAL Attempted to write(2) a list to a cpuset.mems file that included
          a node that held no memory.

   EIO    Attempted  to write(2) a string to a cpuset tasks file that does
          not begin with an ASCII decimal integer.

   EIO    Attempted to rename(2) a cpuset into a different directory.

          Attempted to read(2) a /proc/<pid>/cpuset file for a cpuset path
          that is longer than the kernel page size.

          Attempted  to  create,  using  mkdir(2),  a  cpuset  whose  base
          directory name is longer than 255 characters.

          Attempted  to  create,  using  mkdir(2),  a  cpuset  whose  full
          pathname,  including  the mount point (typically "/dev/cpuset/")
          prefix, is longer than 4095 characters.

   ENODEV The cpuset was removed by another process at the same time as  a
          write(2)  was attempted on one of the pseudo-files in the cpuset

   ENOENT Attempted to create, using mkdir(2), a cpuset in a parent cpuset
          that doesn't exist.

   ENOENT Attempted to access(2) or open(2) a nonexistent file in a cpuset

   ENOMEM Insufficient memory is available within the kernel; can occur on
          a  variety  of  system  calls affecting cpusets, but only if the
          system is extremely short of memory.

   ENOSPC Attempted to write(2) the process ID (PID) of  a  process  to  a
          cpuset  tasks  file  when the cpuset had an empty cpuset.cpus or
          empty cpuset.mems setting.

   ENOSPC Attempted  to  write(2)  an  empty  cpuset.cpus  or  cpuset.mems
          setting to a cpuset that has tasks attached.

          Attempted to rename(2) a nonexistent cpuset.

   EPERM  Attempted to remove a file from a cpuset directory.

   ERANGE Specified  a cpuset.cpus or cpuset.mems list to the kernel which
          included a number too large for the kernel to  set  in  its  bit

   ESRCH  Attempted  to  write(2)  the  process  ID (PID) of a nonexistent
          process to a cpuset tasks file.


   Cpusets appeared in version 2.6.12 of the Linux kernel.


   Despite its name, the pid parameter is actually a thread ID,  and  each
   thread  in a threaded group can be attached to a different cpuset.  The
   value returned from a call to gettid(2) can be passed in  the  argument


   cpuset.memory_pressure   cpuset   files  can  be  opened  for  writing,
   creation, or truncation, but then the write(2) fails with errno set  to
   EACCES,  and  the  creation  and  truncation options on open(2) have no


   The following examples demonstrate querying and setting cpuset  options
   using shell commands.

   Creating and attaching to a cpuset.
   To  create a new cpuset and attach the current command shell to it, the
   steps are:

   1)  mkdir /dev/cpuset (if not already done)
   2)  mount -t cpuset none /dev/cpuset (if not already done)
   3)  Create the new cpuset using mkdir(1).
   4)  Assign CPUs and memory nodes to the new cpuset.
   5)  Attach the shell to the new cpuset.

   For example, the following sequence of commands will set  up  a  cpuset
   named  "Charlie",  containing just CPUs 2 and 3, and memory node 1, and
   then attach the current shell to that cpuset.

       $ mkdir /dev/cpuset
       $ mount -t cpuset cpuset /dev/cpuset
       $ cd /dev/cpuset
       $ mkdir Charlie
       $ cd Charlie
       $ /bin/echo 2-3 > cpuset.cpus
       $ /bin/echo 1 > cpuset.mems
       $ /bin/echo $$ > tasks
       # The current shell is now running in cpuset Charlie
       # The next line should display '/Charlie'
       $ cat /proc/self/cpuset

   Migrating a job to different memory nodes.
   To migrate a job (the  set  of  processes  attached  to  a  cpuset)  to
   different  CPUs  and  memory  nodes in the system, including moving the
   memory pages currently allocated to that  job,  perform  the  following

   1)  Let's  say  we  want  to move the job in cpuset alpha (CPUs 4-7 and
       memory nodes 2-3) to a new cpuset beta (CPUs 16-19 and memory nodes
   2)  First create the new cpuset beta.
   3)  Then allow CPUs 16-19 and memory nodes 8-9 in beta.
   4)  Then enable memory_migration in beta.
   5)  Then move each process from alpha to beta.

   The following sequence of commands accomplishes this.

       $ cd /dev/cpuset
       $ mkdir beta
       $ cd beta
       $ /bin/echo 16-19 > cpuset.cpus
       $ /bin/echo 8-9 > cpuset.mems
       $ /bin/echo 1 > cpuset.memory_migrate
       $ while read i; do /bin/echo $i; done < ../alpha/tasks > tasks

   The  above  should  move any processes in alpha to beta, and any memory
   held by these processes on  memory  nodes  2-3  to  memory  nodes  8-9,

   Notice that the last step of the above sequence did not do:

       $ cp ../alpha/tasks tasks

   The  while  loop,  rather  than  the  seemingly easier use of the cp(1)
   command, was necessary because only one process PID at a  time  may  be
   written to the tasks file.

   The  same  effect  (writing one PID at a time) as the while loop can be
   accomplished more efficiently, in fewer keystrokes and in  syntax  that
   works  on  any  shell,  but  alas  more  obscurely,  by  using  the  -u
   (unbuffered) option of sed(1):

       $ sed -un p < ../alpha/tasks > tasks


   taskset(1),        get_mempolicy(2),        getcpu(2),        mbind(2),
   sched_getaffinity(2),    sched_setaffinity(2),   sched_setscheduler(2),
   set_mempolicy(2), CPU_SET(3), proc(5), cgroups(7),  numa(7),  sched(7),
   migratepages(8), numactl(8)

   Documentation/cpusets.txt in the Linux kernel source tree


   This  page  is  part of release 4.09 of the Linux man-pages project.  A
   description of the project, information about reporting bugs,  and  the
   latest     version     of     this    page,    can    be    found    at


Personal Opportunity - Free software gives you access to billions of dollars of software at no cost. Use this software for your business, personal use or to develop a profitable skill. Access to source code provides access to a level of capabilities/information that companies protect though copyrights. Open source is a core component of the Internet and it is available to you. Leverage the billions of dollars in resources and capabilities to build a career, establish a business or change the world. The potential is endless for those who understand the opportunity.

Business Opportunity - Goldman Sachs, IBM and countless large corporations are leveraging open source to reduce costs, develop products and increase their bottom lines. Learn what these companies know about open source and how open source can give you the advantage.

Free Software

Free Software provides computer programs and capabilities at no cost but more importantly, it provides the freedom to run, edit, contribute to, and share the software. The importance of free software is a matter of access, not price. Software at no cost is a benefit but ownership rights to the software and source code is far more significant.

Free Office Software - The Libre Office suite provides top desktop productivity tools for free. This includes, a word processor, spreadsheet, presentation engine, drawing and flowcharting, database and math applications. Libre Office is available for Linux or Windows.

Free Books

The Free Books Library is a collection of thousands of the most popular public domain books in an online readable format. The collection includes great classical literature and more recent works where the U.S. copyright has expired. These books are yours to read and use without restrictions.

Source Code - Want to change a program or know how it works? Open Source provides the source code for its programs so that anyone can use, modify or learn how to write those programs themselves. Visit the GNU source code repositories to download the source.


Study at Harvard, Stanford or MIT - Open edX provides free online courses from Harvard, MIT, Columbia, UC Berkeley and other top Universities. Hundreds of courses for almost all major subjects and course levels. Open edx also offers some paid courses and selected certifications.

Linux Manual Pages - A man or manual page is a form of software documentation found on Linux/Unix operating systems. Topics covered include computer programs (including library and system calls), formal standards and conventions, and even abstract concepts.