The COLL command

After the server replies to the IMPI commands from the clients, it is ready to start collecting other, opaque, startup information from them. This is done via the COLL command, which instructs the server to collect one payload from each of the clients and return the concatenation (in ascending client order) of all of them.

All COLL payloads sent from the clients to the server begin with the following struct:

        typedef struct {
                IMPI_Int4 label;
        } IMPI_Coll;

The label field marks the payloads as being of a certain kind; only buffers which share the same label will be concatenated by the server.

All COLL payloads sent from the server to the clients begin with the following struct:

        typedef struct {
                IMPI_Int4 label;
                IMPI_Int4 client_mask;
        } IMPI_Chdr;

In addition to the label field, this struct contains the client_mask field, which is a bitmask that identifies which clients have submitted values for this label. Bit i in the mask corresponds to the client with rank i (client rank as specified in the IMPI command, not an MPI_COMM_WORLD rank).

The following IMPI labels are currently defined (full explanations are provided below). C++ style comments are used for convenience.

        #define IMPI_NO_LABEL         0       // reserved for future use

        // Per client labels
        #define IMPI_C_VERSION        0x1000  // IMPI version(s)
        #define IMPI_C_NHOSTS         0x1100  // # of local hosts
        #define IMPI_C_NPROCS         0x1200  // # of local procs
        #define IMPI_C_PKTLEN         0x1300  // maximum packet length
        #define IMPI_C_TAGUB          0x1400  // maximum tag
        #define IMPI_C_COLL_XSIZE     0x1500  // coll. crossover size
        #define IMPI_C_COLL_MAXLINEAR 0x1600  // coll. crossover # of hosts

        // Per host labels
        #define IMPI_H_IPV6           0x2000  // IPv6 address
        #define IMPI_H_PORT           0x2100  // listening port
        #define IMPI_H_NPROCS         0x2200  // # procs per host
        #define IMPI_H_ACKMARK        0x2300  // ackmark flow control
        #define IMPI_H_HIWATER        0x2400  // hiwater flow control

        // Per proc labels
        #define IMPI_P_IPV6           0x3000  // IPv6 address
        #define IMPI_P_PID            0x3100  // pid

The current IMPI version is 0.0. All servers and clients for IMPI version 0.0 must implement all of the above labels, with the exception of IMPI_NO_LABEL.

To simplify server implementations, clients are required to pass labels in ascending numeric order. To simplify client implementations, servers are required to broadcast a concatenated buffer as soon as they receive complete sets of buffers from the clients.

Rationale.Ordering these labels allows the server to identify clients that have not implemented a particular label simply by observing the value of the current label sent from that client. Similarly, clients can ignore buffers they receive from the server with labels they do not understand. The exact order of these labels is not particularly significant except that the IMPI_C_NHOSTS label must precede any per-host labels so that clients can correctly interpret the concatenated buffers they receive for the per-host labels. Similarly, the IMPI_H_NPROCS label must precede any per-proc label.(End of rationale.)