QMP over GM version 1.2.0

Jie Chen, Robert Edwards and William Watson III
Jefferson Lab
Feb 13, 2003

 
QMP (QCD Message Passing) over GM is implemented according to LQCD Message Passing API Specification version 1.0.0 of April. 25, 2002. This document summarizes what routines are implemented and what routines are not supported at this moment. In addition this document explains some of the implementations and describes behavior of some routines.
 
 
unsigned char QMP_u8_t
unsigned short QMP_u16_t
unsigned int QMP_u32_t
char QMP_s8_t
short QMP_s16_t
int QMP_s32_t
int QMP_bool_t
int QMP_status_t
float QMP_float_t
double QMP_double_t
unsigned long (64 bit Arch) QMP_u64_t
long (64 bit Arch) QMP_s64_t
unsigned long long QMP_u64_t
long long  QMP_s64_t
 
QMP_SUCCESS = 0 QMP_ERROR QMP_NOT_INITED QMP_RTENV_ERR
QMP_CPUINFO_ERR QMP_NODEINFO_ERR QMP_NOMEM_ERR QMP_MEMSIZE_ERR
QMP_HOSTNAME_ERR QMP_INITSVC_ERR QMP_TOPOLOGY_EXISTS QMP_CH_TIMEOUT
QMP_NOTSUPPORTED QMP_SVC_BUSY QMP_BAD_MESSAGE QMP_INVALID_ARG
QMP_INVALID_TOPOLOGY QMP_NONEIGHBOR_INFO QMP_MEMSIZE_TOOBIG QMP_BAD_MEMORY
QMP_NO_PORTS QMP_NODE_OUTRANGE QMP_CHDEF_ERR QMP_MEMUSED_ERR
QMP_INVALID_OP QMP_TIMEOUT
  • QMP Interconnect type (QMP_ictype_t):

    •  

     

    QMP_SWITCH=0 QMP_GRID=1 QMP_MESH=1 QMP_FATTREE=2
  • QMP SMP Address Type (QMP_smpaddr_type_t):

    •  
    QMP_SMP_ONE_ADDRESS QMP_SMP_MULTIPLE_ADDRESS
    Done QMP_status_t          QMP_init_msg_passing (int* argc, char*** argv, QMP_smpaddr_type_t smp_type); Done void                          QMP_finalize_msg_passing (void);
    Done void                          QMP_abort (QMP_s32_t error_code);
    Done void                          QMP_verbose  (QMP_bool_t verbose);
    Done const QMP_u32_t    QMP_get_SMP_count (void);
    Done const QMP_ictype_t QMP_get_msg_passing_type (void); Done const QMP_u32_t    QMP_get_number_of_nodes (void);
    Done const QMP_u32_t    QMP_get_allocated_number_of_dimensions (void);
    Done const QMP_u32_t*   QMP_get_allocated_dimensions (void); Done const QMP_u32_t*   QMP_get_allocated_coordinates (void); Done QMP_bool_t            QMP_declare_logical_topology (const QMP_u32_t* dims, QMP_u32_t ndim);
    Done QMP_bool_t            QMP_logical_topology_is_declared (void);
    Done const QMP_u32_t    QMP_get_logical_number_of_dimensions (void);
    Done const QMP_u32_t*   QMP_get_logical_dimensions (void);
    Done const QMP_u32_t    QMP_get_node_number (void);
    Done const QMP_u32_t    QMP_get_number_of_nodes (void);
    Done const QMP_u32_t*   QMP_get_logical_coordinates (void);
    Done const QMP_u32_t*   QMP_get_logical_coordinates_from (QMP_u32_t node);
    Done const QMP_u32_t    QMP_get_node_number_from (const QMP_u32_t* coordinates); Not Done void                          QMP_layout_grid (const QMP_u32_t* latticeDimensions, QMP_u32_t ndimes);
    Not Done const QMP_u32_t*  QMP_get_subgrid_dimensions (void);
      Done void*                         QMP_allocate_aligned_memory (QMP_u32_t nbytes);
    • Allocate memory aligned on 16 byte boundary
    Done void QMP_free_aligned_memory (void* mem);
    Done QMP_msgmem_t    QMP_declare_msgmem (const void* mem,  QMP_u32_t nbytes);
    Done void                          QMP_free_msgmem (QMP_msgmem_t m)
    • User memory is not freed.
    Not Done QMP_msgmem_t    QMP_declare_strided_msg_mem (void* base, QMP_u32_t blksize,QMP_u32_t nblocks, QMP_u32_t stride,);
    Not Done QMP_msgmem_t    QMP_declare_strided_array_msgmem (void** base,QMP_u32_t* blksize, QMP_u32_t* nblocks, QMP_u32_t*stride);
      Done QMP_msghandle_t    QMP_declare_receive_relative (QMP_msgmem_t m, QMP_s32_t axis, QMP_s32_t dir, QMP_s32_t priority);
    Done QMP_msghandle_t    QMP_declare_send_relative    (QMP_msgmem_t m, QMP_s32_t axis, QMP_s32_t dir, QMP_s32_t priority);
    Done QMP_msghandle_t    QMP_declare_send_to (QMP_msgmem_t m, QMP_u32_t rem_node, QMP_s32_t priority);
    Done QMP_msghandle_t    QMP_declare_receive_from(QMP_msgmem_t m,  QMP_u32_t rem_node_rank, QMP_s32_t priority);
    Done QMP_msghandle_t    QMP_declare_multiple (QMP_msghandle_t* msgh, QMP_u32_t num)
    Done void                             QMP_free_msghandle (QMP_msghandle_t h);


    Attention Advisory for QMP GM: Calling QMP_declare_receive_zzz before QMP_declare_send_zzz guarantees connections of desired channels. Otherwise best effort will be made to establish connections. Multiple channels are allowed between a pair of nodes.
     
     

    Done QMP_status_t           QMP_start (QMP_msghandle_t h);
    Done QMP_status_t           QMP_wait (QMP_msghandle_t h);
    Done QMP_bool_t            QMP_is_complete (QMP_msghandle_t h);
    Done QMP_status_t QMP_broadcast (void* buffer, QMP_u32_t nbytes);
    Done QMP_status_t QMP_sum_int    (QMP_s32_t *value);
    Done QMP_status_t QMP_sum_float  (QMP_float_t *value);
    Done QMP_status_t QMP_sum_double (QMP_double_t *value);
    Done QMP_status_t QMP_sum_float_array (QMP_float_t* value, QMP_u32_t length);
    Done QMP_status_t QMP_sum_double_array (QMP_double_t* value, QMP_u32_t length);
    Done QMP_status_t QMP_max_float (QMP_float_t* value);
    Done QMP_status_t QMP_max_double (QMP_double_t* value);
    Done QMP_status_t QMP_min_float (QMP_float_t* value);
    Done QMP_status_t QMP_min_double (QMP_double_t* value);
    Done QMP_status_t QMP_global_xor (long* value);
    Done QMP_status_t QMP_wait_for_barrier (int millieseconds);
    Done QMP_status_t QMP_binary_reduction (void* lbuffer, QMP_u32_t count, QMP_binary_func bfunc);
                     Attention Implemented in QMP GM, but not yet implemented in QMP MPICH.
      Done const char*        QMP_error_string (QMP_status_t code);
    Done const QMP_status_t        QMP_get_error_number (QMP_msghandle_t mh);
    Done const char*                      QMP_get_error_string (QMP_msghandle_t mh);
    Done int QMP_printf (const char* format, ...);
    Not Done ERRFUNCPTR_t QMP_set_error_function (ERRFUNCPTR_t funcptr);
     

    Extension int                             QMP_fprintf  (FILE* stream, const char* formant,...);
    Extension int                             QMP_info  (const char* formant,...);
    Extension int                             QMP_error  (const char* formant,...);
    Extension void                          QMP_error_exit  (const char* formant,...);

    Extension void                          QMP_fatal  (QMP_u32_t rank, const char* formant,...);
    Extension QMP_bool_t            QMP_is_primary_node  (void);
    This version of QMP puts little overhead on top of GM. The latency and bandwidth tests between 2 nodes yield roughly the same performance results between QMP and raw GM.
    There is a compile flag to enable either a single port implementation or multiport implementation. The flag is _QMP_USE_MULTI_PORTS. The single port implementation uses memory copies but has very little overhead in setting up and tearing down communication channels.