| 41 |
|
|
| 42 |
This chapter focuses on describing the {\bf WRAPPER} environment under which |
This chapter focuses on describing the {\bf WRAPPER} environment under which |
| 43 |
both the core numerics and the pluggable packages function. The description |
both the core numerics and the pluggable packages function. The description |
| 44 |
presented here is intended to be a detailed exposistion and contains significant |
presented here is intended to be a detailed exposition and contains significant |
| 45 |
background material, as well as advanced details on working with the WRAPPER. |
background material, as well as advanced details on working with the WRAPPER. |
| 46 |
The examples section of this manual (part \ref{part:example}) contains more |
The examples section of this manual (part \ref{part:example}) contains more |
| 47 |
succinct, step-by-step instructions on running basic numerical |
succinct, step-by-step instructions on running basic numerical |
| 89 |
\caption{ |
\caption{ |
| 90 |
Numerical code is written too fit within a software support |
Numerical code is written too fit within a software support |
| 91 |
infrastructure called WRAPPER. The WRAPPER is portable and |
infrastructure called WRAPPER. The WRAPPER is portable and |
| 92 |
can be sepcialized for a wide range of specific target hardware and |
can be specialized for a wide range of specific target hardware and |
| 93 |
programming environments, without impacting numerical code that fits |
programming environments, without impacting numerical code that fits |
| 94 |
within the WRAPPER. Codes that fit within the WRAPPER can generally be |
within the WRAPPER. Codes that fit within the WRAPPER can generally be |
| 95 |
made to run as fast on a particular platform as codes specially |
made to run as fast on a particular platform as codes specially |
| 166 |
space allocated to a particular logical processor, there will be data |
space allocated to a particular logical processor, there will be data |
| 167 |
structures (arrays, scalar variables etc...) that hold the simulated state of |
structures (arrays, scalar variables etc...) that hold the simulated state of |
| 168 |
that region. We refer to these data structures as being {\bf owned} by the |
that region. We refer to these data structures as being {\bf owned} by the |
| 169 |
pprocessor to which their |
processor to which their |
| 170 |
associated region of physical space has been allocated. Individual |
associated region of physical space has been allocated. Individual |
| 171 |
regions that are allocated to processors are called {\bf tiles}. A |
regions that are allocated to processors are called {\bf tiles}. A |
| 172 |
processor can own more |
processor can own more |
| 789 |
There are six tiles allocated to six separate logical processors ({\em nSx=6}). |
There are six tiles allocated to six separate logical processors ({\em nSx=6}). |
| 790 |
This set of values can be used for a cube sphere calculation. |
This set of values can be used for a cube sphere calculation. |
| 791 |
Each tile of size $32 \times 32$ represents a face of the |
Each tile of size $32 \times 32$ represents a face of the |
| 792 |
cube. Initialising the tile connectivity correctly ( see section |
cube. Initializing the tile connectivity correctly ( see section |
| 793 |
\ref{sec:cube_sphere_communication}. allows the rotations associated with |
\ref{sec:cube_sphere_communication}. allows the rotations associated with |
| 794 |
moving between the six cube faces to be embedded within the |
moving between the six cube faces to be embedded within the |
| 795 |
tile-tile communication code. |
tile-tile communication code. |
| 842 |
\end{figure} |
\end{figure} |
| 843 |
|
|
| 844 |
\subsubsection{Multi-threaded execution} |
\subsubsection{Multi-threaded execution} |
| 845 |
|
\label{sec:multi-threaded-execution} |
| 846 |
Prior to transferring control to the procedure {\em THE\_MODEL\_MAIN()} the |
Prior to transferring control to the procedure {\em THE\_MODEL\_MAIN()} the |
| 847 |
WRAPPER may cause several coarse grain threads to be initialized. The routine |
WRAPPER may cause several coarse grain threads to be initialized. The routine |
| 848 |
{\em THE\_MODEL\_MAIN()} is called once for each thread and is passed a single |
{\em THE\_MODEL\_MAIN()} is called once for each thread and is passed a single |
| 930 |
} \\ |
} \\ |
| 931 |
|
|
| 932 |
\subsubsection{Multi-process execution} |
\subsubsection{Multi-process execution} |
| 933 |
|
\label{sec:multi-process-execution} |
| 934 |
|
|
| 935 |
Despite its appealing programming model, multi-threaded execution remains |
Despite its appealing programming model, multi-threaded execution remains |
| 936 |
less common then multi-process execution. One major reason for this |
less common then multi-process execution. One major reason for this |
| 942 |
|
|
| 943 |
Multi-process execution is more ubiquitous. |
Multi-process execution is more ubiquitous. |
| 944 |
In order to run code in a multi-process configuration a decomposition |
In order to run code in a multi-process configuration a decomposition |
| 945 |
specification is given ( in which the at least one of the |
specification ( see section \ref{sec:specifying_a_decomposition}) |
| 946 |
|
is given ( in which the at least one of the |
| 947 |
parameters {\em nPx} or {\em nPy} will be greater than one) |
parameters {\em nPx} or {\em nPy} will be greater than one) |
| 948 |
and then, as for multi-threaded operation, |
and then, as for multi-threaded operation, |
| 949 |
appropriate compile time and run time steps must be taken. |
appropriate compile time and run time steps must be taken. |
| 1109 |
This latter set of variables can take one of the following values |
This latter set of variables can take one of the following values |
| 1110 |
{\em COMM\_NONE}, {\em COMM\_MSG}, {\em COMM\_PUT} and {\em COMM\_GET}. |
{\em COMM\_NONE}, {\em COMM\_MSG}, {\em COMM\_PUT} and {\em COMM\_GET}. |
| 1111 |
A value of {\em COMM\_NONE} is used to indicate that a tile has no |
A value of {\em COMM\_NONE} is used to indicate that a tile has no |
| 1112 |
neighbor to cummnicate with on a particular face. A value |
neighbor to communicate with on a particular face. A value |
| 1113 |
of {\em COMM\_MSG} is used to indicated that some form of distributed |
of {\em COMM\_MSG} is used to indicated that some form of distributed |
| 1114 |
memory communication is required to communicate between |
memory communication is required to communicate between |
| 1115 |
these tile faces ( see section \ref{sec:distributed_memory_communication}). |
these tile faces ( see section \ref{sec:distributed_memory_communication}). |
| 1166 |
are read from the file {\em eedata}. If the value of {\em nThreads} |
are read from the file {\em eedata}. If the value of {\em nThreads} |
| 1167 |
is inconsistent with the number of threads requested from the |
is inconsistent with the number of threads requested from the |
| 1168 |
operating system (for example by using an environment |
operating system (for example by using an environment |
| 1169 |
varialble as described in section \ref{sec:multi_threaded_execution}) |
variable as described in section \ref{sec:multi_threaded_execution}) |
| 1170 |
then usually an error will be reported by the routine |
then usually an error will be reported by the routine |
| 1171 |
{\em CHECK\_THREADS}.\\ |
{\em CHECK\_THREADS}.\\ |
| 1172 |
|
|
| 1200 |
\begin{verbatim} |
\begin{verbatim} |
| 1201 |
asm("membar #LoadStore|#StoreStore"); |
asm("membar #LoadStore|#StoreStore"); |
| 1202 |
\end{verbatim} |
\end{verbatim} |
| 1203 |
for an Alpha based sytem the euivalent code reads |
for an Alpha based system the equivalent code reads |
| 1204 |
\begin{verbatim} |
\begin{verbatim} |
| 1205 |
asm("mb"); |
asm("mb"); |
| 1206 |
\end{verbatim} |
\end{verbatim} |
| 1367 |
\item The {\em \_GSUM} and {\em \_EXCH} macro definitions are replaced |
\item The {\em \_GSUM} and {\em \_EXCH} macro definitions are replaced |
| 1368 |
with calls to custom routines ( see {\em gsum\_jam.F} and {\em exch\_jam.F}) |
with calls to custom routines ( see {\em gsum\_jam.F} and {\em exch\_jam.F}) |
| 1369 |
\item a highly specialized form of the exchange operator (optimized |
\item a highly specialized form of the exchange operator (optimized |
| 1370 |
for overlap regions of width one) is substitued into the elliptic |
for overlap regions of width one) is substituted into the elliptic |
| 1371 |
solver routine {\em cg2d.F}. |
solver routine {\em cg2d.F}. |
| 1372 |
\end{itemize} |
\end{itemize} |
| 1373 |
Developing specialized code for other libraries follows a similar |
Developing specialized code for other libraries follows a similar |
| 1379 |
a series of template files, for example {\em exch\_rx.template}. |
a series of template files, for example {\em exch\_rx.template}. |
| 1380 |
This is done to allow a large number of variations on the exchange |
This is done to allow a large number of variations on the exchange |
| 1381 |
process to be maintained. One set of variations supports the |
process to be maintained. One set of variations supports the |
| 1382 |
cube sphere grid. Support for a cube sphere gris in MITgcm is based |
cube sphere grid. Support for a cube sphere grid in MITgcm is based |
| 1383 |
on having each face of the cube as a separate tile (or tiles). |
on having each face of the cube as a separate tile (or tiles). |
| 1384 |
The exchage routines are then able to absorb much of the |
The exchange routines are then able to absorb much of the |
| 1385 |
detailed rotation and reorientation required when moving around the |
detailed rotation and reorientation required when moving around the |
| 1386 |
cube grid. The set of {\em \_EXCH} routines that contain the |
cube grid. The set of {\em \_EXCH} routines that contain the |
| 1387 |
word cube in their name perform these transformations. |
word cube in their name perform these transformations. |
| 1388 |
They are invoked when the run-time logical parameter |
They are invoked when the run-time logical parameter |
| 1389 |
{\em useCubedSphereExchange} is set true. To facilitate the |
{\em useCubedSphereExchange} is set true. To facilitate the |
| 1390 |
transformations on a staggered C-grid, exchange operations are defined |
transformations on a staggered C-grid, exchange operations are defined |
| 1391 |
separately for both vector and scalar quantitities and for |
separately for both vector and scalar quantities and for |
| 1392 |
grid-centered and for grid-face and corner quantities. |
grid-centered and for grid-face and corner quantities. |
| 1393 |
Three sets of exchange routines are defined. Routines |
Three sets of exchange routines are defined. Routines |
| 1394 |
with names of the form {\em exch\_rx} are used to exchange |
with names of the form {\em exch\_rx} are used to exchange |
| 1453 |
C | |
C | |
| 1454 |
C |-THE_MODEL_MAIN :: Primary driver for the MITgcm algorithm |
C |-THE_MODEL_MAIN :: Primary driver for the MITgcm algorithm |
| 1455 |
C | :: Called from WRAPPER level numerical |
C | :: Called from WRAPPER level numerical |
| 1456 |
C | :: code innvocation routine. On entry |
C | :: code invocation routine. On entry |
| 1457 |
C | :: to THE_MODEL_MAIN separate thread and |
C | :: to THE_MODEL_MAIN separate thread and |
| 1458 |
C | :: separate processes will have been established. |
C | :: separate processes will have been established. |
| 1459 |
C | :: Each thread and process will have a unique ID |
C | :: Each thread and process will have a unique ID |
| 1467 |
C | | :: By default kernel parameters are read from file |
C | | :: By default kernel parameters are read from file |
| 1468 |
C | | :: "data" in directory in which code executes. |
C | | :: "data" in directory in which code executes. |
| 1469 |
C | | |
C | | |
| 1470 |
C | |-MON_INIT :: Initialises monitor pacakge ( see pkg/monitor ) |
C | |-MON_INIT :: Initializes monitor package ( see pkg/monitor ) |
| 1471 |
C | | |
C | | |
| 1472 |
C | |-INI_GRID :: Control grid array (vert. and hori.) initialisation. |
C | |-INI_GRID :: Control grid array (vert. and hori.) initialization. |
| 1473 |
C | | | :: Grid arrays are held and described in GRID.h. |
C | | | :: Grid arrays are held and described in GRID.h. |
| 1474 |
C | | | |
C | | | |
| 1475 |
C | | |-INI_VERTICAL_GRID :: Initialise vertical grid arrays. |
C | | |-INI_VERTICAL_GRID :: Initialize vertical grid arrays. |
| 1476 |
C | | | |
C | | | |
| 1477 |
C | | |-INI_CARTESIAN_GRID :: Cartesian horiz. grid initialisation |
C | | |-INI_CARTESIAN_GRID :: Cartesian horiz. grid initialization |
| 1478 |
C | | | :: (calculate grid from kernel parameters). |
C | | | :: (calculate grid from kernel parameters). |
| 1479 |
C | | | |
C | | | |
| 1480 |
C | | |-INI_SPHERICAL_POLAR_GRID :: Spherical polar horiz. grid |
C | | |-INI_SPHERICAL_POLAR_GRID :: Spherical polar horiz. grid |
| 1481 |
C | | | :: initialisation (calculate grid from |
C | | | :: initialization (calculate grid from |
| 1482 |
C | | | :: kernel parameters). |
C | | | :: kernel parameters). |
| 1483 |
C | | | |
C | | | |
| 1484 |
C | | |-INI_CURVILINEAR_GRID :: General orthogonal, structured horiz. |
C | | |-INI_CURVILINEAR_GRID :: General orthogonal, structured horiz. |
| 1485 |
C | | :: grid initialisations. ( input from raw |
C | | :: grid initializations. ( input from raw |
| 1486 |
C | | :: grid files, LONC.bin, DXF.bin etc... ) |
C | | :: grid files, LONC.bin, DXF.bin etc... ) |
| 1487 |
C | | |
C | | |
| 1488 |
C | |-INI_DEPTHS :: Read (from "bathyFile") or set bathymetry/orgography. |
C | |-INI_DEPTHS :: Read (from "bathyFile") or set bathymetry/orgography. |
| 1493 |
C | |-INI_LINEAR_PHSURF :: Set ref. surface Bo_surf |
C | |-INI_LINEAR_PHSURF :: Set ref. surface Bo_surf |
| 1494 |
C | | |
C | | |
| 1495 |
C | |-INI_CORI :: Set coriolis term. zero, f-plane, beta-plane, |
C | |-INI_CORI :: Set coriolis term. zero, f-plane, beta-plane, |
| 1496 |
C | | :: sphere optins are coded. |
C | | :: sphere options are coded. |
| 1497 |
C | | |
C | | |
| 1498 |
C | |-PACAKGES_BOOT :: Start up the optional package environment. |
C | |-PACAKGES_BOOT :: Start up the optional package environment. |
| 1499 |
C | | :: Runtime selection of active packages. |
C | | :: Runtime selection of active packages. |
| 1514 |
C | |-PACKAGES_CHECK |
C | |-PACKAGES_CHECK |
| 1515 |
C | | | |
C | | | |
| 1516 |
C | | |-KPP_CHECK :: KPP Package. pkg/kpp |
C | | |-KPP_CHECK :: KPP Package. pkg/kpp |
| 1517 |
C | | |-OBCS_CHECK :: Open bndy Pacakge. pkg/obcs |
C | | |-OBCS_CHECK :: Open bndy Package. pkg/obcs |
| 1518 |
C | | |-GMREDI_CHECK :: GM Package. pkg/gmredi |
C | | |-GMREDI_CHECK :: GM Package. pkg/gmredi |
| 1519 |
C | | |
C | | |
| 1520 |
C | |-PACKAGES_INIT_FIXED |
C | |-PACKAGES_INIT_FIXED |
| 1534 |
C |-CTRL_UNPACK :: Control vector support package. see pkg/ctrl |
C |-CTRL_UNPACK :: Control vector support package. see pkg/ctrl |
| 1535 |
C | |
C | |
| 1536 |
C |-ADTHE_MAIN_LOOP :: Derivative evaluating form of main time stepping loop |
C |-ADTHE_MAIN_LOOP :: Derivative evaluating form of main time stepping loop |
| 1537 |
C ! :: Auotmatically gerenrated by TAMC/TAF. |
C ! :: Auotmatically generated by TAMC/TAF. |
| 1538 |
C | |
C | |
| 1539 |
C |-CTRL_PACK :: Control vector support package. see pkg/ctrl |
C |-CTRL_PACK :: Control vector support package. see pkg/ctrl |
| 1540 |
C | |
C | |
| 1548 |
C | | |-INI_LINEAR_PHISURF :: Set ref. surface Bo_surf |
C | | |-INI_LINEAR_PHISURF :: Set ref. surface Bo_surf |
| 1549 |
C | | | |
C | | | |
| 1550 |
C | | |-INI_CORI :: Set coriolis term. zero, f-plane, beta-plane, |
C | | |-INI_CORI :: Set coriolis term. zero, f-plane, beta-plane, |
| 1551 |
C | | | :: sphere optins are coded. |
C | | | :: sphere options are coded. |
| 1552 |
C | | | |
C | | | |
| 1553 |
C | | |-INI_CG2D :: 2d con. grad solver initialisation. |
C | | |-INI_CG2D :: 2d con. grad solver initialisation. |
| 1554 |
C | | |-INI_CG3D :: 3d con. grad solver initialisation. |
C | | |-INI_CG3D :: 3d con. grad solver initialisation. |
| 1556 |
C | | |-INI_DYNVARS :: Initialise to zero all DYNVARS.h arrays (dynamical |
C | | |-INI_DYNVARS :: Initialise to zero all DYNVARS.h arrays (dynamical |
| 1557 |
C | | | :: fields). |
C | | | :: fields). |
| 1558 |
C | | | |
C | | | |
| 1559 |
C | | |-INI_FIELDS :: Control initialising model fields to non-zero |
C | | |-INI_FIELDS :: Control initializing model fields to non-zero |
| 1560 |
C | | | |-INI_VEL :: Initialize 3D flow field. |
C | | | |-INI_VEL :: Initialize 3D flow field. |
| 1561 |
C | | | |-INI_THETA :: Set model initial temperature field. |
C | | | |-INI_THETA :: Set model initial temperature field. |
| 1562 |
C | | | |-INI_SALT :: Set model initial salinity field. |
C | | | |-INI_SALT :: Set model initial salinity field. |
| 1634 |
C/\ | | |-CALC_SURF_DR :: Calculate the new surface level thickness. |
C/\ | | |-CALC_SURF_DR :: Calculate the new surface level thickness. |
| 1635 |
C/\ | | |-EXF_GETFORCING :: External forcing package. ( pkg/exf ) |
C/\ | | |-EXF_GETFORCING :: External forcing package. ( pkg/exf ) |
| 1636 |
C/\ | | |-EXTERNAL_FIELDS_LOAD :: Control loading time dep. external data. |
C/\ | | |-EXTERNAL_FIELDS_LOAD :: Control loading time dep. external data. |
| 1637 |
C/\ | | | | :: Simple interpolcation between end-points |
C/\ | | | | :: Simple interpolation between end-points |
| 1638 |
C/\ | | | | :: for forcing datasets. |
C/\ | | | | :: for forcing datasets. |
| 1639 |
C/\ | | | | |
C/\ | | | | |
| 1640 |
C/\ | | | |-EXCH :: Sync forcing. in overlap regions. |
C/\ | | | |-EXCH :: Sync forcing. in overlap regions. |