41 |
|
|
42 |
This chapter focuses on describing the {\bf WRAPPER} environment under which |
This chapter focuses on describing the {\bf WRAPPER} environment under which |
43 |
both the core numerics and the pluggable packages function. The description |
both the core numerics and the pluggable packages function. The description |
44 |
presented here is intended to be a detailed exposistion and contains significant |
presented here is intended to be a detailed exposition and contains significant |
45 |
background material, as well as advanced details on working with the WRAPPER. |
background material, as well as advanced details on working with the WRAPPER. |
46 |
The examples section of this manual (part \ref{part:example}) contains more |
The examples section of this manual (part \ref{part:example}) contains more |
47 |
succinct, step-by-step instructions on running basic numerical |
succinct, step-by-step instructions on running basic numerical |
89 |
\caption{ |
\caption{ |
90 |
Numerical code is written too fit within a software support |
Numerical code is written too fit within a software support |
91 |
infrastructure called WRAPPER. The WRAPPER is portable and |
infrastructure called WRAPPER. The WRAPPER is portable and |
92 |
can be sepcialized for a wide range of specific target hardware and |
can be specialized for a wide range of specific target hardware and |
93 |
programming environments, without impacting numerical code that fits |
programming environments, without impacting numerical code that fits |
94 |
within the WRAPPER. Codes that fit within the WRAPPER can generally be |
within the WRAPPER. Codes that fit within the WRAPPER can generally be |
95 |
made to run as fast on a particular platform as codes specially |
made to run as fast on a particular platform as codes specially |
166 |
space allocated to a particular logical processor, there will be data |
space allocated to a particular logical processor, there will be data |
167 |
structures (arrays, scalar variables etc...) that hold the simulated state of |
structures (arrays, scalar variables etc...) that hold the simulated state of |
168 |
that region. We refer to these data structures as being {\bf owned} by the |
that region. We refer to these data structures as being {\bf owned} by the |
169 |
pprocessor to which their |
processor to which their |
170 |
associated region of physical space has been allocated. Individual |
associated region of physical space has been allocated. Individual |
171 |
regions that are allocated to processors are called {\bf tiles}. A |
regions that are allocated to processors are called {\bf tiles}. A |
172 |
processor can own more |
processor can own more |
789 |
There are six tiles allocated to six separate logical processors ({\em nSx=6}). |
There are six tiles allocated to six separate logical processors ({\em nSx=6}). |
790 |
This set of values can be used for a cube sphere calculation. |
This set of values can be used for a cube sphere calculation. |
791 |
Each tile of size $32 \times 32$ represents a face of the |
Each tile of size $32 \times 32$ represents a face of the |
792 |
cube. Initialising the tile connectivity correctly ( see section |
cube. Initializing the tile connectivity correctly ( see section |
793 |
\ref{sec:cube_sphere_communication}. allows the rotations associated with |
\ref{sec:cube_sphere_communication}. allows the rotations associated with |
794 |
moving between the six cube faces to be embedded within the |
moving between the six cube faces to be embedded within the |
795 |
tile-tile communication code. |
tile-tile communication code. |
1109 |
This latter set of variables can take one of the following values |
This latter set of variables can take one of the following values |
1110 |
{\em COMM\_NONE}, {\em COMM\_MSG}, {\em COMM\_PUT} and {\em COMM\_GET}. |
{\em COMM\_NONE}, {\em COMM\_MSG}, {\em COMM\_PUT} and {\em COMM\_GET}. |
1111 |
A value of {\em COMM\_NONE} is used to indicate that a tile has no |
A value of {\em COMM\_NONE} is used to indicate that a tile has no |
1112 |
neighbor to cummnicate with on a particular face. A value |
neighbor to communicate with on a particular face. A value |
1113 |
of {\em COMM\_MSG} is used to indicated that some form of distributed |
of {\em COMM\_MSG} is used to indicated that some form of distributed |
1114 |
memory communication is required to communicate between |
memory communication is required to communicate between |
1115 |
these tile faces ( see section \ref{sec:distributed_memory_communication}). |
these tile faces ( see section \ref{sec:distributed_memory_communication}). |
1166 |
are read from the file {\em eedata}. If the value of {\em nThreads} |
are read from the file {\em eedata}. If the value of {\em nThreads} |
1167 |
is inconsistent with the number of threads requested from the |
is inconsistent with the number of threads requested from the |
1168 |
operating system (for example by using an environment |
operating system (for example by using an environment |
1169 |
varialble as described in section \ref{sec:multi_threaded_execution}) |
variable as described in section \ref{sec:multi_threaded_execution}) |
1170 |
then usually an error will be reported by the routine |
then usually an error will be reported by the routine |
1171 |
{\em CHECK\_THREADS}.\\ |
{\em CHECK\_THREADS}.\\ |
1172 |
|
|
1200 |
\begin{verbatim} |
\begin{verbatim} |
1201 |
asm("membar #LoadStore|#StoreStore"); |
asm("membar #LoadStore|#StoreStore"); |
1202 |
\end{verbatim} |
\end{verbatim} |
1203 |
for an Alpha based sytem the euivalent code reads |
for an Alpha based system the equivalent code reads |
1204 |
\begin{verbatim} |
\begin{verbatim} |
1205 |
asm("mb"); |
asm("mb"); |
1206 |
\end{verbatim} |
\end{verbatim} |
1367 |
\item The {\em \_GSUM} and {\em \_EXCH} macro definitions are replaced |
\item The {\em \_GSUM} and {\em \_EXCH} macro definitions are replaced |
1368 |
with calls to custom routines ( see {\em gsum\_jam.F} and {\em exch\_jam.F}) |
with calls to custom routines ( see {\em gsum\_jam.F} and {\em exch\_jam.F}) |
1369 |
\item a highly specialized form of the exchange operator (optimized |
\item a highly specialized form of the exchange operator (optimized |
1370 |
for overlap regions of width one) is substitued into the elliptic |
for overlap regions of width one) is substituted into the elliptic |
1371 |
solver routine {\em cg2d.F}. |
solver routine {\em cg2d.F}. |
1372 |
\end{itemize} |
\end{itemize} |
1373 |
Developing specialized code for other libraries follows a similar |
Developing specialized code for other libraries follows a similar |
1379 |
a series of template files, for example {\em exch\_rx.template}. |
a series of template files, for example {\em exch\_rx.template}. |
1380 |
This is done to allow a large number of variations on the exchange |
This is done to allow a large number of variations on the exchange |
1381 |
process to be maintained. One set of variations supports the |
process to be maintained. One set of variations supports the |
1382 |
cube sphere grid. Support for a cube sphere gris in MITgcm is based |
cube sphere grid. Support for a cube sphere grid in MITgcm is based |
1383 |
on having each face of the cube as a separate tile (or tiles). |
on having each face of the cube as a separate tile (or tiles). |
1384 |
The exchage routines are then able to absorb much of the |
The exchange routines are then able to absorb much of the |
1385 |
detailed rotation and reorientation required when moving around the |
detailed rotation and reorientation required when moving around the |
1386 |
cube grid. The set of {\em \_EXCH} routines that contain the |
cube grid. The set of {\em \_EXCH} routines that contain the |
1387 |
word cube in their name perform these transformations. |
word cube in their name perform these transformations. |
1388 |
They are invoked when the run-time logical parameter |
They are invoked when the run-time logical parameter |
1389 |
{\em useCubedSphereExchange} is set true. To facilitate the |
{\em useCubedSphereExchange} is set true. To facilitate the |
1390 |
transformations on a staggered C-grid, exchange operations are defined |
transformations on a staggered C-grid, exchange operations are defined |
1391 |
separately for both vector and scalar quantitities and for |
separately for both vector and scalar quantities and for |
1392 |
grid-centered and for grid-face and corner quantities. |
grid-centered and for grid-face and corner quantities. |
1393 |
Three sets of exchange routines are defined. Routines |
Three sets of exchange routines are defined. Routines |
1394 |
with names of the form {\em exch\_rx} are used to exchange |
with names of the form {\em exch\_rx} are used to exchange |
1453 |
C | |
C | |
1454 |
C |-THE_MODEL_MAIN :: Primary driver for the MITgcm algorithm |
C |-THE_MODEL_MAIN :: Primary driver for the MITgcm algorithm |
1455 |
C | :: Called from WRAPPER level numerical |
C | :: Called from WRAPPER level numerical |
1456 |
C | :: code innvocation routine. On entry |
C | :: code invocation routine. On entry |
1457 |
C | :: to THE_MODEL_MAIN separate thread and |
C | :: to THE_MODEL_MAIN separate thread and |
1458 |
C | :: separate processes will have been established. |
C | :: separate processes will have been established. |
1459 |
C | :: Each thread and process will have a unique ID |
C | :: Each thread and process will have a unique ID |
1467 |
C | | :: By default kernel parameters are read from file |
C | | :: By default kernel parameters are read from file |
1468 |
C | | :: "data" in directory in which code executes. |
C | | :: "data" in directory in which code executes. |
1469 |
C | | |
C | | |
1470 |
C | |-MON_INIT :: Initialises monitor pacakge ( see pkg/monitor ) |
C | |-MON_INIT :: Initializes monitor package ( see pkg/monitor ) |
1471 |
C | | |
C | | |
1472 |
C | |-INI_GRID :: Control grid array (vert. and hori.) initialisation. |
C | |-INI_GRID :: Control grid array (vert. and hori.) initialization. |
1473 |
C | | | :: Grid arrays are held and described in GRID.h. |
C | | | :: Grid arrays are held and described in GRID.h. |
1474 |
C | | | |
C | | | |
1475 |
C | | |-INI_VERTICAL_GRID :: Initialise vertical grid arrays. |
C | | |-INI_VERTICAL_GRID :: Initialize vertical grid arrays. |
1476 |
C | | | |
C | | | |
1477 |
C | | |-INI_CARTESIAN_GRID :: Cartesian horiz. grid initialisation |
C | | |-INI_CARTESIAN_GRID :: Cartesian horiz. grid initialization |
1478 |
C | | | :: (calculate grid from kernel parameters). |
C | | | :: (calculate grid from kernel parameters). |
1479 |
C | | | |
C | | | |
1480 |
C | | |-INI_SPHERICAL_POLAR_GRID :: Spherical polar horiz. grid |
C | | |-INI_SPHERICAL_POLAR_GRID :: Spherical polar horiz. grid |
1481 |
C | | | :: initialisation (calculate grid from |
C | | | :: initialization (calculate grid from |
1482 |
C | | | :: kernel parameters). |
C | | | :: kernel parameters). |
1483 |
C | | | |
C | | | |
1484 |
C | | |-INI_CURVILINEAR_GRID :: General orthogonal, structured horiz. |
C | | |-INI_CURVILINEAR_GRID :: General orthogonal, structured horiz. |
1485 |
C | | :: grid initialisations. ( input from raw |
C | | :: grid initializations. ( input from raw |
1486 |
C | | :: grid files, LONC.bin, DXF.bin etc... ) |
C | | :: grid files, LONC.bin, DXF.bin etc... ) |
1487 |
C | | |
C | | |
1488 |
C | |-INI_DEPTHS :: Read (from "bathyFile") or set bathymetry/orgography. |
C | |-INI_DEPTHS :: Read (from "bathyFile") or set bathymetry/orgography. |
1493 |
C | |-INI_LINEAR_PHSURF :: Set ref. surface Bo_surf |
C | |-INI_LINEAR_PHSURF :: Set ref. surface Bo_surf |
1494 |
C | | |
C | | |
1495 |
C | |-INI_CORI :: Set coriolis term. zero, f-plane, beta-plane, |
C | |-INI_CORI :: Set coriolis term. zero, f-plane, beta-plane, |
1496 |
C | | :: sphere optins are coded. |
C | | :: sphere options are coded. |
1497 |
C | | |
C | | |
1498 |
C | |-PACAKGES_BOOT :: Start up the optional package environment. |
C | |-PACAKGES_BOOT :: Start up the optional package environment. |
1499 |
C | | :: Runtime selection of active packages. |
C | | :: Runtime selection of active packages. |
1514 |
C | |-PACKAGES_CHECK |
C | |-PACKAGES_CHECK |
1515 |
C | | | |
C | | | |
1516 |
C | | |-KPP_CHECK :: KPP Package. pkg/kpp |
C | | |-KPP_CHECK :: KPP Package. pkg/kpp |
1517 |
C | | |-OBCS_CHECK :: Open bndy Pacakge. pkg/obcs |
C | | |-OBCS_CHECK :: Open bndy Package. pkg/obcs |
1518 |
C | | |-GMREDI_CHECK :: GM Package. pkg/gmredi |
C | | |-GMREDI_CHECK :: GM Package. pkg/gmredi |
1519 |
C | | |
C | | |
1520 |
C | |-PACKAGES_INIT_FIXED |
C | |-PACKAGES_INIT_FIXED |
1534 |
C |-CTRL_UNPACK :: Control vector support package. see pkg/ctrl |
C |-CTRL_UNPACK :: Control vector support package. see pkg/ctrl |
1535 |
C | |
C | |
1536 |
C |-ADTHE_MAIN_LOOP :: Derivative evaluating form of main time stepping loop |
C |-ADTHE_MAIN_LOOP :: Derivative evaluating form of main time stepping loop |
1537 |
C ! :: Auotmatically gerenrated by TAMC/TAF. |
C ! :: Auotmatically generated by TAMC/TAF. |
1538 |
C | |
C | |
1539 |
C |-CTRL_PACK :: Control vector support package. see pkg/ctrl |
C |-CTRL_PACK :: Control vector support package. see pkg/ctrl |
1540 |
C | |
C | |
1548 |
C | | |-INI_LINEAR_PHISURF :: Set ref. surface Bo_surf |
C | | |-INI_LINEAR_PHISURF :: Set ref. surface Bo_surf |
1549 |
C | | | |
C | | | |
1550 |
C | | |-INI_CORI :: Set coriolis term. zero, f-plane, beta-plane, |
C | | |-INI_CORI :: Set coriolis term. zero, f-plane, beta-plane, |
1551 |
C | | | :: sphere optins are coded. |
C | | | :: sphere options are coded. |
1552 |
C | | | |
C | | | |
1553 |
C | | |-INI_CG2D :: 2d con. grad solver initialisation. |
C | | |-INI_CG2D :: 2d con. grad solver initialisation. |
1554 |
C | | |-INI_CG3D :: 3d con. grad solver initialisation. |
C | | |-INI_CG3D :: 3d con. grad solver initialisation. |
1556 |
C | | |-INI_DYNVARS :: Initialise to zero all DYNVARS.h arrays (dynamical |
C | | |-INI_DYNVARS :: Initialise to zero all DYNVARS.h arrays (dynamical |
1557 |
C | | | :: fields). |
C | | | :: fields). |
1558 |
C | | | |
C | | | |
1559 |
C | | |-INI_FIELDS :: Control initialising model fields to non-zero |
C | | |-INI_FIELDS :: Control initializing model fields to non-zero |
1560 |
C | | | |-INI_VEL :: Initialize 3D flow field. |
C | | | |-INI_VEL :: Initialize 3D flow field. |
1561 |
C | | | |-INI_THETA :: Set model initial temperature field. |
C | | | |-INI_THETA :: Set model initial temperature field. |
1562 |
C | | | |-INI_SALT :: Set model initial salinity field. |
C | | | |-INI_SALT :: Set model initial salinity field. |
1634 |
C/\ | | |-CALC_SURF_DR :: Calculate the new surface level thickness. |
C/\ | | |-CALC_SURF_DR :: Calculate the new surface level thickness. |
1635 |
C/\ | | |-EXF_GETFORCING :: External forcing package. ( pkg/exf ) |
C/\ | | |-EXF_GETFORCING :: External forcing package. ( pkg/exf ) |
1636 |
C/\ | | |-EXTERNAL_FIELDS_LOAD :: Control loading time dep. external data. |
C/\ | | |-EXTERNAL_FIELDS_LOAD :: Control loading time dep. external data. |
1637 |
C/\ | | | | :: Simple interpolcation between end-points |
C/\ | | | | :: Simple interpolation between end-points |
1638 |
C/\ | | | | :: for forcing datasets. |
C/\ | | | | :: for forcing datasets. |
1639 |
C/\ | | | | |
C/\ | | | | |
1640 |
C/\ | | | |-EXCH :: Sync forcing. in overlap regions. |
C/\ | | | |-EXCH :: Sync forcing. in overlap regions. |