| 76 |
\end{figure} |
\end{figure} |
| 77 |
|
|
| 78 |
\section{WRAPPER} |
\section{WRAPPER} |
| 79 |
|
\begin{rawhtml} |
| 80 |
|
<!-- CMIREDIR:wrapper: --> |
| 81 |
|
\end{rawhtml} |
| 82 |
|
|
| 83 |
A significant element of the software architecture utilized in |
A significant element of the software architecture utilized in |
| 84 |
MITgcm is a software superstructure and substructure collectively |
MITgcm is a software superstructure and substructure collectively |
| 153 |
computer architecture currently available to the scientific computing community. |
computer architecture currently available to the scientific computing community. |
| 154 |
|
|
| 155 |
\subsection{Machine model parallelism} |
\subsection{Machine model parallelism} |
| 156 |
|
\begin{rawhtml} |
| 157 |
|
<!-- CMIREDIR:domain_decomp: --> |
| 158 |
|
\end{rawhtml} |
| 159 |
|
|
| 160 |
Codes operating under the WRAPPER target an abstract machine that is assumed to |
Codes operating under the WRAPPER target an abstract machine that is assumed to |
| 161 |
consist of one or more logical processors that can compute concurrently. |
consist of one or more logical processors that can compute concurrently. |
| 983 |
routines must be activated at compile time. Currently MPI libraries are |
routines must be activated at compile time. Currently MPI libraries are |
| 984 |
invoked by |
invoked by |
| 985 |
specifying the appropriate options file with the |
specifying the appropriate options file with the |
| 986 |
\begin{verbatim}-of\end{verbatim} flag when running the {\em genmake2} |
{\tt-of} flag when running the {\em genmake2} |
| 987 |
script, which generates the Makefile for compiling and linking MITgcm. |
script, which generates the Makefile for compiling and linking MITgcm. |
| 988 |
(Previously this was done by setting the {\em ALLOW\_USE\_MPI} and |
(Previously this was done by setting the {\em ALLOW\_USE\_MPI} and |
| 989 |
{\em ALWAYS\_USE\_MPI} flags in the {\em CPP\_EEOPTIONS.h} file.) More |
{\em ALWAYS\_USE\_MPI} flags in the {\em CPP\_EEOPTIONS.h} file.) More |
| 990 |
detailed information about the use of {\em genmake2} for specifying |
detailed information about the use of {\em genmake2} for specifying |
| 991 |
local compiler flags is located in section 3 ??\\ |
local compiler flags is located in section \ref{sect:genmake}.\\ |
| 992 |
|
|
| 993 |
|
|
| 994 |
\fbox{ |
\fbox{ |
| 995 |
\begin{minipage}{4.75in} |
\begin{minipage}{4.75in} |
| 996 |
|
Directory: {\em tools/build\_options}\\ |
| 997 |
File: {\em tools/genmake2} |
File: {\em tools/genmake2} |
| 998 |
\end{minipage} |
\end{minipage} |
| 999 |
} \\ |
} \\ |
| 1012 |
in the file {\em SIZE.h}. The parameter {\em mf} specifies that a text file |
in the file {\em SIZE.h}. The parameter {\em mf} specifies that a text file |
| 1013 |
called ``mf'' will be read to get a list of processor names on |
called ``mf'' will be read to get a list of processor names on |
| 1014 |
which the sixty-four processes will execute. The syntax of this file |
which the sixty-four processes will execute. The syntax of this file |
| 1015 |
is specified by the MPI distribution |
is specified by the MPI distribution. |
| 1016 |
\\ |
\\ |
| 1017 |
|
|
| 1018 |
\fbox{ |
\fbox{ |
| 1063 |
Allocation of processes to tiles in controlled by the routine |
Allocation of processes to tiles in controlled by the routine |
| 1064 |
{\em INI\_PROCS()}. For each process this routine sets |
{\em INI\_PROCS()}. For each process this routine sets |
| 1065 |
the variables {\em myXGlobalLo} and {\em myYGlobalLo}. |
the variables {\em myXGlobalLo} and {\em myYGlobalLo}. |
| 1066 |
These variables specify (in index space) the coordinate |
These variables specify in index space the coordinates |
| 1067 |
of the southern most and western most corner of the |
of the southernmost and westernmost corner of the |
| 1068 |
southern most and western most tile owned by this process. |
southernmost and westernmost tile owned by this process. |
| 1069 |
The variables {\em pidW}, {\em pidE}, {\em pidS} and {\em pidN} |
The variables {\em pidW}, {\em pidE}, {\em pidS} and {\em pidN} |
| 1070 |
are also set in this routine. These are used to identify |
are also set in this routine. These are used to identify |
| 1071 |
processes holding tiles to the west, east, south and north |
processes holding tiles to the west, east, south and north |
| 1072 |
of this process. These values are stored in global storage |
of this process. These values are stored in global storage |
| 1073 |
in the header file {\em EESUPPORT.h} for use by |
in the header file {\em EESUPPORT.h} for use by |
| 1074 |
communication routines. |
communication routines. The above does not hold when the |
| 1075 |
|
exch2 package is used -- exch2 sets its own parameters to |
| 1076 |
|
specify the global indices of tiles and their relationships |
| 1077 |
|
to each other. See the documentation on the exch2 package |
| 1078 |
|
(\ref{sec:exch2}) for |
| 1079 |
|
details. |
| 1080 |
\\ |
\\ |
| 1081 |
|
|
| 1082 |
\fbox{ |
\fbox{ |
| 1102 |
describes the information that is held and used. |
describes the information that is held and used. |
| 1103 |
|
|
| 1104 |
\begin{enumerate} |
\begin{enumerate} |
| 1105 |
\item {\bf Tile-tile connectivity information} For each tile the WRAPPER |
\item {\bf Tile-tile connectivity information} |
| 1106 |
sets a flag that sets the tile number to the north, south, east and |
For each tile the WRAPPER |
| 1107 |
|
sets a flag that sets the tile number to the north, |
| 1108 |
|
south, east and |
| 1109 |
west of that tile. This number is unique over all tiles in a |
west of that tile. This number is unique over all tiles in a |
| 1110 |
configuration. The number is held in the variables {\em tileNo} |
configuration. Except when using the cubed sphere and the exch2 package, |
| 1111 |
|
the number is held in the variables {\em tileNo} |
| 1112 |
( this holds the tiles own number), {\em tileNoN}, {\em tileNoS}, |
( this holds the tiles own number), {\em tileNoN}, {\em tileNoS}, |
| 1113 |
{\em tileNoE} and {\em tileNoW}. A parameter is also stored with each tile |
{\em tileNoE} and {\em tileNoW}. A parameter is also stored with each tile |
| 1114 |
that specifies the type of communication that is used between tiles. |
that specifies the type of communication that is used between tiles. |
| 1131 |
(see figure \ref{fig:communication_primitives}). The routine |
(see figure \ref{fig:communication_primitives}). The routine |
| 1132 |
{\em ini\_communication\_patterns()} is responsible for setting the |
{\em ini\_communication\_patterns()} is responsible for setting the |
| 1133 |
communication mode values for each tile. |
communication mode values for each tile. |
| 1134 |
\\ |
|
| 1135 |
|
When using the cubed sphere configuration with the exch2 package, the |
| 1136 |
|
relationships between tiles and their communication methods are set |
| 1137 |
|
by the package in other variables. See the exch2 package documentation |
| 1138 |
|
(\ref{sec:exch2} for details. |
| 1139 |
|
|
| 1140 |
|
|
| 1141 |
|
|
| 1142 |
\fbox{ |
\fbox{ |
| 1143 |
\begin{minipage}{4.75in} |
\begin{minipage}{4.75in} |
| 1273 |
the cube-sphere grid. In this class of grid a rotation may be required |
the cube-sphere grid. In this class of grid a rotation may be required |
| 1274 |
between tiles. Aligning the coordinate requiring rotation with the |
between tiles. Aligning the coordinate requiring rotation with the |
| 1275 |
tile decomposition, allows the coordinate transformation to |
tile decomposition, allows the coordinate transformation to |
| 1276 |
be embedded within a custom form of the \_EXCH primitive. |
be embedded within a custom form of the \_EXCH primitive. In these |
| 1277 |
|
cases \_EXCH is mapped to exch2 routines, as detailed in the exch2 |
| 1278 |
|
package documentation \ref{sec:exch2}. |
| 1279 |
|
|
| 1280 |
\item {\bf Reverse Mode} |
\item {\bf Reverse Mode} |
| 1281 |
The communication primitives \_EXCH and \_GSUM both employ |
The communication primitives \_EXCH and \_GSUM both employ |
| 1292 |
is set to the value {\em REVERSE\_SIMULATION}. This signifies |
is set to the value {\em REVERSE\_SIMULATION}. This signifies |
| 1293 |
ti the low-level routines that the adjoint forms of the |
ti the low-level routines that the adjoint forms of the |
| 1294 |
appropriate communication operation should be performed. |
appropriate communication operation should be performed. |
| 1295 |
|
|
| 1296 |
\item {\bf MAX\_NO\_THREADS} |
\item {\bf MAX\_NO\_THREADS} |
| 1297 |
The variable {\em MAX\_NO\_THREADS} is used to indicate the |
The variable {\em MAX\_NO\_THREADS} is used to indicate the |
| 1298 |
maximum number of OS threads that a code will use. This |
maximum number of OS threads that a code will use. This |
| 1397 |
This is done to allow a large number of variations on the exchange |
This is done to allow a large number of variations on the exchange |
| 1398 |
process to be maintained. One set of variations supports the |
process to be maintained. One set of variations supports the |
| 1399 |
cube sphere grid. Support for a cube sphere grid in MITgcm is based |
cube sphere grid. Support for a cube sphere grid in MITgcm is based |
| 1400 |
on having each face of the cube as a separate tile (or tiles). |
on having each face of the cube as a separate tile or tiles. |
| 1401 |
The exchange routines are then able to absorb much of the |
The exchange routines are then able to absorb much of the |
| 1402 |
detailed rotation and reorientation required when moving around the |
detailed rotation and reorientation required when moving around the |
| 1403 |
cube grid. The set of {\em \_EXCH} routines that contain the |
cube grid. The set of {\em \_EXCH} routines that contain the |
| 1421 |
|
|
| 1422 |
|
|
| 1423 |
\section{MITgcm execution under WRAPPER} |
\section{MITgcm execution under WRAPPER} |
| 1424 |
|
\begin{rawhtml} |
| 1425 |
|
<!-- CMIREDIR:mitgcm_wrapper: --> |
| 1426 |
|
\end{rawhtml} |
| 1427 |
|
|
| 1428 |
Fitting together the WRAPPER elements, package elements and |
Fitting together the WRAPPER elements, package elements and |
| 1429 |
MITgcm core equation elements of the source code produces calling |
MITgcm core equation elements of the source code produces calling |