76 |
\end{figure} |
\end{figure} |
77 |
|
|
78 |
\section{WRAPPER} |
\section{WRAPPER} |
79 |
|
\begin{rawhtml} |
80 |
|
<!-- CMIREDIR:wrapper --> |
81 |
|
\end{rawhtml} |
82 |
|
|
83 |
A significant element of the software architecture utilized in |
A significant element of the software architecture utilized in |
84 |
MITgcm is a software superstructure and substructure collectively |
MITgcm is a software superstructure and substructure collectively |
153 |
computer architecture currently available to the scientific computing community. |
computer architecture currently available to the scientific computing community. |
154 |
|
|
155 |
\subsection{Machine model parallelism} |
\subsection{Machine model parallelism} |
156 |
|
\begin{rawhtml} |
157 |
|
<!-- CMIREDIR:domain_decomp --> |
158 |
|
\end{rawhtml} |
159 |
|
|
160 |
Codes operating under the WRAPPER target an abstract machine that is assumed to |
Codes operating under the WRAPPER target an abstract machine that is assumed to |
161 |
consist of one or more logical processors that can compute concurrently. |
consist of one or more logical processors that can compute concurrently. |
983 |
routines must be activated at compile time. Currently MPI libraries are |
routines must be activated at compile time. Currently MPI libraries are |
984 |
invoked by |
invoked by |
985 |
specifying the appropriate options file with the |
specifying the appropriate options file with the |
986 |
\begin{verbatim}-of\end{verbatim} flag when running the {\em genmake2} |
{\tt-of} flag when running the {\em genmake2} |
987 |
script, which generates the Makefile for compiling and linking MITgcm. |
script, which generates the Makefile for compiling and linking MITgcm. |
988 |
(Previously this was done by setting the {\em ALLOW\_USE\_MPI} and |
(Previously this was done by setting the {\em ALLOW\_USE\_MPI} and |
989 |
{\em ALWAYS\_USE\_MPI} flags in the {\em CPP\_EEOPTIONS.h} file.) More |
{\em ALWAYS\_USE\_MPI} flags in the {\em CPP\_EEOPTIONS.h} file.) More |
990 |
detailed information about the use of {\em genmake2} for specifying |
detailed information about the use of {\em genmake2} for specifying |
991 |
local compiler flags is located in section 3 ??\\ |
local compiler flags is located in section \ref{sect:genmake}.\\ |
992 |
|
|
993 |
|
|
994 |
\fbox{ |
\fbox{ |
995 |
\begin{minipage}{4.75in} |
\begin{minipage}{4.75in} |
996 |
|
Directory: {\em tools/build\_options}\\ |
997 |
File: {\em tools/genmake2} |
File: {\em tools/genmake2} |
998 |
\end{minipage} |
\end{minipage} |
999 |
} \\ |
} \\ |
1012 |
in the file {\em SIZE.h}. The parameter {\em mf} specifies that a text file |
in the file {\em SIZE.h}. The parameter {\em mf} specifies that a text file |
1013 |
called ``mf'' will be read to get a list of processor names on |
called ``mf'' will be read to get a list of processor names on |
1014 |
which the sixty-four processes will execute. The syntax of this file |
which the sixty-four processes will execute. The syntax of this file |
1015 |
is specified by the MPI distribution |
is specified by the MPI distribution. |
1016 |
\\ |
\\ |
1017 |
|
|
1018 |
\fbox{ |
\fbox{ |
1063 |
Allocation of processes to tiles in controlled by the routine |
Allocation of processes to tiles in controlled by the routine |
1064 |
{\em INI\_PROCS()}. For each process this routine sets |
{\em INI\_PROCS()}. For each process this routine sets |
1065 |
the variables {\em myXGlobalLo} and {\em myYGlobalLo}. |
the variables {\em myXGlobalLo} and {\em myYGlobalLo}. |
1066 |
These variables specify (in index space) the coordinate |
These variables specify in index space the coordinates |
1067 |
of the southern most and western most corner of the |
of the southernmost and westernmost corner of the |
1068 |
southern most and western most tile owned by this process. |
southernmost and westernmost tile owned by this process. |
1069 |
The variables {\em pidW}, {\em pidE}, {\em pidS} and {\em pidN} |
The variables {\em pidW}, {\em pidE}, {\em pidS} and {\em pidN} |
1070 |
are also set in this routine. These are used to identify |
are also set in this routine. These are used to identify |
1071 |
processes holding tiles to the west, east, south and north |
processes holding tiles to the west, east, south and north |
1074 |
communication routines. The above does not hold when the |
communication routines. The above does not hold when the |
1075 |
exch2 package is used -- exch2 sets its own parameters to |
exch2 package is used -- exch2 sets its own parameters to |
1076 |
specify the global indices of tiles and their relationships |
specify the global indices of tiles and their relationships |
1077 |
to each other. See exch2 docs for details??. |
to each other. See the documentation on the exch2 package |
1078 |
|
(\ref{sec:exch2}) for |
1079 |
|
details. |
1080 |
\\ |
\\ |
1081 |
|
|
1082 |
\fbox{ |
\fbox{ |
1102 |
describes the information that is held and used. |
describes the information that is held and used. |
1103 |
|
|
1104 |
\begin{enumerate} |
\begin{enumerate} |
1105 |
\item {\bf Tile-tile connectivity information} For each tile the WRAPPER |
\item {\bf Tile-tile connectivity information} |
1106 |
sets a flag that sets the tile number to the north, south, east and |
For each tile the WRAPPER |
1107 |
|
sets a flag that sets the tile number to the north, |
1108 |
|
south, east and |
1109 |
west of that tile. This number is unique over all tiles in a |
west of that tile. This number is unique over all tiles in a |
1110 |
configuration. Except when using the exch2 package, |
configuration. Except when using the cubed sphere and the exch2 package, |
1111 |
the number is held in the variables {\em tileNo} |
the number is held in the variables {\em tileNo} |
1112 |
( this holds the tiles own number), {\em tileNoN}, {\em tileNoS}, |
( this holds the tiles own number), {\em tileNoN}, {\em tileNoS}, |
1113 |
{\em tileNoE} and {\em tileNoW}. A parameter is also stored with each tile |
{\em tileNoE} and {\em tileNoW}. A parameter is also stored with each tile |
1131 |
(see figure \ref{fig:communication_primitives}). The routine |
(see figure \ref{fig:communication_primitives}). The routine |
1132 |
{\em ini\_communication\_patterns()} is responsible for setting the |
{\em ini\_communication\_patterns()} is responsible for setting the |
1133 |
communication mode values for each tile. |
communication mode values for each tile. |
|
\\ |
|
1134 |
|
|
1135 |
When using the cubed sphere configuration with the exch2 package, the |
When using the cubed sphere configuration with the exch2 package, the |
1136 |
relationships between tiles and their communication methods are set |
relationships between tiles and their communication methods are set |
1137 |
by the package in other variables. See the exch2 docs for details.?? |
by the package in other variables. See the exch2 package documentation |
1138 |
|
(\ref{sec:exch2} for details. |
1139 |
|
|
1140 |
|
|
1141 |
|
|
1273 |
the cube-sphere grid. In this class of grid a rotation may be required |
the cube-sphere grid. In this class of grid a rotation may be required |
1274 |
between tiles. Aligning the coordinate requiring rotation with the |
between tiles. Aligning the coordinate requiring rotation with the |
1275 |
tile decomposition, allows the coordinate transformation to |
tile decomposition, allows the coordinate transformation to |
1276 |
be embedded within a custom form of the \_EXCH primitive. |
be embedded within a custom form of the \_EXCH primitive. In these |
1277 |
|
cases \_EXCH is mapped to exch2 routines, as detailed in the exch2 |
1278 |
|
package documentation \ref{sec:exch2}. |
1279 |
|
|
1280 |
\item {\bf Reverse Mode} |
\item {\bf Reverse Mode} |
1281 |
The communication primitives \_EXCH and \_GSUM both employ |
The communication primitives \_EXCH and \_GSUM both employ |
1292 |
is set to the value {\em REVERSE\_SIMULATION}. This signifies |
is set to the value {\em REVERSE\_SIMULATION}. This signifies |
1293 |
ti the low-level routines that the adjoint forms of the |
ti the low-level routines that the adjoint forms of the |
1294 |
appropriate communication operation should be performed. |
appropriate communication operation should be performed. |
1295 |
|
|
1296 |
\item {\bf MAX\_NO\_THREADS} |
\item {\bf MAX\_NO\_THREADS} |
1297 |
The variable {\em MAX\_NO\_THREADS} is used to indicate the |
The variable {\em MAX\_NO\_THREADS} is used to indicate the |
1298 |
maximum number of OS threads that a code will use. This |
maximum number of OS threads that a code will use. This |
1397 |
This is done to allow a large number of variations on the exchange |
This is done to allow a large number of variations on the exchange |
1398 |
process to be maintained. One set of variations supports the |
process to be maintained. One set of variations supports the |
1399 |
cube sphere grid. Support for a cube sphere grid in MITgcm is based |
cube sphere grid. Support for a cube sphere grid in MITgcm is based |
1400 |
on having each face of the cube as a separate tile (or tiles). |
on having each face of the cube as a separate tile or tiles. |
1401 |
The exchange routines are then able to absorb much of the |
The exchange routines are then able to absorb much of the |
1402 |
detailed rotation and reorientation required when moving around the |
detailed rotation and reorientation required when moving around the |
1403 |
cube grid. The set of {\em \_EXCH} routines that contain the |
cube grid. The set of {\em \_EXCH} routines that contain the |