17 |
features that are employed. |
features that are employed. |
18 |
|
|
19 |
\section{Overall architectural goals} |
\section{Overall architectural goals} |
20 |
|
\begin{rawhtml} |
21 |
|
<!-- CMIREDIR:overall_architectural_goals: --> |
22 |
|
\end{rawhtml} |
23 |
|
|
24 |
Broadly, the goals of the software architecture employed in MITgcm are |
Broadly, the goals of the software architecture employed in MITgcm are |
25 |
three-fold |
three-fold |
79 |
\end{figure} |
\end{figure} |
80 |
|
|
81 |
\section{WRAPPER} |
\section{WRAPPER} |
82 |
|
\begin{rawhtml} |
83 |
|
<!-- CMIREDIR:wrapper: --> |
84 |
|
\end{rawhtml} |
85 |
|
|
86 |
A significant element of the software architecture utilized in |
A significant element of the software architecture utilized in |
87 |
MITgcm is a software superstructure and substructure collectively |
MITgcm is a software superstructure and substructure collectively |
156 |
computer architecture currently available to the scientific computing community. |
computer architecture currently available to the scientific computing community. |
157 |
|
|
158 |
\subsection{Machine model parallelism} |
\subsection{Machine model parallelism} |
159 |
|
\begin{rawhtml} |
160 |
|
<!-- CMIREDIR:domain_decomp: --> |
161 |
|
\end{rawhtml} |
162 |
|
|
163 |
Codes operating under the WRAPPER target an abstract machine that is assumed to |
Codes operating under the WRAPPER target an abstract machine that is assumed to |
164 |
consist of one or more logical processors that can compute concurrently. |
consist of one or more logical processors that can compute concurrently. |
546 |
last 50 years. |
last 50 years. |
547 |
|
|
548 |
\section{Using the WRAPPER} |
\section{Using the WRAPPER} |
549 |
|
\begin{rawhtml} |
550 |
|
<!-- CMIREDIR:using_the_wrapper: --> |
551 |
|
\end{rawhtml} |
552 |
|
|
553 |
In order to support maximum portability the WRAPPER is implemented primarily |
In order to support maximum portability the WRAPPER is implemented primarily |
554 |
in sequential Fortran 77. At a practical level the key steps provided by the |
in sequential Fortran 77. At a practical level the key steps provided by the |
989 |
routines must be activated at compile time. Currently MPI libraries are |
routines must be activated at compile time. Currently MPI libraries are |
990 |
invoked by |
invoked by |
991 |
specifying the appropriate options file with the |
specifying the appropriate options file with the |
992 |
\begin{verbatim}-of\end{verbatim} flag when running the {\em genmake2} |
{\tt-of} flag when running the {\em genmake2} |
993 |
script, which generates the Makefile for compiling and linking MITgcm. |
script, which generates the Makefile for compiling and linking MITgcm. |
994 |
(Previously this was done by setting the {\em ALLOW\_USE\_MPI} and |
(Previously this was done by setting the {\em ALLOW\_USE\_MPI} and |
995 |
{\em ALWAYS\_USE\_MPI} flags in the {\em CPP\_EEOPTIONS.h} file.) More |
{\em ALWAYS\_USE\_MPI} flags in the {\em CPP\_EEOPTIONS.h} file.) More |
996 |
detailed information about the use of {\em genmake2} for specifying |
detailed information about the use of {\em genmake2} for specifying |
997 |
local compiler flags is located in section 3 ??\\ |
local compiler flags is located in section \ref{sect:genmake}.\\ |
998 |
|
|
999 |
|
|
1000 |
\fbox{ |
\fbox{ |
1001 |
\begin{minipage}{4.75in} |
\begin{minipage}{4.75in} |
1002 |
|
Directory: {\em tools/build\_options}\\ |
1003 |
File: {\em tools/genmake2} |
File: {\em tools/genmake2} |
1004 |
\end{minipage} |
\end{minipage} |
1005 |
} \\ |
} \\ |
1018 |
in the file {\em SIZE.h}. The parameter {\em mf} specifies that a text file |
in the file {\em SIZE.h}. The parameter {\em mf} specifies that a text file |
1019 |
called ``mf'' will be read to get a list of processor names on |
called ``mf'' will be read to get a list of processor names on |
1020 |
which the sixty-four processes will execute. The syntax of this file |
which the sixty-four processes will execute. The syntax of this file |
1021 |
is specified by the MPI distribution |
is specified by the MPI distribution. |
1022 |
\\ |
\\ |
1023 |
|
|
1024 |
\fbox{ |
\fbox{ |
1069 |
Allocation of processes to tiles in controlled by the routine |
Allocation of processes to tiles in controlled by the routine |
1070 |
{\em INI\_PROCS()}. For each process this routine sets |
{\em INI\_PROCS()}. For each process this routine sets |
1071 |
the variables {\em myXGlobalLo} and {\em myYGlobalLo}. |
the variables {\em myXGlobalLo} and {\em myYGlobalLo}. |
1072 |
These variables specify (in index space) the coordinate |
These variables specify in index space the coordinates |
1073 |
of the southern most and western most corner of the |
of the southernmost and westernmost corner of the |
1074 |
southern most and western most tile owned by this process. |
southernmost and westernmost tile owned by this process. |
1075 |
The variables {\em pidW}, {\em pidE}, {\em pidS} and {\em pidN} |
The variables {\em pidW}, {\em pidE}, {\em pidS} and {\em pidN} |
1076 |
are also set in this routine. These are used to identify |
are also set in this routine. These are used to identify |
1077 |
processes holding tiles to the west, east, south and north |
processes holding tiles to the west, east, south and north |
1080 |
communication routines. The above does not hold when the |
communication routines. The above does not hold when the |
1081 |
exch2 package is used -- exch2 sets its own parameters to |
exch2 package is used -- exch2 sets its own parameters to |
1082 |
specify the global indices of tiles and their relationships |
specify the global indices of tiles and their relationships |
1083 |
to each other. See exch2 docs for details??. |
to each other. See the documentation on the exch2 package |
1084 |
|
(\ref{sec:exch2}) for |
1085 |
|
details. |
1086 |
\\ |
\\ |
1087 |
|
|
1088 |
\fbox{ |
\fbox{ |
1108 |
describes the information that is held and used. |
describes the information that is held and used. |
1109 |
|
|
1110 |
\begin{enumerate} |
\begin{enumerate} |
1111 |
\item {\bf Tile-tile connectivity information} For each tile the WRAPPER |
\item {\bf Tile-tile connectivity information} |
1112 |
sets a flag that sets the tile number to the north, south, east and |
For each tile the WRAPPER |
1113 |
|
sets a flag that sets the tile number to the north, |
1114 |
|
south, east and |
1115 |
west of that tile. This number is unique over all tiles in a |
west of that tile. This number is unique over all tiles in a |
1116 |
configuration. Except when using the exch2 package, |
configuration. Except when using the cubed sphere and the exch2 package, |
1117 |
the number is held in the variables {\em tileNo} |
the number is held in the variables {\em tileNo} |
1118 |
( this holds the tiles own number), {\em tileNoN}, {\em tileNoS}, |
( this holds the tiles own number), {\em tileNoN}, {\em tileNoS}, |
1119 |
{\em tileNoE} and {\em tileNoW}. A parameter is also stored with each tile |
{\em tileNoE} and {\em tileNoW}. A parameter is also stored with each tile |
1137 |
(see figure \ref{fig:communication_primitives}). The routine |
(see figure \ref{fig:communication_primitives}). The routine |
1138 |
{\em ini\_communication\_patterns()} is responsible for setting the |
{\em ini\_communication\_patterns()} is responsible for setting the |
1139 |
communication mode values for each tile. |
communication mode values for each tile. |
|
\\ |
|
1140 |
|
|
1141 |
When using the cubed sphere configuration with the exch2 package, the |
When using the cubed sphere configuration with the exch2 package, the |
1142 |
relationships between tiles and their communication methods are set |
relationships between tiles and their communication methods are set |
1143 |
by the package in other variables. See the exch2 docs for details.?? |
by the package in other variables. See the exch2 package documentation |
1144 |
|
(\ref{sec:exch2} for details. |
1145 |
|
|
1146 |
|
|
1147 |
|
|
1279 |
the cube-sphere grid. In this class of grid a rotation may be required |
the cube-sphere grid. In this class of grid a rotation may be required |
1280 |
between tiles. Aligning the coordinate requiring rotation with the |
between tiles. Aligning the coordinate requiring rotation with the |
1281 |
tile decomposition, allows the coordinate transformation to |
tile decomposition, allows the coordinate transformation to |
1282 |
be embedded within a custom form of the \_EXCH primitive. |
be embedded within a custom form of the \_EXCH primitive. In these |
1283 |
|
cases \_EXCH is mapped to exch2 routines, as detailed in the exch2 |
1284 |
|
package documentation \ref{sec:exch2}. |
1285 |
|
|
1286 |
\item {\bf Reverse Mode} |
\item {\bf Reverse Mode} |
1287 |
The communication primitives \_EXCH and \_GSUM both employ |
The communication primitives \_EXCH and \_GSUM both employ |
1298 |
is set to the value {\em REVERSE\_SIMULATION}. This signifies |
is set to the value {\em REVERSE\_SIMULATION}. This signifies |
1299 |
ti the low-level routines that the adjoint forms of the |
ti the low-level routines that the adjoint forms of the |
1300 |
appropriate communication operation should be performed. |
appropriate communication operation should be performed. |
1301 |
|
|
1302 |
\item {\bf MAX\_NO\_THREADS} |
\item {\bf MAX\_NO\_THREADS} |
1303 |
The variable {\em MAX\_NO\_THREADS} is used to indicate the |
The variable {\em MAX\_NO\_THREADS} is used to indicate the |
1304 |
maximum number of OS threads that a code will use. This |
maximum number of OS threads that a code will use. This |
1403 |
This is done to allow a large number of variations on the exchange |
This is done to allow a large number of variations on the exchange |
1404 |
process to be maintained. One set of variations supports the |
process to be maintained. One set of variations supports the |
1405 |
cube sphere grid. Support for a cube sphere grid in MITgcm is based |
cube sphere grid. Support for a cube sphere grid in MITgcm is based |
1406 |
on having each face of the cube as a separate tile (or tiles). |
on having each face of the cube as a separate tile or tiles. |
1407 |
The exchange routines are then able to absorb much of the |
The exchange routines are then able to absorb much of the |
1408 |
detailed rotation and reorientation required when moving around the |
detailed rotation and reorientation required when moving around the |
1409 |
cube grid. The set of {\em \_EXCH} routines that contain the |
cube grid. The set of {\em \_EXCH} routines that contain the |
1427 |
|
|
1428 |
|
|
1429 |
\section{MITgcm execution under WRAPPER} |
\section{MITgcm execution under WRAPPER} |
1430 |
|
\begin{rawhtml} |
1431 |
|
<!-- CMIREDIR:mitgcm_wrapper: --> |
1432 |
|
\end{rawhtml} |
1433 |
|
|
1434 |
Fitting together the WRAPPER elements, package elements and |
Fitting together the WRAPPER elements, package elements and |
1435 |
MITgcm core equation elements of the source code produces calling |
MITgcm core equation elements of the source code produces calling |