661 |
computation is performed concurrently over as many processes and threads |
computation is performed concurrently over as many processes and threads |
662 |
as there are physical processors available to compute. |
as there are physical processors available to compute. |
663 |
|
|
664 |
|
An exception to the the use of {\em bi} and {\em bj} in loops arises in the |
665 |
|
exchange routines used when the exch2 package is used with the cubed |
666 |
|
sphere. In this case {\em bj} is generally set to 1 and the loop runs from |
667 |
|
1,{\em bi}. Within the loop {\em bi} is used to retrieve the tile number, |
668 |
|
which is then used to reference exchange parameters. |
669 |
|
|
670 |
The amount of computation that can be embedded |
The amount of computation that can be embedded |
671 |
a single loop over {\em bi} and {\em bj} varies for different parts of the |
a single loop over {\em bi} and {\em bj} varies for different parts of the |
672 |
MITgcm algorithm. Figure \ref{fig:bibj_extract} shows a code extract |
MITgcm algorithm. Figure \ref{fig:bibj_extract} shows a code extract |
935 |
File: {\em eesupp/inc/MAIN\_PDIRECTIVES2.h}\\ |
File: {\em eesupp/inc/MAIN\_PDIRECTIVES2.h}\\ |
936 |
File: {\em model/src/THE\_MODEL\_MAIN.F}\\ |
File: {\em model/src/THE\_MODEL\_MAIN.F}\\ |
937 |
File: {\em eesupp/src/MAIN.F}\\ |
File: {\em eesupp/src/MAIN.F}\\ |
938 |
File: {\em tools/genmake}\\ |
File: {\em tools/genmake2}\\ |
939 |
File: {\em eedata}\\ |
File: {\em eedata}\\ |
940 |
CPP: {\em TARGET\_SUN}\\ |
CPP: {\em TARGET\_SUN}\\ |
941 |
CPP: {\em TARGET\_DEC}\\ |
CPP: {\em TARGET\_DEC}\\ |
974 |
of controlling and coordinating the start up of a large number |
of controlling and coordinating the start up of a large number |
975 |
(hundreds and possibly even thousands) of copies of the same |
(hundreds and possibly even thousands) of copies of the same |
976 |
program, MPI is used. The calls to the MPI multi-process startup |
program, MPI is used. The calls to the MPI multi-process startup |
977 |
routines must be activated at compile time. This is done |
routines must be activated at compile time. Currently MPI libraries are |
978 |
by setting the {\em ALLOW\_USE\_MPI} and {\em ALWAYS\_USE\_MPI} |
invoked by |
979 |
flags in the {\em CPP\_EEOPTIONS.h} file.\\ |
specifying the appropriate options file with the |
980 |
|
\begin{verbatim}-of\end{verbatim} flag when running the {\em genmake2} |
981 |
\fbox{ |
script, which generates the Makefile for compiling and linking MITgcm. |
982 |
\begin{minipage}{4.75in} |
(Previously this was done by setting the {\em ALLOW\_USE\_MPI} and |
983 |
File: {\em eesupp/inc/CPP\_EEOPTIONS.h}\\ |
{\em ALWAYS\_USE\_MPI} flags in the {\em CPP\_EEOPTIONS.h} file.) More |
984 |
CPP: {\em ALLOW\_USE\_MPI}\\ |
detailed information about the use of {\em genmake2} for specifying |
985 |
CPP: {\em ALWAYS\_USE\_MPI}\\ |
local compiler flags is located in section 3 ??\\ |
|
Parameter: {\em nPx}\\ |
|
|
Parameter: {\em nPy} |
|
|
\end{minipage} |
|
|
} \\ |
|
986 |
|
|
|
Additionally, compile time options are required to link in the |
|
|
MPI libraries and header files. Examples of these options |
|
|
can be found in the {\em genmake} script that creates makefiles |
|
|
for compilation. When this script is executed with the {bf -mpi} |
|
|
flag it will generate a makefile that includes |
|
|
paths for search for MPI head files and for linking in |
|
|
MPI libraries. For example the {\bf -mpi} flag on a |
|
|
Silicon Graphics IRIX system causes a |
|
|
Makefile with the compilation command |
|
|
Graphics IRIX system \begin{verbatim} |
|
|
mpif77 -I/usr/local/mpi/include -DALLOW_USE_MPI -DALWAYS_USE_MPI |
|
|
\end{verbatim} |
|
|
to be generated. |
|
|
This is the correct set of options for using the MPICH open-source |
|
|
version of MPI, when it has been installed under the subdirectory |
|
|
/usr/local/mpi. |
|
|
However, on many systems there may be several |
|
|
versions of MPI installed. For example many systems have both |
|
|
the open source MPICH set of libraries and a vendor specific native form |
|
|
of the MPI libraries. The correct setup to use will depend on the |
|
|
local configuration of your system.\\ |
|
987 |
|
|
988 |
\fbox{ |
\fbox{ |
989 |
\begin{minipage}{4.75in} |
\begin{minipage}{4.75in} |
990 |
File: {\em tools/genmake} |
File: {\em tools/genmake2} |
991 |
\end{minipage} |
\end{minipage} |
992 |
} \\ |
} \\ |
993 |
\paragraph{\bf Execution} The mechanics of starting a program in |
\paragraph{\bf Execution} The mechanics of starting a program in |
1064 |
processes holding tiles to the west, east, south and north |
processes holding tiles to the west, east, south and north |
1065 |
of this process. These values are stored in global storage |
of this process. These values are stored in global storage |
1066 |
in the header file {\em EESUPPORT.h} for use by |
in the header file {\em EESUPPORT.h} for use by |
1067 |
communication routines. |
communication routines. The above does not hold when the |
1068 |
|
exch2 package is used -- exch2 sets its own parameters to |
1069 |
|
specify the global indices of tiles and their relationships |
1070 |
|
to each other. See exch2 docs for details??. |
1071 |
\\ |
\\ |
1072 |
|
|
1073 |
\fbox{ |
\fbox{ |
1096 |
\item {\bf Tile-tile connectivity information} For each tile the WRAPPER |
\item {\bf Tile-tile connectivity information} For each tile the WRAPPER |
1097 |
sets a flag that sets the tile number to the north, south, east and |
sets a flag that sets the tile number to the north, south, east and |
1098 |
west of that tile. This number is unique over all tiles in a |
west of that tile. This number is unique over all tiles in a |
1099 |
configuration. The number is held in the variables {\em tileNo} |
configuration. Except when using the exch2 package, |
1100 |
|
the number is held in the variables {\em tileNo} |
1101 |
( this holds the tiles own number), {\em tileNoN}, {\em tileNoS}, |
( this holds the tiles own number), {\em tileNoN}, {\em tileNoS}, |
1102 |
{\em tileNoE} and {\em tileNoW}. A parameter is also stored with each tile |
{\em tileNoE} and {\em tileNoW}. A parameter is also stored with each tile |
1103 |
that specifies the type of communication that is used between tiles. |
that specifies the type of communication that is used between tiles. |
1122 |
communication mode values for each tile. |
communication mode values for each tile. |
1123 |
\\ |
\\ |
1124 |
|
|
1125 |
|
When using the cubed sphere configuration with the exch2 package, the |
1126 |
|
relationships between tiles and their communication methods are set |
1127 |
|
by the package in other variables. See the exch2 docs for details.?? |
1128 |
|
|
1129 |
|
|
1130 |
|
|
1131 |
\fbox{ |
\fbox{ |
1132 |
\begin{minipage}{4.75in} |
\begin{minipage}{4.75in} |
1133 |
File: {\em eesupp/src/ini\_communication\_patterns.F}\\ |
File: {\em eesupp/src/ini\_communication\_patterns.F}\\ |