/[MITgcm]/manual/s_software/text/sarch.tex

Diff of /manual/s_software/text/sarch.tex

Parent Directory | Revision Log | View Revision Graph Revision Graph | View Patch Patch

-revision 1.9 by adcroft,
Wed Apr 24 19:23:58 2002 UTC
+revision 1.20 by edhill,
Sat Oct 16 03:40:16 2004 UTC
 Line 17 
 details of the MITgcm implementation and
  features that are employed.
  \section{Overall architectural goals}
+ \begin{rawhtml}
+ <!-- CMIREDIR:overall_architectural_goals: -->
+ \end{rawhtml}
  Broadly, the goals of the software architecture employed in MITgcm are
  three-fold
-Line 76 
 floating point operations.}
+Line 79 
 floating point operations.}
  \end{figure}
  \section{WRAPPER}
+ \begin{rawhtml}
+ <!-- CMIREDIR:wrapper: -->
+ \end{rawhtml}
  A significant element of the software architecture utilized in
  MITgcm is a software superstructure and substructure collectively
-Line 97 
 and operating systems. This allows numer
+Line 103 
 and operating systems. This allows numer
  \resizebox{!}{4.5in}{\includegraphics{part4/fit_in_wrapper.eps}}
  \end{center}
  \caption{
- Numerical code is written too fit within a software support
+ Numerical code is written to fit within a software support
  infrastructure called WRAPPER. The WRAPPER is portable and
  can be specialized for a wide range of specific target hardware and
  programming environments, without impacting numerical code that fits
-Line 120 
 uniprocessor and multi-processor Sun sys
+Line 126 
 uniprocessor and multi-processor Sun sys
  (UMA) and non-uniform memory access (NUMA) designs. Significant work has also
  been undertaken on x86 cluster systems, Alpha processor based clustered SMP
  systems, and on cache-coherent NUMA (CC-NUMA) systems from Silicon Graphics.
- The MITgcm code, operating within the WRAPPER, is also used routinely used on
+ The MITgcm code, operating within the WRAPPER, is also routinely used on
  large scale MPP systems (for example T3E systems and IBM SP systems). In all
  cases numerical code, operating within the WRAPPER, performs and scales very
  competitively with equivalent numerical code that has been modified to contain
-Line 150 
 easily be specialized to fit, in a compu
+Line 156 
 easily be specialized to fit, in a compu
  computer architecture currently available to the scientific computing community.
  \subsection{Machine model parallelism}
+ \begin{rawhtml}
+ <!-- CMIREDIR:domain_decomp: -->
+ \end{rawhtml}
   Codes operating under the WRAPPER target an abstract machine that is assumed to
  consist of one or more logical processors that can compute concurrently.
-Line 537 
 of almost all successful scientific comp
+Line 546 
 of almost all successful scientific comp
  last 50 years.
  \section{Using the WRAPPER}
+ \begin{rawhtml}
+ <!-- CMIREDIR:using_the_wrapper: -->
+ \end{rawhtml}
  In order to support maximum portability the WRAPPER is implemented primarily
  in sequential Fortran 77. At a practical level the key steps provided by the
-Line 661 
 Within a {\em bi}, {\em bj} loop
+Line 673 
 Within a {\em bi}, {\em bj} loop
  computation is performed concurrently over as many processes and threads
  as there are physical processors available to compute.
+ An exception to the the use of {\em bi} and {\em bj} in loops arises in the
+ exchange routines used when the exch2 package is used with the cubed
+ sphere.  In this case {\em bj} is generally set to 1 and the loop runs from
+,{\em bi}.  Within the loop {\em bi} is used to retrieve the tile number,
+ which is then used to reference exchange parameters.
  The amount of computation that can be embedded
  a single loop over {\em bi} and {\em bj} varies for different parts of the
  MITgcm algorithm. Figure \ref{fig:bibj_extract} shows a code extract
-Line 781 
 The global domain size is again ninety g
+Line 799 
 The global domain size is again ninety g
  forty grid points in y. The two sub-domains in each process will be computed
  sequentially if they are given to a single thread within a single process.
  Alternatively if the code is invoked with multiple threads per process
- the two domains in y may be computed on concurrently.
+ the two domains in y may be computed concurrently.
  \item
  \begin{verbatim}
        PARAMETER (
-Line 929 
 File: {\em eesupp/inc/MAIN\_PDIRECTIVES1
+Line 947 
 File: {\em eesupp/inc/MAIN\_PDIRECTIVES1
  File: {\em eesupp/inc/MAIN\_PDIRECTIVES2.h}\\
  File: {\em model/src/THE\_MODEL\_MAIN.F}\\
  File: {\em eesupp/src/MAIN.F}\\
- File: {\em tools/genmake}\\
+ File: {\em tools/genmake2}\\
  File: {\em eedata}\\
  CPP:  {\em TARGET\_SUN}\\
  CPP:  {\em TARGET\_DEC}\\
-Line 968 
 critical communication. However, in orde
+Line 986 
 critical communication. However, in orde
  of controlling and coordinating the start up of a large number
  (hundreds and possibly even thousands) of copies of the same
  program, MPI is used. The calls to the MPI multi-process startup
- routines must be activated at compile time. This is done
+ routines must be activated at compile time.  Currently MPI libraries are
- by setting the {\em ALLOW\_USE\_MPI} and {\em ALWAYS\_USE\_MPI}
+ invoked by
- flags in the {\em CPP\_EEOPTIONS.h} file.\\
+ specifying the appropriate options file with the
+ {\tt-of} flag when running the {\em genmake2}
- \fbox{
+ script, which generates the Makefile for compiling and linking MITgcm.
- \begin{minipage}{4.75in}
+ (Previously this was done by setting the {\em ALLOW\_USE\_MPI} and
- File: {\em eesupp/inc/CPP\_EEOPTIONS.h}\\
+ {\em ALWAYS\_USE\_MPI} flags in the {\em CPP\_EEOPTIONS.h} file.)  More
- CPP:  {\em ALLOW\_USE\_MPI}\\
+ detailed information about the use of {\em genmake2} for specifying
- CPP:  {\em ALWAYS\_USE\_MPI}\\
+ local compiler flags is located in section \ref{sect:genmake}.\\
- Parameter:  {\em nPx}\\
- Parameter:  {\em nPy}
- \end{minipage}
- } \\
- Additionally, compile time options are required to link in the
- MPI libraries and header files. Examples of these options
- can be found in the {\em genmake} script that creates makefiles
- for compilation. When this script is executed with the {bf -mpi}
- flag it will generate a makefile that includes
- paths for search for MPI head files and for linking in
- MPI libraries. For example the {\bf -mpi} flag on a
-  Silicon Graphics IRIX system causes a
- Makefile with the compilation command
- Graphics IRIX system \begin{verbatim}
- mpif77 -I/usr/local/mpi/include -DALLOW_USE_MPI -DALWAYS_USE_MPI
- \end{verbatim}
- to be generated.
- This is the correct set of options for using the MPICH open-source
- version of MPI, when it has been installed under the subdirectory
- /usr/local/mpi.
- However, on many systems there may be several
- versions of MPI installed. For example many systems have both
- the open source MPICH set of libraries and a vendor specific native form
- of the MPI libraries. The correct setup to use will depend on the
- local configuration of your system.\\
  \fbox{
  \begin{minipage}{4.75in}
- File: {\em tools/genmake}
+ Directory: {\em tools/build\_options}\\
+ File: {\em tools/genmake2}
  \end{minipage}
  } \\
  \paragraph{\bf Execution} The mechanics of starting a program in
-Line 1024 
 product of the processor grid settings o
+Line 1018 
 product of the processor grid settings o
  in the file {\em SIZE.h}. The parameter {\em mf} specifies that a text file
  called ``mf'' will be read to get a list of processor names on
  which the sixty-four processes will execute. The syntax of this file
- is specified by the MPI distribution
+ is specified by the MPI distribution.
  \\
  \fbox{
-Line 1075 
 to processor identification are are used
+Line 1069 
 to processor identification are are used
  Allocation of processes to tiles in controlled by the routine
  {\em INI\_PROCS()}. For each process this routine sets
  the variables {\em myXGlobalLo} and {\em myYGlobalLo}.
- These variables specify (in index space) the coordinate
+ These variables specify in index space the coordinates
- of the southern most and western most corner of the
+ of the southernmost and westernmost corner of the
- southern most and western most tile owned by this process.
+ southernmost and westernmost tile owned by this process.
  The variables {\em pidW}, {\em pidE}, {\em pidS} and {\em pidN}
  are also set in this routine. These are used to identify
  processes holding tiles to the west, east, south and north
  of this process. These values are stored in global storage
  in the header file {\em EESUPPORT.h} for use by
- communication routines.
+ communication routines.  The above does not hold when the
+ exch2 package is used -- exch2 sets its own parameters to
+ specify the global indices of tiles and their relationships
+ to each other.  See the documentation on the exch2 package
+ (\ref{sec:exch2})  for
+ details.
  \\
  \fbox{
-Line 1109 
 operations and that can be customized fo
+Line 1108 
 operations and that can be customized fo
  describes the information that is held and used.
  \begin{enumerate}
- \item {\bf Tile-tile connectivity information} For each tile the WRAPPER
+ \item {\bf Tile-tile connectivity information}
- sets a flag that sets the tile number to the north, south, east and
+ For each tile the WRAPPER
+ sets a flag that sets the tile number to the north,
+ south, east and
  west of that tile. This number is unique over all tiles in a
- configuration. The number is held in the variables {\em tileNo}
+ configuration. Except when using the cubed sphere and the exch2 package,
+ the number is held in the variables {\em tileNo}
  ( this holds the tiles own number), {\em tileNoN}, {\em tileNoS},
  {\em tileNoE} and {\em tileNoW}. A parameter is also stored with each tile
  that specifies the type of communication that is used between tiles.
-Line 1135 
 of the WRAPPER exchange primitive
+Line 1137 
 of the WRAPPER exchange primitive
  (see figure \ref{fig:communication_primitives}). The routine
  {\em ini\_communication\_patterns()} is responsible for setting the
  communication mode values for each tile.
- \\
+ When using the cubed sphere configuration with the exch2 package, the
+ relationships between tiles and their communication methods are set
+ by the package in other variables.  See the exch2 package documentation
+ (\ref{sec:exch2} for details.
  \fbox{
  \begin{minipage}{4.75in}
-Line 1271 
 The \_EXCH operation is also central to
+Line 1279 
 The \_EXCH operation is also central to
  the cube-sphere grid. In this class of grid a rotation may be required
  between tiles. Aligning the coordinate requiring rotation with the
  tile decomposition, allows the coordinate transformation to
- be embedded within a custom form of the \_EXCH primitive.
+ be embedded within a custom form of the \_EXCH primitive.  In these
+ cases \_EXCH is mapped to exch2 routines, as detailed in the exch2
+ package documentation  \ref{sec:exch2}.
  \item {\bf Reverse Mode}
  The communication primitives \_EXCH and \_GSUM both employ
-Line 1288 
 operations. However, the routine argumen
+Line 1298 
 operations. However, the routine argumen
  is set to the value {\em REVERSE\_SIMULATION}. This signifies
  ti the low-level routines that the adjoint forms of the
  appropriate communication operation should be performed.
  \item {\bf MAX\_NO\_THREADS}
  The variable {\em MAX\_NO\_THREADS} is used to indicate the
  maximum number of OS threads that a code will use. This
-Line 1392 
 a series of template files, for example
+Line 1403 
 a series of template files, for example
  This is done to allow a large number of variations on the exchange
  process to be maintained. One set of variations supports the
  cube sphere grid. Support for a cube sphere grid in MITgcm is based
- on having each face of the cube as a separate tile (or tiles).
+ on having each face of the cube as a separate tile or tiles.
  The exchange routines are then able to absorb much of the
  detailed rotation and reorientation required when moving around the
  cube grid. The set of {\em \_EXCH} routines that contain the
-Line 1416 
 quantities at the C-grid vorticity point
+Line 1427 
 quantities at the C-grid vorticity point
  \section{MITgcm execution under WRAPPER}
+ \begin{rawhtml}
+ <!-- CMIREDIR:mitgcm_wrapper: -->
+ \end{rawhtml}
  Fitting together the WRAPPER elements, package elements and
  MITgcm core equation elements of the source code produces calling

 Legend:



Removed from v.1.9
 


changed lines


 
Added in v.1.20
 Legend:



Removed from v.1.9
 


changed lines


 
Added in v.1.20
-Removed from v.1.9
+Added in v.1.20

	ViewVC Help
Powered by ViewVC 1.1.22