/[MITgcm]/manual/s_software/text/sarch.tex

Diff of /manual/s_software/text/sarch.tex

Parent Directory | Revision Log | View Revision Graph Revision Graph | View Patch Patch

-revision 1.1 by cnh,
Tue Oct  9 10:33:17 2001 UTC
+revision 1.4 by cnh,
Thu Oct 25 18:36:55 2001 UTC
 Line 1
+ % $Header$
  In this chapter we describe the software architecture and
  implementation strategy for the MITgcm code. The first part of this
-Line 11 
 Broadly, the goals of the software archi
+Line 12 
 Broadly, the goals of the software archi
  three-fold
  \begin{itemize}
  \item We wish to be able to study a very broad range
  of interesting and challenging rotating fluids problems.
  \item We wish the model code to be readily targeted to
  a wide range of platforms
  \item On any given platform we would like to be
  able to achieve performance comparable to an implementation
  developed and specialized specifically for that platform.
  \end{itemize}
  These points are summarized in figure \ref{fig:mitgcm_architecture_goals}
-Line 30 
 a software architecture which at the hig
+Line 27 
 a software architecture which at the hig
  of
  \begin{enumerate}
  \item A core set of numerical and support code. This is discussed in detail in
  section \ref{sec:partII}.
  \item A scheme for supporting optional "pluggable" {\bf packages} (containing
  for example mixed-layer schemes, biogeochemical schemes, atmospheric physics).
  These packages are used both to overlay alternate dynamics and to introduce
  specialized physical content onto the core numerical code. An overview of
  the {\bf package} scheme is given at the start of part \ref{part:packages}.
  \item A support framework called {\bf WRAPPER} (Wrappable Application Parallel
  Programming Environment Resource), within which the core numerics and pluggable
  packages operate.
  \end{enumerate}
  This chapter focuses on describing the {\bf WRAPPER} environment under which
  both the core numerics and the pluggable packages function. The description
- presented here is intended to be a detailed exposistion and contains significant
+ presented here is intended to be a detailed exposition and contains significant
  background material, as well as advanced details on working with the WRAPPER.
  The examples section of this manual (part \ref{part:example}) contains more
  succinct, step-by-step instructions on running basic numerical
-Line 57 
 experiments both sequentially and in par
+Line 49 
 experiments both sequentially and in par
  starting from an example code and adapting it to suit a particular situation
  will be all that is required.
  \begin{figure}
  \begin{center}
-  \resizebox{!}{2.5in}{
+ \resizebox{!}{2.5in}{\includegraphics{part4/mitgcm_goals.eps}}
-   \includegraphics*[1.5in,2.4in][9.5in,6.3in]{part4/mitgcm_goals.eps}
-  }
  \end{center}
- \caption{The MITgcm architecture is designed to allow simulation of a wide
+ \caption{
+ The MITgcm architecture is designed to allow simulation of a wide
  range of physical problems on a wide range of hardware. The computational
  resource requirements of the applications targeted range from around
  $10^7$ bytes ( $\approx 10$ megabytes ) of memory to $10^{11}$ bytes
  ( $\approx 100$ gigabytes). Arithmetic operation counts for the applications of
  interest range from $10^{9}$ floating point operations to more than $10^{17}$
- floating point operations.} \label{fig:mitgcm_architecture_goals}
+ floating point operations.}
+ \label{fig:mitgcm_architecture_goals}
  \end{figure}
  \section{WRAPPER}
-Line 87 
 The approach taken by the WRAPPER is ill
+Line 80 
 The approach taken by the WRAPPER is ill
  \ref{fig:fit_in_wrapper} which shows how the WRAPPER serves to insulate code
  that fits within it from architectural differences between hardware platforms
  and operating systems. This allows numerical code to be easily retargetted.
  \begin{figure}
  \begin{center}
-  \resizebox{6in}{4.5in}{
+ \resizebox{!}{4.5in}{\includegraphics{part4/fit_in_wrapper.eps}}
-   \includegraphics*[0.6in,0.7in][9.0in,8.5in]{part4/fit_in_wrapper.eps}
-  }
  \end{center}
- \caption{ Numerical code is written too fit within a software support
+ \caption{
+ Numerical code is written too fit within a software support
  infrastructure called WRAPPER. The WRAPPER is portable and
- can be sepcialized for a wide range of specific target hardware and
+ can be specialized for a wide range of specific target hardware and
  programming environments, without impacting numerical code that fits
  within the WRAPPER. Codes that fit within the WRAPPER can generally be
  made to run as fast on a particular platform as codes specially
- optimized for that platform.
+ optimized for that platform.}
- } \label{fig:fit_in_wrapper}
+ \label{fig:fit_in_wrapper}
  \end{figure}
  \subsection{Target hardware}
-Line 172 
 Computationally, associated with each re
+Line 166 
 Computationally, associated with each re
  space allocated to a particular logical processor, there will be data
  structures (arrays, scalar variables etc...) that hold the simulated state of
  that region. We refer to these data structures as being {\bf owned} by the
- pprocessor to which their
+ processor to which their
  associated region of physical space has been allocated. Individual
  regions that are allocated to processors are called {\bf tiles}. A
  processor can own more
-Line 186 
 independently of the other tiles, in a s
+Line 180 
 independently of the other tiles, in a s
  \begin{figure}
  \begin{center}
-  \resizebox{7in}{3in}{
+  \resizebox{5in}{!}{
-   \includegraphics*[0.5in,2.7in][12.5in,6.4in]{part4/domain_decomp.eps}
+   \includegraphics{part4/domain_decomp.eps}
   }
  \end{center}
  \caption{ The WRAPPER provides support for one and two dimensional
-Line 222 
 variety of different mechanisms to commu
+Line 216 
 variety of different mechanisms to commu
  \begin{figure}
  \begin{center}
-  \resizebox{7in}{3in}{
+  \resizebox{5in}{!}{
-   \includegraphics*[4.5in,3.7in][12.5in,6.7in]{part4/tiled-world.eps}
+   \includegraphics{part4/tiled-world.eps}
   }
  \end{center}
  \caption{ A global grid subdivided into tiles.
-Line 404 
 highly optimized library.
+Line 398 
 highly optimized library.
  \begin{figure}
  \begin{center}
-  \resizebox{5in}{3in}{
+  \resizebox{5in}{!}{
-   \includegraphics*[1.5in,0.7in][7.9in,4.4in]{part4/comm-primm.eps}
+   \includegraphics{part4/comm-primm.eps}
   }
  \end{center}
  \caption{Three performance critical parallel primititives are provided
-Line 485 
 sub-domains.
+Line 479 
 sub-domains.
  \begin{figure}
  \begin{center}
-  \resizebox{5in}{3in}{
+  \resizebox{5in}{!}{
-   \includegraphics*[0.5in,1.3in][7.9in,5.7in]{part4/tiling_detail.eps}
+   \includegraphics{part4/tiling_detail.eps}
   }
  \end{center}
  \caption{The tiling strategy that the WRAPPER supports allows tiles
-Line 589 
 not cause any other problems.
+Line 583 
 not cause any other problems.
  \begin{figure}
  \begin{center}
-  \resizebox{5in}{7in}{
+  \resizebox{5in}{!}{
-   \includegraphics*[0.5in,0.3in][7.9in,10.7in]{part4/size_h.eps}
+   \includegraphics{part4/size_h.eps}
   }
  \end{center}
  \caption{ The three level domain decomposition hierarchy employed by the
-Line 795 
 thirty-two grid points, and x and y over
+Line 789 
 thirty-two grid points, and x and y over
  There are six tiles allocated to six separate logical processors ({\em nSx=6}).
  This set of values can be used for a cube sphere calculation.
  Each tile of size $32 \times 32$ represents a face of the
- cube. Initialising the tile connectivity correctly ( see section
+ cube. Initializing the tile connectivity correctly ( see section
  \ref{sec:cube_sphere_communication}. allows the rotations associated with
  moving between the six cube faces to be embedded within the
  tile-tile communication code.
-Line 812 
 to support subsequent calls to communica
+Line 806 
 to support subsequent calls to communica
  by the application code. The startup calling sequence followed by the
  WRAPPER is shown in figure \ref{fig:wrapper_startup}.
  \begin{figure}
  \begin{verbatim}
-Line 849 
 occurs through the procedure {\em THE\_M
+Line 842 
 occurs through the procedure {\em THE\_M
  \end{figure}
  \subsubsection{Multi-threaded execution}
+ \label{sec:multi-threaded-execution}
  Prior to transferring control to the procedure {\em THE\_MODEL\_MAIN()} the
  WRAPPER may cause several coarse grain threads to be initialized. The routine
  {\em THE\_MODEL\_MAIN()} is called once for each thread and is passed a single
-Line 904 
 parallelization the compiler may otherwi
+Line 898 
 parallelization the compiler may otherwi
  \end{enumerate}
- \paragraph{Environment variables}
- On most systems multi-threaded execution also requires the setting
- of a special environment variable. On many machines this variable
- is called PARALLEL and its values should be set to the number
- of parallel threads required. Generally the help pages associated
- with the multi-threaded compiler on a machine will explain
- how to set the required environment variables for that machines.
- \paragraph{Runtime input parameters}
- Finally the file {\em eedata} needs to be configured to indicate
- the number of threads to be used in the x and y directions.
- The variables {\em nTx} and {\em nTy} in this file are used to
- specify the information required. The product of {\em nTx} and
- {\em nTy} must be equal to the number of threads spawned i.e.
- the setting of the environment variable PARALLEL.
- The value of {\em nTx} must subdivide the number of sub-domains
- in x ({\em nSx}) exactly. The value of {\em nTy} must subdivide the
- number of sub-domains in y ({\em nSy}) exactly.
  An example of valid settings for the {\em eedata} file for a
  domain with two subdomains in y and running with two threads is shown
  below
-Line 955 
 Parameter:  {\em nTy}
+Line 930 
 Parameter:  {\em nTy}
  } \\
  \subsubsection{Multi-process execution}
+ \label{sec:multi-process-execution}
  Despite its appealing programming model, multi-threaded execution remains
  less common then multi-process execution. One major reason for this
-Line 966 
 models varies between systems.
+Line 942 
 models varies between systems.
  Multi-process execution is more ubiquitous.
  In order to run code in a multi-process configuration a decomposition
- specification is given ( in which the at least one of the
+ specification ( see section \ref{sec:specifying_a_decomposition})
+ is given ( in which the at least one of the
  parameters {\em nPx} or {\em nPy} will be greater than one)
  and then, as for multi-threaded operation,
  appropriate compile time and run time steps must be taken.
-Line 1046 
 Parameter: {\em nPy}
+Line 1023 
 Parameter: {\em nPy}
  \end{minipage}
  } \\
+ \paragraph{Environment variables}
+ On most systems multi-threaded execution also requires the setting
+ of a special environment variable. On many machines this variable
+ is called PARALLEL and its values should be set to the number
+ of parallel threads required. Generally the help pages associated
+ with the multi-threaded compiler on a machine will explain
+ how to set the required environment variables for that machines.
+ \paragraph{Runtime input parameters}
+ Finally the file {\em eedata} needs to be configured to indicate
+ the number of threads to be used in the x and y directions.
+ The variables {\em nTx} and {\em nTy} in this file are used to
+ specify the information required. The product of {\em nTx} and
+ {\em nTy} must be equal to the number of threads spawned i.e.
+ the setting of the environment variable PARALLEL.
+ The value of {\em nTx} must subdivide the number of sub-domains
+ in x ({\em nSx}) exactly. The value of {\em nTy} must subdivide the
+ number of sub-domains in y ({\em nSy}) exactly.
  The multiprocess startup of the MITgcm executable {\em mitgcmuv}
  is controlled by the routines {\em EEBOOT\_MINIMAL()} and
  {\em INI\_PROCS()}. The first routine performs basic steps required
-Line 1058 
 number so that process number 0 will cre
+Line 1054 
 number so that process number 0 will cre
  output files {\bf STDOUT.0001} and {\bf STDERR.0001} etc... These files
  are used for reporting status and configuration information and
  for reporting error conditions on a process by process basis.
- The {{\em EEBOOT\_MINIMAL()} procedure also sets the variables
+ The {\em EEBOOT\_MINIMAL()} procedure also sets the variables
  {\em myProcId} and {\em MPI\_COMM\_MODEL}.
  These variables are related
  to processor identification are are used later in the routine
-Line 1099 
 Parameter: {\em pidN       }
+Line 1095 
 Parameter: {\em pidN       }
  The WRAPPER maintains internal information that is used for communication
  operations and that can be customized for different platforms. This section
  describes the information that is held and used.
  \begin{enumerate}
  \item {\bf Tile-tile connectivity information} For each tile the WRAPPER
  sets a flag that sets the tile number to the north, south, east and
-Line 1112 
 This information is held in the variable
+Line 1109 
 This information is held in the variable
  This latter set of variables can take one of the following values
  {\em COMM\_NONE}, {\em COMM\_MSG}, {\em COMM\_PUT} and {\em COMM\_GET}.
  A value of {\em COMM\_NONE} is used to indicate that a tile has no
- neighbor to cummnicate with on a particular face. A value
+ neighbor to communicate with on a particular face. A value
  of {\em COMM\_MSG} is used to indicated that some form of distributed
  memory communication is required to communicate between
  these tile faces ( see section \ref{sec:distributed_memory_communication}).
-Line 1169 
 the product of the parameters {\em nTx}
+Line 1166 
 the product of the parameters {\em nTx}
  are read from the file {\em eedata}. If the value of {\em nThreads}
  is inconsistent with the number of threads requested from the
  operating system (for example by using an environment
- varialble as described in section \ref{sec:multi_threaded_execution})
+ variable as described in section \ref{sec:multi_threaded_execution})
  then usually an error will be reported by the routine
  {\em CHECK\_THREADS}.\\
-Line 1186 
 Parameter: {\em nTy} \\
+Line 1183 
 Parameter: {\em nTy} \\
  \end{minipage}
  }
- \begin{figure}
- \begin{verbatim}
- C--
- C--  Parallel directives for MIPS Pro Fortran compiler
- C--
- C      Parallel compiler directives for SGI with IRIX
- C$PAR  PARALLEL DO
- C$PAR&  CHUNK=1,MP_SCHEDTYPE=INTERLEAVE,
- C$PAR&  SHARE(nThreads),LOCAL(myThid,I)
- C
-       DO I=1,nThreads
-         myThid = I
- C--     Invoke nThreads instances of the numerical model
-         CALL THE_MODEL_MAIN(myThid)
-       ENDDO
- \end{verbatim}
- \caption{Prior to transferring control to
- the procedure {\em THE\_MODEL\_MAIN()} the WRAPPER may use
- MP directives to spawn multiple threads.
- } \label{fig:mp_directives}
- \end{figure}
  \item {\bf memsync flags}
  As discussed in section \ref{sec:memory_consistency}, when using shared memory,
  a low-level system function may be need to force memory consistency.
-Line 1228 
 For an Ultra Sparc system the following
+Line 1200 
 For an Ultra Sparc system the following
  \begin{verbatim}
  asm("membar #LoadStore|#StoreStore");
  \end{verbatim}
- for an Alpha based sytem the euivalent code reads
+ for an Alpha based system the equivalent code reads
  \begin{verbatim}
  asm("mb");
  \end{verbatim}
-Line 1344 
 In the case of permanent state informati
+Line 1316 
 In the case of permanent state informati
  because there has to be enough storage allocated for all tiles.
  However, the technique can sometimes be a useful scheme for reducing memory
  requirements in complex physical paramterisations.
  \end{enumerate}
+ \begin{figure}
+ \begin{verbatim}
+ C--
+ C--  Parallel directives for MIPS Pro Fortran compiler
+ C--
+ C      Parallel compiler directives for SGI with IRIX
+ C$PAR  PARALLEL DO
+ C$PAR&  CHUNK=1,MP_SCHEDTYPE=INTERLEAVE,
+ C$PAR&  SHARE(nThreads),LOCAL(myThid,I)
+ C
+       DO I=1,nThreads
+         myThid = I
+ C--     Invoke nThreads instances of the numerical model
+         CALL THE_MODEL_MAIN(myThid)
+       ENDDO
+ \end{verbatim}
+ \caption{Prior to transferring control to
+ the procedure {\em THE\_MODEL\_MAIN()} the WRAPPER may use
+ MP directives to spawn multiple threads.
+ } \label{fig:mp_directives}
+ \end{figure}
  \subsubsection{Specializing the Communication Code}
  The isolation of performance critical communication primitives and the
-Line 1371 
 communications library ( see {\em ini\_j
+Line 1367 
 communications library ( see {\em ini\_j
  \item The {\em \_GSUM} and {\em \_EXCH} macro definitions are replaced
  with calls to custom routines ( see {\em gsum\_jam.F} and {\em exch\_jam.F})
  \item a highly specialized form of the exchange operator (optimized
- for overlap regions of width one) is substitued into the elliptic
+ for overlap regions of width one) is substituted into the elliptic
  solver routine {\em cg2d.F}.
  \end{itemize}
  Developing specialized code for other libraries follows a similar
-Line 1383 
 Actual {\em \_EXCH} routine code is gene
+Line 1379 
 Actual {\em \_EXCH} routine code is gene
  a series of template files, for example {\em exch\_rx.template}.
  This is done to allow a large number of variations on the exchange
  process to be maintained. One set of variations supports the
- cube sphere grid. Support for a cube sphere gris in MITgcm is based
+ cube sphere grid. Support for a cube sphere grid in MITgcm is based
  on having each face of the cube as a separate tile (or tiles).
- The exchage routines are then able to absorb much of the
+ The exchange routines are then able to absorb much of the
  detailed rotation and reorientation required when moving around the
  cube grid. The set of {\em \_EXCH} routines that contain the
  word cube in their name perform these transformations.
  They are invoked when the run-time logical parameter
  {\em useCubedSphereExchange} is set true. To facilitate the
  transformations on a staggered C-grid, exchange operations are defined
- separately for both vector and scalar quantitities and for
+ separately for both vector and scalar quantities and for
  grid-centered and for grid-face and corner quantities.
  Three sets of exchange routines are defined. Routines
  with names of the form {\em exch\_rx} are used to exchange
-Line 1457 
 C  :
+Line 1453 
 C  :
  C  |
  C  |-THE_MODEL_MAIN :: Primary driver for the MITgcm algorithm
  C    |              :: Called from WRAPPER level numerical
- C    |              :: code innvocation routine. On entry
+ C    |              :: code invocation routine. On entry
  C    |              :: to THE_MODEL_MAIN separate thread and
  C    |              :: separate processes will have been established.
  C    |              :: Each thread and process will have a unique ID
-Line 1471 
 C    | |-INI_PARMS :: Routine to set ker
+Line 1467 
 C    | |-INI_PARMS :: Routine to set ker
  C    | |           :: By default kernel parameters are read from file
  C    | |           :: "data" in directory in which code executes.
  C    | |
- C    | |-MON_INIT :: Initialises monitor pacakge ( see pkg/monitor )
+ C    | |-MON_INIT :: Initializes monitor package ( see pkg/monitor )
  C    | |
- C    | |-INI_GRID :: Control grid array (vert. and hori.) initialisation.
+ C    | |-INI_GRID :: Control grid array (vert. and hori.) initialization.
  C    | | |        :: Grid arrays are held and described in GRID.h.
  C    | | |
- C    | | |-INI_VERTICAL_GRID        :: Initialise vertical grid arrays.
+ C    | | |-INI_VERTICAL_GRID        :: Initialize vertical grid arrays.
  C    | | |
- C    | | |-INI_CARTESIAN_GRID       :: Cartesian horiz. grid initialisation
+ C    | | |-INI_CARTESIAN_GRID       :: Cartesian horiz. grid initialization
  C    | | |                          :: (calculate grid from kernel parameters).
  C    | | |
  C    | | |-INI_SPHERICAL_POLAR_GRID :: Spherical polar horiz. grid
- C    | | |                          :: initialisation (calculate grid from
+ C    | | |                          :: initialization (calculate grid from
  C    | | |                          :: kernel parameters).
  C    | | |
  C    | | |-INI_CURVILINEAR_GRID     :: General orthogonal, structured horiz.
- C    | |                            :: grid initialisations. ( input from raw
+ C    | |                            :: grid initializations. ( input from raw
  C    | |                            :: grid files, LONC.bin, DXF.bin etc... )
  C    | |
  C    | |-INI_DEPTHS    :: Read (from "bathyFile") or set bathymetry/orgography.
-Line 1497 
 C    | |
+Line 1493 
 C    | |
  C    | |-INI_LINEAR_PHSURF :: Set ref. surface Bo_surf
  C    | |
  C    | |-INI_CORI          :: Set coriolis term. zero, f-plane, beta-plane,
- C    | |                   :: sphere optins are coded.
+ C    | |                   :: sphere options are coded.
  C    | |
  C    | |-PACAKGES_BOOT      :: Start up the optional package environment.
  C    | |                    :: Runtime selection of active packages.
-Line 1518 
 C    | |
+Line 1514 
 C    | |
  C    | |-PACKAGES_CHECK
  C    | | |
  C    | | |-KPP_CHECK           :: KPP Package. pkg/kpp
- C    | | |-OBCS_CHECK          :: Open bndy Pacakge. pkg/obcs
+ C    | | |-OBCS_CHECK          :: Open bndy Package. pkg/obcs
  C    | | |-GMREDI_CHECK        :: GM Package. pkg/gmredi
  C    | |
  C    | |-PACKAGES_INIT_FIXED
-Line 1538 
 C    |
+Line 1534 
 C    |
  C    |-CTRL_UNPACK :: Control vector support package. see pkg/ctrl
  C    |
  C    |-ADTHE_MAIN_LOOP :: Derivative evaluating form of main time stepping loop
- C    !                 :: Auotmatically gerenrated by TAMC/TAF.
+ C    !                 :: Auotmatically generated by TAMC/TAF.
  C    |
  C    |-CTRL_PACK   :: Control vector support package. see pkg/ctrl
  C    |
-Line 1552 
 C    | | |
+Line 1548 
 C    | | |
  C    | | |-INI_LINEAR_PHISURF :: Set ref. surface Bo_surf
  C    | | |
  C    | | |-INI_CORI     :: Set coriolis term. zero, f-plane, beta-plane,
- C    | | |              :: sphere optins are coded.
+ C    | | |              :: sphere options are coded.
  C    | | |
  C    | | |-INI_CG2D     :: 2d con. grad solver initialisation.
  C    | | |-INI_CG3D     :: 3d con. grad solver initialisation.
-Line 1560 
 C    | | |-INI_MIXING   :: Initialise di
+Line 1556 
 C    | | |-INI_MIXING   :: Initialise di
  C    | | |-INI_DYNVARS  :: Initialise to zero all DYNVARS.h arrays (dynamical
  C    | | |              :: fields).
  C    | | |
- C    | | |-INI_FIELDS   :: Control initialising model fields to non-zero
+ C    | | |-INI_FIELDS   :: Control initializing model fields to non-zero
  C    | | | |-INI_VEL    :: Initialize 3D flow field.
  C    | | | |-INI_THETA  :: Set model initial temperature field.
  C    | | | |-INI_SALT   :: Set model initial salinity field.
-Line 1638 
 C/\  | | |-CALC_EXACT_ETA :: Change SSH
+Line 1634 
 C/\  | | |-CALC_EXACT_ETA :: Change SSH
  C/\  | | |-CALC_SURF_DR   :: Calculate the new surface level thickness.
  C/\  | | |-EXF_GETFORCING :: External forcing package. ( pkg/exf )
  C/\  | | |-EXTERNAL_FIELDS_LOAD :: Control loading time dep. external data.
- C/\  | | | |                    :: Simple interpolcation between end-points
+ C/\  | | | |                    :: Simple interpolation between end-points
  C/\  | | | |                    :: for forcing datasets.
  C/\  | | | |
  C/\  | | | |-EXCH :: Sync forcing. in overlap regions.

 Legend:



Removed from v.1.1
 


changed lines


 
Added in v.1.4
 Legend:



Removed from v.1.1
 


changed lines


 
Added in v.1.4
-Removed from v.1.1
+Added in v.1.4

	ViewVC Help
Powered by ViewVC 1.1.22