--- manual/s_software/text/sarch.tex	2006/04/04 15:54:55	1.21
+++ manual/s_software/text/sarch.tex	2010/08/27 13:17:59	1.25
@@ -1,20 +1,20 @@
-% $Header: /home/ubuntu/mnt/e9_copy/manual/s_software/text/sarch.tex,v 1.21 2006/04/04 15:54:55 edhill Exp $
+% $Header: /home/ubuntu/mnt/e9_copy/manual/s_software/text/sarch.tex,v 1.25 2010/08/27 13:17:59 jmc Exp $
 
-This chapter focuses on describing the {\bf WRAPPER} environment within which
-both the core numerics and the pluggable packages operate. The description
-presented here is intended to be a detailed exposition and contains significant
-background material, as well as advanced details on working with the WRAPPER. 
-The tutorial sections of this manual (see sections
-\ref{sect:tutorials}  and \ref{sect:tutorialIII}) 
-contain more succinct, step-by-step instructions on running basic numerical
-experiments, of varous types, both sequentially and in parallel. For many 
-projects simply starting from an example code and adapting it to suit a 
-particular situation 
-will be all that is required.
-The first part of this chapter discusses the MITgcm architecture at an 
-abstract level. In the second part of the chapter we described practical 
-details of the MITgcm implementation and of current tools and operating system 
-features that are employed.
+This chapter focuses on describing the {\bf WRAPPER} environment
+within which both the core numerics and the pluggable packages
+operate. The description presented here is intended to be a detailed
+exposition and contains significant background material, as well as
+advanced details on working with the WRAPPER.  The tutorial sections
+of this manual (see sections \ref{sect:tutorials} and
+\ref{sect:tutorialIII}) contain more succinct, step-by-step
+instructions on running basic numerical experiments, of varous types,
+both sequentially and in parallel. For many projects simply starting
+from an example code and adapting it to suit a particular situation
+will be all that is required.  The first part of this chapter
+discusses the MITgcm architecture at an abstract level. In the second
+part of the chapter we described practical details of the MITgcm
+implementation and of current tools and operating system features that
+are employed.
 
 \section{Overall architectural goals}
 \begin{rawhtml}
@@ -25,19 +25,19 @@
 three-fold
  
 \begin{itemize}
-\item We wish to be able to study a very broad range
-of interesting and challenging rotating fluids problems.
-\item We wish the model code to be readily targeted to
-a wide range of platforms
-\item On any given platform we would like to be
-able to achieve performance comparable to an implementation 
-developed and specialized specifically for that platform.
+\item We wish to be able to study a very broad range of interesting
+  and challenging rotating fluids problems.
+\item We wish the model code to be readily targeted to a wide range of
+  platforms
+\item On any given platform we would like to be able to achieve
+  performance comparable to an implementation developed and
+  specialized specifically for that platform.
 \end{itemize}
 
-These points are summarized in figure \ref{fig:mitgcm_architecture_goals} 
-which conveys the goals of the MITgcm design. The goals lead to
-a software architecture which at the high-level can be viewed as consisting 
-of
+These points are summarized in figure
+\ref{fig:mitgcm_architecture_goals} which conveys the goals of the
+MITgcm design. The goals lead to a software architecture which at the
+high-level can be viewed as consisting of
 
 \begin{enumerate}
 \item A core set of numerical and support code. This is discussed in
@@ -69,16 +69,15 @@
 
 \begin{figure}
 \begin{center}
-\resizebox{!}{2.5in}{\includegraphics{part4/mitgcm_goals.eps}}
+\resizebox{!}{2.5in}{\includegraphics{s_software/figs/mitgcm_goals.eps}}
 \end{center}
-\caption{
-The MITgcm architecture is designed to allow simulation of a wide 
-range of physical problems on a wide range of hardware. The computational 
-resource requirements of the applications targeted range from around 
-$10^7$ bytes ( $\approx 10$ megabytes ) of memory to $10^{11}$ bytes
-( $\approx 100$ gigabytes). Arithmetic operation counts for the applications of 
-interest range from $10^{9}$ floating point operations to more than $10^{17}$ 
-floating point operations.}
+\caption{ The MITgcm architecture is designed to allow simulation of a
+  wide range of physical problems on a wide range of hardware. The
+  computational resource requirements of the applications targeted
+  range from around $10^7$ bytes ($\approx 10$ megabytes) of memory to
+  $10^{11}$ bytes ($\approx 100$ gigabytes). Arithmetic operation
+  counts for the applications of interest range from $10^{9}$ floating
+  point operations to more than $10^{17}$ floating point operations.}
 \label{fig:mitgcm_architecture_goals}
 \end{figure}
 
@@ -87,24 +86,25 @@
 <!-- CMIREDIR:wrapper: -->
 \end{rawhtml}
 
-A significant element of the software architecture utilized in
-MITgcm is a software superstructure and substructure collectively
-called the WRAPPER (Wrappable Application Parallel Programming 
-Environment Resource). All numerical and support code in MITgcm is written 
-to ``fit'' within the WRAPPER infrastructure. Writing code to ``fit'' within 
-the WRAPPER means that coding has to follow certain, relatively
-straightforward, rules and conventions (these are discussed further in 
+A significant element of the software architecture utilized in MITgcm
+is a software superstructure and substructure collectively called the
+WRAPPER (Wrappable Application Parallel Programming Environment
+Resource). All numerical and support code in MITgcm is written to
+``fit'' within the WRAPPER infrastructure. Writing code to ``fit''
+within the WRAPPER means that coding has to follow certain, relatively
+straightforward, rules and conventions (these are discussed further in
 section \ref{sect:specifying_a_decomposition}).
 
-The approach taken by the WRAPPER is illustrated in figure 
-\ref{fig:fit_in_wrapper} which shows how the WRAPPER serves to insulate code 
-that fits within it from architectural differences between hardware platforms 
-and operating systems. This allows numerical code to be easily retargetted. 
+The approach taken by the WRAPPER is illustrated in figure
+\ref{fig:fit_in_wrapper} which shows how the WRAPPER serves to
+insulate code that fits within it from architectural differences
+between hardware platforms and operating systems. This allows
+numerical code to be easily retargetted.
 
 
 \begin{figure}
 \begin{center}
-\resizebox{!}{4.5in}{\includegraphics{part4/fit_in_wrapper.eps}}
+\resizebox{!}{4.5in}{\includegraphics{s_software/figs/fit_in_wrapper.eps}}
 \end{center}
 \caption{
 Numerical code is written to fit within a software support
@@ -165,6 +165,7 @@
 scientific computing community.
 
 \subsection{Machine model parallelism}
+\label{sect:domain_decomposition}
 \begin{rawhtml}
 <!-- CMIREDIR:domain_decomp: -->
 \end{rawhtml}
@@ -210,7 +211,7 @@
 \begin{figure}
 \begin{center}
  \resizebox{5in}{!}{
-  \includegraphics{part4/domain_decomp.eps}
+  \includegraphics{s_software/figs/domain_decomp.eps}
  }
 \end{center}
 \caption{ The WRAPPER provides support for one and two dimensional
@@ -245,7 +246,7 @@
 \begin{figure}
 \begin{center}
  \resizebox{5in}{!}{
-  \includegraphics{part4/tiled-world.eps}
+  \includegraphics{s_software/figs/tiled-world.eps}
  }
 \end{center}
 \caption{ A global grid subdivided into tiles.
@@ -416,16 +417,17 @@
 \subsection{Distributed memory communication}
 \label{sect:distributed_memory_communication}
 Many parallel systems are not constructed in a way where it is
-possible or practical for an application to use shared memory
-for communication. For example cluster systems consist of individual computers
-connected by a fast network. On such systems there is no notion of shared memory
-at the system level. For this sort of system the WRAPPER provides support
-for communication based on a bespoke communication library 
-(see figure \ref{fig:comm_msg}).  The default communication library used is MPI 
-\ref{mpi}. However, it is relatively straightforward to implement bindings to 
-optimized platform specific communication libraries. For example the work 
-described in \ref{hoe-hill:99} substituted standard MPI communication for a 
-highly optimized library.
+possible or practical for an application to use shared memory for
+communication. For example cluster systems consist of individual
+computers connected by a fast network. On such systems there is no
+notion of shared memory at the system level. For this sort of system
+the WRAPPER provides support for communication based on a bespoke
+communication library (see figure \ref{fig:comm_msg}).  The default
+communication library used is MPI \cite{MPI-std-20}. However, it is
+relatively straightforward to implement bindings to optimized platform
+specific communication libraries. For example the work described in
+\ref{hoe-hill:99} substituted standard MPI communication for a highly
+optimized library.
 
 \subsection{Communication primitives}
 \label{sect:communication_primitives}
@@ -433,7 +435,7 @@
 \begin{figure}
 \begin{center}
  \resizebox{5in}{!}{
-  \includegraphics{part4/comm-primm.eps}
+  \includegraphics{s_software/figs/comm-primm.eps}
  }
 \end{center}
 \caption{Three performance critical parallel primitives are provided
@@ -516,7 +518,7 @@
 \begin{figure}
 \begin{center}
  \resizebox{5in}{!}{
-  \includegraphics{part4/tiling_detail.eps}
+  \includegraphics{s_software/figs/tiling_detail.eps}
  }
 \end{center}
 \caption{The tiling strategy that the WRAPPER supports allows tiles
@@ -622,7 +624,7 @@
 \begin{figure}
 \begin{center}
  \resizebox{5in}{!}{
-  \includegraphics{part4/size_h.eps}
+  \includegraphics{s_software/figs/size_h.eps}
  }
 \end{center}
 \caption{ The three level domain decomposition hierarchy employed by the
@@ -1480,7 +1482,6 @@
 {\footnotesize
 \begin{verbatim}
 C
-C
 C Invocation from WRAPPER level...
 C  :
 C  :
@@ -1544,22 +1545,25 @@
 C    | | |-OPTIM_READPARMS     :: Optimisation support package. see pkg/ctrl 
 C    | | |-GRDCHK_READPARMS    :: Gradient check package. see pkg/grdchk
 C    | | |-ECCO_READPARMS      :: ECCO Support Package. see pkg/ecco
+C    | | |-PTRACERS_READPARMS  :: multiple tracer package, see pkg/ptracers
+C    | | |-GCHEM_READPARMS     :: tracer interface package, see pkg/gchem
 C    | |
 C    | |-PACKAGES_CHECK
 C    | | |
 C    | | |-KPP_CHECK           :: KPP Package. pkg/kpp
-C    | | |-OBCS_CHECK          :: Open bndy Package. pkg/obcs
+C    | | |-OBCS_CHECK          :: Open bndy Pacakge. pkg/obcs
 C    | | |-GMREDI_CHECK        :: GM Package. pkg/gmredi
 C    | |
 C    | |-PACKAGES_INIT_FIXED
 C    | | |-OBCS_INIT_FIXED     :: Open bndy Package. see pkg/obcs
 C    | | |-FLT_INIT            :: Floats Package. see pkg/flt
+C    | | |-GCHEM_INIT_FIXED    :: tracer interface pachage, see pkg/gchem
 C    | |
 C    | |-ZONAL_FILT_INIT       :: FFT filter Package. see pkg/zonal_filt
 C    | |
-C    | |-INI_CG2D              :: 2d con. grad solver initialisation.
+C    | |-INI_CG2D              :: 2d con. grad solver initialization.
 C    | |
-C    | |-INI_CG3D              :: 3d con. grad solver initialisation.
+C    | |-INI_CG3D              :: 3d con. grad solver initialization.
 C    | |
 C    | |-CONFIG_SUMMARY        :: Provide synopsis of kernel setup.
 C    |                         :: Includes annotated table of kernel 
@@ -1584,10 +1588,10 @@
 C    | | |-INI_CORI     :: Set coriolis term. zero, f-plane, beta-plane,
 C    | | |              :: sphere options are coded.
 C    | | |
-C    | | |-INI_CG2D     :: 2d con. grad solver initialisation.
-C    | | |-INI_CG3D     :: 3d con. grad solver initialisation.
-C    | | |-INI_MIXING   :: Initialise diapycnal diffusivity.
-C    | | |-INI_DYNVARS  :: Initialise to zero all DYNVARS.h arrays (dynamical
+C    | | |-INI_CG2D     :: 2d con. grad solver initialization.
+C    | | |-INI_CG3D     :: 3d con. grad solver initialization.
+C    | | |-INI_MIXING   :: Initialize diapycnal diffusivity.
+C    | | |-INI_DYNVARS  :: Initialize to zero all DYNVARS.h arrays (dynamical
 C    | | |              :: fields).
 C    | | |
 C    | | |-INI_FIELDS   :: Control initializing model fields to non-zero
@@ -1595,8 +1599,8 @@
 C    | | | |-INI_THETA  :: Set model initial temperature field.
 C    | | | |-INI_SALT   :: Set model initial salinity field.
 C    | | | |-INI_PSURF  :: Set model initial free-surface height/pressure.
-C    | | |
-C    | | |-INI_TR1      :: Set initial tracer 1 distribution.
+C    | | | |-INI_PRESSURE :: Compute model initial hydrostatic pressure
+C    | | | |-READ_CHECKPOINT :: Read the checkpoint
 C    | | |
 C    | | |-THE_CORRECTION_STEP :: Step forward to next time step.
 C    | | | |                   :: Here applied to move restart conditions
@@ -1623,13 +1627,15 @@
 C    | | | |-CONVECT   :: Mix static instability.
 C    | | | |-TIMEAVE_CUMULATE :: Update convection statistics.
 C    | | |
-C    | | |-PACKAGES_INIT_VARIABLES :: Does initialisation of time evolving 
+C    | | |-PACKAGES_INIT_VARIABLES :: Does initialization of time evolving 
 C    | | | |                       :: package data.
 C    | | | |
 C    | | | |-GMREDI_INIT          :: GM package. ( see pkg/gmredi )
 C    | | | |-KPP_INIT             :: KPP package. ( see pkg/kpp )
 C    | | | |-KPP_OPEN_DIAGS    
 C    | | | |-OBCS_INIT_VARIABLES  :: Open bndy. package. ( see pkg/obcs )
+C    | | | |-PTRACERS_INIT        :: multi. tracer package,(see pkg/ptracers)
+C    | | | |-GCHEM_INIT           :: tracer interface pkg (see pkh/gchem)
 C    | | | |-AIM_INIT             :: Interm. atmos package. ( see pkg/aim )
 C    | | | |-CTRL_MAP_INI         :: Control vector package.( see pkg/ctrl )
 C    | | | |-COST_INIT            :: Cost function package. ( see pkg/cost )
@@ -1672,18 +1678,22 @@
 C/\  | | | |                    :: for forcing datasets.
 C/\  | | | |                  
 C/\  | | | |-EXCH :: Sync forcing. in overlap regions.
+C/\  | | |-SEAICE_MODEL   :: Compute sea-ice terms. ( pkg/seaice )
+C/\  | | |-FREEZE         :: Limit surface temperature.
+C/\  | | |-GCHEM_FIELD_LOAD :: load tracer forcing fields (pkg/gchem)
 C/\  | | |
 C/\  | | |-THERMODYNAMICS :: theta, salt + tracer equations driver.
 C/\  | | | |
 C/\  | | | |-INTEGRATE_FOR_W :: Integrate for vertical velocity.
 C/\  | | | |-OBCS_APPLY_W    :: Open bndy. package ( see pkg/obcs ).
-C/\  | | | |-FIND_RHO        :: Calculates [rho(S,T,z)-Rhonil] of a slice
+C/\  | | | |-FIND_RHO        :: Calculates [rho(S,T,z)-RhoConst] of a slice
 C/\  | | | |-GRAD_SIGMA      :: Calculate isoneutral gradients
 C/\  | | | |-CALC_IVDC       :: Set Implicit Vertical Diffusivity for Convection
 C/\  | | | |
 C/\  | | | |-OBCS_CALC            :: Open bndy. package ( see pkg/obcs ).
 C/\  | | | |-EXTERNAL_FORCING_SURF:: Accumulates appropriately dimensioned 
-C/\  | | | |                      :: forcing terms.
+C/\  | | | | |                    :: forcing terms.
+C/\  | | | | |-PTRACERS_FORCING_SURF :: Tracer package ( see pkg/ptracers ).
 C/\  | | | |
 C/\  | | | |-GMREDI_CALC_TENSOR   :: GM package ( see pkg/gmredi ).
 C/\  | | | |-GMREDI_CALC_TENSOR_DUMMY :: GM package ( see pkg/gmredi ). 
@@ -1701,7 +1711,9 @@
 C/\  | | | |-CALC_GT              :: Calculate the temperature tendency terms
 C/\  | | | | |
 C/\  | | | | |-GAD_CALC_RHS       :: Generalised advection package 
-C/\  | | | | |                    :: ( see pkg/gad )
+C/\  | | | | | |                  :: ( see pkg/gad )
+C/\  | | | | | |-KPP_TRANSPORT_T  :: KPP non-local transport ( see pkg/kpp ).
+C/\  | | | | |
 C/\  | | | | |-EXTERNAL_FORCING_T :: Problem specific forcing for temperature.
 C/\  | | | | |-ADAMS_BASHFORTH2   :: Extrapolate tendencies forward in time.
 C/\  | | | | |-FREESURF_RESCALE_G :: Re-scale Gt for free-surface height.
@@ -1711,24 +1723,31 @@
 C/\  | | | |-CALC_GS              :: Calculate the salinity tendency terms
 C/\  | | | | |
 C/\  | | | | |-GAD_CALC_RHS       :: Generalised advection package 
-C/\  | | | | |                    :: ( see pkg/gad )
+C/\  | | | | | |                  :: ( see pkg/gad )
+C/\  | | | | | |-KPP_TRANSPORT_S  :: KPP non-local transport ( see pkg/kpp ).
+C/\  | | | | |
 C/\  | | | | |-EXTERNAL_FORCING_S :: Problem specific forcing for salt.
 C/\  | | | | |-ADAMS_BASHFORTH2   :: Extrapolate tendencies forward in time.
 C/\  | | | | |-FREESURF_RESCALE_G :: Re-scale Gs for free-surface height.
 C/\  | | | |
 C/\  | | | |-TIMESTEP_TRACER      :: Step tracer field forward in time
 C/\  | | | |
-C/\  | | | |-CALC_GTR1            :: Calculate other tracer(s) tendency terms
+C/\  | | | |-TIMESTEP_TRACER      :: Step tracer field forward in time
+C/\  | | | |
+C/\  | | | |-PTRACERS_INTEGRATE   :: Integrate other tracer(s) (see pkg/ptracers).
 C/\  | | | | |
 C/\  | | | | |-GAD_CALC_RHS       :: Generalised advection package 
-C/\  | | | | |                    :: ( see pkg/gad )
-C/\  | | | | |-EXTERNAL_FORCING_TR:: Problem specific forcing for tracer.
+C/\  | | | | | |                  :: ( see pkg/gad )
+C/\  | | | | | |-KPP_TRANSPORT_PTR:: KPP non-local transport ( see pkg/kpp ).
+C/\  | | | | |
+C/\  | | | | |-PTRACERS_FORCING   :: Problem specific forcing for tracer.
+C/\  | | | | |-GCHEM_FORCING_INT  :: tracer forcing for gchem pkg (if all
+C/\  | | | | |                       tendancy terms calcualted together)
 C/\  | | | | |-ADAMS_BASHFORTH2   :: Extrapolate tendencies forward in time.
 C/\  | | | | |-FREESURF_RESCALE_G :: Re-scale Gs for free-surface height.
+C/\  | | | | |-TIMESTEP_TRACER    :: Step tracer field forward in time
 C/\  | | | |
-C/\  | | | |-TIMESTEP_TRACER      :: Step tracer field forward in time
 C/\  | | | |-OBCS_APPLY_TS        :: Open bndy. package (see pkg/obcs ).
-C/\  | | | |-FREEZE               :: Limit range of temperature.
 C/\  | | | |
 C/\  | | | |-IMPLDIFF             :: Solve vertical implicit diffusion equation.
 C/\  | | | |-OBCS_APPLY_TS        :: Open bndy. package (see pkg/obcs ). 
@@ -1787,6 +1806,10 @@
 C/\  | | |-DO_FIELDS_BLOCKING_EXCHANGES :: Sync up overlap regions.
 C/\  | | | |-EXCH                                                   
 C/\  | | |
+C/\  | | |-GCHEM_FORCING_SEP :: tracer forcing for gchem pkg (if
+C/\  | | |                      tracer dependent tendencies calculated
+C/\  | | |                      separatly)
+C/\  | | |
 C/\  | | |-FLT_MAIN         :: Float package ( pkg/flt ).
 C/\  | | |
 C/\  | | |-MONITOR          :: Monitor package ( pkg/monitor ).
@@ -1797,6 +1820,10 @@
 C/\  | | | |-AIM_WRITE_DIAGS     :: Intermed. atmos diags. see pkg/aim
 C/\  | | | |-GMREDI_DIAGS        :: GM diags. see pkg/gmredi
 C/\  | | | |-KPP_DO_DIAGS        :: KPP diags. see pkg/kpp
+C/\  | | | |-SBO_CALC            :: SBO diags. see pkg/sbo
+C/\  | | | |-SBO_DIAGS           :: SBO diags. see pkg/sbo
+C/\  | | | |-SEAICE_DO_DIAGS     :: SEAICE diags. see pkg/seaice
+C/\  | | | |-GCHEM_DIAGS         :: gchem diags. see pkg/gchem
 C/\  | | |
 C/\  | | |-WRITE_CHECKPOINT :: Do I/O for restart files.
 C/\  | |
@@ -1814,7 +1841,7 @@
 C    |
 C    |-COMM_STATS     :: Summarise inter-proc and inter-thread communication
 C                     :: events.
-C 
+C
 \end{verbatim}
 }
 
