--- manual/s_software/text/sarch.tex 2006/04/04 15:54:55 1.21 +++ manual/s_software/text/sarch.tex 2006/04/05 01:12:02 1.24 @@ -1,20 +1,20 @@ -% $Header: /home/ubuntu/mnt/e9_copy/manual/s_software/text/sarch.tex,v 1.21 2006/04/04 15:54:55 edhill Exp $ +% $Header: /home/ubuntu/mnt/e9_copy/manual/s_software/text/sarch.tex,v 1.24 2006/04/05 01:12:02 jmc Exp $ -This chapter focuses on describing the {\bf WRAPPER} environment within which -both the core numerics and the pluggable packages operate. The description -presented here is intended to be a detailed exposition and contains significant -background material, as well as advanced details on working with the WRAPPER. -The tutorial sections of this manual (see sections -\ref{sect:tutorials} and \ref{sect:tutorialIII}) -contain more succinct, step-by-step instructions on running basic numerical -experiments, of varous types, both sequentially and in parallel. For many -projects simply starting from an example code and adapting it to suit a -particular situation -will be all that is required. -The first part of this chapter discusses the MITgcm architecture at an -abstract level. In the second part of the chapter we described practical -details of the MITgcm implementation and of current tools and operating system -features that are employed. +This chapter focuses on describing the {\bf WRAPPER} environment +within which both the core numerics and the pluggable packages +operate. The description presented here is intended to be a detailed +exposition and contains significant background material, as well as +advanced details on working with the WRAPPER. The tutorial sections +of this manual (see sections \ref{sect:tutorials} and +\ref{sect:tutorialIII}) contain more succinct, step-by-step +instructions on running basic numerical experiments, of varous types, +both sequentially and in parallel. For many projects simply starting +from an example code and adapting it to suit a particular situation +will be all that is required. The first part of this chapter +discusses the MITgcm architecture at an abstract level. In the second +part of the chapter we described practical details of the MITgcm +implementation and of current tools and operating system features that +are employed. \section{Overall architectural goals} \begin{rawhtml} @@ -25,19 +25,19 @@ three-fold \begin{itemize} -\item We wish to be able to study a very broad range -of interesting and challenging rotating fluids problems. -\item We wish the model code to be readily targeted to -a wide range of platforms -\item On any given platform we would like to be -able to achieve performance comparable to an implementation -developed and specialized specifically for that platform. +\item We wish to be able to study a very broad range of interesting + and challenging rotating fluids problems. +\item We wish the model code to be readily targeted to a wide range of + platforms +\item On any given platform we would like to be able to achieve + performance comparable to an implementation developed and + specialized specifically for that platform. \end{itemize} -These points are summarized in figure \ref{fig:mitgcm_architecture_goals} -which conveys the goals of the MITgcm design. The goals lead to -a software architecture which at the high-level can be viewed as consisting -of +These points are summarized in figure +\ref{fig:mitgcm_architecture_goals} which conveys the goals of the +MITgcm design. The goals lead to a software architecture which at the +high-level can be viewed as consisting of \begin{enumerate} \item A core set of numerical and support code. This is discussed in @@ -71,14 +71,13 @@ \begin{center} \resizebox{!}{2.5in}{\includegraphics{part4/mitgcm_goals.eps}} \end{center} -\caption{ -The MITgcm architecture is designed to allow simulation of a wide -range of physical problems on a wide range of hardware. The computational -resource requirements of the applications targeted range from around -$10^7$ bytes ( $\approx 10$ megabytes ) of memory to $10^{11}$ bytes -( $\approx 100$ gigabytes). Arithmetic operation counts for the applications of -interest range from $10^{9}$ floating point operations to more than $10^{17}$ -floating point operations.} +\caption{ The MITgcm architecture is designed to allow simulation of a + wide range of physical problems on a wide range of hardware. The + computational resource requirements of the applications targeted + range from around $10^7$ bytes ($\approx 10$ megabytes) of memory to + $10^{11}$ bytes ($\approx 100$ gigabytes). Arithmetic operation + counts for the applications of interest range from $10^{9}$ floating + point operations to more than $10^{17}$ floating point operations.} \label{fig:mitgcm_architecture_goals} \end{figure} @@ -87,19 +86,20 @@ \end{rawhtml} -A significant element of the software architecture utilized in -MITgcm is a software superstructure and substructure collectively -called the WRAPPER (Wrappable Application Parallel Programming -Environment Resource). All numerical and support code in MITgcm is written -to ``fit'' within the WRAPPER infrastructure. Writing code to ``fit'' within -the WRAPPER means that coding has to follow certain, relatively -straightforward, rules and conventions (these are discussed further in +A significant element of the software architecture utilized in MITgcm +is a software superstructure and substructure collectively called the +WRAPPER (Wrappable Application Parallel Programming Environment +Resource). All numerical and support code in MITgcm is written to +``fit'' within the WRAPPER infrastructure. Writing code to ``fit'' +within the WRAPPER means that coding has to follow certain, relatively +straightforward, rules and conventions (these are discussed further in section \ref{sect:specifying_a_decomposition}). -The approach taken by the WRAPPER is illustrated in figure -\ref{fig:fit_in_wrapper} which shows how the WRAPPER serves to insulate code -that fits within it from architectural differences between hardware platforms -and operating systems. This allows numerical code to be easily retargetted. +The approach taken by the WRAPPER is illustrated in figure +\ref{fig:fit_in_wrapper} which shows how the WRAPPER serves to +insulate code that fits within it from architectural differences +between hardware platforms and operating systems. This allows +numerical code to be easily retargetted. \begin{figure} @@ -165,6 +165,7 @@ scientific computing community. \subsection{Machine model parallelism} +\label{sect:domain_decomposition} \begin{rawhtml} \end{rawhtml} @@ -416,16 +417,17 @@ \subsection{Distributed memory communication} \label{sect:distributed_memory_communication} Many parallel systems are not constructed in a way where it is -possible or practical for an application to use shared memory -for communication. For example cluster systems consist of individual computers -connected by a fast network. On such systems there is no notion of shared memory -at the system level. For this sort of system the WRAPPER provides support -for communication based on a bespoke communication library -(see figure \ref{fig:comm_msg}). The default communication library used is MPI -\ref{mpi}. However, it is relatively straightforward to implement bindings to -optimized platform specific communication libraries. For example the work -described in \ref{hoe-hill:99} substituted standard MPI communication for a -highly optimized library. +possible or practical for an application to use shared memory for +communication. For example cluster systems consist of individual +computers connected by a fast network. On such systems there is no +notion of shared memory at the system level. For this sort of system +the WRAPPER provides support for communication based on a bespoke +communication library (see figure \ref{fig:comm_msg}). The default +communication library used is MPI \cite{MPI-std-20}. However, it is +relatively straightforward to implement bindings to optimized platform +specific communication libraries. For example the work described in +\ref{hoe-hill:99} substituted standard MPI communication for a highly +optimized library. \subsection{Communication primitives} \label{sect:communication_primitives} @@ -1480,7 +1482,6 @@ {\footnotesize \begin{verbatim} C -C C Invocation from WRAPPER level... C : C : @@ -1544,22 +1545,25 @@ C | | |-OPTIM_READPARMS :: Optimisation support package. see pkg/ctrl C | | |-GRDCHK_READPARMS :: Gradient check package. see pkg/grdchk C | | |-ECCO_READPARMS :: ECCO Support Package. see pkg/ecco +C | | |-PTRACERS_READPARMS :: multiple tracer package, see pkg/ptracers +C | | |-GCHEM_READPARMS :: tracer interface package, see pkg/gchem C | | C | |-PACKAGES_CHECK C | | | C | | |-KPP_CHECK :: KPP Package. pkg/kpp -C | | |-OBCS_CHECK :: Open bndy Package. pkg/obcs +C | | |-OBCS_CHECK :: Open bndy Pacakge. pkg/obcs C | | |-GMREDI_CHECK :: GM Package. pkg/gmredi C | | C | |-PACKAGES_INIT_FIXED C | | |-OBCS_INIT_FIXED :: Open bndy Package. see pkg/obcs C | | |-FLT_INIT :: Floats Package. see pkg/flt +C | | |-GCHEM_INIT_FIXED :: tracer interface pachage, see pkg/gchem C | | C | |-ZONAL_FILT_INIT :: FFT filter Package. see pkg/zonal_filt C | | -C | |-INI_CG2D :: 2d con. grad solver initialisation. +C | |-INI_CG2D :: 2d con. grad solver initialization. C | | -C | |-INI_CG3D :: 3d con. grad solver initialisation. +C | |-INI_CG3D :: 3d con. grad solver initialization. C | | C | |-CONFIG_SUMMARY :: Provide synopsis of kernel setup. C | :: Includes annotated table of kernel @@ -1584,10 +1588,10 @@ C | | |-INI_CORI :: Set coriolis term. zero, f-plane, beta-plane, C | | | :: sphere options are coded. C | | | -C | | |-INI_CG2D :: 2d con. grad solver initialisation. -C | | |-INI_CG3D :: 3d con. grad solver initialisation. -C | | |-INI_MIXING :: Initialise diapycnal diffusivity. -C | | |-INI_DYNVARS :: Initialise to zero all DYNVARS.h arrays (dynamical +C | | |-INI_CG2D :: 2d con. grad solver initialization. +C | | |-INI_CG3D :: 3d con. grad solver initialization. +C | | |-INI_MIXING :: Initialize diapycnal diffusivity. +C | | |-INI_DYNVARS :: Initialize to zero all DYNVARS.h arrays (dynamical C | | | :: fields). C | | | C | | |-INI_FIELDS :: Control initializing model fields to non-zero @@ -1595,8 +1599,8 @@ C | | | |-INI_THETA :: Set model initial temperature field. C | | | |-INI_SALT :: Set model initial salinity field. C | | | |-INI_PSURF :: Set model initial free-surface height/pressure. -C | | | -C | | |-INI_TR1 :: Set initial tracer 1 distribution. +C | | | |-INI_PRESSURE :: Compute model initial hydrostatic pressure +C | | | |-READ_CHECKPOINT :: Read the checkpoint C | | | C | | |-THE_CORRECTION_STEP :: Step forward to next time step. C | | | | :: Here applied to move restart conditions @@ -1623,13 +1627,15 @@ C | | | |-CONVECT :: Mix static instability. C | | | |-TIMEAVE_CUMULATE :: Update convection statistics. C | | | -C | | |-PACKAGES_INIT_VARIABLES :: Does initialisation of time evolving +C | | |-PACKAGES_INIT_VARIABLES :: Does initialization of time evolving C | | | | :: package data. C | | | | C | | | |-GMREDI_INIT :: GM package. ( see pkg/gmredi ) C | | | |-KPP_INIT :: KPP package. ( see pkg/kpp ) C | | | |-KPP_OPEN_DIAGS C | | | |-OBCS_INIT_VARIABLES :: Open bndy. package. ( see pkg/obcs ) +C | | | |-PTRACERS_INIT :: multi. tracer package,(see pkg/ptracers) +C | | | |-GCHEM_INIT :: tracer interface pkg (see pkh/gchem) C | | | |-AIM_INIT :: Interm. atmos package. ( see pkg/aim ) C | | | |-CTRL_MAP_INI :: Control vector package.( see pkg/ctrl ) C | | | |-COST_INIT :: Cost function package. ( see pkg/cost ) @@ -1672,18 +1678,22 @@ C/\ | | | | :: for forcing datasets. C/\ | | | | C/\ | | | |-EXCH :: Sync forcing. in overlap regions. +C/\ | | |-SEAICE_MODEL :: Compute sea-ice terms. ( pkg/seaice ) +C/\ | | |-FREEZE :: Limit surface temperature. +C/\ | | |-GCHEM_FIELD_LOAD :: load tracer forcing fields (pkg/gchem) C/\ | | | C/\ | | |-THERMODYNAMICS :: theta, salt + tracer equations driver. C/\ | | | | C/\ | | | |-INTEGRATE_FOR_W :: Integrate for vertical velocity. C/\ | | | |-OBCS_APPLY_W :: Open bndy. package ( see pkg/obcs ). -C/\ | | | |-FIND_RHO :: Calculates [rho(S,T,z)-Rhonil] of a slice +C/\ | | | |-FIND_RHO :: Calculates [rho(S,T,z)-RhoConst] of a slice C/\ | | | |-GRAD_SIGMA :: Calculate isoneutral gradients C/\ | | | |-CALC_IVDC :: Set Implicit Vertical Diffusivity for Convection C/\ | | | | C/\ | | | |-OBCS_CALC :: Open bndy. package ( see pkg/obcs ). C/\ | | | |-EXTERNAL_FORCING_SURF:: Accumulates appropriately dimensioned -C/\ | | | | :: forcing terms. +C/\ | | | | | :: forcing terms. +C/\ | | | | |-PTRACERS_FORCING_SURF :: Tracer package ( see pkg/ptracers ). C/\ | | | | C/\ | | | |-GMREDI_CALC_TENSOR :: GM package ( see pkg/gmredi ). C/\ | | | |-GMREDI_CALC_TENSOR_DUMMY :: GM package ( see pkg/gmredi ). @@ -1701,7 +1711,9 @@ C/\ | | | |-CALC_GT :: Calculate the temperature tendency terms C/\ | | | | | C/\ | | | | |-GAD_CALC_RHS :: Generalised advection package -C/\ | | | | | :: ( see pkg/gad ) +C/\ | | | | | | :: ( see pkg/gad ) +C/\ | | | | | |-KPP_TRANSPORT_T :: KPP non-local transport ( see pkg/kpp ). +C/\ | | | | | C/\ | | | | |-EXTERNAL_FORCING_T :: Problem specific forcing for temperature. C/\ | | | | |-ADAMS_BASHFORTH2 :: Extrapolate tendencies forward in time. C/\ | | | | |-FREESURF_RESCALE_G :: Re-scale Gt for free-surface height. @@ -1711,24 +1723,31 @@ C/\ | | | |-CALC_GS :: Calculate the salinity tendency terms C/\ | | | | | C/\ | | | | |-GAD_CALC_RHS :: Generalised advection package -C/\ | | | | | :: ( see pkg/gad ) +C/\ | | | | | | :: ( see pkg/gad ) +C/\ | | | | | |-KPP_TRANSPORT_S :: KPP non-local transport ( see pkg/kpp ). +C/\ | | | | | C/\ | | | | |-EXTERNAL_FORCING_S :: Problem specific forcing for salt. C/\ | | | | |-ADAMS_BASHFORTH2 :: Extrapolate tendencies forward in time. C/\ | | | | |-FREESURF_RESCALE_G :: Re-scale Gs for free-surface height. C/\ | | | | C/\ | | | |-TIMESTEP_TRACER :: Step tracer field forward in time C/\ | | | | -C/\ | | | |-CALC_GTR1 :: Calculate other tracer(s) tendency terms +C/\ | | | |-TIMESTEP_TRACER :: Step tracer field forward in time +C/\ | | | | +C/\ | | | |-PTRACERS_INTEGRATE :: Integrate other tracer(s) (see pkg/ptracers). C/\ | | | | | C/\ | | | | |-GAD_CALC_RHS :: Generalised advection package -C/\ | | | | | :: ( see pkg/gad ) -C/\ | | | | |-EXTERNAL_FORCING_TR:: Problem specific forcing for tracer. +C/\ | | | | | | :: ( see pkg/gad ) +C/\ | | | | | |-KPP_TRANSPORT_PTR:: KPP non-local transport ( see pkg/kpp ). +C/\ | | | | | +C/\ | | | | |-PTRACERS_FORCING :: Problem specific forcing for tracer. +C/\ | | | | |-GCHEM_FORCING_INT :: tracer forcing for gchem pkg (if all +C/\ | | | | | tendancy terms calcualted together) C/\ | | | | |-ADAMS_BASHFORTH2 :: Extrapolate tendencies forward in time. C/\ | | | | |-FREESURF_RESCALE_G :: Re-scale Gs for free-surface height. +C/\ | | | | |-TIMESTEP_TRACER :: Step tracer field forward in time C/\ | | | | -C/\ | | | |-TIMESTEP_TRACER :: Step tracer field forward in time C/\ | | | |-OBCS_APPLY_TS :: Open bndy. package (see pkg/obcs ). -C/\ | | | |-FREEZE :: Limit range of temperature. C/\ | | | | C/\ | | | |-IMPLDIFF :: Solve vertical implicit diffusion equation. C/\ | | | |-OBCS_APPLY_TS :: Open bndy. package (see pkg/obcs ). @@ -1787,6 +1806,10 @@ C/\ | | |-DO_FIELDS_BLOCKING_EXCHANGES :: Sync up overlap regions. C/\ | | | |-EXCH C/\ | | | +C/\ | | |-GCHEM_FORCING_SEP :: tracer forcing for gchem pkg (if +C/\ | | | tracer dependent tendencies calculated +C/\ | | | separatly) +C/\ | | | C/\ | | |-FLT_MAIN :: Float package ( pkg/flt ). C/\ | | | C/\ | | |-MONITOR :: Monitor package ( pkg/monitor ). @@ -1797,6 +1820,10 @@ C/\ | | | |-AIM_WRITE_DIAGS :: Intermed. atmos diags. see pkg/aim C/\ | | | |-GMREDI_DIAGS :: GM diags. see pkg/gmredi C/\ | | | |-KPP_DO_DIAGS :: KPP diags. see pkg/kpp +C/\ | | | |-SBO_CALC :: SBO diags. see pkg/sbo +C/\ | | | |-SBO_DIAGS :: SBO diags. see pkg/sbo +C/\ | | | |-SEAICE_DO_DIAGS :: SEAICE diags. see pkg/seaice +C/\ | | | |-GCHEM_DIAGS :: gchem diags. see pkg/gchem C/\ | | | C/\ | | |-WRITE_CHECKPOINT :: Do I/O for restart files. C/\ | | @@ -1814,7 +1841,7 @@ C | C |-COMM_STATS :: Summarise inter-proc and inter-thread communication C :: events. -C +C \end{verbatim} }