--- manual/s_phys_pkgs/text/exch2.tex 2004/02/17 21:58:56 1.8 +++ manual/s_phys_pkgs/text/exch2.tex 2004/03/15 20:11:56 1.10 @@ -1,4 +1,4 @@ -% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.8 2004/02/17 21:58:56 edhill Exp $ +% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.10 2004/03/15 20:11:56 afe Exp $ % $Name: $ %% * Introduction @@ -10,31 +10,157 @@ %% o automatically inserted at \section{Reference} -\section{Extended Cubed Sphere Exchange} +\section{exch2: Extended Cubed Sphere \mbox{Topology}} \label{sec:exch2} \subsection{Introduction} The \texttt{exch2} package is an extension to the original cubed -sphere exchanges to allow more flexible domain decomposition and -parallelization. Cube faces (subdomains) may be divided into whatever -number of tiles that divide evenly into the grid point dimensions of -the subdomain. Furthermore, the individual tiles may be run on -separate processors in different combinations, and whether exchanges -between particular tiles occur between different processors is -determined at runtime. +sphere topological configuration that allows more flexible domain +decomposition and parallelization. Cube faces (also called +subdomains) may be divided into any number of tiles that divide evenly +into the grid point dimensions of the subdomain. Furthermore, the +individual tiles may be run on separate processors in different +combinations, and whether exchanges between particular tiles occur +between different processors is determined at runtime. This +flexibility provides for manual compile-time load balancing across a +relatively arbitrary number of processors. \\ The exchange parameters are declared in \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} and assigned in -\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}, both in -the \texttt{pkg/exch2} directory. The validity of the cube topology -depends on the \texttt{SIZE.h} file as detailed below. Both files are -generated by Matlab scripts and should not be edited. The default -files provided in the release set up a cube sphere arrangement of six -tiles, one per subdomain, each with 32x32 grid points, running on a -single processor. +\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The +validity of the cube topology depends on the \texttt{SIZE.h} file as +detailed below. Both files are generated by Matlab scripts in +\texttt{utils/exch2/matlab-topology-generator}; see Section +\ref{sec:topogen} for details on creating alternate topologies. The +default files provided in the release configure a cubed sphere +topology of six tiles, one per subdomain, each with 32$\times$32 grid +points, all running on a single processor. Pregenerated examples of +these files with alternate topologies are provided under +\texttt{utils/exch2/code-mods} along with the appropriate +\texttt{SIZE.h} file for single-processor execution. + +\subsection{Invoking exch2} + +To use exch2 with the cubed sphere, the following conditions must be +met: \\ + +$\bullet$ The exch2 package is included when \texttt{genmake2} is run. The + easiest way to do this is to add the line \texttt{exch2} to the + \texttt{profile.conf} file -- see Section \ref{sect:buildingCode} + for general details. \\ + +$\bullet$ An example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and + \texttt{w2\_e2setup.F} must reside in a directory containing code + linked when \texttt{genmake2} runs. The safest place to put these + is the directory indicated in the \texttt{-mods=DIR} command line + modifier (typically \texttt{../code}), or the build directory. The + default versions of these files reside in \texttt{pkg/exch2} and are + linked automatically if no other versions exist elsewhere in the + link path, but they should be left untouched to avoid breaking + configurations other than the one you intend to modify.\\ + +$\bullet$ Files containing grid parameters, named + \texttt{tile}???\texttt{.mitgrid} where ??? is \texttt{001} through + \texttt{006} (one per subdomain), must be in the working directory + when the MITgcm executable is run. These files are provided in the + example experiments for cubed sphere configurations with + 32$\times$32 cube sides and are non-trivial to generate -- please + contact MITgcm support if you want to generate files for other + configurations. \\ + +$\bullet$ As always when compiling MITgcm, the file \texttt{SIZE.h} + must be placed where \texttt{genmake2} will find it. In particular + for the exch2, the domain decompositin specified in \texttt{SIZE.h} + must correspond with the particular configuration's topology + specified in \texttt{W2\_EXCH2\_TOPOLOGY.h} and + \texttt{w2\_e2setup.F}. Domain decomposition issues particular to + exch2 are addressed in Section \ref{sec:topogen}: ``Generating + Topology Files for exch2''; a more general background on the subject + relvant to MITgcm is presented in Section + \ref{sect:specifying_a_decomposition}: ``Specifying a + decomposition''.\\ + +As of the time of writing the following examples use exch2 and may be +used for guidance: + +\begin{verbatim} +verification/adjust_nlfs.cs-32x32x1 +verification/adjustment.cs-32x32x1 +verification/aim.5l_cs +verification/global_ocean.cs32x15 +verification/hs94.cs-32x32x5 +\end{verbatim} + + + + +\subsection{Generating Topology Files for exch2} +\label{sec:topogen} + +Alternate cubed sphere topologies may be created using the Matlab +scripts in \texttt{utils/exch2/matlab-topology-generator}. Running the +m-file \texttt{driver} from the Matlab prompt (without passing any +function parameters) generates exch2 topology files +\texttt{W2\_EXCH2\_TOPOLOGY.h} and \texttt{w2\_e2setup.F} in the +working directory and displays via Matlab a figure of the topology. +The other m-files in the directory are subroutines of \texttt{driver} +and should not be run except for development purposes. \\ + +The parameters that determine the dimensions and topology of the +generated configuration are nr, nb, ng, tnx and tny, and all are +assigned early in the script. + +The first three determine the size of the subdomains (cube faces) and +hence the size of the overall domain. Each one determines the number +of grid points, and therefore the resolution, along the subdomain +sides in a ``great circle'' around each axis of the cube. At the time +of this writing MITgcm requires these three parameters to be equal, +but they provide for future releases of MITgcm to accomodate different +resolutions around the axes to allow (for example) greater resolution +around the equator.\\ + +The parameters tnx and tny determine the dimensions of the tiles into +which the subdomains are decomposed, and must evenly divide the +integer assigned to nr, nb and ng. The result is a rectangular tiling +of the subdomain. Figure \ref{fig:24tile} shows one possible topology +for a twenty-four tile cube, and figure \ref{fig:12tile} shows one for +twelve tiles. \\ + +\begin{figure} +\begin{center} + \resizebox{4in}{!}{ + \includegraphics{part6/s24t_16x16.ps} + } +\end{center} +\caption{Plot of cubed sphere topology with a 32$\times$32 grid and +twenty-four tiles (tnx=16, tny=16) +} \label{fig:24tile} +\end{figure} + +\begin{figure} +\begin{center} + \resizebox{4in}{!}{ + \includegraphics{part6/s12t_16x32.ps} + } +\end{center} +\caption{Plot of cubed sphere topology with a 32$\times$32 grid and +twelve tiles (tnx=16, tny=32) +} \label{fig:12tile} +\end{figure} + +Tiles can be selected from the topology to be omitted from being +allocated memory and processors. This kind of tuning is useful in +ocean modeling for omitting tiles that fall entirely on land. The +tiles omitted are specified in the file \texttt{blanklist.txt} by +their tile number in the topology, separated by a newline. \\ + + + + + \subsection{Key Variables} @@ -52,7 +178,7 @@ \texttt{NTILES}, and the maximum number of neighbors of any tiles by \texttt{MAX\_NEIGHBOURS}. These parameters are used for defining the size of the various one and two dimensional arrays that store tile -parameters indexed to the tile number. +parameters indexed to the tile number.\\ The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt} and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number