--- manual/s_phys_pkgs/text/exch2.tex 2004/03/12 20:58:19 1.9 +++ manual/s_phys_pkgs/text/exch2.tex 2004/03/15 22:39:28 1.11 @@ -1,4 +1,4 @@ -% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.9 2004/03/12 20:58:19 afe Exp $ +% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.11 2004/03/15 22:39:28 afe Exp $ % $Name: $ %% * Introduction @@ -10,7 +10,7 @@ %% o automatically inserted at \section{Reference} -\section{Extended Cubed Sphere Exchange} +\section{exch2: Extended Cubed Sphere \mbox{Topology}} \label{sec:exch2} @@ -24,47 +24,66 @@ individual tiles may be run on separate processors in different combinations, and whether exchanges between particular tiles occur between different processors is determined at runtime. This -flexibility provides for manual load balancing across a relatively -arbitrary number of processors. +flexibility provides for manual compile-time load balancing across a +relatively arbitrary number of processors. \\ The exchange parameters are declared in \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} and assigned in \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The -validity of the cube topology depends on the \texttt{SIZE.h} file as -detailed below. Both files are generated by Matlab scripts in ?? -check these in already! and should not be edited. The default files -provided in the release configure a cubed sphere arrangement of six +validity of the cube topology depends on the \file{SIZE.h} file as +detailed below. Both files are generated by Matlab scripts in +\file{utils/exch2/matlab-topology-generator}; see Section +\ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2} +for details on creating alternate topologies. The default files +provided in the release configure a cubed sphere topology of six tiles, one per subdomain, each with 32$\times$32 grid points, all -running on a single processor. Pregenerated examples of these files -with alternate topologies are provided in ??. +running on a single processor. Pregenerated examples of these files +with alternate topologies are provided under +\file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h} +file for single-processor execution. \subsection{Invoking exch2} -To use exch2 with the cubed sphere, the following conditions must be met: +To use exch2 with the cubed sphere, the following conditions must be +met: \\ -- the exch2 package is included when \texttt{genmake2} is run. The - easiest way to do this is to add the line \texttt{exch2} to the - \texttt{profile.conf} file -- see Section \ref{sect:buildingCode} - for general details. \\ - -- an example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and - \texttt{w2\_e2setup.F} must reside in a directory containing code - linked when \texttt{genmake2} runs. The safest place to put these - is the directory indicated in the \texttt{-mods=DIR} command line - modifier (typically \texttt{../code}), or the build directory. The - default versions of these files reside in \texttt{pkg/exch2}, but - they should be left untouched to avoid breaking configurations other - than the one you intend to modify.\\ - -- files containing grid parameters, named - \texttt{tile}xxx\texttt{.mitgrid} where xxx is \texttt{001} through - \texttt{006}, must be in the working directory when the MITgcm - executable is run. These files are provided in the example - experiments for cubed sphere configurations with 32$\times$32 cube - sides and are non-trivial to generate -- please contact MITgcm - support if you want to generate files for other configurations. - This is lame. ?? \\ +$\bullet$ The exch2 package is included when \file{genmake2} is run. + The easiest way to do this is to add the line \code{exch2} to the + \file{profile.conf} file -- see Section + \ref{sect:buildingCode}\sectiontitle{Building the code} for general + details. \\ + +$\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and + \file{w2\_e2setup.F} must reside in a directory containing code + linked when \file{genmake2} runs. The safest place to put these + is the directory indicated in the \code{-mods=DIR} command line + modifier (typically \file{../code}), or the build directory. The + default versions of these files reside in \file{pkg/exch2} and are + linked automatically if no other versions exist elsewhere in the + link path, but they should be left untouched to avoid breaking + configurations other than the one you intend to modify.\\ + +$\bullet$ Files containing grid parameters, named + \file{tile???.mitgrid} where \file{???} is \file{001} through + \file{006} (one per subdomain), must be in the working directory + when the MITgcm executable is run. These files are provided in the + example experiments for cubed sphere configurations with + 32$\times$32 cube sides and are non-trivial to generate -- please + contact MITgcm support if you want to generate files for other + configurations. \\ + +$\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} + must be placed where \file{genmake2} will find it. In particular + for the exch2, the domain decomposition specified in \file{SIZE.h} + must correspond with the particular configuration's topology + specified in \file{W2\_EXCH2\_TOPOLOGY.h} and + \file{w2\_e2setup.F}. Domain decomposition issues particular to + exch2 are addressed in Section \ref{sec:topogen} \sectiontitle{Generating + Topology Files for exch2}; a more general background on the subject + relvant to MITgcm is presented in Section + \ref{sect:specifying_a_decomposition}\sectiontitle{Specifying a + decomposition}.\\ As of the time of writing the following examples use exch2 and may be used for guidance: @@ -80,7 +99,70 @@ -\subsection{Generating Topology Files} +\subsection{Generating Topology Files for exch2} +\label{sec:topogen} + +Alternate cubed sphere topologies may be created using the Matlab +scripts in \file{utils/exch2/matlab-topology-generator}. Running the +m-file \file{driver.m} from the Matlab prompt (there are no parameters +to pass) generates exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} +and \file{w2\_e2setup.F} in the working directory and displays a +figure of the topology via Matlab. The other m-files in the directory +are subroutines of \file{driver.m} and should not be run except for +development purposes. \\ + +The parameters that determine the dimensions and topology of the +generated configuration are \code{nr}, \code{nb}, \code{ng}, +\code{tnx} and \code{tny}, and all are assigned early in the script. + +The first three determine the size of the subdomains (cube faces) and +hence the size of the overall domain. Each one determines the number +of grid points, and therefore the resolution, along the subdomain +sides in a ``great circle'' around each axis of the cube. At the time +of this writing MITgcm requires these three parameters to be equal, +but they provide for future releases of MITgcm to accomodate different +resolutions around the axes to allow (for example) greater resolution +around the equator.\\ + +The parameters \code{tnx} and \code{tny} determine the dimensions of +the tiles into which the subdomains are decomposed, and must evenly +divide the integer assigned to \code{nr}, \code{nb} and \code{ng}. +The result is a rectangular tiling of the subdomain. Figure +\ref{fig:24tile} shows one possible topology for a twenty-four tile +cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\ + +\begin{figure} +\begin{center} + \resizebox{4in}{!}{ + \includegraphics{part6/s24t_16x16.ps} + } +\end{center} +\caption{Plot of cubed sphere topology with a 32$\times$32 grid and +twenty-four tiles (\code{tnx=16, tny=16}) +} \label{fig:24tile} +\end{figure} + +\begin{figure} +\begin{center} + \resizebox{4in}{!}{ + \includegraphics{part6/s12t_16x32.ps} + } +\end{center} +\caption{Plot of cubed sphere topology with a 32$\times$32 grid and +twelve tiles (\code{tnx=16, tny=32}) +} \label{fig:12tile} +\end{figure} + +Tiles can be selected from the topology to be omitted from being +allocated memory and processors. This kind otuning is useful in +ocean modeling for omitting tiles that fall entirely on land. The +tiles omitted are specified in the file \file{blanklist.txt} by +their tile number in the topology, separated by a newline. \\ + + + + + \subsection{Key Variables}