--- manual/s_phys_pkgs/text/exch2.tex 2004/05/10 21:39:11 1.19 +++ manual/s_phys_pkgs/text/exch2.tex 2005/08/09 21:52:09 1.25 @@ -1,4 +1,4 @@ -% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.19 2004/05/10 21:39:11 afe Exp $ +% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.25 2005/08/09 21:52:09 edhill Exp $ % $Name: $ %% * Introduction @@ -10,11 +10,11 @@ %% o automatically inserted at \section{Reference} -\section{exch2: Extended Cubed Sphere \mbox{Topology}} +\subsection{exch2: Extended Cubed Sphere \mbox{Topology}} \label{sec:exch2} -\subsection{Introduction} +\subsubsection{Introduction} The \texttt{exch2} package extends the original cubed sphere topology configuration to allow more flexible domain decomposition and @@ -23,7 +23,7 @@ dimensions of the subdomain. Furthermore, the tiles can run on separate processors individually or in groups, which provides for manual compile-time load balancing across a relatively arbitrary -number of processors. \\ +number of processors. The exchange parameters are declared in \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} @@ -41,46 +41,49 @@ \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h} file for single-processor execution. -\subsection{Invoking exch2} +\subsubsection{Invoking exch2} To use exch2 with the cubed sphere, the following conditions must be -met: \\ +met: -$\bullet$ The exch2 package is included when \file{genmake2} is run. - The easiest way to do this is to add the line \code{exch2} to the - \file{profile.conf} file -- see Section - \ref{sect:buildingCode} \sectiontitle{Building the code} for general - details. \\ +\begin{itemize} +\item The exch2 package is included when \file{genmake2} is run. The + easiest way to do this is to add the line \code{exch2} to the + \file{packages.conf} file -- see Section \ref{sect:buildingCode} + \sectiontitle{Building the code} for general + details. -$\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and +\item An example of \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F} must reside in a directory containing files - symbolically linked by the \file{genmake2} script. The safest place to - put these is the directory indicated in the \code{-mods=DIR} command - line modifier (typically \file{../code}), or the build directory. - The default versions of these files reside in \file{pkg/exch2} and - are linked automatically if no other versions exist elsewhere in the - build path, but they should be left untouched to avoid breaking - configurations other than the one you intend to modify.\\ - -$\bullet$ Files containing grid parameters, named - \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain), - must be in the working directory when the MITgcm executable is run. - These files are provided in the example experiments for cubed sphere - configurations with 32$\times$32 cube sides - -- please contact MITgcm support if you want to generate - files for other configurations. \\ - -$\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must - be placed where \file{genmake2} will find it. In particular for - exch2, the domain decomposition specified in \file{SIZE.h} must - correspond with the particular configuration's topology specified in + symbolically linked by the \file{genmake2} script. The safest place + to put these is the directory indicated in the \code{-mods=DIR} + command line modifier (typically \file{../code}), or the build + directory. The default versions of these files reside in + \file{pkg/exch2} and are linked automatically if no other versions + exist elsewhere in the build path, but they should be left untouched + to avoid breaking configurations other than the one you intend to + modify. + +\item Files containing grid parameters, named \file{tile00$n$.mitgrid} + where $n$=\code{(1:6)} (one per subdomain), must be in the working + directory when the MITgcm executable is run. These files are + provided in the example experiments for cubed sphere configurations + with 32$\times$32 cube sides -- please contact MITgcm support if you + want to generate files for other configurations. + +\item As always when compiling MITgcm, the file \file{SIZE.h} must be + placed where \file{genmake2} will find it. In particular for exch2, + the domain decomposition specified in \file{SIZE.h} must correspond + with the particular configuration's topology specified in \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}. Domain decomposition issues particular to exch2 are addressed in Section \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2} - and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and Multiprocessing}; a more - general background on the subject relevant to MITgcm is presented in - Section \ref{sect:specifying_a_decomposition} - \sectiontitle{Specifying a decomposition}.\\ + and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and + Multiprocessing}; a more general background on the subject + relevant to MITgcm is presented in Section + \ref{sect:specifying_a_decomposition} + \sectiontitle{Specifying a decomposition}. +\end{itemize} At the time of this writing the following examples use exch2 and may be used for guidance: @@ -96,7 +99,7 @@ -\subsection{Generating Topology Files for exch2} +\subsubsection{Generating Topology Files for exch2} \label{sec:topogen} Alternate cubed sphere topologies may be created using the Matlab @@ -182,27 +185,26 @@ -\subsection{exch2, SIZE.h, and Multiprocessing} +\subsubsection{exch2, SIZE.h, and Multiprocessing} \label{sec:exch2mpi} Once the topology configuration files are created, the Fortran \code{PARAMETER}s in \file{SIZE.h} must be configured to match. Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying -a decomposition} provides a general description of domain + a decomposition} provides a general description of domain decomposition within MITgcm and its relation to \file{SIZE.h}. The -current section specifies constraints that the exch2 package -imposes and describes how to enable parallel execution with -MPI. \\ +current section specifies constraints that the exch2 package imposes +and describes how to enable parallel execution with MPI. As in the general case, the parameters \varlink{sNx}{sNx} and \varlink{sNy}{sNy} define the size of the individual tiles, and so must be assigned the same respective values as \code{tnx} and -\code{tny} in \file{driver.m}.\\ +\code{tny} in \file{driver.m}. The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy} have no special bearing on exch2 and may be assigned as in the general -case. The same holds for \varlink{Nr}{Nr}, the number of vertical -levels in the model.\\ +case. The same holds for \varlink{Nr}{Nr}, the number of vertical +levels in the model. The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy}, \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of @@ -210,21 +212,28 @@ the tiles are stored in the $x$ dimension, and so \code{\varlink{nSy}{nSy}=1} in all cases. Since the tiles as configured by exch2 cannot be split up accross processors without -regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\ +regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. The number of tiles MITgcm allocates and how they are distributed between processors depends on \varlink{nPx}{nPx} and \varlink{nSx}{nSx}. \varlink{nSx}{nSx} is the number of tiles per -processor and \varlink{nPx}{nPx} is the number of processors. The total -number of tiles in the topology minus those listed in -\file{blanklist.txt} must equal \code{nSx*nPx}. Note that in order to +processor and \varlink{nPx}{nPx} is the number of processors. The +total number of tiles in the topology minus those listed in +\file{blanklist.txt} must equal \code{nSx*nPx}. Note that in order to obtain maximum usage from a given number of processors in some cases, -this restriction might entail sharing a processor with a tile that would -otherwise be excluded. \\ +this restriction might entail sharing a processor with a tile that +would otherwise be excluded because it is topographically outside of +the domain and therefore in \file{blanklist.txt}. For example, +suppose you have five processors and a domain decomposition of +thirty-six tiles that allows you to exclude seven tiles. To evenly +distribute the remaining twenty-nine tiles among five processors, you +would have to run one ``dummy'' tile to make an even six tiles per +processor. Such dummy tiles are \emph{not} listed in +\file{blanklist.txt}. The following is an example of \file{SIZE.h} for the twelve-tile -configuration illustrated in figure \ref{fig:12tile} running on -one processor: \\ +configuration illustrated in figure \ref{fig:12tile} running on one +processor: \begin{verbatim} PARAMETER ( @@ -260,10 +269,7 @@ \end{verbatim} - - - -\subsection{Key Variables} +\subsubsection{Key Variables} The descriptions of the variables are divided up into scalars, one-dimensional arrays indexed to the tile number, and two and @@ -273,7 +279,7 @@ arrays to individual tiles, and the arrays indexed by tile and neighbor to relationships between tiles and their neighbors. \\ -\subsubsection{Scalars} +Scalars: The number of tiles in a particular topology is set with the parameter \code{NTILES}, and the maximum number of neighbors of any tiles by @@ -296,7 +302,7 @@ $x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to equal \code{1} throughout the package. \\ -\subsubsection{Arrays indexed to tile number} +Arrays indexed to tile number: The following arrays are of length \code{NTILES} and are indexed to the tile number, which is indicated in the diagrams with the notation @@ -349,7 +355,7 @@ corners of the cube. \\ -\subsubsection{Arrays Indexed to Tile Number and Neighbor} +Arrays Indexed to Tile Number and Neighbor: The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and \code{NTILES} and describe the orientations between the the tiles. \\ @@ -493,7 +499,7 @@ width of \code{T}'s northern edge, expanded by one into the halo. \\ -\subsection{Key Routines} +\subsubsection{Key Routines} Most of the subroutines particular to exch2 handle the exchanges themselves and are of the same format as those described in