--- manual/s_phys_pkgs/text/exch2.tex 2004/10/12 15:47:40 1.20 +++ manual/s_phys_pkgs/text/exch2.tex 2010/08/30 23:09:21 1.29 @@ -1,4 +1,4 @@ -% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.20 2004/10/12 15:47:40 edhill Exp $ +% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.29 2010/08/30 23:09:21 jmc Exp $ % $Name: $ %% * Introduction @@ -10,11 +10,11 @@ %% o automatically inserted at \section{Reference} -\section{exch2: Extended Cubed Sphere \mbox{Topology}} +\subsection{exch2: Extended Cubed Sphere \mbox{Topology}} \label{sec:exch2} -\subsection{Introduction} +\subsubsection{Introduction} The \texttt{exch2} package extends the original cubed sphere topology configuration to allow more flexible domain decomposition and @@ -23,7 +23,7 @@ dimensions of the subdomain. Furthermore, the tiles can run on separate processors individually or in groups, which provides for manual compile-time load balancing across a relatively arbitrary -number of processors. \\ +number of processors. The exchange parameters are declared in \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} @@ -41,7 +41,7 @@ \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h} file for single-processor execution. -\subsection{Invoking exch2} +\subsubsection{Invoking exch2} To use exch2 with the cubed sphere, the following conditions must be met: @@ -49,9 +49,10 @@ \begin{itemize} \item The exch2 package is included when \file{genmake2} is run. The easiest way to do this is to add the line \code{exch2} to the - \file{profile.conf} file -- see Section \ref{sect:buildingCode} - \sectiontitle{Building the code} for general details. - + \file{packages.conf} file -- see Section \ref{sec:buildingCode} + \sectiontitle{Building the code} for general + details. + \item An example of \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F} must reside in a directory containing files symbolically linked by the \file{genmake2} script. The safest place @@ -62,21 +63,14 @@ exist elsewhere in the build path, but they should be left untouched to avoid breaking configurations other than the one you intend to modify. - + \item Files containing grid parameters, named \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain), must be in the working directory when the MITgcm executable is run. These files are provided in the example experiments for cubed sphere configurations - with 32$\times$32 cube sides -- please contact - \begin{rawhtml} - - \end{rawhtml} -\begin{verbatim} -MITgcm-support@mitgcm.org -\end{verbatim} - \begin{rawhtml} \end{rawhtml} - if you want to generate files for other configurations. - + with 32$\times$32 cube sides -- please contact MITgcm support if you + want to generate files for other configurations. + \item As always when compiling MITgcm, the file \file{SIZE.h} must be placed where \file{genmake2} will find it. In particular for exch2, the domain decomposition specified in \file{SIZE.h} must correspond @@ -87,12 +81,10 @@ and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and Multiprocessing}; a more general background on the subject relevant to MITgcm is presented in Section - \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying a - decomposition}. + \ref{sec:specifying_a_decomposition} + \sectiontitle{Specifying a decomposition}. \end{itemize} - - At the time of this writing the following examples use exch2 and may be used for guidance: @@ -107,7 +99,7 @@ -\subsection{Generating Topology Files for exch2} +\subsubsection{Generating Topology Files for exch2} \label{sec:topogen} Alternate cubed sphere topologies may be created using the Matlab @@ -117,8 +109,8 @@ from the Matlab prompt (there are no parameters to pass) generates exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F} in the working directory and displays a figure of -the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:12tile}, -and \ref{fig:24tile} are examples of the generated diagrams. The other +the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:18tile}, +and \ref{fig:48tile} are examples of the generated diagrams. The other m-files in the directory are subroutines called from \file{driver.m} and should not be run ``bare'' except for development purposes. \\ @@ -139,40 +131,49 @@ the tiles into which the subdomains are decomposed, and must evenly divide the integer assigned to \code{nr}, \code{nb} and \code{ng}. The result is a rectangular tiling of the subdomain. Figure -\ref{fig:24tile} shows one possible topology for a twenty-four-tile -cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\ +\ref{fig:48tile} shows one possible topology for a twenty-four-tile +cube, and figure \ref{fig:6tile} shows one for six tiles. \\ \begin{figure} \begin{center} - \resizebox{4in}{!}{ - \includegraphics{part6/s24t_16x16.ps} + \resizebox{6in}{!}{ +% \includegraphics{s_phys_pkgs/figs/s24t_16x16.ps} + \includegraphics{s_phys_pkgs/figs/adjust_cs.ps} } \end{center} \caption{Plot of a cubed sphere topology with a 32$\times$192 domain divided into six 32$\times$32 subdomains, each of which is divided -into four tiles of width \code{tnx=16} and height \code{tny=16} for a -total of twenty-four tiles. The colored borders of the subdomains -represent the parameters \code{nr} (red), \code{nb} (blue), and -\code{ng} (green). } \label{fig:24tile} +into eight tiles of width \code{tnx=16} and height \code{tny=8} for a +total of forty-eight tiles. The colored borders of the subdomains +represent the parameters \code{nr} (red), \code{ng} (green), and +\code{nb} (blue). +This tiling is used in the example +verification/adjustment.cs-32x32x1/ +with the option (blanklist.txt) to remove the land-only 4 tiles +(11,12,13,14) which are filled in red on the plot. +} \label{fig:48tile} \end{figure} \begin{figure} \begin{center} - \resizebox{4in}{!}{ - \includegraphics{part6/s12t_16x32.ps} + \resizebox{6in}{!}{ +% \includegraphics{s_phys_pkgs/figs/s12t_16x32.ps} + \includegraphics{s_phys_pkgs/figs/polarcap.ps} } \end{center} -\caption{Plot of a cubed sphere topology with a 32$\times$192 domain -divided into six 32$\times$32 subdomains of two tiles each - (\code{tnx=16, tny=32}). -} \label{fig:12tile} +\caption{Plot of a non-square cubed sphere topology with +6 subdomains of different size (nr=90,ng=360,nb=90), +divided into one to four tiles each + (\code{tnx=90, tny=90}), resulting in a total of 18 tiles. +} \label{fig:18tile} \end{figure} \begin{figure} \begin{center} \resizebox{4in}{!}{ - \includegraphics{part6/s6t_32x32.ps} +% \includegraphics{s_phys_pkgs/figs/s6t_32x32.ps} + \includegraphics{s_phys_pkgs/figs/s6t_32x32.ps} } \end{center} \caption{Plot of a cubed sphere topology with a 32$\times$192 domain @@ -193,27 +194,26 @@ -\subsection{exch2, SIZE.h, and Multiprocessing} +\subsubsection{exch2, SIZE.h, and Multiprocessing} \label{sec:exch2mpi} Once the topology configuration files are created, the Fortran \code{PARAMETER}s in \file{SIZE.h} must be configured to match. -Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying -a decomposition} provides a general description of domain +Section \ref{sec:specifying_a_decomposition} \sectiontitle{Specifying + a decomposition} provides a general description of domain decomposition within MITgcm and its relation to \file{SIZE.h}. The -current section specifies constraints that the exch2 package -imposes and describes how to enable parallel execution with -MPI. \\ +current section specifies constraints that the exch2 package imposes +and describes how to enable parallel execution with MPI. As in the general case, the parameters \varlink{sNx}{sNx} and \varlink{sNy}{sNy} define the size of the individual tiles, and so must be assigned the same respective values as \code{tnx} and -\code{tny} in \file{driver.m}.\\ +\code{tny} in \file{driver.m}. The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy} have no special bearing on exch2 and may be assigned as in the general -case. The same holds for \varlink{Nr}{Nr}, the number of vertical -levels in the model.\\ +case. The same holds for \varlink{Nr}{Nr}, the number of vertical +levels in the model. The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy}, \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of @@ -221,29 +221,36 @@ the tiles are stored in the $x$ dimension, and so \code{\varlink{nSy}{nSy}=1} in all cases. Since the tiles as configured by exch2 cannot be split up accross processors without -regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\ +regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. The number of tiles MITgcm allocates and how they are distributed between processors depends on \varlink{nPx}{nPx} and \varlink{nSx}{nSx}. \varlink{nSx}{nSx} is the number of tiles per -processor and \varlink{nPx}{nPx} is the number of processors. The total -number of tiles in the topology minus those listed in -\file{blanklist.txt} must equal \code{nSx*nPx}. Note that in order to +processor and \varlink{nPx}{nPx} is the number of processors. The +total number of tiles in the topology minus those listed in +\file{blanklist.txt} must equal \code{nSx*nPx}. Note that in order to obtain maximum usage from a given number of processors in some cases, -this restriction might entail sharing a processor with a tile that would -otherwise be excluded. \\ - -The following is an example of \file{SIZE.h} for the twelve-tile -configuration illustrated in figure \ref{fig:12tile} running on -one processor: \\ +this restriction might entail sharing a processor with a tile that +would otherwise be excluded because it is topographically outside of +the domain and therefore in \file{blanklist.txt}. For example, +suppose you have five processors and a domain decomposition of +thirty-six tiles that allows you to exclude seven tiles. To evenly +distribute the remaining twenty-nine tiles among five processors, you +would have to run one ``dummy'' tile to make an even six tiles per +processor. Such dummy tiles are \emph{not} listed in +\file{blanklist.txt}. + +The following is an example of \file{SIZE.h} for the six-tile +configuration illustrated in figure \ref{fig:6tile} +running on one processor: \begin{verbatim} PARAMETER ( - & sNx = 16, + & sNx = 32, & sNy = 32, & OLx = 2, & OLy = 2, - & nSx = 12, + & nSx = 6, & nSy = 1, & nPx = 1, & nPy = 1, @@ -252,16 +259,16 @@ & Nr = 5) \end{verbatim} -The following is an example for the twenty-four-tile topology in -figure \ref{fig:24tile} running on six processors: +The following is an example for the forty-eight-tile topology in +figure \ref{fig:48tile} running on six processors: \begin{verbatim} PARAMETER ( & sNx = 16, - & sNy = 16, + & sNy = 8, & OLx = 2, & OLy = 2, - & nSx = 4, + & nSx = 8, & nSy = 1, & nPx = 6, & nPy = 1, @@ -271,10 +278,7 @@ \end{verbatim} - - - -\subsection{Key Variables} +\subsubsection{Key Variables} The descriptions of the variables are divided up into scalars, one-dimensional arrays indexed to the tile number, and two and @@ -284,7 +288,7 @@ arrays to individual tiles, and the arrays indexed by tile and neighbor to relationships between tiles and their neighbors. \\ -\subsubsection{Scalars} +Scalars: The number of tiles in a particular topology is set with the parameter \code{NTILES}, and the maximum number of neighbors of any tiles by @@ -298,20 +302,20 @@ of tiles in the $x$ and $y$ global indices. For example, the default setup of six tiles (Fig. \ref{fig:6tile}) has \code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}. A -topology of twenty-four square tiles, four per subdomain (as in figure -\ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and -\code{exch2\_domain\_nyt=2}. Note that these parameters express the +topology of forty-eight tiles, eight per subdomain (as in figure +\ref{fig:48tile}), will have \code{exch2\_domain\_nxt=12} and +\code{exch2\_domain\_nyt=4}. Note that these parameters express the tile layout in order to allow global data files that are tile-layout-neutral. They have no bearing on the internal storage of the arrays. The tiles are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the $x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to equal \code{1} throughout the package. \\ -\subsubsection{Arrays indexed to tile number} +Arrays indexed to tile number: The following arrays are of length \code{NTILES} and are indexed to the tile number, which is indicated in the diagrams with the notation -\code{tn}. The indices are omitted in the descriptions. \\ +\textsf{t}$n$. The indices are omitted in the descriptions. \\ The arrays \varlink{exch2\_tnx}{exch2_tnx} and \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of @@ -339,11 +343,11 @@ The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of the subdomain of each tile, in a range \code{(1:6)} in the case of the -standard cube topology and indicated by \textbf{\textsf{fn}} in -figures \ref{fig:12tile} and \ref{fig:24tile}. The -\varlink{exch2\_nNeighbours}{exch2_nNeighbours} variable contains a -count of the neighboring tiles each tile has, and sets the bounds for -looping over neighboring tiles. And +standard cube topology and indicated by \textbf{\textsf{f}}$n$ in +figures \ref{fig:6tile} and +\ref{fig:48tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours} +contains a count of the neighboring tiles each tile has, and sets +the bounds for looping over neighboring tiles. \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each tile, and is used in interprocess communication. \\ @@ -360,7 +364,7 @@ corners of the cube. \\ -\subsubsection{Arrays Indexed to Tile Number and Neighbor} +Arrays Indexed to Tile Number and Neighbor: The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and \code{NTILES} and describe the orientations between the the tiles. \\ @@ -463,8 +467,8 @@ \varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index bounds of the edge segment of the neighbor tile \code{N}'s subdomain that gets exchanged with the local tile \code{T}. To take the example -of tile \code{T=2} in the twelve-tile topology -(Fig. \ref{fig:12tile}): \\ +of tile \code{T=2} in the forty-eight-tile topology +(Fig. \ref{fig:48tile}): \\ \begin{verbatim} exch2_itlo_c(4,2)=17 @@ -504,11 +508,11 @@ width of \code{T}'s northern edge, expanded by one into the halo. \\ -\subsection{Key Routines} +\subsubsection{Key Routines} Most of the subroutines particular to exch2 handle the exchanges themselves and are of the same format as those described in -\ref{sect:cube_sphere_communication} \sectiontitle{Cube sphere +\ref{sec:cube_sphere_communication} \sectiontitle{Cube sphere communication}. Like the original routines, they are written as templates which the local Makefile converts from \code{RX} into \code{RL} and \code{RS} forms. \\ @@ -534,3 +538,10 @@ \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of the work using the variables discussed above. \\ +\subsubsection{Experiments and tutorials that use exch2} +\label{sec:pkg:exch2:experiments} + +\begin{itemize} +\item{Held Suarez tutorial, in tutorial\_held\_suarez\_cs verification directory, +described in section \ref{sec:eg-hs} } +\end{itemize}