--- manual/s_phys_pkgs/text/exch2.tex 2004/05/06 15:21:01 1.18 +++ manual/s_phys_pkgs/text/exch2.tex 2004/05/10 21:39:11 1.19 @@ -1,4 +1,4 @@ -% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.18 2004/05/06 15:21:01 afe Exp $ +% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.19 2004/05/10 21:39:11 afe Exp $ % $Name: $ %% * Introduction @@ -54,7 +54,7 @@ $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F} must reside in a directory containing files - symbolically linked when \file{genmake2} runs. The safest place to + symbolically linked by the \file{genmake2} script. The safest place to put these is the directory indicated in the \code{-mods=DIR} command line modifier (typically \file{../code}), or the build directory. The default versions of these files reside in \file{pkg/exch2} and @@ -77,7 +77,7 @@ \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}. Domain decomposition issues particular to exch2 are addressed in Section \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2} - and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and MPI}; a more + and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and Multiprocessing}; a more general background on the subject relevant to MITgcm is presented in Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying a decomposition}.\\ @@ -107,22 +107,22 @@ exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F} in the working directory and displays a figure of the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:12tile}, -and \ref{fig:24tile} are examples. The other m-files in the directory are -subroutines of \file{driver.m} and should not be run ``bare'' except +and \ref{fig:24tile} are examples of the generated diagrams. The other +m-files in the directory are +subroutines called from \file{driver.m} and should not be run ``bare'' except for development purposes. \\ The parameters that determine the dimensions and topology of the generated configuration are \code{nr}, \code{nb}, \code{ng}, \code{tnx} and \code{tny}, and all are assigned early in the script. \\ -The first three determine the size of the subdomains and +The first three determine the height and width of the subdomains and hence the size of the overall domain. Each one determines the number of grid points, and therefore the resolution, along the subdomain sides in a ``great circle'' around each the three spatial axes of the cube. At the time of this writing MITgcm requires these three parameters to be equal, but they provide for future releases to accomodate different -resolutions around the axes to allow (for example) greater resolution -around the equator.\\ +resolutions around the axes to allow subdomains with differing resolutions.\\ The parameters \code{tnx} and \code{tny} determine the width and height of the tiles into which the subdomains are decomposed, and must evenly @@ -182,7 +182,7 @@ -\subsection{exch2, SIZE.h, and multiprocessing} +\subsection{exch2, SIZE.h, and Multiprocessing} \label{sec:exch2mpi} Once the topology configuration files are created, the Fortran @@ -190,8 +190,8 @@ Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying a decomposition} provides a general description of domain decomposition within MITgcm and its relation to \file{SIZE.h}. The -current section specifies certain constraints the exch2 package -imposes as well as describes how to enable parallel execution with +current section specifies constraints that the exch2 package +imposes and describes how to enable parallel execution with MPI. \\ As in the general case, the parameters \varlink{sNx}{sNx} and @@ -207,7 +207,7 @@ The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy}, \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of tiles and how they are distributed on processors. When using exch2, -the tiles are stored in a single dimension, and so +the tiles are stored in the $x$ dimension, and so \code{\varlink{nSy}{nSy}=1} in all cases. Since the tiles as configured by exch2 cannot be split up accross processors without regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\ @@ -215,9 +215,12 @@ The number of tiles MITgcm allocates and how they are distributed between processors depends on \varlink{nPx}{nPx} and \varlink{nSx}{nSx}. \varlink{nSx}{nSx} is the number of tiles per -processor and \varlink{nPx}{nPx} the number of processors. The total +processor and \varlink{nPx}{nPx} is the number of processors. The total number of tiles in the topology minus those listed in -\file{blanklist.txt} must equal \code{nSx*nPx}. \\ +\file{blanklist.txt} must equal \code{nSx*nPx}. Note that in order to +obtain maximum usage from a given number of processors in some cases, +this restriction might entail sharing a processor with a tile that would +otherwise be excluded. \\ The following is an example of \file{SIZE.h} for the twelve-tile configuration illustrated in figure \ref{fig:12tile} running on @@ -287,13 +290,13 @@ topology of twenty-four square tiles, four per subdomain (as in figure \ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and \code{exch2\_domain\_nyt=2}. Note that these parameters express the -tile layout to allow global data files that are tile-layout-neutral -and have no bearing on the internal storage of the arrays. The tiles -are stored internally in a range from \code{(1:\varlink{bi}{bi})} the +tile layout in order to allow global data files that are tile-layout-neutral. +They have no bearing on the internal storage of the arrays. The tiles +are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the $x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to equal \code{1} throughout the package. \\ -\subsubsection{Arrays Indexed to Tile Number} +\subsubsection{Arrays indexed to tile number} The following arrays are of length \code{NTILES} and are indexed to the tile number, which is indicated in the diagrams with the notation @@ -303,23 +306,23 @@ \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of each tile. At present for each tile \texttt{exch2\_tnx=sNx} and \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in -section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and -multiprocessing}. Future releases of MITgcm may allow varying tile +Section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and +Multiprocessing}. Future releases of MITgcm may allow varying tile sizes. \\ -The location of the tiles' Cartesian origin within a subdomain are -determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and -\varlink{exch2\_tbasey}{exch2_tbasey}. These variables are used to -relate the location of the edges of different tiles to each other. As +The arrays \varlink{exch2\_tbasex}{exch2_tbasex} and +\varlink{exch2\_tbasey}{exch2_tbasey} determine the tiles' +Cartesian origin within a subdomain +and locate the edges of different tiles relative to each other. As an example, in the default six-tile topology (Fig. \ref{fig:6tile}) each index in these arrays is set to \code{0} since a tile occupies its entire subdomain. The twenty-four-tile case discussed above will -have values of \code{0} or \code{16}, depending on the quadrant the -tile falls within the subdomain. The elements of the arrays +have values of \code{0} or \code{16}, depending on the quadrant of the +tile within the subdomain. The elements of the arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to \varlink{exch2\_tbasex}{exch2_tbasex} and -\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the +\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tile edges within the global address space, similar to that used by global output and input files. \\ @@ -328,8 +331,8 @@ standard cube topology and indicated by \textbf{\textsf{f}}$n$ in figures \ref{fig:12tile} and \ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours} -contains a count of the neighboring tiles each tile has, and is used -for setting bounds for looping over neighboring tiles. +contains a count of the neighboring tiles each tile has, and sets +the bounds for looping over neighboring tiles. \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each tile, and is used in interprocess communication. \\ @@ -338,7 +341,7 @@ \varlink{exch2\_isEedge}{exch2_isEedge}, \varlink{exch2\_isSedge}{exch2_isSedge}, and \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the -indexed tile lies on the respective edge of a subdomain, \code{0} if +indexed tile lies on the edge of its subdomain, \code{0} if not. The values are used within the topology generator to determine the orientation of neighboring tiles, and to indicate whether a tile lies on the corner of a subdomain. The latter case requires special @@ -371,9 +374,10 @@ The arrays \varlink{exch2\_pi}{exch2_pi} and \varlink{exch2\_pj}{exch2_pj} specify the transformations of indices in exchanges between the neighboring tiles. These transformations are -necessary in exchanges between subdomains because the array index in -one dimension may map to the other index in an adjacent subdomain, and -may be have its indexing reversed. This swapping arises from the +necessary in exchanges between subdomains because a horizontal dimension +in one subdomain +may map to other horizonal dimension in an adjacent subdomain, and +may also have its indexing reversed. This swapping arises from the ``folding'' of two-dimensional arrays into a three-dimensional cube. \\ @@ -381,8 +385,8 @@ are the neighbor ID \code{N} and the tile number \code{T} as explained above, plus a vector of length \code{2} containing transformation factors \code{t}. The first element of the transformation vector -holds the factor to multiply the index in the same axis, and the -second element holds the the same for the orthogonal index. To +holds the factor to multiply the index in the same dimension, and the +second element holds the the same for the orthogonal dimension. To clarify, \code{exch2\_pi(1,N,T)} holds the mapping of the $x$ axis index of tile \code{T} to the $x$ axis of tile \code{T}'s neighbor \code{N}, and \code{exch2\_pi(2,N,T)} holds the mapping of \code{T}'s @@ -397,7 +401,7 @@ \code{(1,0)}, since all tiles on the same subdomain are oriented identically. An axis that corresponds to the orthogonal dimension with the same index direction in a particular tile-neighbor -orientation will have \code{(0,1)}. Those in the opposite index +orientation will have \code{(0,1)}. Those with the opposite index direction will have \code{(0,-1)} in order to reverse the ordering. \\ The arrays \varlink{exch2\_oi}{exch2_oi}, @@ -483,9 +487,10 @@ \code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis, \code{T}'s northern edge exchanges with \code{Tn}'s western edge. The western edge of the tiles corresponds to the lower bound of the $x$ axis, so -\code{exch2\_itlo\_c} \code{exch2\_ithi\_c} are \code{0}. The range of +\code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} are \code{0}, in the +western halo region of \code{Tn}. The range of \code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the -width of \code{T}'s northern edge, plus the halo. \\ +width of \code{T}'s northern edge, expanded by one into the halo. \\ \subsection{Key Routines} @@ -494,8 +499,8 @@ themselves and are of the same format as those described in \ref{sect:cube_sphere_communication} \sectiontitle{Cube sphere communication}. Like the original routines, they are written as -templates which the local Makefile converts from RX into RL and RS -forms. \\ +templates which the local Makefile converts from \code{RX} into +\code{RL} and \code{RS} forms. \\ The interfaces with the core model subroutines are \code{EXCH\_UV\_XY\_RX}, \code{EXCH\_UV\_XYZ\_RX} and @@ -510,11 +515,11 @@ the singularities at the cube corners. \\ The separate scalar and vector forms of \code{EXCH2\_RX1\_CUBE} and -\code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subrouine -needs to pass both the $u$ and $v$ components of the phsical vectors. -This arises from the topological folding discussed above, where the -$x$ and $y$ axes get swapped in some cases. This swapping is not an -issue with the scalar version. These subroutines call +\code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subroutine +needs to pass both the $u$ and $v$ components of the physical vectors. +This swapping arises from the topological folding discussed above, where the +$x$ and $y$ axes get swapped in some cases, and is not an +issue with the scalar case. These subroutines call \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of the work using the variables discussed above. \\