--- manual/s_phys_pkgs/text/exch2.tex 2004/02/11 20:48:14 1.7 +++ manual/s_phys_pkgs/text/exch2.tex 2004/02/17 21:58:56 1.8 @@ -1,4 +1,4 @@ -% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.7 2004/02/11 20:48:14 afe Exp $ +% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.8 2004/02/17 21:58:56 edhill Exp $ % $Name: $ %% * Introduction @@ -10,151 +10,159 @@ %% o automatically inserted at \section{Reference} -\section{exch2: Extended Cubed Sphere Exchange} +\section{Extended Cubed Sphere Exchange} \label{sec:exch2} \subsection{Introduction} -The exch2 package is an extension to the original cubed sphere exchanges -to allow more flexible domain decomposition and parallelization. Cube faces -(subdomains) may be divided into whatever number of tiles that divide evenly -into the grid point dimensions of the subdomain. Furthermore, the individual -tiles may be run on separate processors in different combinations, -and whether exchanges between particular tiles occur between different -processors is determined at runtime. - -The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and -assigned in {\em w2\_e2setup.F}, both in the -{\em pkg/exch2} directory. The validity of the cube topology depends -on the {\em SIZE.h} file as detailed below. Both files are generated by -Matlab scripts and -should not be edited. The default files provided in the release set up -a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid -points, running on a single processor. +The \texttt{exch2} package is an extension to the original cubed +sphere exchanges to allow more flexible domain decomposition and +parallelization. Cube faces (subdomains) may be divided into whatever +number of tiles that divide evenly into the grid point dimensions of +the subdomain. Furthermore, the individual tiles may be run on +separate processors in different combinations, and whether exchanges +between particular tiles occur between different processors is +determined at runtime. + +The exchange parameters are declared in +\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} +and assigned in +\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}, both in +the \texttt{pkg/exch2} directory. The validity of the cube topology +depends on the \texttt{SIZE.h} file as detailed below. Both files are +generated by Matlab scripts and should not be edited. The default +files provided in the release set up a cube sphere arrangement of six +tiles, one per subdomain, each with 32x32 grid points, running on a +single processor. \subsection{Key Variables} The descriptions of the variables are divided up into scalars, -one-dimensional arrays indexed to the tile number, and two and three -dimensional -arrays indexed to tile number and neighboring tile. This division -actually reflects the functionality of these variables: the scalars -are common to every part of the topology, the tile-indexed arrays to -individual tiles, and the arrays indexed to tile and neighbor to -relationships between tiles and their neighbors. +one-dimensional arrays indexed to the tile number, and two and three +dimensional arrays indexed to tile number and neighboring tile. This +division actually reflects the functionality of these variables: the +scalars are common to every part of the topology, the tile-indexed +arrays to individual tiles, and the arrays indexed to tile and +neighbor to relationships between tiles and their neighbors. \subsubsection{Scalars} The number of tiles in a particular topology is set with the parameter -{\em NTILES}, and the maximum number of neighbors of any tiles by -{\em MAX\_NEIGHBOURS}. These parameters are used for defining the size of -the various one and two dimensional arrays that store tile parameters -indexed to the tile number. - -The scalar parameters {\em exch2\_domain\_nxt} and -{\em exch2\_domain\_nyt} express the number of tiles in the x and y global -indices. For example, the default setup of six tiles has -{\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}. A topology of -twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will -have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}. Note -that these parameters express the tile layout to allow global data files that -are tile-layout-neutral and have no bearing on the internal storage of the -arrays. The tiles are internally stored in a range from {\em 1,bi} (in the -x axis) and y-axis variable {\em bj} is generally ignored within the package. +\texttt{NTILES}, and the maximum number of neighbors of any tiles by +\texttt{MAX\_NEIGHBOURS}. These parameters are used for defining the +size of the various one and two dimensional arrays that store tile +parameters indexed to the tile number. + +The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt} +and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number +of tiles in the x and y global indices. For example, the default +setup of six tiles has \texttt{exch2\_domain\_nxt=6} and +\texttt{exch2\_domain\_nyt=1}. A topology of twenty-four square (in +gridpoints) tiles, four (2x2) per subdomain, will have +\texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}. +Note that these parameters express the tile layout to allow global +data files that are tile-layout-neutral and have no bearing on the +internal storage of the arrays. The tiles are internally stored in a +range from \texttt{1,bi} (in the x axis) and y-axis variable +\texttt{bj} is generally ignored within the package. \subsubsection{Arrays Indexed to Tile Number} -The following arrays are of size {\em NTILES}, are indexed to the tile number, -and the indices are omitted in their descriptions. +The following arrays are of size \texttt{NTILES}, are indexed to the +tile number, and the indices are omitted in their descriptions. -The arrays {\em exch2\_tnx} and {\em exch2\_tny} -express the x and y dimensions of each tile. At present for each tile -{\em exch2\_tnx = sNx} -and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}. Future releases of -MITgcm are to allow varying tile sizes. - -The location of the tiles' Cartesian origin within a subdomain are determined -by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}. These variables -are used to relate the location of the edges of the tiles to each other. As -an example, in the default six-tile topology (the degenerate case) -each index in these arrays are -set to 0. The twenty-four, 32x32 cube face case discussed above will have -values of 0 or 16, depending on the quadrant the tile falls within the -subdomain. {\em exch2\_myFace} contains the number of the -cubeface/subdomain of each tile, numbered 1-6 in the case of the standard -cube topology. - -The arrays {\em exch2\_txglobalo} and {\em exch2\_txglobalo} are similar to -{\em exch2\_tbasex} and {\em exch2\_tbasey}, but locate the tiles within -the global address space, similar to that used by global files. - -The arrays {\em exch2\_isWedge}, {\em exch2\_isEedge}, {\em exch2\_isSedge}, -and {\em exch2\_isNedge} are set to 1 if the indexed tile lies on the edge -of a subdomain, 0 if not. The values are used within the topology generator -to determine the orientation of neighboring tiles and to indicate whether -a tile lies on the corner of a subdomain. The latter case indicates -special exchange and numerical handling for the singularities at the eight -corners of the cube. {\em exch2\_isNedge} contains a count of how many -neighboring tiles each tile has, and is used for setting bounds for looping -over neighboring tiles. {\em exch2\_tProc} holds the process rank of each tile, -and is used in interprocess communication. +The arrays \varlink{exch2\_tnx}{exch2_tnx} and +\varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each +tile. At present for each tile \texttt{exch2\_tnx=sNx} and +\texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}. Future +releases of MITgcm are to allow varying tile sizes. + +The location of the tiles' Cartesian origin within a subdomain are +determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and +\varlink{exch2\_tbasey}{exch2_tbasey}. These variables are used to +relate the location of the edges of the tiles to each other. As an +example, in the default six-tile topology (the degenerate case) each +index in these arrays are set to 0. The twenty-four, 32x32 cube face +case discussed above will have values of 0 or 16, depending on the +quadrant the tile falls within the subdomain. The array +\varlink{exch2\_myFace}{exch2_myFace} contains the number of the +cubeface/subdomain of each tile, numbered 1-6 in the case of the +standard cube topology. + +The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and +\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to +\varlink{exch2\_tbasex}{exch2_tbasex} and +\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the +global address space, similar to that used by global files. + +The arrays \varlink{exch2\_isWedge}{exch2_isWedge}, +\varlink{exch2\_isEedge}{exch2_isEedge}, +\varlink{exch2\_isSedge}{exch2_isSedge}, and +\varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed +tile lies on the edge of a subdomain, 0 if not. The values are used +within the topology generator to determine the orientation of +neighboring tiles and to indicate whether a tile lies on the corner of +a subdomain. The latter case indicates special exchange and numerical +handling for the singularities at the eight corners of the cube. +\varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of +how many neighboring tiles each tile has, and is used for setting +bounds for looping over neighboring tiles. +\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each +tile, and is used in interprocess communication. \subsubsection{Arrays Indexed to Tile Number and Neighbor} -The following arrays are all of size {\em MAX\_NEIGHBOURS}x{\em NTILES} and -describe the orientations between the the tiles. +The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$ +\texttt{NTILES} and describe the orientations between the the tiles. -The array {\em exch2\_neighbourId(a,T)} holds the tile number $T_{n}$ for each tile -{\em T}'s neighbor tile {\em a}. The neighbor tiles are indexed {\em 1,MAX\_NEIGHBOURS } -in the order right to left on the north then south edges, and then top to bottom on the east -and west edges. maybe throw in a fig here, eh? - -{\em exch2\_opposingSend\_record(a,T)} holds -the index c in {\em exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T. -In other words, - -\begin{verbatim} -exch2_neighbourId( exch2_opposingSend_record(a,T), - exch2_neighbourId(a,T) ) = T +The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for +each of the $n$ neighboring tiles. The neighbor tiles are indexed +\texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north +then south edges, and then top to bottom on the east and west edges. +Maybe throw in a fig here, eh? + +The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c +in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T. +In other words, +\begin{verbatim} + exch2_neighbourId( exch2_opposingSend_record(a,T), + exch2_neighbourId(a,T) ) = T \end{verbatim} +and this provides a back-reference from the neighbor tiles. -% {\em exch2\_neighbourId(exch2\_opposingSend\_record(a,T),exch2\_neighbourId(a,T))=T}. -% alternate version - -This is to provide a backreference from the neighbor tiles. - -The arrays {\em exch2\_pi }, {\em exch2\_pj }, {\em exch2\_oi }, -{\em exch2\_oj }, {\em exch2\_oi\_f }, and {\em exch2\_oj\_f } specify -the transformations in exchanges between the neighboring tiles. The dimensions -of {\em exch2\_pi(t,N,T) } and {\em exch2\_pj(t,N,T) } are the neighbor ID -{ \em N } and the tile number {\em T } as explained above, plus the transformation -vector {\em t }, of length two. The first element of the transformation vector indicates -the factor by which variables representing the same vector component of a tile -will be multiplied, and the second element indicates the transform to the -variable in the other direction. As an example, {\em exch2\_pi(1,N,T) } holds the -transform of the i-component of a vector variable in tile {\em T } to the i-component of -tile {\em T }'s neighbor {\em N }, and {\em exch2\_pi(2,N,T) } hold the component -of neighbor {\em N }'s j-component. - -Under the current cube topology, one of the two elements of {\em exch2\_pi } or {\em exch2\_pj } -for a given tile {\em T } and neighbor {\em N } will be 0, reflecting the fact that -the vector components are orthogonal. The other element will be 1 or -1, depending on whether -the components are indexed in the same or opposite directions. For example, the transform dimension -of the arrays for all tile neighbors on the same subdomain will be {\em [1 , 0] }, since all tiles on -the same subdomain are oriented identically. Vectors that correspond to the orthogonal dimension with the -same index direction will have {\em [0 , 1] }, whereas those in the opposite index direction will have -{\em [0 , -1] }. - - +The arrays \varlink{exch2\_pi}{exch2_pi}, +\varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi}, +\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and +\varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in +exchanges between the neighboring tiles. The dimensions of +\texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the +neighbor ID \textit{N} and the tile number \textit{T} as explained +above, plus the transformation vector {\em t }, of length two. The +first element of the transformation vector indicates the factor by +which variables representing the same vector component of a tile will +be multiplied, and the second element indicates the transform to the +variable in the other direction. As an example, +\texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a +vector variable in tile \texttt{T} to the i-component of tile +\texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold +the component of neighbor \texttt{N}'s j-component. + +Under the current cube topology, one of the two elements of +\texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T} +and neighbor \texttt{N} will be 0, reflecting the fact that the vector +components are orthogonal. The other element will be 1 or -1, +depending on whether the components are indexed in the same or +opposite directions. For example, the transform dimension of the +arrays for all tile neighbors on the same subdomain will be [1,0], +since all tiles on the same subdomain are oriented identically. +Vectors that correspond to the orthogonal dimension with the same +index direction will have [0,1], whereas those in the opposite index +direction will have [0,-1]. -// - +{\footnotesize \begin{verbatim} - - - C exch2_pi :: X index row of target to source permutation C :: matrix for each neighbour entry. C exch2_pj :: Y index row of target to source permutation @@ -172,7 +180,7 @@ C :: offset vector for face quantities C :: of each neighbor entry. \end{verbatim} - +}