--- manual/s_phys_pkgs/text/exch2.tex 2004/01/28 18:08:22 1.2 +++ manual/s_phys_pkgs/text/exch2.tex 2004/02/11 20:48:14 1.7 @@ -1,4 +1,4 @@ -% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.2 2004/01/28 18:08:22 afe Exp $ +% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.7 2004/02/11 20:48:14 afe Exp $ % $Name: $ %% * Introduction @@ -11,19 +11,167 @@ \section{exch2: Extended Cubed Sphere Exchange} +\label{sec:exch2} + \subsection{Introduction} The exch2 package is an extension to the original cubed sphere exchanges to allow more flexible domain decomposition and parallelization. Cube faces -(subdomain) may be divided into whatever number of tiles that divide evenly +(subdomains) may be divided into whatever number of tiles that divide evenly into the grid point dimensions of the subdomain. Furthermore, the individual -tiles may be run on different processors in any combination, (tone this down -a bit), and whether exchanges between particular tiles occur between different -processors is decided at runtime. - - - +tiles may be run on separate processors in different combinations, +and whether exchanges between particular tiles occur between different +processors is determined at runtime. + +The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and +assigned in {\em w2\_e2setup.F}, both in the +{\em pkg/exch2} directory. The validity of the cube topology depends +on the {\em SIZE.h} file as detailed below. Both files are generated by +Matlab scripts and +should not be edited. The default files provided in the release set up +a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid +points, running on a single processor. + +\subsection{Key Variables} + +The descriptions of the variables are divided up into scalars, +one-dimensional arrays indexed to the tile number, and two and three +dimensional +arrays indexed to tile number and neighboring tile. This division +actually reflects the functionality of these variables: the scalars +are common to every part of the topology, the tile-indexed arrays to +individual tiles, and the arrays indexed to tile and neighbor to +relationships between tiles and their neighbors. + +\subsubsection{Scalars} + +The number of tiles in a particular topology is set with the parameter +{\em NTILES}, and the maximum number of neighbors of any tiles by +{\em MAX\_NEIGHBOURS}. These parameters are used for defining the size of +the various one and two dimensional arrays that store tile parameters +indexed to the tile number. + +The scalar parameters {\em exch2\_domain\_nxt} and +{\em exch2\_domain\_nyt} express the number of tiles in the x and y global +indices. For example, the default setup of six tiles has +{\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}. A topology of +twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will +have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}. Note +that these parameters express the tile layout to allow global data files that +are tile-layout-neutral and have no bearing on the internal storage of the +arrays. The tiles are internally stored in a range from {\em 1,bi} (in the +x axis) and y-axis variable {\em bj} is generally ignored within the package. + +\subsubsection{Arrays Indexed to Tile Number} + +The following arrays are of size {\em NTILES}, are indexed to the tile number, +and the indices are omitted in their descriptions. + +The arrays {\em exch2\_tnx} and {\em exch2\_tny} +express the x and y dimensions of each tile. At present for each tile +{\em exch2\_tnx = sNx} +and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}. Future releases of +MITgcm are to allow varying tile sizes. + +The location of the tiles' Cartesian origin within a subdomain are determined +by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}. These variables +are used to relate the location of the edges of the tiles to each other. As +an example, in the default six-tile topology (the degenerate case) +each index in these arrays are +set to 0. The twenty-four, 32x32 cube face case discussed above will have +values of 0 or 16, depending on the quadrant the tile falls within the +subdomain. {\em exch2\_myFace} contains the number of the +cubeface/subdomain of each tile, numbered 1-6 in the case of the standard +cube topology. + +The arrays {\em exch2\_txglobalo} and {\em exch2\_txglobalo} are similar to +{\em exch2\_tbasex} and {\em exch2\_tbasey}, but locate the tiles within +the global address space, similar to that used by global files. + +The arrays {\em exch2\_isWedge}, {\em exch2\_isEedge}, {\em exch2\_isSedge}, +and {\em exch2\_isNedge} are set to 1 if the indexed tile lies on the edge +of a subdomain, 0 if not. The values are used within the topology generator +to determine the orientation of neighboring tiles and to indicate whether +a tile lies on the corner of a subdomain. The latter case indicates +special exchange and numerical handling for the singularities at the eight +corners of the cube. {\em exch2\_isNedge} contains a count of how many +neighboring tiles each tile has, and is used for setting bounds for looping +over neighboring tiles. {\em exch2\_tProc} holds the process rank of each tile, +and is used in interprocess communication. + +\subsubsection{Arrays Indexed to Tile Number and Neighbor} + +The following arrays are all of size {\em MAX\_NEIGHBOURS}x{\em NTILES} and +describe the orientations between the the tiles. + +The array {\em exch2\_neighbourId(a,T)} holds the tile number $T_{n}$ for each tile +{\em T}'s neighbor tile {\em a}. The neighbor tiles are indexed {\em 1,MAX\_NEIGHBOURS } +in the order right to left on the north then south edges, and then top to bottom on the east +and west edges. maybe throw in a fig here, eh? + +{\em exch2\_opposingSend\_record(a,T)} holds +the index c in {\em exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T. +In other words, + +\begin{verbatim} +exch2_neighbourId( exch2_opposingSend_record(a,T), + exch2_neighbourId(a,T) ) = T +\end{verbatim} + +% {\em exch2\_neighbourId(exch2\_opposingSend\_record(a,T),exch2\_neighbourId(a,T))=T}. +% alternate version + +This is to provide a backreference from the neighbor tiles. + +The arrays {\em exch2\_pi }, {\em exch2\_pj }, {\em exch2\_oi }, +{\em exch2\_oj }, {\em exch2\_oi\_f }, and {\em exch2\_oj\_f } specify +the transformations in exchanges between the neighboring tiles. The dimensions +of {\em exch2\_pi(t,N,T) } and {\em exch2\_pj(t,N,T) } are the neighbor ID +{ \em N } and the tile number {\em T } as explained above, plus the transformation +vector {\em t }, of length two. The first element of the transformation vector indicates +the factor by which variables representing the same vector component of a tile +will be multiplied, and the second element indicates the transform to the +variable in the other direction. As an example, {\em exch2\_pi(1,N,T) } holds the +transform of the i-component of a vector variable in tile {\em T } to the i-component of +tile {\em T }'s neighbor {\em N }, and {\em exch2\_pi(2,N,T) } hold the component +of neighbor {\em N }'s j-component. + +Under the current cube topology, one of the two elements of {\em exch2\_pi } or {\em exch2\_pj } +for a given tile {\em T } and neighbor {\em N } will be 0, reflecting the fact that +the vector components are orthogonal. The other element will be 1 or -1, depending on whether +the components are indexed in the same or opposite directions. For example, the transform dimension +of the arrays for all tile neighbors on the same subdomain will be {\em [1 , 0] }, since all tiles on +the same subdomain are oriented identically. Vectors that correspond to the orthogonal dimension with the +same index direction will have {\em [0 , 1] }, whereas those in the opposite index direction will have +{\em [0 , -1] }. + + + + +// + +\begin{verbatim} + + + +C exch2_pi :: X index row of target to source permutation +C :: matrix for each neighbour entry. +C exch2_pj :: Y index row of target to source permutation +C :: matrix for each neighbour entry. +C exch2_oi :: X index element of target to source +C :: offset vector for cell-centered quantities +C :: of each neighbor entry. +C exch2_oj :: Y index element of target to source +C :: offset vector for cell-centered quantities +C :: of each neighbor entry. +C exch2_oi_f :: X index element of target to source +C :: offset vector for face quantities +C :: of each neighbor entry. +C exch2_oj_f :: Y index element of target to source +C :: offset vector for face quantities +C :: of each neighbor entry. +\end{verbatim}