/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory | Revision Log | View Revision Graph Revision Graph | View Patch Patch

-revision 1.12 by afe,
Tue Mar 16 21:52:15 2004 UTC
+revision 1.14 by afe,
Wed Mar 17 21:44:02 2004 UTC
 Line 21 
 sphere topology configuration to allow m
  decomposition and parallelization.  Cube faces (also called
  subdomains) may be divided into any number of tiles that divide evenly
  into the grid point dimensions of the subdomain.  Furthermore, the
- individual tiles may be run on separate processors in different
+ individual tiles can run on separate processors in different
  combinations, and whether exchanges between particular tiles occur
  between different processors is determined at runtime.  This
  flexibility provides for manual compile-time load balancing across a
 Line 65 
 $\bullet$ An example of \file{W2\_EXCH2\
    configurations other than the one you intend to modify.\\
  $\bullet$ Files containing grid parameters, named
-   \file{tile00$n$.mitgrid} where $n$=[1,6] (one per subdomain), must
+   \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain),
-   be in the working directory when the MITgcm executable is run.
+   must be in the working directory when the MITgcm executable is run.
    These files are provided in the example experiments for cubed sphere
    configurations with 32$\times$32 cube sides and are non-trivial to
    generate -- please contact MITgcm support if you want to generate
    files for other configurations. \\
  $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must
-   be placed where \file{genmake2} will find it.  In particular for the
+   be placed where \file{genmake2} will find it.  In particular for
    exch2, the domain decomposition specified in \file{SIZE.h} must
    correspond with the particular configuration's topology specified in
    \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain
 Line 119 
 generated configuration are \code{nr}, \
  The first three determine the size of the subdomains and
  hence the size of the overall domain.  Each one determines the number
  of grid points, and therefore the resolution, along the subdomain
- sides in a ``great circle'' around each axis of the cube.  At the time
+ sides in a ``great circle'' around an axis of the cube.  At the time
  of this writing MITgcm requires these three parameters to be equal,
  but they provide for future releases  to accomodate different
  resolutions around the axes to allow (for example) greater resolution
 Line 129 
 The parameters \code{tnx} and \code{tny}
  the tiles into which the subdomains are decomposed, and must evenly
  divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
  The result is a rectangular tiling of the subdomain.  Figure
- \ref{fig:24tile} shows one possible topology for a twenty-four tile
+ \ref{fig:24tile} shows one possible topology for a twentyfour-tile
  cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
  \begin{figure}
 Line 139 
 cube, and figure \ref{fig:12tile} shows
   }
  \end{center}
- \caption{Plot of cubed sphere topology with a 32$\times$192 domain
+ \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
  divided into six 32$\times$32 subdomains, each of which is divided into four tiles
- (\code{tnx=16, tny=16}) for a total of twenty-four tiles.
+ (\code{tnx=16, tny=16}) for a total of twentyfour tiles.
  } \label{fig:24tile}
  \end{figure}
 Line 151 
 divided into six 32$\times$32 subdomains
    \includegraphics{part6/s12t_16x32.ps}
   }
  \end{center}
- \caption{Plot of cubed sphere topology with a 32$\times$192 domain
+ \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
  divided into six 32$\times$32 subdomains of two tiles each
   (\code{tnx=16, tny=32}).
  } \label{fig:12tile}
  \end{figure}
+ \begin{figure}
+ \begin{center}
+  \resizebox{4in}{!}{
+   \includegraphics{part6/s6t_32x32.ps}
+  }
+ \end{center}
+ \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
+ divided into six 32$\times$32 subdomains with one tile each
+ (\code{tnx=32, tny=32}).  This is the default configuration.
+   }
+ \label{fig:6tile}
+ \end{figure}
  Tiles can be selected from the topology to be omitted from being
  allocated memory and processors.  This tuning is useful in ocean
  modeling for omitting tiles that fall entirely on land.  The tiles
-Line 171 
 by their tile number in the topology, se
+Line 185 
 by their tile number in the topology, se
  \label{sec:exch2mpi}
  Once the topology configuration files are created, the Fortran
- parameters in \file{SIZE.h} must be configured to match.  Section
+ \code{PARAMETER}s in \file{SIZE.h} must be configured to match.
- \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying a
+ Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying
- decomposition} provides a general description of domain decomposition
+ a decomposition} provides a general description of domain
- within MITgcm and its relation to \file{SIZE.h}. The current section
+ decomposition within MITgcm and its relation to \file{SIZE.h}. The
- specifies certain constraints the exch2 package imposes as well as
+ current section specifies certain constraints the exch2 package
- describes how to enable parallel execution with MPI. \\
+ imposes as well as describes how to enable parallel execution with
+ MPI. \\
  As in the general case, the parameters \varlink{sNx}{sNx} and
  \varlink{sNy}{sNy} define the size of the individual tiles, and so
-Line 266 
 generated by \file{driver.m}.\\
+Line 281 
 generated by \file{driver.m}.\\
  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
  of tiles in the $x$ and $y$ global indices.  For example, the default
- setup of six tiles has \code{exch2\_domain\_nxt=6} and
+ setup of six tiles (Fig. \ref{fig:6tile}) has \code{exch2\_domain\_nxt=6} and
  \code{exch2\_domain\_nyt=1}.  A topology of twenty-four square tiles,
  four per subdomain (as in figure \ref{fig:24tile}), will have
  \code{exch2\_domain\_nxt=12} and \code{exch2\_domain\_nyt=2}.  Note
  that these parameters express the tile layout to allow global data
  files that are tile-layout-neutral and have no bearing on the internal
  storage of the arrays.  The tiles are internally stored in a range
- from [1,\varlink{bi}{bi}] the $x$ axis and $y$ axis variable
+ from \code{(1:\varlink{bi}{bi})} the $x$ axis, and $y$ axis variable
  \varlink{bj}{bj} is generally ignored within the package. \\
  \subsubsection{Arrays Indexed to Tile Number}
- The following arrays are of size \code{NTILES}, are indexed to the
+ The following arrays are of length \code{NTILES}, are indexed to the
  tile number, and the indices are omitted in their descriptions. \\
  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
-Line 293 
 The location of the tiles' Cartesian ori
+Line 308 
 The location of the tiles' Cartesian ori
  determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
  \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
  relate the location of the edges of different tiles to each other.  As
- an example, in the default six-tile topology ??  each index in these
+ an example, in the default six-tile topology (Fig. \ref{fig:6tile})
- arrays are set to \code{0}.  The twentyfour-tile case discussed above
+ each index in these arrays is set to \code{0} since a tile occupies
- will have values of \code{0} or \code{16}, depending on the quadrant
+ its entire subdomain.  The twentyfour-tile case discussed above will
- the tile falls within the subdomain.  The array
+ have values of \code{0} or \code{16}, depending on the quadrant the
- \varlink{exch2\_myFace}{exch2_myFace} contains the number of the
+ tile falls within the subdomain.  The elements of the arrays
- subdomain of each tile, numbered \code{(1:6)} in the case of the
+ \varlink{exch2\_txglobalo}{exch2_txglobalo} and
- standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
+ \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
- figures \ref{fig:12tile}) and \ref{fig:24tile}). \\
- The elements of the arrays \varlink{exch2\_txglobalo}{exch2_txglobalo}
- and \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
  \varlink{exch2\_tbasex}{exch2_tbasex} and
  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
  global address space, similar to that used by global files. \\
+ The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of
+ the subdomain of each tile, in a range \code{(1:6)} in the case of the
+ standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
+ figures \ref{fig:12tile} and
+ \ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
+ contains a count of how many neighboring tiles each tile has, and is
+ used for setting bounds for looping over neighboring tiles.
+ \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
+ tile, and is used in interprocess communication.  \\
  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
  \varlink{exch2\_isEedge}{exch2_isEedge},
  \varlink{exch2\_isSedge}{exch2_isSedge}, and
-Line 317 
 values are used within the topology gene
+Line 339 
 values are used within the topology gene
  orientation of neighboring tiles, and to indicate whether a tile lies
  on the corner of a subdomain.  The latter case requires special
  exchange and numerical handling for the singularities at the eight
- corners of the cube.  \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
+ corners of the cube. \\
- contains a count of how many neighboring tiles each tile has, and is
- used for setting bounds for looping over neighboring tiles.
- \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
- tile, and is used in interprocess communication.  \\
  \subsubsection{Arrays Indexed to Tile Number and Neighbor}
-Line 336 
 in the order right to left on the north
+Line 355 
 in the order right to left on the north
  to bottom on the east and west edges.  Maybe throw in a fig here, eh?
  \\
+ \sloppy
  The \code{exch2\_opposingSend\_record(a,T)} array holds the index
  \code{b} in \texttt{exch2\_neighbourId(b,Tn)} that holds the tile
  number \code{T}.  In other words,
-Line 345 
 number \code{T}.  In other words,
+Line 365 
 number \code{T}.  In other words,
  \end{verbatim}
  This provides a back-reference from the neighbor tiles. \\
- The arrays \varlink{exch2\_pi}{exch2_pi},
+ The arrays \varlink{exch2\_pi}{exch2_pi} and
- \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
+ \varlink{exch2\_pj}{exch2_pj} specify the transformations of variables
- \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
+ in exchanges between the neighboring tiles.  These transformations are
- \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
+ necessary in exchanges between subdomains because a physical vector
- exchanges between the neighboring tiles.  The dimensions of
+ component in one direction may map to one in a different direction in
- \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)} are the neighbor
+ an adjacent subdomain, and may be have its indexing reversed. This
- ID \code{N} and the tile number \code{T} as explained above, plus a
+ swapping arises from the ``folding'' of two-dimensional arrays into a
- vector of length 2 containing transformation factors \code{t}.  The
+ three-dimensional cube.
- first element of the transformation vector indicates the factor
- \code{t} by which variables representing the same vector component of
+ The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)}
- a tile \code{T} will be multiplied in exchanges with neighbor
+ are the neighbor ID \code{N} and the tile number \code{T} as explained
- \code{N}, and the second element indicates the transform to the
+ above, plus a vector of length 2 containing transformation factors
- variable in the other direction.  As an example,
+ \code{t}.  The first element of the transformation vector indicates
- \code{exch2\_pi(1,N,T)} holds the transform of the $i$ component of a
+ the factor \code{t} by which variables representing the same
- vector variable in tile \code{T} to the $i$ component of tile
+ \emph{physical} vector component of a tile \code{T} will be multiplied
- \code{T}'s neighbor \code{N}, and \code{exch2\_pi(2,N,T)} hold the
+ in exchanges with neighbor \code{N}, and the second element indicates
- component of neighbor \code{N}'s $j$ component. \\
+ the transform to the physical vector in the other direction.  To
+ clarify (hopefully), \code{exch2\_pi(1,N,T)} holds the transform of
+ the $i$ component of a vector variable in tile \code{T} to the $i$
+ component of tile \code{T}'s neighbor \code{N}, and
+ \code{exch2\_pi(2,N,T)} holds the transform of \code{T}'s $i$
+ components to the neighbor \code{N}'s $j$ component. \\
  Under the current cube topology, one of the two elements of
  \code{exch2\_pi} or \code{exch2\_pj} for a given tile \code{T} and
  neighbor \code{N} will be \code{0}, reflecting the fact that the two
- vector components are orthogonal.  The other element will be 1 or -1,
+ vector components are orthogonal.  The other element will be \code{1}
- depending on whether the components are indexed in the same or
+ or \code{-1}, depending on whether the components are indexed in the
- opposite directions.  For example, the transform vector of the arrays
+ same or opposite directions.  For example, the transform vector of the
- for all tile neighbors on the same subdomain will be \code{(1,0)},
+ arrays for all tile neighbors on the same subdomain will be
- since all tiles on the same subdomain are oriented identically.  A
+ \code{(1,0)}, since all tiles on the same subdomain are oriented
- vector direction that corresponds to the orthogonal dimension with the
+ identically.  A vector direction that corresponds to the orthogonal
- same index direction in a particular tile-neighbor orientation will
+ dimension with the same index direction in a particular tile-neighbor
- have \code{(0,1)}, whereas those in the opposite index direction will
+ orientation will have \code{(0,1)}, whereas those in the opposite
- have \code{(0,-1)}.  This needs some diagrams.
+ index direction will have \code{(0,-1)}. \\
+ The arrays \varlink{exch2\_oi}{exch2_oi},
+ \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
+ \varlink{exch2\_oj\_f}{exch2_oj_f} are indexed to tile number and
+ neighbor and specify the relative offset within the subdomain of the
+ array index of a variable going from a neighboring tile $N$ to a local
+ tile $T$.  Consider the six-tile case (Fig. \ref{fig:6tile}), where
+ \code{exch2\_oi(1,1)=33}, \code{exch2\_oi(2,1)=0},
+ \code{exch2\_oi(3,1)=32}, and \code{exch2\_oi(4,1)=-32}.  Each of these
+ indicates the offset in the $x$ direction \\
+ Finally, \varlink{exch2\_itlo\_c}{exch2_itlo_c},
+ \varlink{exch2\_ithi\_c}{exch2_ithi_c},
+ \varlink{exch2\_jtlo\_c}{exch2_jtlo_c} and
+ \varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index
+ bounds of the edge segment of the neighbor tile \code{N}'s subdomain
+ that gets exchanged with the local tile \code{T}.  To take the example
+ of tile \code{T=2} in the twelve-tile topology
+ (Fig. \ref{fig:12tile}): \\
+ \begin{verbatim}
+        exch2_itlo_c(4,2)=17
+        exch2_ithi_c(4,2)=17
+        exch2_jtlo_c(4,2)=0
+        exch2_jthi_c(4,2)=33
+ \end{verbatim}
+ Here \code{N=4}, indicating the western neighbor, which is \code{Tn=1}.
+ \code{Tn=1} resides on the same subdomain as \code{T=2}, so the tiles
+ have the same orientation and the same $x$ and $y$ axes.  The $i$
+ component is orthogonal to the western edge and the tile is 16 points
+ wide, so \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} indicate the
+ column beyond \code{Tn=1}'s eastern edge, in that tile's halo
+ region. Since the border of the tiles extends through the entire
+ height of the subdomain, the $y$ axis bounds \code{exch2\_jtlo\_c} to
+ \code{exch2\_jthi\_c} cover the height, plus 1 in either direction to
+ cover part of the halo. \\
+ For the north edge of the same tile \code{T=2} where \code{N=1} and
+ the neighbor tile is \code{Tn=5}:
- {\footnotesize
  \begin{verbatim}
- C      exch2_pi          :: X index row of target to source permutation
+        exch2_itlo_c(1,2)=0
- C                        :: matrix for each neighbour entry.
+        exch2_ithi_c(1,2)=0
- C      exch2_pj          :: Y index row of target to source permutation
+        exch2_jtlo_c(1,2)=0
- C                        :: matrix for each neighbour entry.
+        exch2_jthi_c(1,2)=17
- C      exch2_oi          :: X index element of target to source
- C                        :: offset vector for cell-centered quantities
- C                        :: of each neighbor entry.
- C      exch2_oj          :: Y index element of target to source
- C                        :: offset vector for cell-centered quantities
- C                        :: of each neighbor entry.
- C      exch2_oi_f        :: X index element of target to source
- C                        :: offset vector for face quantities
- C                        :: of each neighbor entry.
- C      exch2_oj_f        :: Y index element of target to source
- C                        :: offset vector for face quantities
- C                        :: of each neighbor entry.
  \end{verbatim}
- }
+ \code{T}'s northern edge is parallel to the $x$ axis, but since
+ \code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis,
+ \code{T}'s northern edge exchanges with \code{Tn}'s western edge.
+ The western edge of the tiles corresponds to the lower bound of the
+ $x$ axis, so \code{exch2\_itlo\_c} \code{exch2\_ithi\_c} are \code{0}. The
+ range of \code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the
+ width of \code{T}'s northern edge, plus the halo. \\
+ This needs some diagrams. \\

 Legend:



Removed from v.1.12
 


changed lines


 
Added in v.1.14
 Legend:



Removed from v.1.12
 


changed lines


 
Added in v.1.14
-Removed from v.1.12
+Added in v.1.14

	ViewVC Help
Powered by ViewVC 1.1.22