/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory | Revision Log | View Revision Graph Revision Graph | View Patch Patch

-revision 1.12 by afe,
Tue Mar 16 21:52:15 2004 UTC
+revision 1.22 by edhill,
Tue Oct 12 18:16:03 2004 UTC
 Line 12
  \section{exch2: Extended Cubed Sphere \mbox{Topology}}
  \label{sec:exch2}
+ \label{sec:pkg:exch2}
+ \begin{rawhtml}
+ <!-- CMIREDIR:package_exch2: -->
+ \end{rawhtml}
  \subsection{Introduction}
- The \texttt{exch2} package extends the original cubed
+ The \texttt{exch2} package extends the original cubed sphere topology
- sphere topology configuration to allow more flexible domain
+ configuration to allow more flexible domain decomposition and
- decomposition and parallelization.  Cube faces (also called
+ parallelization.  Cube faces (also called subdomains) may be divided
- subdomains) may be divided into any number of tiles that divide evenly
+ into any number of tiles that divide evenly into the grid point
- into the grid point dimensions of the subdomain.  Furthermore, the
+ dimensions of the subdomain.  Furthermore, the tiles can run on
- individual tiles may be run on separate processors in different
+ separate processors individually or in groups, which provides for
- combinations, and whether exchanges between particular tiles occur
+ manual compile-time load balancing across a relatively arbitrary
- between different processors is determined at runtime.  This
+ number of processors. \\
- flexibility provides for manual compile-time load balancing across a
- relatively arbitrary number of processors. \\
  The exchange parameters are declared in
  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
-Line 34 
 and assigned in
+Line 36 
 and assigned in
  validity of the cube topology depends on the \file{SIZE.h} file as
  detailed below.  The default files provided in the release configure a
  cubed sphere topology of six tiles, one per subdomain, each with
-$\times$32 grid points, all running on a single processor.  Both
+$\times$32 grid points, with all tiles running on a single processor.  Both
  files are generated by Matlab scripts in
  \file{utils/exch2/matlab-topology-generator}; see Section
  \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
-Line 46 
 file for single-processor execution.
+Line 48 
 file for single-processor execution.
  \subsection{Invoking exch2}
  To use exch2 with the cubed sphere, the following conditions must be
- met: \\
+ met:
- $\bullet$ The exch2 package is included when \file{genmake2} is run.
+ \begin{itemize}
-   The easiest way to do this is to add the line \code{exch2} to the
+ \item The exch2 package is included when \file{genmake2} is run.  The
-   \file{profile.conf} file -- see Section
+   easiest way to do this is to add the line \code{exch2} to the
-   \ref{sect:buildingCode} \sectiontitle{Building the code} for general
+   \file{profile.conf} file -- see Section \ref{sect:buildingCode}
-   details. \\
+   \sectiontitle{Building the code} for general details.
- $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
+ \item An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
-   \file{w2\_e2setup.F} must reside in a directory containing code
+   \file{w2\_e2setup.F} must reside in a directory containing files
-   linked when \file{genmake2} runs.  The safest place to put these
+   symbolically linked by the \file{genmake2} script.  The safest place
-   is the directory indicated in the \code{-mods=DIR} command line
+   to put these is the directory indicated in the \code{-mods=DIR}
-   modifier (typically \file{../code}), or the build directory.  The
+   command line modifier (typically \file{../code}), or the build
-   default versions of these files reside in \file{pkg/exch2} and are
+   directory.  The default versions of these files reside in
-   linked automatically if no other versions exist elsewhere in the
+   \file{pkg/exch2} and are linked automatically if no other versions
-   link path, but they should be left untouched to avoid breaking
+   exist elsewhere in the build path, but they should be left untouched
-   configurations other than the one you intend to modify.\\
+   to avoid breaking configurations other than the one you intend to
+   modify.
- $\bullet$ Files containing grid parameters, named
-   \file{tile00$n$.mitgrid} where $n$=[1,6] (one per subdomain), must
+ \item Files containing grid parameters, named \file{tile00$n$.mitgrid}
-   be in the working directory when the MITgcm executable is run.
+   where $n$=\code{(1:6)} (one per subdomain), must be in the working
-   These files are provided in the example experiments for cubed sphere
+   directory when the MITgcm executable is run.  These files are
-   configurations with 32$\times$32 cube sides and are non-trivial to
+   provided in the example experiments for cubed sphere configurations
-   generate -- please contact MITgcm support if you want to generate
+   with 32$\times$32 cube sides -- please contact
-   files for other configurations. \\
+   \begin{rawhtml}
+     <A href="mailto:mitgcm-support@dev.mitgcm.org">
- $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must
+   \end{rawhtml}
-   be placed where \file{genmake2} will find it.  In particular for the
+ \begin{verbatim}
-   exch2, the domain decomposition specified in \file{SIZE.h} must
+ MITgcm-support@mitgcm.org
-   correspond with the particular configuration's topology specified in
+ \end{verbatim}
+   \begin{rawhtml} </A> \end{rawhtml}
+   if you want to generate files for other configurations.
+ \item As always when compiling MITgcm, the file \file{SIZE.h} must be
+   placed where \file{genmake2} will find it.  In particular for exch2,
+   the domain decomposition specified in \file{SIZE.h} must correspond
+   with the particular configuration's topology specified in
    \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain
    decomposition issues particular to exch2 are addressed in Section
    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
-   and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and MPI}; a more
+   and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
-   general background on the subject relevant to MITgcm is presented in
+     Multiprocessing}; a more general background on the subject
-   Section \ref{sect:specifying_a_decomposition}
+   relevant to MITgcm is presented in Section
-   \sectiontitle{Specifying a decomposition}.\\
+   \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying a
+     decomposition}.
+ \end{itemize}
- As of the time of writing the following examples use exch2 and may be
- used for guidance:
+ At the time of this writing the following examples use exch2 and may
+ be used for guidance:
  \begin{verbatim}
  verification/adjust_nlfs.cs-32x32x1
-Line 108 
 m-file
+Line 121 
 m-file
  from the Matlab prompt (there are no parameters to pass) generates
  exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and
  \file{w2\_e2setup.F} in the working directory and displays a figure of
- the topology via Matlab.  The other m-files in the directory are
+ the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:12tile},
- subroutines of \file{driver.m} and should not be run ``bare'' except
+ and \ref{fig:24tile} are examples of the generated diagrams.  The other
+ m-files in the directory are
+ subroutines called from \file{driver.m} and should not be run ``bare'' except
  for development purposes. \\
  The parameters that determine the dimensions and topology of the
  generated configuration are \code{nr}, \code{nb}, \code{ng},
  \code{tnx} and \code{tny}, and all are assigned early in the script. \\
- The first three determine the size of the subdomains and
+ The first three determine the height and width of the subdomains and
  hence the size of the overall domain.  Each one determines the number
  of grid points, and therefore the resolution, along the subdomain
- sides in a ``great circle'' around each axis of the cube.  At the time
+ sides in a ``great circle'' around each the three spatial axes of the cube.  At the time
  of this writing MITgcm requires these three parameters to be equal,
  but they provide for future releases  to accomodate different
- resolutions around the axes to allow (for example) greater resolution
+ resolutions around the axes to allow subdomains with differing resolutions.\\
- around the equator.\\
- The parameters \code{tnx} and \code{tny} determine the dimensions of
+ The parameters \code{tnx} and \code{tny} determine the width and height of
  the tiles into which the subdomains are decomposed, and must evenly
  divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
  The result is a rectangular tiling of the subdomain.  Figure
- \ref{fig:24tile} shows one possible topology for a twenty-four tile
+ \ref{fig:24tile} shows one possible topology for a twenty-four-tile
  cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
  \begin{figure}
-Line 139 
 cube, and figure \ref{fig:12tile} shows
+Line 153 
 cube, and figure \ref{fig:12tile} shows
   }
  \end{center}
- \caption{Plot of cubed sphere topology with a 32$\times$192 domain
+ \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
- divided into six 32$\times$32 subdomains, each of which is divided into four tiles
+ divided into six 32$\times$32 subdomains, each of which is divided
- (\code{tnx=16, tny=16}) for a total of twenty-four tiles.
+ into four tiles of width \code{tnx=16} and height \code{tny=16} for a
- } \label{fig:24tile}
+ total of twenty-four tiles.  The colored borders of the subdomains
+ represent the parameters \code{nr} (red), \code{nb} (blue), and
+ \code{ng} (green).  } \label{fig:24tile}
  \end{figure}
  \begin{figure}
-Line 151 
 divided into six 32$\times$32 subdomains
+Line 167 
 divided into six 32$\times$32 subdomains
    \includegraphics{part6/s12t_16x32.ps}
   }
  \end{center}
- \caption{Plot of cubed sphere topology with a 32$\times$192 domain
+ \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
  divided into six 32$\times$32 subdomains of two tiles each
   (\code{tnx=16, tny=32}).
  } \label{fig:12tile}
  \end{figure}
+ \begin{figure}
+ \begin{center}
+  \resizebox{4in}{!}{
+   \includegraphics{part6/s6t_32x32.ps}
+  }
+ \end{center}
+ \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
+ divided into six 32$\times$32 subdomains with one tile each
+ (\code{tnx=32, tny=32}).  This is the default configuration.
+   }
+ \label{fig:6tile}
+ \end{figure}
  Tiles can be selected from the topology to be omitted from being
  allocated memory and processors.  This tuning is useful in ocean
  modeling for omitting tiles that fall entirely on land.  The tiles
-Line 167 
 by their tile number in the topology, se
+Line 197 
 by their tile number in the topology, se
- \subsection{exch2, SIZE.h, and multiprocessing}
+ \subsection{exch2, SIZE.h, and Multiprocessing}
  \label{sec:exch2mpi}
  Once the topology configuration files are created, the Fortran
- parameters in \file{SIZE.h} must be configured to match.  Section
+ \code{PARAMETER}s in \file{SIZE.h} must be configured to match.
- \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying a
+ Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying
- decomposition} provides a general description of domain decomposition
+ a decomposition} provides a general description of domain
- within MITgcm and its relation to \file{SIZE.h}. The current section
+ decomposition within MITgcm and its relation to \file{SIZE.h}. The
- specifies certain constraints the exch2 package imposes as well as
+ current section specifies constraints that the exch2 package
- describes how to enable parallel execution with MPI. \\
+ imposes and describes how to enable parallel execution with
+ MPI. \\
  As in the general case, the parameters \varlink{sNx}{sNx} and
  \varlink{sNy}{sNy} define the size of the individual tiles, and so
-Line 191 
 levels in the model.\\
+Line 222 
 levels in the model.\\
  The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},
  \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of
  tiles and how they are distributed on processors.  When using exch2,
- the tiles are stored in single dimension, and so
+ the tiles are stored in the $x$ dimension, and so
  \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as
  configured by exch2 cannot be split up accross processors without
  regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\
-Line 199 
 regenerating the topology, \code{\varlin
+Line 230 
 regenerating the topology, \code{\varlin
  The number of tiles MITgcm allocates and how they are distributed
  between processors depends on \varlink{nPx}{nPx} and
  \varlink{nSx}{nSx}.  \varlink{nSx}{nSx} is the number of tiles per
- processor and \varlink{nPx}{nPx} the number of processors.  The total
+ processor and \varlink{nPx}{nPx} is the number of processors.  The total
  number of tiles in the topology minus those listed in
- \file{blanklist.txt} must equal \code{nSx*nPx}. \\
+ \file{blanklist.txt} must equal \code{nSx*nPx}.  Note that in order to
+ obtain maximum usage from a given number of processors in some cases,
+ this restriction might entail sharing a processor with a tile that would
+ otherwise be excluded. \\
  The following is an example of \file{SIZE.h} for the twelve-tile
  configuration illustrated in figure \ref{fig:12tile} running on
-Line 222 
 one processor: \\
+Line 256 
 one processor: \\
       &           Nr  =   5)
  \end{verbatim}
- The following is an example for the twentyfour-tile topology in figure
+ The following is an example for the twenty-four-tile topology in
- \ref{fig:24tile} running on six processors:
+ figure \ref{fig:24tile} running on six processors:
  \begin{verbatim}
        PARAMETER (
-Line 247 
 The following is an example for the twen
+Line 281 
 The following is an example for the twen
  \subsection{Key Variables}
  The descriptions of the variables are divided up into scalars,
- one-dimensional arrays indexed to the tile number, and two and three
+ one-dimensional arrays indexed to the tile number, and two and
- dimensional arrays indexed to tile number and neighboring tile.  This
+ three-dimensional arrays indexed to tile number and neighboring tile.
- division reflects the functionality of these variables: The
+ This division reflects the functionality of these variables: The
  scalars are common to every part of the topology, the tile-indexed
  arrays to individual tiles, and the arrays indexed by tile and
  neighbor to relationships between tiles and their neighbors. \\
-Line 266 
 generated by \file{driver.m}.\\
+Line 300 
 generated by \file{driver.m}.\\
  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
  of tiles in the $x$ and $y$ global indices.  For example, the default
- setup of six tiles has \code{exch2\_domain\_nxt=6} and
+ setup of six tiles (Fig. \ref{fig:6tile}) has
- \code{exch2\_domain\_nyt=1}.  A topology of twenty-four square tiles,
+ \code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}.  A
- four per subdomain (as in figure \ref{fig:24tile}), will have
+ topology of twenty-four square tiles, four per subdomain (as in figure
- \code{exch2\_domain\_nxt=12} and \code{exch2\_domain\_nyt=2}.  Note
+ \ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and
- that these parameters express the tile layout to allow global data
+ \code{exch2\_domain\_nyt=2}.  Note that these parameters express the
- files that are tile-layout-neutral and have no bearing on the internal
+ tile layout in order to allow global data files that are tile-layout-neutral.
- storage of the arrays.  The tiles are internally stored in a range
+ They have no bearing on the internal storage of the arrays.  The tiles
- from [1,\varlink{bi}{bi}] the $x$ axis and $y$ axis variable
+ are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the
- \varlink{bj}{bj} is generally ignored within the package. \\
+ $x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to
+ equal \code{1} throughout the package. \\
- \subsubsection{Arrays Indexed to Tile Number}
+ \subsubsection{Arrays indexed to tile number}
- The following arrays are of size \code{NTILES}, are indexed to the
- tile number, and the indices are omitted in their descriptions. \\
+ The following arrays are of length \code{NTILES} and are indexed to
+ the tile number, which is indicated in the diagrams with the notation
+ \code{tn}.  The indices are omitted in the descriptions. \\
  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
  \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of
  each tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
  \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in
- section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
+ Section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
- multiprocessing}.  Future releases of MITgcm are to allow varying tile
+ Multiprocessing}.  Future releases of MITgcm may allow varying tile
  sizes. \\
- The location of the tiles' Cartesian origin within a subdomain are
+ The arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
- determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
+ \varlink{exch2\_tbasey}{exch2_tbasey} determine the tiles'
- \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
+ Cartesian origin within a subdomain
- relate the location of the edges of different tiles to each other.  As
+ and locate the edges of different tiles relative to each other.  As
- an example, in the default six-tile topology ??  each index in these
+ an example, in the default six-tile topology (Fig. \ref{fig:6tile})
- arrays are set to \code{0}.  The twentyfour-tile case discussed above
+ each index in these arrays is set to \code{0} since a tile occupies
- will have values of \code{0} or \code{16}, depending on the quadrant
+ its entire subdomain.  The twenty-four-tile case discussed above will
- the tile falls within the subdomain.  The array
+ have values of \code{0} or \code{16}, depending on the quadrant of the
- \varlink{exch2\_myFace}{exch2_myFace} contains the number of the
+ tile within the subdomain.  The elements of the arrays
- subdomain of each tile, numbered \code{(1:6)} in the case of the
+ \varlink{exch2\_txglobalo}{exch2_txglobalo} and
- standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
+ \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
- figures \ref{fig:12tile}) and \ref{fig:24tile}). \\
- The elements of the arrays \varlink{exch2\_txglobalo}{exch2_txglobalo}
- and \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
  \varlink{exch2\_tbasex}{exch2_tbasex} and
- \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
+ \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tile edges within the
- global address space, similar to that used by global files. \\
+ global address space, similar to that used by global output and input
+ files. \\
+ The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of
+ the subdomain of each tile, in a range \code{(1:6)} in the case of the
+ standard cube topology and indicated by \textbf{\textsf{fn}} in
+ figures \ref{fig:12tile} and \ref{fig:24tile}. The
+ \varlink{exch2\_nNeighbours}{exch2_nNeighbours} variable contains a
+ count of the neighboring tiles each tile has, and sets the bounds for
+ looping over neighboring tiles.  And
+ \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
+ tile, and is used in interprocess communication.  \\
  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
  \varlink{exch2\_isEedge}{exch2_isEedge},
  \varlink{exch2\_isSedge}{exch2_isSedge}, and
  \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the
- indexed tile lies on the edge of a subdomain, \code{0} if not.  The
+ indexed tile lies on the edge of its subdomain, \code{0} if
- values are used within the topology generator to determine the
+ not.  The values are used within the topology generator to determine
- orientation of neighboring tiles, and to indicate whether a tile lies
+ the orientation of neighboring tiles, and to indicate whether a tile
- on the corner of a subdomain.  The latter case requires special
+ lies on the corner of a subdomain.  The latter case requires special
  exchange and numerical handling for the singularities at the eight
- corners of the cube.  \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
+ corners of the cube. \\
- contains a count of how many neighboring tiles each tile has, and is
- used for setting bounds for looping over neighboring tiles.
- \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
- tile, and is used in interprocess communication.  \\
  \subsubsection{Arrays Indexed to Tile Number and Neighbor}
- The following arrays are all of size
+ The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and
- \code{MAX\_NEIGHBOURS}$\times$\code{NTILES} and describe the
+ \code{NTILES} and describe the orientations between the the tiles. \\
- orientations between the the tiles. \\
  The array \code{exch2\_neighbourId(a,T)} holds the tile number
  \code{Tn} for each of the tile number \code{T}'s neighboring tiles
- \code{a}.  The neighbor tiles are indexed \code{(1:MAX\_NEIGHBOURS)}
+ \code{a}.  The neighbor tiles are indexed
- in the order right to left on the north then south edges, and then top
+ \code{(1:exch2\_nNeighbours(T))} in the order right to left on the
- to bottom on the east and west edges.  Maybe throw in a fig here, eh?
+ north then south edges, and then top to bottom on the east then west
- \\
+ edges.  \\
- The \code{exch2\_opposingSend\_record(a,T)} array holds the index
+  The \code{exch2\_opposingSend\_record(a,T)} array holds the
- \code{b} in \texttt{exch2\_neighbourId(b,Tn)} that holds the tile
+ index \code{b} of the element in \texttt{exch2\_neighbourId(b,Tn)}
- number \code{T}.  In other words,
+ that holds the tile number \code{T}, given
+ \code{Tn=exch2\_neighborId(a,T)}.  In other words,
  \begin{verbatim}
     exch2_neighbourId( exch2_opposingSend_record(a,T),
                        exch2_neighbourId(a,T) ) = T
  \end{verbatim}
  This provides a back-reference from the neighbor tiles. \\
- The arrays \varlink{exch2\_pi}{exch2_pi},
+ The arrays \varlink{exch2\_pi}{exch2_pi} and
- \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
+ \varlink{exch2\_pj}{exch2_pj} specify the transformations of indices
- \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
+ in exchanges between the neighboring tiles.  These transformations are
- \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
+ necessary in exchanges between subdomains because a horizontal dimension
- exchanges between the neighboring tiles.  The dimensions of
+ in one subdomain
- \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)} are the neighbor
+ may map to other horizonal dimension in an adjacent subdomain, and
- ID \code{N} and the tile number \code{T} as explained above, plus a
+ may also have its indexing reversed. This swapping arises from the
- vector of length 2 containing transformation factors \code{t}.  The
+ ``folding'' of two-dimensional arrays into a three-dimensional
- first element of the transformation vector indicates the factor
+ cube. \\
- \code{t} by which variables representing the same vector component of
- a tile \code{T} will be multiplied in exchanges with neighbor
+ The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)}
- \code{N}, and the second element indicates the transform to the
+ are the neighbor ID \code{N} and the tile number \code{T} as explained
- variable in the other direction.  As an example,
+ above, plus a vector of length \code{2} containing transformation
- \code{exch2\_pi(1,N,T)} holds the transform of the $i$ component of a
+ factors \code{t}.  The first element of the transformation vector
- vector variable in tile \code{T} to the $i$ component of tile
+ holds the factor to multiply the index in the same dimension, and the
- \code{T}'s neighbor \code{N}, and \code{exch2\_pi(2,N,T)} hold the
+ second element holds the the same for the orthogonal dimension.  To
- component of neighbor \code{N}'s $j$ component. \\
+ clarify, \code{exch2\_pi(1,N,T)} holds the mapping of the $x$ axis
+ index of tile \code{T} to the $x$ axis of tile \code{T}'s neighbor
+ \code{N}, and \code{exch2\_pi(2,N,T)} holds the mapping of \code{T}'s
+ $x$ index to the neighbor \code{N}'s $y$ index. \\
- Under the current cube topology, one of the two elements of
+ One of the two elements of \code{exch2\_pi} or \code{exch2\_pj} for a
- \code{exch2\_pi} or \code{exch2\_pj} for a given tile \code{T} and
+ given tile \code{T} and neighbor \code{N} will be \code{0}, reflecting
- neighbor \code{N} will be \code{0}, reflecting the fact that the two
+ the fact that the two axes are orthogonal.  The other element will be
- vector components are orthogonal.  The other element will be 1 or -1,
+ \code{1} or \code{-1}, depending on whether the axes are indexed in
- depending on whether the components are indexed in the same or
+ the same or opposite directions.  For example, the transform vector of
- opposite directions.  For example, the transform vector of the arrays
+ the arrays for all tile neighbors on the same subdomain will be
- for all tile neighbors on the same subdomain will be \code{(1,0)},
+ \code{(1,0)}, since all tiles on the same subdomain are oriented
- since all tiles on the same subdomain are oriented identically.  A
+ identically.  An axis that corresponds to the orthogonal dimension
- vector direction that corresponds to the orthogonal dimension with the
+ with the same index direction in a particular tile-neighbor
- same index direction in a particular tile-neighbor orientation will
+ orientation will have \code{(0,1)}.  Those with the opposite index
- have \code{(0,1)}, whereas those in the opposite index direction will
+ direction will have \code{(0,-1)} in order to reverse the ordering. \\
- have \code{(0,-1)}.  This needs some diagrams.
+ The arrays \varlink{exch2\_oi}{exch2_oi},
+ \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
+ \varlink{exch2\_oj\_f}{exch2_oj_f} are indexed to tile number and
+ neighbor and specify the relative offset within the subdomain of the
+ array index of a variable going from a neighboring tile \code{N} to a
+ local tile \code{T}.  Consider \code{T=1} in the six-tile topology
+ (Fig. \ref{fig:6tile}), where
- {\footnotesize
  \begin{verbatim}
- C      exch2_pi          :: X index row of target to source permutation
+        exch2_oi(1,1)=33
- C                        :: matrix for each neighbour entry.
+        exch2_oi(2,1)=0
- C      exch2_pj          :: Y index row of target to source permutation
+        exch2_oi(3,1)=32
- C                        :: matrix for each neighbour entry.
+        exch2_oi(4,1)=-32
- C      exch2_oi          :: X index element of target to source
- C                        :: offset vector for cell-centered quantities
- C                        :: of each neighbor entry.
- C      exch2_oj          :: Y index element of target to source
- C                        :: offset vector for cell-centered quantities
- C                        :: of each neighbor entry.
- C      exch2_oi_f        :: X index element of target to source
- C                        :: offset vector for face quantities
- C                        :: of each neighbor entry.
- C      exch2_oj_f        :: Y index element of target to source
- C                        :: offset vector for face quantities
- C                        :: of each neighbor entry.
  \end{verbatim}
- }
+ The simplest case is \code{exch2\_oi(2,1)}, the southern neighbor,
+ which is \code{Tn=6}.  The axes of \code{T} and \code{Tn} have the
+ same orientation and their $x$ axes have the same origin, and so an
+ exchange between the two requires no changes to the $x$ index.  For
+ the western neighbor (\code{Tn=5}), \code{code\_oi(3,1)=32} since the
+ \code{x=0} vector on \code{T} corresponds to the \code{y=32} vector on
+ \code{Tn}.  The eastern edge of \code{T} shows the reverse case
+ (\code{exch2\_oi(4,1)=-32)}), where \code{x=32} on \code{T} exchanges
+ with \code{x=0} on \code{Tn=2}. \\
+  The most interesting case, where \code{exch2\_oi(1,1)=33} and
+ \code{Tn=3}, involves a reversal of indices.  As in every case, the
+ offset \code{exch2\_oi} is added to the original $x$ index of \code{T}
+ multiplied by the transformation factor \code{exch2\_pi(t,N,T)}.  Here
+ \code{exch2\_pi(1,1,1)=0} since the $x$ axis of \code{T} is orthogonal
+ to the $x$ axis of \code{Tn}.  \code{exch2\_pi(2,1,1)=-1} since the
+ $x$ axis of \code{T} corresponds to the $y$ axis of \code{Tn}, but the
+ index is reversed.  The result is that the index of the northern edge
+ of \code{T}, which runs \code{(1:32)}, is transformed to
+ \code{(-1:-32)}. \code{exch2\_oi(1,1)} is then added to this range to
+ get back \code{(32:1)} -- the index of the $y$ axis of \code{Tn}
+ relative to \code{T}.  This transformation may seem overly convoluted
+ for the six-tile case, but it is necessary to provide a general
+ solution for various topologies. \\
- \subsection{Key Routines}
+ Finally, \varlink{exch2\_itlo\_c}{exch2_itlo_c},
+ \varlink{exch2\_ithi\_c}{exch2_ithi_c},
+ \varlink{exch2\_jtlo\_c}{exch2_jtlo_c} and
+ \varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index
+ bounds of the edge segment of the neighbor tile \code{N}'s subdomain
+ that gets exchanged with the local tile \code{T}.  To take the example
+ of tile \code{T=2} in the twelve-tile topology
+ (Fig. \ref{fig:12tile}): \\
+ \begin{verbatim}
+        exch2_itlo_c(4,2)=17
+        exch2_ithi_c(4,2)=17
+        exch2_jtlo_c(4,2)=0
+        exch2_jthi_c(4,2)=33
+ \end{verbatim}
+ Here \code{N=4}, indicating the western neighbor, which is
+ \code{Tn=1}.  \code{Tn} resides on the same subdomain as \code{T}, so
+ the tiles have the same orientation and the same $x$ and $y$ axes.
+ The $x$ axis is orthogonal to the western edge and the tile is 16
+ points wide, so \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c}
+ indicate the column beyond \code{Tn}'s eastern edge, in that tile's
+ halo region. Since the border of the tiles extends through the entire
+ height of the subdomain, the $y$ axis bounds \code{exch2\_jtlo\_c} to
+ \code{exch2\_jthi\_c} cover the height of \code{(1:32)}, plus 1 in
+ either direction to cover part of the halo. \\
+ For the north edge of the same tile \code{T=2} where \code{N=1} and
+ the neighbor tile is \code{Tn=5}:
+ \begin{verbatim}
+        exch2_itlo_c(1,2)=0
+        exch2_ithi_c(1,2)=0
+        exch2_jtlo_c(1,2)=0
+        exch2_jthi_c(1,2)=17
+ \end{verbatim}
+ \code{T}'s northern edge is parallel to the $x$ axis, but since
+ \code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis, \code{T}'s
+ northern edge exchanges with \code{Tn}'s western edge.  The western
+ edge of the tiles corresponds to the lower bound of the $x$ axis, so
+ \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} are \code{0}, in the
+ western halo region of \code{Tn}. The range of
+ \code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the
+ width of \code{T}'s northern edge, expanded by one into the halo. \\
+ \subsection{Key Routines}
+ Most of the subroutines particular to exch2 handle the exchanges
+ themselves and are of the same format as those described in
+ \ref{sect:cube_sphere_communication} \sectiontitle{Cube sphere
+ communication}.  Like the original routines, they are written as
+ templates which the local Makefile converts from \code{RX} into
+ \code{RL} and \code{RS} forms. \\
+ The interfaces with the core model subroutines are
+ \code{EXCH\_UV\_XY\_RX}, \code{EXCH\_UV\_XYZ\_RX} and
+ \code{EXCH\_XY\_RX}.  They override the standard exchange routines
+ when \code{genmake2} is run with \code{exch2} option.  They in turn
+ call the local exch2 subroutines \code{EXCH2\_UV\_XY\_RX} and
+ \code{EXCH2\_UV\_XYZ\_RX} for two and three-dimensional vector
+ quantities, and \code{EXCH2\_XY\_RX} and \code{EXCH2\_XYZ\_RX} for two
+ and three-dimensional scalar quantities.  These subroutines set the
+ dimensions of the area to be exchanged, call \code{EXCH2\_RX1\_CUBE}
+ for scalars and \code{EXCH2\_RX2\_CUBE} for vectors, and then handle
+ the singularities at the cube corners. \\
+ The separate scalar and vector forms of \code{EXCH2\_RX1\_CUBE} and
+ \code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subroutine
+ needs to pass both the $u$ and $v$ components of the physical vectors.
+ This swapping arises from the topological folding discussed above, where the
+ $x$ and $y$ axes get swapped in some cases, and is not an
+ issue with the scalar case. These subroutines call
+ \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of
+ the work using the variables discussed above. \\
- \subsection{References}

 Legend:



Removed from v.1.12
 


changed lines


 
Added in v.1.22
 Legend:



Removed from v.1.12
 


changed lines


 
Added in v.1.22
-Removed from v.1.12
+Added in v.1.22

	ViewVC Help
Powered by ViewVC 1.1.22