/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.8 by edhill, Tue Feb 17 21:58:56 2004 UTC revision 1.24 by molod, Mon Jul 18 20:45:27 2005 UTC
# Line 10  Line 10 
10  %%    o automatically inserted at \section{Reference}  %%    o automatically inserted at \section{Reference}
11    
12    
13  \section{Extended Cubed Sphere Exchange}  \subsection{exch2: Extended Cubed Sphere \mbox{Topology}}
14  \label{sec:exch2}  \label{sec:exch2}
15    
16    
17  \subsection{Introduction}  \subsubsection{Introduction}
18    
19  The \texttt{exch2} package is an extension to the original cubed  The \texttt{exch2} package extends the original cubed sphere topology
20  sphere exchanges to allow more flexible domain decomposition and  configuration to allow more flexible domain decomposition and
21  parallelization.  Cube faces (subdomains) may be divided into whatever  parallelization.  Cube faces (also called subdomains) may be divided
22  number of tiles that divide evenly into the grid point dimensions of  into any number of tiles that divide evenly into the grid point
23  the subdomain.  Furthermore, the individual tiles may be run on  dimensions of the subdomain.  Furthermore, the tiles can run on
24  separate processors in different combinations, and whether exchanges  separate processors individually or in groups, which provides for
25  between particular tiles occur between different processors is  manual compile-time load balancing across a relatively arbitrary
26  determined at runtime.  number of processors. \\
27    
28  The exchange parameters are declared in  The exchange parameters are declared in
29  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
30  and assigned in  and assigned in
31  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}, both in  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
32  the \texttt{pkg/exch2} directory.  The validity of the cube topology  validity of the cube topology depends on the \file{SIZE.h} file as
33  depends on the \texttt{SIZE.h} file as detailed below.  Both files are  detailed below.  The default files provided in the release configure a
34  generated by Matlab scripts and should not be edited.  The default  cubed sphere topology of six tiles, one per subdomain, each with
35  files provided in the release set up a cube sphere arrangement of six  32$\times$32 grid points, with all tiles running on a single processor.  Both
36  tiles, one per subdomain, each with 32x32 grid points, running on a  files are generated by Matlab scripts in
37  single processor.  \file{utils/exch2/matlab-topology-generator}; see Section
38    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
39    for details on creating alternate topologies.  Pregenerated examples
40    of these files with alternate topologies are provided under
41    \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
42    file for single-processor execution.
43    
44    \subsubsection{Invoking exch2}
45    
46    To use exch2 with the cubed sphere, the following conditions must be
47    met: \\
48    
49    $\bullet$ The exch2 package is included when \file{genmake2} is run.
50      The easiest way to do this is to add the line \code{exch2} to the
51      \file{profile.conf} file -- see Section
52      \ref{sect:buildingCode} \sectiontitle{Building the code} for general
53      details. \\
54    
55    $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
56      \file{w2\_e2setup.F} must reside in a directory containing files
57      symbolically linked by the \file{genmake2} script.  The safest place to
58      put these is the directory indicated in the \code{-mods=DIR} command
59      line modifier (typically \file{../code}), or the build directory.
60      The default versions of these files reside in \file{pkg/exch2} and
61      are linked automatically if no other versions exist elsewhere in the
62      build path, but they should be left untouched to avoid breaking
63      configurations other than the one you intend to modify.\\
64    
65    $\bullet$ Files containing grid parameters, named
66      \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain),
67      must be in the working directory when the MITgcm executable is run.
68      These files are provided in the example experiments for cubed sphere
69      configurations with 32$\times$32 cube sides
70      -- please contact MITgcm support if you want to generate
71      files for other configurations. \\
72    
73    $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must
74      be placed where \file{genmake2} will find it.  In particular for
75      exch2, the domain decomposition specified in \file{SIZE.h} must
76      correspond with the particular configuration's topology specified in
77      \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain
78      decomposition issues particular to exch2 are addressed in Section
79      \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
80      and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and Multiprocessing}; a more
81      general background on the subject relevant to MITgcm is presented in
82      Section \ref{sect:specifying_a_decomposition}
83      \sectiontitle{Specifying a decomposition}.\\
84    
85  \subsection{Key Variables}  At the time of this writing the following examples use exch2 and may
86    be used for guidance:
87    
88    \begin{verbatim}
89    verification/adjust_nlfs.cs-32x32x1
90    verification/adjustment.cs-32x32x1
91    verification/aim.5l_cs
92    verification/global_ocean.cs32x15
93    verification/hs94.cs-32x32x5
94    \end{verbatim}
95    
96    
97    
98    
99    \subsubsection{Generating Topology Files for exch2}
100    \label{sec:topogen}
101    
102    Alternate cubed sphere topologies may be created using the Matlab
103    scripts in \file{utils/exch2/matlab-topology-generator}. Running the
104    m-file
105    \filelink{driver.m}{utils-exch2-matlab-topology-generator_driver.m}
106    from the Matlab prompt (there are no parameters to pass) generates
107    exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and
108    \file{w2\_e2setup.F} in the working directory and displays a figure of
109    the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:12tile},
110    and \ref{fig:24tile} are examples of the generated diagrams.  The other
111    m-files in the directory are
112    subroutines called from \file{driver.m} and should not be run ``bare'' except
113    for development purposes. \\
114    
115    The parameters that determine the dimensions and topology of the
116    generated configuration are \code{nr}, \code{nb}, \code{ng},
117    \code{tnx} and \code{tny}, and all are assigned early in the script. \\
118    
119    The first three determine the height and width of the subdomains and
120    hence the size of the overall domain.  Each one determines the number
121    of grid points, and therefore the resolution, along the subdomain
122    sides in a ``great circle'' around each the three spatial axes of the cube.  At the time
123    of this writing MITgcm requires these three parameters to be equal,
124    but they provide for future releases  to accomodate different
125    resolutions around the axes to allow subdomains with differing resolutions.\\
126    
127    The parameters \code{tnx} and \code{tny} determine the width and height of
128    the tiles into which the subdomains are decomposed, and must evenly
129    divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
130    The result is a rectangular tiling of the subdomain.  Figure
131    \ref{fig:24tile} shows one possible topology for a twenty-four-tile
132    cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
133    
134    \begin{figure}
135    \begin{center}
136     \resizebox{4in}{!}{
137      \includegraphics{part6/s24t_16x16.ps}
138     }
139    \end{center}
140    
141    \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
142    divided into six 32$\times$32 subdomains, each of which is divided
143    into four tiles of width \code{tnx=16} and height \code{tny=16} for a
144    total of twenty-four tiles.  The colored borders of the subdomains
145    represent the parameters \code{nr} (red), \code{nb} (blue), and
146    \code{ng} (green).  } \label{fig:24tile}
147    \end{figure}
148    
149    \begin{figure}
150    \begin{center}
151     \resizebox{4in}{!}{
152      \includegraphics{part6/s12t_16x32.ps}
153     }
154    \end{center}
155    \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
156    divided into six 32$\times$32 subdomains of two tiles each
157     (\code{tnx=16, tny=32}).
158    } \label{fig:12tile}
159    \end{figure}
160    
161    \begin{figure}
162    \begin{center}
163     \resizebox{4in}{!}{
164      \includegraphics{part6/s6t_32x32.ps}
165     }
166    \end{center}
167    \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
168    divided into six 32$\times$32 subdomains with one tile each
169    (\code{tnx=32, tny=32}).  This is the default configuration.
170      }
171    \label{fig:6tile}
172    \end{figure}
173    
174    
175    Tiles can be selected from the topology to be omitted from being
176    allocated memory and processors.  This tuning is useful in ocean
177    modeling for omitting tiles that fall entirely on land.  The tiles
178    omitted are specified in the file
179    \filelink{blanklist.txt}{utils-exch2-matlab-topology-generator_blanklist.txt}
180    by their tile number in the topology, separated by a newline. \\
181    
182    
183    
184    
185    \subsubsection{exch2, SIZE.h, and Multiprocessing}
186    \label{sec:exch2mpi}
187    
188    Once the topology configuration files are created, the Fortran
189    \code{PARAMETER}s in \file{SIZE.h} must be configured to match.
190    Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying
191    a decomposition} provides a general description of domain
192    decomposition within MITgcm and its relation to \file{SIZE.h}. The
193    current section specifies constraints that the exch2 package
194    imposes and describes how to enable parallel execution with
195    MPI. \\
196    
197    As in the general case, the parameters \varlink{sNx}{sNx} and
198    \varlink{sNy}{sNy} define the size of the individual tiles, and so
199    must be assigned the same respective values as \code{tnx} and
200    \code{tny} in \file{driver.m}.\\
201    
202    The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy}
203    have no special bearing on exch2 and may be assigned as in the general
204    case. The same holds for \varlink{Nr}{Nr}, the number of vertical
205    levels in the model.\\
206    
207    The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},
208    \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of
209    tiles and how they are distributed on processors.  When using exch2,
210    the tiles are stored in the $x$ dimension, and so
211    \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as
212    configured by exch2 cannot be split up accross processors without
213    regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\
214    
215    The number of tiles MITgcm allocates and how they are distributed
216    between processors depends on \varlink{nPx}{nPx} and
217    \varlink{nSx}{nSx}.  \varlink{nSx}{nSx} is the number of tiles per
218    processor and \varlink{nPx}{nPx} is the number of processors.  The
219    total number of tiles in the topology minus those listed in
220    \file{blanklist.txt} must equal \code{nSx*nPx}.  Note that in order to
221    obtain maximum usage from a given number of processors in some cases,
222    this restriction might entail sharing a processor with a tile that
223    would otherwise be excluded because it is topographically outside of
224    the domain and therefore in \file{blanklist.txt}.  For example,
225    suppose you have five processors and a domain decomposition of
226    thirty-six tiles that allows you to exclude seven tiles.  To evenly
227    distribute the remaining twenty-nine tiles among five processors, you
228    would have to run one ``dummy'' tile to make an even six tiles per
229    processor.  Such dummy tiles are \emph{not} listed in
230    \file{blanklist.txt}.\\
231    
232    
233    The following is an example of \file{SIZE.h} for the twelve-tile
234    configuration illustrated in figure \ref{fig:12tile} running on
235    one processor: \\
236    
237    \begin{verbatim}
238          PARAMETER (
239         &           sNx =  16,
240         &           sNy =  32,
241         &           OLx =   2,
242         &           OLy =   2,
243         &           nSx =  12,
244         &           nSy =   1,
245         &           nPx =   1,
246         &           nPy =   1,
247         &           Nx  = sNx*nSx*nPx,
248         &           Ny  = sNy*nSy*nPy,
249         &           Nr  =   5)
250    \end{verbatim}
251    
252    The following is an example for the twenty-four-tile topology in
253    figure \ref{fig:24tile} running on six processors:
254    
255    \begin{verbatim}
256          PARAMETER (
257         &           sNx =  16,
258         &           sNy =  16,
259         &           OLx =   2,
260         &           OLy =   2,
261         &           nSx =   4,
262         &           nSy =   1,
263         &           nPx =   6,
264         &           nPy =   1,
265         &           Nx  = sNx*nSx*nPx,
266         &           Ny  = sNy*nSy*nPy,
267         &           Nr  =   5)
268    \end{verbatim}
269    
270    
271    
272    
273    
274    \subsubsection{Key Variables}
275    
276  The descriptions of the variables are divided up into scalars,  The descriptions of the variables are divided up into scalars,
277  one-dimensional arrays indexed to the tile number, and two and three  one-dimensional arrays indexed to the tile number, and two and
278  dimensional arrays indexed to tile number and neighboring tile.  This  three-dimensional arrays indexed to tile number and neighboring tile.
279  division actually reflects the functionality of these variables: the  This division reflects the functionality of these variables: The
280  scalars are common to every part of the topology, the tile-indexed  scalars are common to every part of the topology, the tile-indexed
281  arrays to individual tiles, and the arrays indexed to tile and  arrays to individual tiles, and the arrays indexed by tile and
282  neighbor to relationships between tiles and their neighbors.  neighbor to relationships between tiles and their neighbors. \\
283    
284  \subsubsection{Scalars}  Scalars:
285    
286  The number of tiles in a particular topology is set with the parameter  The number of tiles in a particular topology is set with the parameter
287  \texttt{NTILES}, and the maximum number of neighbors of any tiles by  \code{NTILES}, and the maximum number of neighbors of any tiles by
288  \texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the  \code{MAX\_NEIGHBOURS}.  These parameters are used for defining the
289  size of the various one and two dimensional arrays that store tile  size of the various one and two dimensional arrays that store tile
290  parameters indexed to the tile number.  parameters indexed to the tile number and are assigned in the files
291    generated by \file{driver.m}.\\
292    
293  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
294  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
295  of tiles in the x and y global indices.  For example, the default  of tiles in the $x$ and $y$ global indices.  For example, the default
296  setup of six tiles has \texttt{exch2\_domain\_nxt=6} and  setup of six tiles (Fig. \ref{fig:6tile}) has
297  \texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in  \code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}.  A
298  gridpoints) tiles, four (2x2) per subdomain, will have  topology of twenty-four square tiles, four per subdomain (as in figure
299  \texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.  \ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and
300  Note that these parameters express the tile layout to allow global  \code{exch2\_domain\_nyt=2}.  Note that these parameters express the
301  data files that are tile-layout-neutral and have no bearing on the  tile layout in order to allow global data files that are tile-layout-neutral.
302  internal storage of the arrays.  The tiles are internally stored in a  They have no bearing on the internal storage of the arrays.  The tiles
303  range from \texttt{1,bi} (in the x axis) and y-axis variable  are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the
304  \texttt{bj} is generally ignored within the package.  $x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to
305    equal \code{1} throughout the package. \\
306  \subsubsection{Arrays Indexed to Tile Number}  
307    Arrays indexed to tile number:
308  The following arrays are of size \texttt{NTILES}, are indexed to the  
309  tile number, and the indices are omitted in their descriptions.  The following arrays are of length \code{NTILES} and are indexed to
310    the tile number, which is indicated in the diagrams with the notation
311    \textsf{t}$n$.  The indices are omitted in the descriptions. \\
312    
313  The arrays \varlink{exch2\_tnx}{exch2_tnx} and  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
314  \varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each  \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of
315  tile.  At present for each tile \texttt{exch2\_tnx=sNx} and  each tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
316  \texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future  \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in
317  releases of MITgcm are to allow varying tile sizes.  Section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
318    Multiprocessing}.  Future releases of MITgcm may allow varying tile
319  The location of the tiles' Cartesian origin within a subdomain are  sizes. \\
320  determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and  
321  \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to  The arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
322  relate the location of the edges of the tiles to each other.  As an  \varlink{exch2\_tbasey}{exch2_tbasey} determine the tiles'
323  example, in the default six-tile topology (the degenerate case) each  Cartesian origin within a subdomain  
324  index in these arrays are set to 0.  The twenty-four, 32x32 cube face  and locate the edges of different tiles relative to each other.  As
325  case discussed above will have values of 0 or 16, depending on the  an example, in the default six-tile topology (Fig. \ref{fig:6tile})
326  quadrant the tile falls within the subdomain.  The array  each index in these arrays is set to \code{0} since a tile occupies
327  \varlink{exch2\_myFace}{exch2_myFace} contains the number of the  its entire subdomain.  The twenty-four-tile case discussed above will
328  cubeface/subdomain of each tile, numbered 1-6 in the case of the  have values of \code{0} or \code{16}, depending on the quadrant of the
329  standard cube topology.  tile within the subdomain.  The elements of the arrays
330    \varlink{exch2\_txglobalo}{exch2_txglobalo} and
 The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and  
331  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
332  \varlink{exch2\_tbasex}{exch2_tbasex} and  \varlink{exch2\_tbasex}{exch2_tbasex} and
333  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tile edges within the
334  global address space, similar to that used by global files.  global address space, similar to that used by global output and input
335    files. \\
336    
337    The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of
338    the subdomain of each tile, in a range \code{(1:6)} in the case of the
339    standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
340    figures \ref{fig:12tile} and
341    \ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
342    contains a count of the neighboring tiles each tile has, and sets
343    the bounds for looping over neighboring tiles.
344    \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
345    tile, and is used in interprocess communication.  \\
346    
347    
348  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
349  \varlink{exch2\_isEedge}{exch2_isEedge},  \varlink{exch2\_isEedge}{exch2_isEedge},
350  \varlink{exch2\_isSedge}{exch2_isSedge}, and  \varlink{exch2\_isSedge}{exch2_isSedge}, and
351  \varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed  \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the
352  tile lies on the edge of a subdomain, 0 if not.  The values are used  indexed tile lies on the edge of its subdomain, \code{0} if
353  within the topology generator to determine the orientation of  not.  The values are used within the topology generator to determine
354  neighboring tiles and to indicate whether a tile lies on the corner of  the orientation of neighboring tiles, and to indicate whether a tile
355  a subdomain.  The latter case indicates special exchange and numerical  lies on the corner of a subdomain.  The latter case requires special
356  handling for the singularities at the eight corners of the cube.  exchange and numerical handling for the singularities at the eight
357  \varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of  corners of the cube. \\
358  how many neighboring tiles each tile has, and is used for setting  
359  bounds for looping over neighboring tiles.  
360  \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each  Arrays Indexed to Tile Number and Neighbor:
361  tile, and is used in interprocess communication.  
362    The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and
363  \subsubsection{Arrays Indexed to Tile Number and Neighbor}  \code{NTILES} and describe the orientations between the the tiles. \\
364    
365  The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$  The array \code{exch2\_neighbourId(a,T)} holds the tile number
366  \texttt{NTILES} and describe the orientations between the the tiles.  \code{Tn} for each of the tile number \code{T}'s neighboring tiles
367    \code{a}.  The neighbor tiles are indexed
368  The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for  \code{(1:exch2\_nNeighbours(T))} in the order right to left on the
369  each of the $n$ neighboring tiles.  The neighbor tiles are indexed  north then south edges, and then top to bottom on the east then west
370  \texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north  edges.  \\
371  then south edges, and then top to bottom on the east and west edges.  
372  Maybe throw in a fig here, eh?   The \code{exch2\_opposingSend\_record(a,T)} array holds the
373    index \code{b} of the element in \texttt{exch2\_neighbourId(b,Tn)}
374  The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c  that holds the tile number \code{T}, given
375  in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.  \code{Tn=exch2\_neighborId(a,T)}.  In other words,
 In other words,  
376  \begin{verbatim}  \begin{verbatim}
377     exch2_neighbourId( exch2_opposingSend_record(a,T),     exch2_neighbourId( exch2_opposingSend_record(a,T),
378                        exch2_neighbourId(a,T) ) = T                        exch2_neighbourId(a,T) ) = T
379  \end{verbatim}  \end{verbatim}
380  and this provides a back-reference from the neighbor tiles.  This provides a back-reference from the neighbor tiles. \\
381    
382    The arrays \varlink{exch2\_pi}{exch2_pi} and
383    \varlink{exch2\_pj}{exch2_pj} specify the transformations of indices
384    in exchanges between the neighboring tiles.  These transformations are
385    necessary in exchanges between subdomains because a horizontal dimension
386    in one subdomain
387    may map to other horizonal dimension in an adjacent subdomain, and
388    may also have its indexing reversed. This swapping arises from the
389    ``folding'' of two-dimensional arrays into a three-dimensional
390    cube. \\
391    
392    The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)}
393    are the neighbor ID \code{N} and the tile number \code{T} as explained
394    above, plus a vector of length \code{2} containing transformation
395    factors \code{t}.  The first element of the transformation vector
396    holds the factor to multiply the index in the same dimension, and the
397    second element holds the the same for the orthogonal dimension.  To
398    clarify, \code{exch2\_pi(1,N,T)} holds the mapping of the $x$ axis
399    index of tile \code{T} to the $x$ axis of tile \code{T}'s neighbor
400    \code{N}, and \code{exch2\_pi(2,N,T)} holds the mapping of \code{T}'s
401    $x$ index to the neighbor \code{N}'s $y$ index. \\
402    
403    One of the two elements of \code{exch2\_pi} or \code{exch2\_pj} for a
404    given tile \code{T} and neighbor \code{N} will be \code{0}, reflecting
405    the fact that the two axes are orthogonal.  The other element will be
406    \code{1} or \code{-1}, depending on whether the axes are indexed in
407    the same or opposite directions.  For example, the transform vector of
408    the arrays for all tile neighbors on the same subdomain will be
409    \code{(1,0)}, since all tiles on the same subdomain are oriented
410    identically.  An axis that corresponds to the orthogonal dimension
411    with the same index direction in a particular tile-neighbor
412    orientation will have \code{(0,1)}.  Those with the opposite index
413    direction will have \code{(0,-1)} in order to reverse the ordering. \\
414    
415  The arrays \varlink{exch2\_pi}{exch2_pi},  The arrays \varlink{exch2\_oi}{exch2_oi},
 \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},  
416  \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and  \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
417  \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in  \varlink{exch2\_oj\_f}{exch2_oj_f} are indexed to tile number and
418  exchanges between the neighboring tiles.  The dimensions of  neighbor and specify the relative offset within the subdomain of the
419  \texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the  array index of a variable going from a neighboring tile \code{N} to a
420  neighbor ID \textit{N} and the tile number \textit{T} as explained  local tile \code{T}.  Consider \code{T=1} in the six-tile topology
421  above, plus the transformation vector {\em t }, of length two.  The  (Fig. \ref{fig:6tile}), where
 first element of the transformation vector indicates the factor by  
 which variables representing the same vector component of a tile will  
 be multiplied, and the second element indicates the transform to the  
 variable in the other direction.  As an example,  
 \texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a  
 vector variable in tile \texttt{T} to the i-component of tile  
 \texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold  
 the component of neighbor \texttt{N}'s j-component.  
   
 Under the current cube topology, one of the two elements of  
 \texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}  
 and neighbor \texttt{N} will be 0, reflecting the fact that the vector  
 components are orthogonal.  The other element will be 1 or -1,  
 depending on whether the components are indexed in the same or  
 opposite directions.  For example, the transform dimension of the  
 arrays for all tile neighbors on the same subdomain will be [1,0],  
 since all tiles on the same subdomain are oriented identically.  
 Vectors that correspond to the orthogonal dimension with the same  
 index direction will have [0,1], whereas those in the opposite index  
 direction will have [0,-1].  
   
   
 {\footnotesize  
 \begin{verbatim}  
 C      exch2_pi          :: X index row of target to source permutation  
 C                        :: matrix for each neighbour entry.              
 C      exch2_pj          :: Y index row of target to source permutation  
 C                        :: matrix for each neighbour entry.              
 C      exch2_oi          :: X index element of target to source  
 C                        :: offset vector for cell-centered quantities    
 C                        :: of each neighbor entry.                      
 C      exch2_oj          :: Y index element of target to source  
 C                        :: offset vector for cell-centered quantities    
 C                        :: of each neighbor entry.                      
 C      exch2_oi_f        :: X index element of target to source  
 C                        :: offset vector for face quantities            
 C                        :: of each neighbor entry.                      
 C      exch2_oj_f        :: Y index element of target to source  
 C                        :: offset vector for face quantities            
 C                        :: of each neighbor entry.                      
 \end{verbatim}  
 }  
422    
423    \begin{verbatim}
424           exch2_oi(1,1)=33
425           exch2_oi(2,1)=0
426           exch2_oi(3,1)=32
427           exch2_oi(4,1)=-32
428    \end{verbatim}
429    
430    The simplest case is \code{exch2\_oi(2,1)}, the southern neighbor,
431    which is \code{Tn=6}.  The axes of \code{T} and \code{Tn} have the
432    same orientation and their $x$ axes have the same origin, and so an
433    exchange between the two requires no changes to the $x$ index.  For
434    the western neighbor (\code{Tn=5}), \code{code\_oi(3,1)=32} since the
435    \code{x=0} vector on \code{T} corresponds to the \code{y=32} vector on
436    \code{Tn}.  The eastern edge of \code{T} shows the reverse case
437    (\code{exch2\_oi(4,1)=-32)}), where \code{x=32} on \code{T} exchanges
438    with \code{x=0} on \code{Tn=2}. \\
439    
440     The most interesting case, where \code{exch2\_oi(1,1)=33} and
441    \code{Tn=3}, involves a reversal of indices.  As in every case, the
442    offset \code{exch2\_oi} is added to the original $x$ index of \code{T}
443    multiplied by the transformation factor \code{exch2\_pi(t,N,T)}.  Here
444    \code{exch2\_pi(1,1,1)=0} since the $x$ axis of \code{T} is orthogonal
445    to the $x$ axis of \code{Tn}.  \code{exch2\_pi(2,1,1)=-1} since the
446    $x$ axis of \code{T} corresponds to the $y$ axis of \code{Tn}, but the
447    index is reversed.  The result is that the index of the northern edge
448    of \code{T}, which runs \code{(1:32)}, is transformed to
449    \code{(-1:-32)}. \code{exch2\_oi(1,1)} is then added to this range to
450    get back \code{(32:1)} -- the index of the $y$ axis of \code{Tn}
451    relative to \code{T}.  This transformation may seem overly convoluted
452    for the six-tile case, but it is necessary to provide a general
453    solution for various topologies. \\
454    
455    
 \subsection{Key Routines}  
456    
457    Finally, \varlink{exch2\_itlo\_c}{exch2_itlo_c},
458    \varlink{exch2\_ithi\_c}{exch2_ithi_c},
459    \varlink{exch2\_jtlo\_c}{exch2_jtlo_c} and
460    \varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index
461    bounds of the edge segment of the neighbor tile \code{N}'s subdomain
462    that gets exchanged with the local tile \code{T}.  To take the example
463    of tile \code{T=2} in the twelve-tile topology
464    (Fig. \ref{fig:12tile}): \\
465    
466    \begin{verbatim}
467           exch2_itlo_c(4,2)=17
468           exch2_ithi_c(4,2)=17
469           exch2_jtlo_c(4,2)=0
470           exch2_jthi_c(4,2)=33
471    \end{verbatim}
472    
473    Here \code{N=4}, indicating the western neighbor, which is
474    \code{Tn=1}.  \code{Tn} resides on the same subdomain as \code{T}, so
475    the tiles have the same orientation and the same $x$ and $y$ axes.
476    The $x$ axis is orthogonal to the western edge and the tile is 16
477    points wide, so \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c}
478    indicate the column beyond \code{Tn}'s eastern edge, in that tile's
479    halo region. Since the border of the tiles extends through the entire
480    height of the subdomain, the $y$ axis bounds \code{exch2\_jtlo\_c} to
481    \code{exch2\_jthi\_c} cover the height of \code{(1:32)}, plus 1 in
482    either direction to cover part of the halo. \\
483    
484    For the north edge of the same tile \code{T=2} where \code{N=1} and
485    the neighbor tile is \code{Tn=5}:
486    
487    \begin{verbatim}
488           exch2_itlo_c(1,2)=0
489           exch2_ithi_c(1,2)=0
490           exch2_jtlo_c(1,2)=0
491           exch2_jthi_c(1,2)=17
492    \end{verbatim}
493    
494    \code{T}'s northern edge is parallel to the $x$ axis, but since
495    \code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis, \code{T}'s
496    northern edge exchanges with \code{Tn}'s western edge.  The western
497    edge of the tiles corresponds to the lower bound of the $x$ axis, so
498    \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} are \code{0}, in the
499    western halo region of \code{Tn}. The range of
500    \code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the
501    width of \code{T}'s northern edge, expanded by one into the halo. \\
502    
503    
504    \subsubsection{Key Routines}
505    
506    Most of the subroutines particular to exch2 handle the exchanges
507    themselves and are of the same format as those described in
508    \ref{sect:cube_sphere_communication} \sectiontitle{Cube sphere
509    communication}.  Like the original routines, they are written as
510    templates which the local Makefile converts from \code{RX} into
511    \code{RL} and \code{RS} forms. \\
512    
513    The interfaces with the core model subroutines are
514    \code{EXCH\_UV\_XY\_RX}, \code{EXCH\_UV\_XYZ\_RX} and
515    \code{EXCH\_XY\_RX}.  They override the standard exchange routines
516    when \code{genmake2} is run with \code{exch2} option.  They in turn
517    call the local exch2 subroutines \code{EXCH2\_UV\_XY\_RX} and
518    \code{EXCH2\_UV\_XYZ\_RX} for two and three-dimensional vector
519    quantities, and \code{EXCH2\_XY\_RX} and \code{EXCH2\_XYZ\_RX} for two
520    and three-dimensional scalar quantities.  These subroutines set the
521    dimensions of the area to be exchanged, call \code{EXCH2\_RX1\_CUBE}
522    for scalars and \code{EXCH2\_RX2\_CUBE} for vectors, and then handle
523    the singularities at the cube corners. \\
524    
525    The separate scalar and vector forms of \code{EXCH2\_RX1\_CUBE} and
526    \code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subroutine
527    needs to pass both the $u$ and $v$ components of the physical vectors.
528    This swapping arises from the topological folding discussed above, where the
529    $x$ and $y$ axes get swapped in some cases, and is not an
530    issue with the scalar case. These subroutines call
531    \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of
532    the work using the variables discussed above. \\
533    
 \subsection{References}  
Legend:
Removed from v.1.8  
changed lines
  Added in v.1.24
  ViewVC Help
Powered by ViewVC 1.1.22