/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.13 by afe, Wed Mar 17 19:49:22 2004 UTC revision 1.18 by afe, Thu May 6 15:21:01 2004 UTC
# Line 16  Line 16 
16    
17  \subsection{Introduction}  \subsection{Introduction}
18    
19  The \texttt{exch2} package extends the original cubed  The \texttt{exch2} package extends the original cubed sphere topology
20  sphere topology configuration to allow more flexible domain  configuration to allow more flexible domain decomposition and
21  decomposition and parallelization.  Cube faces (also called  parallelization.  Cube faces (also called subdomains) may be divided
22  subdomains) may be divided into any number of tiles that divide evenly  into any number of tiles that divide evenly into the grid point
23  into the grid point dimensions of the subdomain.  Furthermore, the  dimensions of the subdomain.  Furthermore, the tiles can run on
24  individual tiles can run on separate processors in different  separate processors individually or in groups, which provides for
25  combinations, and whether exchanges between particular tiles occur  manual compile-time load balancing across a relatively arbitrary
26  between different processors is determined at runtime.  This  number of processors. \\
 flexibility provides for manual compile-time load balancing across a  
 relatively arbitrary number of processors. \\  
27    
28  The exchange parameters are declared in  The exchange parameters are declared in
29  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
# Line 34  and assigned in Line 32  and assigned in
32  validity of the cube topology depends on the \file{SIZE.h} file as  validity of the cube topology depends on the \file{SIZE.h} file as
33  detailed below.  The default files provided in the release configure a  detailed below.  The default files provided in the release configure a
34  cubed sphere topology of six tiles, one per subdomain, each with  cubed sphere topology of six tiles, one per subdomain, each with
35  32$\times$32 grid points, all running on a single processor.  Both  32$\times$32 grid points, with all tiles running on a single processor.  Both
36  files are generated by Matlab scripts in  files are generated by Matlab scripts in
37  \file{utils/exch2/matlab-topology-generator}; see Section  \file{utils/exch2/matlab-topology-generator}; see Section
38  \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}  \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
# Line 55  $\bullet$ The exch2 package is included Line 53  $\bullet$ The exch2 package is included
53    details. \\    details. \\
54    
55  $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and  $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
56    \file{w2\_e2setup.F} must reside in a directory containing code    \file{w2\_e2setup.F} must reside in a directory containing files
57    linked when \file{genmake2} runs.  The safest place to put these    symbolically linked when \file{genmake2} runs.  The safest place to
58    is the directory indicated in the \code{-mods=DIR} command line    put these is the directory indicated in the \code{-mods=DIR} command
59    modifier (typically \file{../code}), or the build directory.  The    line modifier (typically \file{../code}), or the build directory.
60    default versions of these files reside in \file{pkg/exch2} and are    The default versions of these files reside in \file{pkg/exch2} and
61    linked automatically if no other versions exist elsewhere in the    are linked automatically if no other versions exist elsewhere in the
62    link path, but they should be left untouched to avoid breaking    build path, but they should be left untouched to avoid breaking
63    configurations other than the one you intend to modify.\\    configurations other than the one you intend to modify.\\
64    
65  $\bullet$ Files containing grid parameters, named  $\bullet$ Files containing grid parameters, named
66    \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain),    \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain),
67    must be in the working directory when the MITgcm executable is run.    must be in the working directory when the MITgcm executable is run.
68    These files are provided in the example experiments for cubed sphere    These files are provided in the example experiments for cubed sphere
69    configurations with 32$\times$32 cube sides and are non-trivial to    configurations with 32$\times$32 cube sides
70    generate -- please contact MITgcm support if you want to generate    -- please contact MITgcm support if you want to generate
71    files for other configurations. \\    files for other configurations. \\
72    
73  $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must  $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must
# Line 84  $\bullet$ As always when compiling MITgc Line 82  $\bullet$ As always when compiling MITgc
82    Section \ref{sect:specifying_a_decomposition}    Section \ref{sect:specifying_a_decomposition}
83    \sectiontitle{Specifying a decomposition}.\\    \sectiontitle{Specifying a decomposition}.\\
84    
85  As of the time of writing the following examples use exch2 and may be  At the time of this writing the following examples use exch2 and may
86  used for guidance:  be used for guidance:
87    
88  \begin{verbatim}  \begin{verbatim}
89  verification/adjust_nlfs.cs-32x32x1  verification/adjust_nlfs.cs-32x32x1
# Line 108  m-file Line 106  m-file
106  from the Matlab prompt (there are no parameters to pass) generates  from the Matlab prompt (there are no parameters to pass) generates
107  exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and  exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and
108  \file{w2\_e2setup.F} in the working directory and displays a figure of  \file{w2\_e2setup.F} in the working directory and displays a figure of
109  the topology via Matlab.  The other m-files in the directory are  the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:12tile},
110    and \ref{fig:24tile} are examples.  The other m-files in the directory are
111  subroutines of \file{driver.m} and should not be run ``bare'' except  subroutines of \file{driver.m} and should not be run ``bare'' except
112  for development purposes. \\  for development purposes. \\
113    
# Line 119  generated configuration are \code{nr}, \ Line 118  generated configuration are \code{nr}, \
118  The first three determine the size of the subdomains and  The first three determine the size of the subdomains and
119  hence the size of the overall domain.  Each one determines the number  hence the size of the overall domain.  Each one determines the number
120  of grid points, and therefore the resolution, along the subdomain  of grid points, and therefore the resolution, along the subdomain
121  sides in a ``great circle'' around an axis of the cube.  At the time  sides in a ``great circle'' around each the three spatial axes of the cube.  At the time
122  of this writing MITgcm requires these three parameters to be equal,  of this writing MITgcm requires these three parameters to be equal,
123  but they provide for future releases  to accomodate different  but they provide for future releases  to accomodate different
124  resolutions around the axes to allow (for example) greater resolution  resolutions around the axes to allow (for example) greater resolution
125  around the equator.\\  around the equator.\\
126    
127  The parameters \code{tnx} and \code{tny} determine the dimensions of  The parameters \code{tnx} and \code{tny} determine the width and height of
128  the tiles into which the subdomains are decomposed, and must evenly  the tiles into which the subdomains are decomposed, and must evenly
129  divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.  divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
130  The result is a rectangular tiling of the subdomain.  Figure  The result is a rectangular tiling of the subdomain.  Figure
131  \ref{fig:24tile} shows one possible topology for a twentyfour-tile  \ref{fig:24tile} shows one possible topology for a twenty-four-tile
132  cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\  cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
133    
134  \begin{figure}  \begin{figure}
# Line 140  cube, and figure \ref{fig:12tile} shows Line 139  cube, and figure \ref{fig:12tile} shows
139  \end{center}  \end{center}
140    
141  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
142  divided into six 32$\times$32 subdomains, each of which is divided into four tiles  divided into six 32$\times$32 subdomains, each of which is divided
143  (\code{tnx=16, tny=16}) for a total of twentyfour tiles.  into four tiles of width \code{tnx=16} and height \code{tny=16} for a
144  } \label{fig:24tile}  total of twenty-four tiles.  The colored borders of the subdomains
145    represent the parameters \code{nr} (red), \code{nb} (blue), and
146    \code{ng} (green).  } \label{fig:24tile}
147  \end{figure}  \end{figure}
148    
149  \begin{figure}  \begin{figure}
# Line 206  levels in the model.\\ Line 207  levels in the model.\\
207  The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},  The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},
208  \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of  \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of
209  tiles and how they are distributed on processors.  When using exch2,  tiles and how they are distributed on processors.  When using exch2,
210  the tiles are stored in single dimension, and so  the tiles are stored in a single dimension, and so
211  \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as  \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as
212  configured by exch2 cannot be split up accross processors without  configured by exch2 cannot be split up accross processors without
213  regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\  regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\
# Line 237  one processor: \\ Line 238  one processor: \\
238       &           Nr  =   5)       &           Nr  =   5)
239  \end{verbatim}  \end{verbatim}
240    
241  The following is an example for the twentyfour-tile topology in figure  The following is an example for the twenty-four-tile topology in
242  \ref{fig:24tile} running on six processors:  figure \ref{fig:24tile} running on six processors:
243    
244  \begin{verbatim}  \begin{verbatim}
245        PARAMETER (        PARAMETER (
# Line 262  The following is an example for the twen Line 263  The following is an example for the twen
263  \subsection{Key Variables}  \subsection{Key Variables}
264    
265  The descriptions of the variables are divided up into scalars,  The descriptions of the variables are divided up into scalars,
266  one-dimensional arrays indexed to the tile number, and two and three  one-dimensional arrays indexed to the tile number, and two and
267  dimensional arrays indexed to tile number and neighboring tile.  This  three-dimensional arrays indexed to tile number and neighboring tile.
268  division reflects the functionality of these variables: The  This division reflects the functionality of these variables: The
269  scalars are common to every part of the topology, the tile-indexed  scalars are common to every part of the topology, the tile-indexed
270  arrays to individual tiles, and the arrays indexed by tile and  arrays to individual tiles, and the arrays indexed by tile and
271  neighbor to relationships between tiles and their neighbors. \\  neighbor to relationships between tiles and their neighbors. \\
# Line 281  generated by \file{driver.m}.\\ Line 282  generated by \file{driver.m}.\\
282  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
283  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
284  of tiles in the $x$ and $y$ global indices.  For example, the default  of tiles in the $x$ and $y$ global indices.  For example, the default
285  setup of six tiles (Fig. \ref{fig:6tile}) has \code{exch2\_domain\_nxt=6} and  setup of six tiles (Fig. \ref{fig:6tile}) has
286  \code{exch2\_domain\_nyt=1}.  A topology of twenty-four square tiles,  \code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}.  A
287  four per subdomain (as in figure \ref{fig:24tile}), will have  topology of twenty-four square tiles, four per subdomain (as in figure
288  \code{exch2\_domain\_nxt=12} and \code{exch2\_domain\_nyt=2}.  Note  \ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and
289  that these parameters express the tile layout to allow global data  \code{exch2\_domain\_nyt=2}.  Note that these parameters express the
290  files that are tile-layout-neutral and have no bearing on the internal  tile layout to allow global data files that are tile-layout-neutral
291  storage of the arrays.  The tiles are internally stored in a range  and have no bearing on the internal storage of the arrays.  The tiles
292  from \code{(1:\varlink{bi}{bi})} the $x$ axis, and $y$ axis variable  are stored internally in a range from \code{(1:\varlink{bi}{bi})} the
293  \varlink{bj}{bj} is generally ignored within the package. \\  $x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to
294    equal \code{1} throughout the package. \\
295    
296  \subsubsection{Arrays Indexed to Tile Number}  \subsubsection{Arrays Indexed to Tile Number}
297    
298  The following arrays are of length \code{NTILES}, are indexed to the  The following arrays are of length \code{NTILES} and are indexed to
299  tile number, and the indices are omitted in their descriptions. \\  the tile number, which is indicated in the diagrams with the notation
300    \textsf{t}$n$.  The indices are omitted in the descriptions. \\
301    
302  The arrays \varlink{exch2\_tnx}{exch2_tnx} and  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
303  \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of  \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of
304  each tile.  At present for each tile \texttt{exch2\_tnx=sNx} and  each tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
305  \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in  \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in
306  section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and  section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
307  multiprocessing}.  Future releases of MITgcm are to allow varying tile  multiprocessing}.  Future releases of MITgcm may allow varying tile
308  sizes. \\  sizes. \\
309    
310  The location of the tiles' Cartesian origin within a subdomain are  The location of the tiles' Cartesian origin within a subdomain are
# Line 310  determined by the arrays \varlink{exch2\ Line 313  determined by the arrays \varlink{exch2\
313  relate the location of the edges of different tiles to each other.  As  relate the location of the edges of different tiles to each other.  As
314  an example, in the default six-tile topology (Fig. \ref{fig:6tile})  an example, in the default six-tile topology (Fig. \ref{fig:6tile})
315  each index in these arrays is set to \code{0} since a tile occupies  each index in these arrays is set to \code{0} since a tile occupies
316  its entire subdomain.  The twentyfour-tile case discussed above will  its entire subdomain.  The twenty-four-tile case discussed above will
317  have values of \code{0} or \code{16}, depending on the quadrant the  have values of \code{0} or \code{16}, depending on the quadrant the
318  tile falls within the subdomain.  The elements of the arrays  tile falls within the subdomain.  The elements of the arrays
319  \varlink{exch2\_txglobalo}{exch2_txglobalo} and  \varlink{exch2\_txglobalo}{exch2_txglobalo} and
320  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
321  \varlink{exch2\_tbasex}{exch2_tbasex} and  \varlink{exch2\_tbasex}{exch2_tbasex} and
322  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
323  global address space, similar to that used by global files. \\  global address space, similar to that used by global output and input
324    files. \\
325    
326  The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of  The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of
327  the subdomain of each tile, in a range \code{(1:6)} in the case of the  the subdomain of each tile, in a range \code{(1:6)} in the case of the
328  standard cube topology and indicated by \textbf{\textsf{f}}$n$ in  standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
329  figures \ref{fig:12tile} and  figures \ref{fig:12tile} and
330  \ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}  \ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
331  contains a count of how many neighboring tiles each tile has, and is  contains a count of the neighboring tiles each tile has, and is used
332  used for setting bounds for looping over neighboring tiles.  for setting bounds for looping over neighboring tiles.
333  \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each  \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
334  tile, and is used in interprocess communication.  \\  tile, and is used in interprocess communication.  \\
335    
# Line 334  The arrays \varlink{exch2\_isWedge}{exch Line 338  The arrays \varlink{exch2\_isWedge}{exch
338  \varlink{exch2\_isEedge}{exch2_isEedge},  \varlink{exch2\_isEedge}{exch2_isEedge},
339  \varlink{exch2\_isSedge}{exch2_isSedge}, and  \varlink{exch2\_isSedge}{exch2_isSedge}, and
340  \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the  \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the
341  indexed tile lies on the edge of a subdomain, \code{0} if not.  The  indexed tile lies on the respective edge of a subdomain, \code{0} if
342  values are used within the topology generator to determine the  not.  The values are used within the topology generator to determine
343  orientation of neighboring tiles, and to indicate whether a tile lies  the orientation of neighboring tiles, and to indicate whether a tile
344  on the corner of a subdomain.  The latter case requires special  lies on the corner of a subdomain.  The latter case requires special
345  exchange and numerical handling for the singularities at the eight  exchange and numerical handling for the singularities at the eight
346  corners of the cube. \\  corners of the cube. \\
347    
348    
349  \subsubsection{Arrays Indexed to Tile Number and Neighbor}  \subsubsection{Arrays Indexed to Tile Number and Neighbor}
350    
351  The following arrays are all of size  The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and
352  \code{MAX\_NEIGHBOURS}$\times$\code{NTILES} and describe the  \code{NTILES} and describe the orientations between the the tiles. \\
 orientations between the the tiles. \\  
353    
354  The array \code{exch2\_neighbourId(a,T)} holds the tile number  The array \code{exch2\_neighbourId(a,T)} holds the tile number
355  \code{Tn} for each of the tile number \code{T}'s neighboring tiles  \code{Tn} for each of the tile number \code{T}'s neighboring tiles
356  \code{a}.  The neighbor tiles are indexed \code{(1:MAX\_NEIGHBOURS)}  \code{a}.  The neighbor tiles are indexed
357  in the order right to left on the north then south edges, and then top  \code{(1:exch2\_nNeighbours(T))} in the order right to left on the
358  to bottom on the east and west edges.  Maybe throw in a fig here, eh?  north then south edges, and then top to bottom on the east then west
359  \\  edges.  \\
360    
361  \sloppy   The \code{exch2\_opposingSend\_record(a,T)} array holds the
362  The \code{exch2\_opposingSend\_record(a,T)} array holds the index  index \code{b} of the element in \texttt{exch2\_neighbourId(b,Tn)}
363  \code{b} in \texttt{exch2\_neighbourId(b,Tn)} that holds the tile  that holds the tile number \code{T}, given
364  number \code{T}.  In other words,  \code{Tn=exch2\_neighborId(a,T)}.  In other words,
365  \begin{verbatim}  \begin{verbatim}
366     exch2_neighbourId( exch2_opposingSend_record(a,T),     exch2_neighbourId( exch2_opposingSend_record(a,T),
367                        exch2_neighbourId(a,T) ) = T                        exch2_neighbourId(a,T) ) = T
# Line 366  number \code{T}.  In other words, Line 369  number \code{T}.  In other words,
369  This provides a back-reference from the neighbor tiles. \\  This provides a back-reference from the neighbor tiles. \\
370    
371  The arrays \varlink{exch2\_pi}{exch2_pi} and  The arrays \varlink{exch2\_pi}{exch2_pi} and
372  \varlink{exch2\_pj}{exch2_pj} specify the transformations of variables  \varlink{exch2\_pj}{exch2_pj} specify the transformations of indices
373  in exchanges between the neighboring tiles.  These transformations are  in exchanges between the neighboring tiles.  These transformations are
374  necessary in exchanges between subdomains because a physical vector  necessary in exchanges between subdomains because the array index in
375  component in one direction may map to one in a different direction in  one dimension may map to the other index in an adjacent subdomain, and
376  an adjacent subdomain, and may be have its indexing reversed. This  may be have its indexing reversed. This swapping arises from the
377  swapping arises from the ``folding'' of two-dimensional arrays into a  ``folding'' of two-dimensional arrays into a three-dimensional
378  three-dimensional cube.  cube. \\
379    
380  The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)}  The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)}
381  are the neighbor ID \code{N} and the tile number \code{T} as explained  are the neighbor ID \code{N} and the tile number \code{T} as explained
382  above, plus a vector of length 2 containing transformation factors  above, plus a vector of length \code{2} containing transformation
383  \code{t}.  The first element of the transformation vector indicates  factors \code{t}.  The first element of the transformation vector
384  the factor \code{t} by which variables representing the same  holds the factor to multiply the index in the same axis, and the
385  \emph{physical} vector component of a tile \code{T} will be multiplied  second element holds the the same for the orthogonal index.  To
386  in exchanges with neighbor \code{N}, and the second element indicates  clarify, \code{exch2\_pi(1,N,T)} holds the mapping of the $x$ axis
387  the transform to the physical vector in the other direction.  To  index of tile \code{T} to the $x$ axis of tile \code{T}'s neighbor
388  clarify (hopefully), \code{exch2\_pi(1,N,T)} holds the transform of  \code{N}, and \code{exch2\_pi(2,N,T)} holds the mapping of \code{T}'s
389  the $i$ component of a vector variable in tile \code{T} to the $i$  $x$ index to the neighbor \code{N}'s $y$ index. \\
 component of tile \code{T}'s neighbor \code{N}, and  
 \code{exch2\_pi(2,N,T)} holds the transform of \code{T}'s $i$  
 components to the neighbor \code{N}'s $j$ component. \\  
390    
391  Under the current cube topology, one of the two elements of  One of the two elements of \code{exch2\_pi} or \code{exch2\_pj} for a
392  \code{exch2\_pi} or \code{exch2\_pj} for a given tile \code{T} and  given tile \code{T} and neighbor \code{N} will be \code{0}, reflecting
393  neighbor \code{N} will be \code{0}, reflecting the fact that the two  the fact that the two axes are orthogonal.  The other element will be
394  vector components are orthogonal.  The other element will be \code{1}  \code{1} or \code{-1}, depending on whether the axes are indexed in
395  or \code{-1}, depending on whether the components are indexed in the  the same or opposite directions.  For example, the transform vector of
396  same or opposite directions.  For example, the transform vector of the  the arrays for all tile neighbors on the same subdomain will be
 arrays for all tile neighbors on the same subdomain will be  
397  \code{(1,0)}, since all tiles on the same subdomain are oriented  \code{(1,0)}, since all tiles on the same subdomain are oriented
398  identically.  A vector direction that corresponds to the orthogonal  identically.  An axis that corresponds to the orthogonal dimension
399  dimension with the same index direction in a particular tile-neighbor  with the same index direction in a particular tile-neighbor
400  orientation will have \code{(0,1)}, whereas those in the opposite  orientation will have \code{(0,1)}.  Those in the opposite index
401  index direction will have \code{(0,-1)}. \\  direction will have \code{(0,-1)} in order to reverse the ordering. \\
402    
403    The arrays \varlink{exch2\_oi}{exch2_oi},
 \varlink{exch2\_oi}{exch2_oi},  
404  \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and  \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
405  \varlink{exch2\_oj\_f}{exch2_oj_f}  \varlink{exch2\_oj\_f}{exch2_oj_f} are indexed to tile number and
406    neighbor and specify the relative offset within the subdomain of the
407    array index of a variable going from a neighboring tile \code{N} to a
408    local tile \code{T}.  Consider \code{T=1} in the six-tile topology
409    (Fig. \ref{fig:6tile}), where
410    
411    \begin{verbatim}
412           exch2_oi(1,1)=33
413           exch2_oi(2,1)=0
414           exch2_oi(3,1)=32
415           exch2_oi(4,1)=-32
416    \end{verbatim}
417    
418    The simplest case is \code{exch2\_oi(2,1)}, the southern neighbor,
419    which is \code{Tn=6}.  The axes of \code{T} and \code{Tn} have the
420    same orientation and their $x$ axes have the same origin, and so an
421    exchange between the two requires no changes to the $x$ index.  For
422    the western neighbor (\code{Tn=5}), \code{code\_oi(3,1)=32} since the
423    \code{x=0} vector on \code{T} corresponds to the \code{y=32} vector on
424    \code{Tn}.  The eastern edge of \code{T} shows the reverse case
425    (\code{exch2\_oi(4,1)=-32)}), where \code{x=32} on \code{T} exchanges
426    with \code{x=0} on \code{Tn=2}. \\
427    
428     The most interesting case, where \code{exch2\_oi(1,1)=33} and
429    \code{Tn=3}, involves a reversal of indices.  As in every case, the
430    offset \code{exch2\_oi} is added to the original $x$ index of \code{T}
431    multiplied by the transformation factor \code{exch2\_pi(t,N,T)}.  Here
432    \code{exch2\_pi(1,1,1)=0} since the $x$ axis of \code{T} is orthogonal
433    to the $x$ axis of \code{Tn}.  \code{exch2\_pi(2,1,1)=-1} since the
434    $x$ axis of \code{T} corresponds to the $y$ axis of \code{Tn}, but the
435    index is reversed.  The result is that the index of the northern edge
436    of \code{T}, which runs \code{(1:32)}, is transformed to
437    \code{(-1:-32)}. \code{exch2\_oi(1,1)} is then added to this range to
438    get back \code{(32:1)} -- the index of the $y$ axis of \code{Tn}
439    relative to \code{T}.  This transformation may seem overly convoluted
440    for the six-tile case, but it is necessary to provide a general
441    solution for various topologies. \\
442    
443    
 This needs some diagrams. \\  
444    
445    Finally, \varlink{exch2\_itlo\_c}{exch2_itlo_c},
446    \varlink{exch2\_ithi\_c}{exch2_ithi_c},
447    \varlink{exch2\_jtlo\_c}{exch2_jtlo_c} and
448    \varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index
449    bounds of the edge segment of the neighbor tile \code{N}'s subdomain
450    that gets exchanged with the local tile \code{T}.  To take the example
451    of tile \code{T=2} in the twelve-tile topology
452    (Fig. \ref{fig:12tile}): \\
453    
 {\footnotesize  
454  \begin{verbatim}  \begin{verbatim}
455  C      exch2_pi          :: X index row of target to source permutation         exch2_itlo_c(4,2)=17
456  C                        :: matrix for each neighbour entry.                     exch2_ithi_c(4,2)=17
457  C      exch2_pj          :: Y index row of target to source permutation         exch2_jtlo_c(4,2)=0
458  C                        :: matrix for each neighbour entry.                     exch2_jthi_c(4,2)=33
 C      exch2_oi          :: X index element of target to source  
 C                        :: offset vector for cell-centered quantities    
 C                        :: of each neighbor entry.                      
 C      exch2_oj          :: Y index element of target to source  
 C                        :: offset vector for cell-centered quantities    
 C                        :: of each neighbor entry.                      
 C      exch2_oi_f        :: X index element of target to source  
 C                        :: offset vector for face quantities            
 C                        :: of each neighbor entry.                      
 C      exch2_oj_f        :: Y index element of target to source  
 C                        :: offset vector for face quantities            
 C                        :: of each neighbor entry.                      
459  \end{verbatim}  \end{verbatim}
460  }  
461    Here \code{N=4}, indicating the western neighbor, which is
462    \code{Tn=1}.  \code{Tn} resides on the same subdomain as \code{T}, so
463    the tiles have the same orientation and the same $x$ and $y$ axes.
464    The $x$ axis is orthogonal to the western edge and the tile is 16
465    points wide, so \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c}
466    indicate the column beyond \code{Tn}'s eastern edge, in that tile's
467    halo region. Since the border of the tiles extends through the entire
468    height of the subdomain, the $y$ axis bounds \code{exch2\_jtlo\_c} to
469    \code{exch2\_jthi\_c} cover the height of \code{(1:32)}, plus 1 in
470    either direction to cover part of the halo. \\
471    
472    For the north edge of the same tile \code{T=2} where \code{N=1} and
473    the neighbor tile is \code{Tn=5}:
474    
475    \begin{verbatim}
476           exch2_itlo_c(1,2)=0
477           exch2_ithi_c(1,2)=0
478           exch2_jtlo_c(1,2)=0
479           exch2_jthi_c(1,2)=17
480    \end{verbatim}
481    
482    \code{T}'s northern edge is parallel to the $x$ axis, but since
483    \code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis, \code{T}'s
484    northern edge exchanges with \code{Tn}'s western edge.  The western
485    edge of the tiles corresponds to the lower bound of the $x$ axis, so
486    \code{exch2\_itlo\_c} \code{exch2\_ithi\_c} are \code{0}. The range of
487    \code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the
488    width of \code{T}'s northern edge, plus the halo. \\
489    
 \subsection{Key Routines}  
490    
491    \subsection{Key Routines}
492    
493    Most of the subroutines particular to exch2 handle the exchanges
494    themselves and are of the same format as those described in
495    \ref{sect:cube_sphere_communication} \sectiontitle{Cube sphere
496    communication}.  Like the original routines, they are written as
497    templates which the local Makefile converts from RX into RL and RS
498    forms. \\
499    
500    The interfaces with the core model subroutines are
501    \code{EXCH\_UV\_XY\_RX}, \code{EXCH\_UV\_XYZ\_RX} and
502    \code{EXCH\_XY\_RX}.  They override the standard exchange routines
503    when \code{genmake2} is run with \code{exch2} option.  They in turn
504    call the local exch2 subroutines \code{EXCH2\_UV\_XY\_RX} and
505    \code{EXCH2\_UV\_XYZ\_RX} for two and three-dimensional vector
506    quantities, and \code{EXCH2\_XY\_RX} and \code{EXCH2\_XYZ\_RX} for two
507    and three-dimensional scalar quantities.  These subroutines set the
508    dimensions of the area to be exchanged, call \code{EXCH2\_RX1\_CUBE}
509    for scalars and \code{EXCH2\_RX2\_CUBE} for vectors, and then handle
510    the singularities at the cube corners. \\
511    
512    The separate scalar and vector forms of \code{EXCH2\_RX1\_CUBE} and
513    \code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subrouine
514    needs to pass both the $u$ and $v$ components of the phsical vectors.
515    This arises from the topological folding discussed above, where the
516    $x$ and $y$ axes get swapped in some cases.  This swapping is not an
517    issue with the scalar version. These subroutines call
518    \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of
519    the work using the variables discussed above. \\
520    
 \subsection{References}  

Legend:
Removed from v.1.13  
changed lines
  Added in v.1.18

  ViewVC Help
Powered by ViewVC 1.1.22