/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.16 by afe, Thu Mar 18 22:20:38 2004 UTC revision 1.24 by molod, Mon Jul 18 20:45:27 2005 UTC
# Line 10  Line 10 
10  %%    o automatically inserted at \section{Reference}  %%    o automatically inserted at \section{Reference}
11    
12    
13  \section{exch2: Extended Cubed Sphere \mbox{Topology}}  \subsection{exch2: Extended Cubed Sphere \mbox{Topology}}
14  \label{sec:exch2}  \label{sec:exch2}
15    
16    
17  \subsection{Introduction}  \subsubsection{Introduction}
18    
19  The \texttt{exch2} package extends the original cubed  The \texttt{exch2} package extends the original cubed sphere topology
20  sphere topology configuration to allow more flexible domain  configuration to allow more flexible domain decomposition and
21  decomposition and parallelization.  Cube faces (also called  parallelization.  Cube faces (also called subdomains) may be divided
22  subdomains) may be divided into any number of tiles that divide evenly  into any number of tiles that divide evenly into the grid point
23  into the grid point dimensions of the subdomain.  Furthermore, the  dimensions of the subdomain.  Furthermore, the tiles can run on
24  individual tiles can run on separate processors in different  separate processors individually or in groups, which provides for
25  combinations, and whether exchanges between particular tiles occur  manual compile-time load balancing across a relatively arbitrary
26  between different processors is determined at runtime.  This  number of processors. \\
 flexibility provides for manual compile-time load balancing across a  
 relatively arbitrary number of processors. \\  
27    
28  The exchange parameters are declared in  The exchange parameters are declared in
29  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
# Line 34  and assigned in Line 32  and assigned in
32  validity of the cube topology depends on the \file{SIZE.h} file as  validity of the cube topology depends on the \file{SIZE.h} file as
33  detailed below.  The default files provided in the release configure a  detailed below.  The default files provided in the release configure a
34  cubed sphere topology of six tiles, one per subdomain, each with  cubed sphere topology of six tiles, one per subdomain, each with
35  32$\times$32 grid points, all running on a single processor.  Both  32$\times$32 grid points, with all tiles running on a single processor.  Both
36  files are generated by Matlab scripts in  files are generated by Matlab scripts in
37  \file{utils/exch2/matlab-topology-generator}; see Section  \file{utils/exch2/matlab-topology-generator}; see Section
38  \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}  \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
# Line 43  of these files with alternate topologies Line 41  of these files with alternate topologies
41  \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}  \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
42  file for single-processor execution.  file for single-processor execution.
43    
44  \subsection{Invoking exch2}  \subsubsection{Invoking exch2}
45    
46  To use exch2 with the cubed sphere, the following conditions must be  To use exch2 with the cubed sphere, the following conditions must be
47  met: \\  met: \\
# Line 55  $\bullet$ The exch2 package is included Line 53  $\bullet$ The exch2 package is included
53    details. \\    details. \\
54    
55  $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and  $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
56    \file{w2\_e2setup.F} must reside in a directory containing code    \file{w2\_e2setup.F} must reside in a directory containing files
57    linked when \file{genmake2} runs.  The safest place to put these    symbolically linked by the \file{genmake2} script.  The safest place to
58    is the directory indicated in the \code{-mods=DIR} command line    put these is the directory indicated in the \code{-mods=DIR} command
59    modifier (typically \file{../code}), or the build directory.  The    line modifier (typically \file{../code}), or the build directory.
60    default versions of these files reside in \file{pkg/exch2} and are    The default versions of these files reside in \file{pkg/exch2} and
61    linked automatically if no other versions exist elsewhere in the    are linked automatically if no other versions exist elsewhere in the
62    link path, but they should be left untouched to avoid breaking    build path, but they should be left untouched to avoid breaking
63    configurations other than the one you intend to modify.\\    configurations other than the one you intend to modify.\\
64    
65  $\bullet$ Files containing grid parameters, named  $\bullet$ Files containing grid parameters, named
66    \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain),    \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain),
67    must be in the working directory when the MITgcm executable is run.    must be in the working directory when the MITgcm executable is run.
68    These files are provided in the example experiments for cubed sphere    These files are provided in the example experiments for cubed sphere
69    configurations with 32$\times$32 cube sides and are non-trivial to    configurations with 32$\times$32 cube sides
70    generate -- please contact MITgcm support if you want to generate    -- please contact MITgcm support if you want to generate
71    files for other configurations. \\    files for other configurations. \\
72    
73  $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must  $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must
# Line 79  $\bullet$ As always when compiling MITgc Line 77  $\bullet$ As always when compiling MITgc
77    \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain    \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain
78    decomposition issues particular to exch2 are addressed in Section    decomposition issues particular to exch2 are addressed in Section
79    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
80    and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and MPI}; a more    and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and Multiprocessing}; a more
81    general background on the subject relevant to MITgcm is presented in    general background on the subject relevant to MITgcm is presented in
82    Section \ref{sect:specifying_a_decomposition}    Section \ref{sect:specifying_a_decomposition}
83    \sectiontitle{Specifying a decomposition}.\\    \sectiontitle{Specifying a decomposition}.\\
84    
85  As of the time of writing the following examples use exch2 and may be  At the time of this writing the following examples use exch2 and may
86  used for guidance:  be used for guidance:
87    
88  \begin{verbatim}  \begin{verbatim}
89  verification/adjust_nlfs.cs-32x32x1  verification/adjust_nlfs.cs-32x32x1
# Line 98  verification/hs94.cs-32x32x5 Line 96  verification/hs94.cs-32x32x5
96    
97    
98    
99  \subsection{Generating Topology Files for exch2}  \subsubsection{Generating Topology Files for exch2}
100  \label{sec:topogen}  \label{sec:topogen}
101    
102  Alternate cubed sphere topologies may be created using the Matlab  Alternate cubed sphere topologies may be created using the Matlab
# Line 108  m-file Line 106  m-file
106  from the Matlab prompt (there are no parameters to pass) generates  from the Matlab prompt (there are no parameters to pass) generates
107  exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and  exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and
108  \file{w2\_e2setup.F} in the working directory and displays a figure of  \file{w2\_e2setup.F} in the working directory and displays a figure of
109  the topology via Matlab.  The other m-files in the directory are  the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:12tile},
110  subroutines of \file{driver.m} and should not be run ``bare'' except  and \ref{fig:24tile} are examples of the generated diagrams.  The other
111    m-files in the directory are
112    subroutines called from \file{driver.m} and should not be run ``bare'' except
113  for development purposes. \\  for development purposes. \\
114    
115  The parameters that determine the dimensions and topology of the  The parameters that determine the dimensions and topology of the
116  generated configuration are \code{nr}, \code{nb}, \code{ng},  generated configuration are \code{nr}, \code{nb}, \code{ng},
117  \code{tnx} and \code{tny}, and all are assigned early in the script. \\  \code{tnx} and \code{tny}, and all are assigned early in the script. \\
118    
119  The first three determine the size of the subdomains and  The first three determine the height and width of the subdomains and
120  hence the size of the overall domain.  Each one determines the number  hence the size of the overall domain.  Each one determines the number
121  of grid points, and therefore the resolution, along the subdomain  of grid points, and therefore the resolution, along the subdomain
122  sides in a ``great circle'' around an axis of the cube.  At the time  sides in a ``great circle'' around each the three spatial axes of the cube.  At the time
123  of this writing MITgcm requires these three parameters to be equal,  of this writing MITgcm requires these three parameters to be equal,
124  but they provide for future releases  to accomodate different  but they provide for future releases  to accomodate different
125  resolutions around the axes to allow (for example) greater resolution  resolutions around the axes to allow subdomains with differing resolutions.\\
 around the equator.\\  
126    
127  The parameters \code{tnx} and \code{tny} determine the dimensions of  The parameters \code{tnx} and \code{tny} determine the width and height of
128  the tiles into which the subdomains are decomposed, and must evenly  the tiles into which the subdomains are decomposed, and must evenly
129  divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.  divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
130  The result is a rectangular tiling of the subdomain.  Figure  The result is a rectangular tiling of the subdomain.  Figure
131  \ref{fig:24tile} shows one possible topology for a twentyfour-tile  \ref{fig:24tile} shows one possible topology for a twenty-four-tile
132  cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\  cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
133    
134  \begin{figure}  \begin{figure}
# Line 140  cube, and figure \ref{fig:12tile} shows Line 139  cube, and figure \ref{fig:12tile} shows
139  \end{center}  \end{center}
140    
141  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
142  divided into six 32$\times$32 subdomains, each of which is divided into four tiles  divided into six 32$\times$32 subdomains, each of which is divided
143  (\code{tnx=16, tny=16}) for a total of twentyfour tiles.  into four tiles of width \code{tnx=16} and height \code{tny=16} for a
144  } \label{fig:24tile}  total of twenty-four tiles.  The colored borders of the subdomains
145    represent the parameters \code{nr} (red), \code{nb} (blue), and
146    \code{ng} (green).  } \label{fig:24tile}
147  \end{figure}  \end{figure}
148    
149  \begin{figure}  \begin{figure}
# Line 181  by their tile number in the topology, se Line 182  by their tile number in the topology, se
182    
183    
184    
185  \subsection{exch2, SIZE.h, and multiprocessing}  \subsubsection{exch2, SIZE.h, and Multiprocessing}
186  \label{sec:exch2mpi}  \label{sec:exch2mpi}
187    
188  Once the topology configuration files are created, the Fortran  Once the topology configuration files are created, the Fortran
# Line 189  Once the topology configuration files ar Line 190  Once the topology configuration files ar
190  Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying  Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying
191  a decomposition} provides a general description of domain  a decomposition} provides a general description of domain
192  decomposition within MITgcm and its relation to \file{SIZE.h}. The  decomposition within MITgcm and its relation to \file{SIZE.h}. The
193  current section specifies certain constraints the exch2 package  current section specifies constraints that the exch2 package
194  imposes as well as describes how to enable parallel execution with  imposes and describes how to enable parallel execution with
195  MPI. \\  MPI. \\
196    
197  As in the general case, the parameters \varlink{sNx}{sNx} and  As in the general case, the parameters \varlink{sNx}{sNx} and
# Line 206  levels in the model.\\ Line 207  levels in the model.\\
207  The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},  The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},
208  \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of  \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of
209  tiles and how they are distributed on processors.  When using exch2,  tiles and how they are distributed on processors.  When using exch2,
210  the tiles are stored in single dimension, and so  the tiles are stored in the $x$ dimension, and so
211  \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as  \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as
212  configured by exch2 cannot be split up accross processors without  configured by exch2 cannot be split up accross processors without
213  regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\  regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\
# Line 214  regenerating the topology, \code{\varlin Line 215  regenerating the topology, \code{\varlin
215  The number of tiles MITgcm allocates and how they are distributed  The number of tiles MITgcm allocates and how they are distributed
216  between processors depends on \varlink{nPx}{nPx} and  between processors depends on \varlink{nPx}{nPx} and
217  \varlink{nSx}{nSx}.  \varlink{nSx}{nSx} is the number of tiles per  \varlink{nSx}{nSx}.  \varlink{nSx}{nSx} is the number of tiles per
218  processor and \varlink{nPx}{nPx} the number of processors.  The total  processor and \varlink{nPx}{nPx} is the number of processors.  The
219  number of tiles in the topology minus those listed in  total number of tiles in the topology minus those listed in
220  \file{blanklist.txt} must equal \code{nSx*nPx}. \\  \file{blanklist.txt} must equal \code{nSx*nPx}.  Note that in order to
221    obtain maximum usage from a given number of processors in some cases,
222    this restriction might entail sharing a processor with a tile that
223    would otherwise be excluded because it is topographically outside of
224    the domain and therefore in \file{blanklist.txt}.  For example,
225    suppose you have five processors and a domain decomposition of
226    thirty-six tiles that allows you to exclude seven tiles.  To evenly
227    distribute the remaining twenty-nine tiles among five processors, you
228    would have to run one ``dummy'' tile to make an even six tiles per
229    processor.  Such dummy tiles are \emph{not} listed in
230    \file{blanklist.txt}.\\
231    
232    
233  The following is an example of \file{SIZE.h} for the twelve-tile  The following is an example of \file{SIZE.h} for the twelve-tile
234  configuration illustrated in figure \ref{fig:12tile} running on  configuration illustrated in figure \ref{fig:12tile} running on
# Line 237  one processor: \\ Line 249  one processor: \\
249       &           Nr  =   5)       &           Nr  =   5)
250  \end{verbatim}  \end{verbatim}
251    
252  The following is an example for the twentyfour-tile topology in figure  The following is an example for the twenty-four-tile topology in
253  \ref{fig:24tile} running on six processors:  figure \ref{fig:24tile} running on six processors:
254    
255  \begin{verbatim}  \begin{verbatim}
256        PARAMETER (        PARAMETER (
# Line 259  The following is an example for the twen Line 271  The following is an example for the twen
271    
272    
273    
274  \subsection{Key Variables}  \subsubsection{Key Variables}
275    
276  The descriptions of the variables are divided up into scalars,  The descriptions of the variables are divided up into scalars,
277  one-dimensional arrays indexed to the tile number, and two and three  one-dimensional arrays indexed to the tile number, and two and
278  dimensional arrays indexed to tile number and neighboring tile.  This  three-dimensional arrays indexed to tile number and neighboring tile.
279  division reflects the functionality of these variables: The  This division reflects the functionality of these variables: The
280  scalars are common to every part of the topology, the tile-indexed  scalars are common to every part of the topology, the tile-indexed
281  arrays to individual tiles, and the arrays indexed by tile and  arrays to individual tiles, and the arrays indexed by tile and
282  neighbor to relationships between tiles and their neighbors. \\  neighbor to relationships between tiles and their neighbors. \\
283    
284  \subsubsection{Scalars}  Scalars:
285    
286  The number of tiles in a particular topology is set with the parameter  The number of tiles in a particular topology is set with the parameter
287  \code{NTILES}, and the maximum number of neighbors of any tiles by  \code{NTILES}, and the maximum number of neighbors of any tiles by
# Line 286  setup of six tiles (Fig. \ref{fig:6tile} Line 298  setup of six tiles (Fig. \ref{fig:6tile}
298  topology of twenty-four square tiles, four per subdomain (as in figure  topology of twenty-four square tiles, four per subdomain (as in figure
299  \ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and  \ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and
300  \code{exch2\_domain\_nyt=2}.  Note that these parameters express the  \code{exch2\_domain\_nyt=2}.  Note that these parameters express the
301  tile layout to allow global data files that are tile-layout-neutral  tile layout in order to allow global data files that are tile-layout-neutral.
302  and have no bearing on the internal storage of the arrays.  The tiles  They have no bearing on the internal storage of the arrays.  The tiles
303  are internally stored in a range from \code{(1:\varlink{bi}{bi})} the  are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the
304  $x$ axis, and the $y$ axis variable \varlink{bj}{bj} is generally  $x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to
305  ignored within the package. \\  equal \code{1} throughout the package. \\
306    
307  \subsubsection{Arrays Indexed to Tile Number}  Arrays indexed to tile number:
308    
309  The following arrays are of length \code{NTILES}and are indexed to the  The following arrays are of length \code{NTILES} and are indexed to
310  tile number, which is indicated in the diagrams with the notation  the tile number, which is indicated in the diagrams with the notation
311  \textsf{t}$n$.  The indices are omitted in the descriptions. \\  \textsf{t}$n$.  The indices are omitted in the descriptions. \\
312    
313  The arrays \varlink{exch2\_tnx}{exch2_tnx} and  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
314  \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of  \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of
315  each tile.  At present for each tile \texttt{exch2\_tnx=sNx} and  each tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
316  \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in  \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in
317  section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and  Section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
318  multiprocessing}.  Future releases of MITgcm are to allow varying tile  Multiprocessing}.  Future releases of MITgcm may allow varying tile
319  sizes. \\  sizes. \\
320    
321  The location of the tiles' Cartesian origin within a subdomain are  The arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
322  determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and  \varlink{exch2\_tbasey}{exch2_tbasey} determine the tiles'
323  \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to  Cartesian origin within a subdomain  
324  relate the location of the edges of different tiles to each other.  As  and locate the edges of different tiles relative to each other.  As
325  an example, in the default six-tile topology (Fig. \ref{fig:6tile})  an example, in the default six-tile topology (Fig. \ref{fig:6tile})
326  each index in these arrays is set to \code{0} since a tile occupies  each index in these arrays is set to \code{0} since a tile occupies
327  its entire subdomain.  The twentyfour-tile case discussed above will  its entire subdomain.  The twenty-four-tile case discussed above will
328  have values of \code{0} or \code{16}, depending on the quadrant the  have values of \code{0} or \code{16}, depending on the quadrant of the
329  tile falls within the subdomain.  The elements of the arrays  tile within the subdomain.  The elements of the arrays
330  \varlink{exch2\_txglobalo}{exch2_txglobalo} and  \varlink{exch2\_txglobalo}{exch2_txglobalo} and
331  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
332  \varlink{exch2\_tbasex}{exch2_tbasex} and  \varlink{exch2\_tbasex}{exch2_tbasex} and
333  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tile edges within the
334  global address space, similar to that used by global files. \\  global address space, similar to that used by global output and input
335    files. \\
336    
337  The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of  The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of
338  the subdomain of each tile, in a range \code{(1:6)} in the case of the  the subdomain of each tile, in a range \code{(1:6)} in the case of the
339  standard cube topology and indicated by \textbf{\textsf{f}}$n$ in  standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
340  figures \ref{fig:12tile} and  figures \ref{fig:12tile} and
341  \ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}  \ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
342  contains a count the  neighboring tiles each tile has, and is  contains a count of the neighboring tiles each tile has, and sets
343  used for setting bounds for looping over neighboring tiles.  the bounds for looping over neighboring tiles.
344  \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each  \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
345  tile, and is used in interprocess communication.  \\  tile, and is used in interprocess communication.  \\
346    
# Line 336  The arrays \varlink{exch2\_isWedge}{exch Line 349  The arrays \varlink{exch2\_isWedge}{exch
349  \varlink{exch2\_isEedge}{exch2_isEedge},  \varlink{exch2\_isEedge}{exch2_isEedge},
350  \varlink{exch2\_isSedge}{exch2_isSedge}, and  \varlink{exch2\_isSedge}{exch2_isSedge}, and
351  \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the  \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the
352  indexed tile lies on the respective edge of a subdomain, \code{0} if  indexed tile lies on the edge of its subdomain, \code{0} if
353  not.  The values are used within the topology generator to determine  not.  The values are used within the topology generator to determine
354  the orientation of neighboring tiles, and to indicate whether a tile  the orientation of neighboring tiles, and to indicate whether a tile
355  lies on the corner of a subdomain.  The latter case requires special  lies on the corner of a subdomain.  The latter case requires special
# Line 344  exchange and numerical handling for the Line 357  exchange and numerical handling for the
357  corners of the cube. \\  corners of the cube. \\
358    
359    
360  \subsubsection{Arrays Indexed to Tile Number and Neighbor}  Arrays Indexed to Tile Number and Neighbor:
361    
362  The following arrays are all of size  The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and
363  \code{MAX\_NEIGHBOURS}$\times$\code{NTILES} and describe the  \code{NTILES} and describe the orientations between the the tiles. \\
 orientations between the the tiles. \\  
364    
365  The array \code{exch2\_neighbourId(a,T)} holds the tile number  The array \code{exch2\_neighbourId(a,T)} holds the tile number
366  \code{Tn} for each of the tile number \code{T}'s neighboring tiles  \code{Tn} for each of the tile number \code{T}'s neighboring tiles
367  \code{a}.  The neighbor tiles are indexed  \code{a}.  The neighbor tiles are indexed
368  \code{(1:exch2\_NNeighbours(T))} in the order right to left on the  \code{(1:exch2\_nNeighbours(T))} in the order right to left on the
369  north then south edges, and then top to bottom on the east and west  north then south edges, and then top to bottom on the east then west
370  edges.  Maybe throw in a fig here, eh?  \\  edges.  \\
371    
372  \sloppy The \code{exch2\_opposingSend\_record(a,T)} array holds the   The \code{exch2\_opposingSend\_record(a,T)} array holds the
373  index \code{b} of the element in \texttt{exch2\_neighbourId(b,Tn)}  index \code{b} of the element in \texttt{exch2\_neighbourId(b,Tn)}
374  that holds the tile number \code{T}, given  that holds the tile number \code{T}, given
375  \code{Tn=exch2\_neighborId(a,T)}.  In other words,  \code{Tn=exch2\_neighborId(a,T)}.  In other words,
# Line 370  This provides a back-reference from the Line 382  This provides a back-reference from the
382  The arrays \varlink{exch2\_pi}{exch2_pi} and  The arrays \varlink{exch2\_pi}{exch2_pi} and
383  \varlink{exch2\_pj}{exch2_pj} specify the transformations of indices  \varlink{exch2\_pj}{exch2_pj} specify the transformations of indices
384  in exchanges between the neighboring tiles.  These transformations are  in exchanges between the neighboring tiles.  These transformations are
385  necessary in exchanges between subdomains because the array index in  necessary in exchanges between subdomains because a horizontal dimension
386  one dimension may map to the other index in an adjacent subdomain, and  in one subdomain
387  may be have its indexing reversed. This swapping arises from the  may map to other horizonal dimension in an adjacent subdomain, and
388  ``folding'' of two-dimensional arrays into a three-dimensional cube.  may also have its indexing reversed. This swapping arises from the
389    ``folding'' of two-dimensional arrays into a three-dimensional
390    cube. \\
391    
392  The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)}  The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)}
393  are the neighbor ID \code{N} and the tile number \code{T} as explained  are the neighbor ID \code{N} and the tile number \code{T} as explained
394  above, plus a vector of length \code{2} containing transformation  above, plus a vector of length \code{2} containing transformation
395  factors \code{t}.  The first element of the transformation vector  factors \code{t}.  The first element of the transformation vector
396  holds the factor to multiply the index in the same axis, and the  holds the factor to multiply the index in the same dimension, and the
397  second element holds the the same for the orthogonal index.  To  second element holds the the same for the orthogonal dimension.  To
398  clarify, \code{exch2\_pi(1,N,T)} holds the mapping of the $x$ axis  clarify, \code{exch2\_pi(1,N,T)} holds the mapping of the $x$ axis
399  index of tile \code{T} to the $x$ axis of tile \code{T}'s neighbor  index of tile \code{T} to the $x$ axis of tile \code{T}'s neighbor
400  \code{N}, and \code{exch2\_pi(2,N,T)} holds the mapping of \code{T}'s  \code{N}, and \code{exch2\_pi(2,N,T)} holds the mapping of \code{T}'s
# Line 395  the arrays for all tile neighbors on the Line 409  the arrays for all tile neighbors on the
409  \code{(1,0)}, since all tiles on the same subdomain are oriented  \code{(1,0)}, since all tiles on the same subdomain are oriented
410  identically.  An axis that corresponds to the orthogonal dimension  identically.  An axis that corresponds to the orthogonal dimension
411  with the same index direction in a particular tile-neighbor  with the same index direction in a particular tile-neighbor
412  orientation will have \code{(0,1)}.  Those in the opposite index  orientation will have \code{(0,1)}.  Those with the opposite index
413  direction will have \code{(0,-1)} in order to reverse the ordering. \\  direction will have \code{(0,-1)} in order to reverse the ordering. \\
414    
415  The arrays \varlink{exch2\_oi}{exch2_oi},  The arrays \varlink{exch2\_oi}{exch2_oi},
416  \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and  \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
417  \varlink{exch2\_oj\_f}{exch2_oj_f} are indexed to tile number and  \varlink{exch2\_oj\_f}{exch2_oj_f} are indexed to tile number and
418  neighbor and specify the relative offset within the subdomain of the  neighbor and specify the relative offset within the subdomain of the
419  array index of a variable going from a neighboring tile $N$ to a local  array index of a variable going from a neighboring tile \code{N} to a
420  tile $T$.  Consider \code{T=1} in the six-tile topology  local tile \code{T}.  Consider \code{T=1} in the six-tile topology
421  (Fig. \ref{fig:6tile}), where  (Fig. \ref{fig:6tile}), where
422    
423  \begin{verbatim}  \begin{verbatim}
# Line 420  exchange between the two requires no cha Line 434  exchange between the two requires no cha
434  the western neighbor (\code{Tn=5}), \code{code\_oi(3,1)=32} since the  the western neighbor (\code{Tn=5}), \code{code\_oi(3,1)=32} since the
435  \code{x=0} vector on \code{T} corresponds to the \code{y=32} vector on  \code{x=0} vector on \code{T} corresponds to the \code{y=32} vector on
436  \code{Tn}.  The eastern edge of \code{T} shows the reverse case  \code{Tn}.  The eastern edge of \code{T} shows the reverse case
437  (\code{exch2\_oi(4,1)=-32)}, where \code{x=32} on \code{T} exchanges  (\code{exch2\_oi(4,1)=-32)}), where \code{x=32} on \code{T} exchanges
438  with \code{x=0} on \code{Tn=2}.  The most interesting case, where  with \code{x=0} on \code{Tn=2}. \\
439  \code{exch2\_oi(1,1)=33} and \code{Tn=3}, involves a reversal of  
440  indices.  As in every case, the offset \code{exch2\_oi} is added to   The most interesting case, where \code{exch2\_oi(1,1)=33} and
441  the original $x$ index of \code{T} multiplied by the transformation  \code{Tn=3}, involves a reversal of indices.  As in every case, the
442  factor \code{exch2\_pi(t,N,T)}.  Here \code{exch2\_pi(1,1,1)=0} since  offset \code{exch2\_oi} is added to the original $x$ index of \code{T}
443  the $x$ axis of \code{T} is orthogonal to the $x$ axis of \code{Tn}.  multiplied by the transformation factor \code{exch2\_pi(t,N,T)}.  Here
444  \code{exch2\_pi(2,1,1)=-1} since the $x$ axis of \code{T} corresponds  \code{exch2\_pi(1,1,1)=0} since the $x$ axis of \code{T} is orthogonal
445  to the $y$ axis of \code{Tn}, but the axes are reversed.  The result  to the $x$ axis of \code{Tn}.  \code{exch2\_pi(2,1,1)=-1} since the
446  is that the index of the northern edge of \code{T}, which runs  $x$ axis of \code{T} corresponds to the $y$ axis of \code{Tn}, but the
447  \code{(1:32)}, is transformed to  index is reversed.  The result is that the index of the northern edge
448    of \code{T}, which runs \code{(1:32)}, is transformed to
449  \code{(-1:-32)}. \code{exch2\_oi(1,1)} is then added to this range to  \code{(-1:-32)}. \code{exch2\_oi(1,1)} is then added to this range to
450  get back \code{(1:32)} -- the index of the $y$ axis of \code{Tn}.  get back \code{(32:1)} -- the index of the $y$ axis of \code{Tn}
451  This transformation may seem overly convoluted for the six-tile case,  relative to \code{T}.  This transformation may seem overly convoluted
452  but it is necessary to provide a general solution for various  for the six-tile case, but it is necessary to provide a general
453  topologies. \\  solution for various topologies. \\
454    
455    
456    
# Line 455  of tile \code{T=2} in the twelve-tile to Line 470  of tile \code{T=2} in the twelve-tile to
470         exch2_jthi_c(4,2)=33         exch2_jthi_c(4,2)=33
471  \end{verbatim}  \end{verbatim}
472    
473  Here \code{N=4}, indicating the western neighbor, which is \code{Tn=1}.  Here \code{N=4}, indicating the western neighbor, which is
474  \code{Tn=1} resides on the same subdomain as \code{T=2}, so the tiles  \code{Tn=1}.  \code{Tn} resides on the same subdomain as \code{T}, so
475  have the same orientation and the same $x$ and $y$ axes.  The $i$  the tiles have the same orientation and the same $x$ and $y$ axes.
476  component is orthogonal to the western edge and the tile is 16 points  The $x$ axis is orthogonal to the western edge and the tile is 16
477  wide, so \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} indicate the  points wide, so \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c}
478  column beyond \code{Tn=1}'s eastern edge, in that tile's halo  indicate the column beyond \code{Tn}'s eastern edge, in that tile's
479  region. Since the border of the tiles extends through the entire  halo region. Since the border of the tiles extends through the entire
480  height of the subdomain, the $y$ axis bounds \code{exch2\_jtlo\_c} to  height of the subdomain, the $y$ axis bounds \code{exch2\_jtlo\_c} to
481  \code{exch2\_jthi\_c} cover the height, plus 1 in either direction to  \code{exch2\_jthi\_c} cover the height of \code{(1:32)}, plus 1 in
482  cover part of the halo. \\  either direction to cover part of the halo. \\
483    
484  For the north edge of the same tile \code{T=2} where \code{N=1} and  For the north edge of the same tile \code{T=2} where \code{N=1} and
485  the neighbor tile is \code{Tn=5}:  the neighbor tile is \code{Tn=5}:
# Line 477  the neighbor tile is \code{Tn=5}: Line 492  the neighbor tile is \code{Tn=5}:
492  \end{verbatim}  \end{verbatim}
493    
494  \code{T}'s northern edge is parallel to the $x$ axis, but since  \code{T}'s northern edge is parallel to the $x$ axis, but since
495  \code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis,  \code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis, \code{T}'s
496  \code{T}'s northern edge exchanges with \code{Tn}'s western edge.  northern edge exchanges with \code{Tn}'s western edge.  The western
497  The western edge of the tiles corresponds to the lower bound of the  edge of the tiles corresponds to the lower bound of the $x$ axis, so
498  $x$ axis, so \code{exch2\_itlo\_c} \code{exch2\_ithi\_c} are \code{0}. The  \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} are \code{0}, in the
499  range of \code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the  western halo region of \code{Tn}. The range of
500  width of \code{T}'s northern edge, plus the halo. \\  \code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the
501    width of \code{T}'s northern edge, expanded by one into the halo. \\
   
   
   
   
   
   
   
   
   
   
 This needs some diagrams. \\  
   
502    
503    
504  \subsection{Key Routines}  \subsubsection{Key Routines}
505    
506  Most of the subroutines particular to exch2 handle the exchanges  Most of the subroutines particular to exch2 handle the exchanges
507  themselves and are of the same format as those described in  themselves and are of the same format as those described in
508  \ref{sect:cube_sphere_communication} \sectiontitle{Cube sphere  \ref{sect:cube_sphere_communication} \sectiontitle{Cube sphere
509  communication}.  Like the original routines, they are written as  communication}.  Like the original routines, they are written as
510  templates which the local Makefile converts from RX into RL and RS  templates which the local Makefile converts from \code{RX} into
511  forms. \\  \code{RL} and \code{RS} forms. \\
512    
513  The interfaces with the core model subroutines are  The interfaces with the core model subroutines are
514  \code{EXCH\_UV\_XY\_RX}, \code{EXCH\_UV\_XYZ\_RX} and \code{EXCH\_XY\_RX}.  \code{EXCH\_UV\_XY\_RX}, \code{EXCH\_UV\_XYZ\_RX} and
515  They override the standard exchange routines when \code{genmake2} is  \code{EXCH\_XY\_RX}.  They override the standard exchange routines
516  run with \code{exch2} option.  They in turn call the local exch2  when \code{genmake2} is run with \code{exch2} option.  They in turn
517  subroutines \code{EXCH2\_UV\_XY\_RX} and \code{EXCH2\_UV\_XYZ\_RX} for two  call the local exch2 subroutines \code{EXCH2\_UV\_XY\_RX} and
518  and three dimensional vector quantities, and \code{EXCH2\_XY\_RX} and  \code{EXCH2\_UV\_XYZ\_RX} for two and three-dimensional vector
519  \code{EXCH2\_XYZ\_RX} for two and three dimensional scalar quantities.  quantities, and \code{EXCH2\_XY\_RX} and \code{EXCH2\_XYZ\_RX} for two
520  These subroutines set the dimensions of the area to be exchanged, call  and three-dimensional scalar quantities.  These subroutines set the
521  \code{EXCH2\_RX1\_CUBE} for scalars and \code{EXCH2\_RX2\_CUBE} for  dimensions of the area to be exchanged, call \code{EXCH2\_RX1\_CUBE}
522  vectors, and then handle the singularities at the cube corners. \\  for scalars and \code{EXCH2\_RX2\_CUBE} for vectors, and then handle
523    the singularities at the cube corners. \\
524    
525  The separate scalar and vector forms of \code{EXCH2\_RX1\_CUBE} and  The separate scalar and vector forms of \code{EXCH2\_RX1\_CUBE} and
526  \code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subrouine needs  \code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subroutine
527  to pass both the $x$ and $y$ components of the vectors.  This arises  needs to pass both the $u$ and $v$ components of the physical vectors.
528  from the topological folding discussed above, where the $x$ and $y$  This swapping arises from the topological folding discussed above, where the
529  axes get swapped in some cases.  This swapping is not an issue with  $x$ and $y$ axes get swapped in some cases, and is not an
530  the scalar version. These subroutines call \code{EXCH2\_SEND\_RX1} and  issue with the scalar case. These subroutines call
531  \code{EXCH2\_SEND\_RX2}, which do most of the work using the variables  \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of
532  discussed above. \\  the work using the variables discussed above. \\
533    

Legend:
Removed from v.1.16  
changed lines
  Added in v.1.24

  ViewVC Help
Powered by ViewVC 1.1.22