/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.10 by afe, Mon Mar 15 20:11:56 2004 UTC revision 1.19 by afe, Mon May 10 21:39:11 2004 UTC
# Line 16  Line 16 
16    
17  \subsection{Introduction}  \subsection{Introduction}
18    
19  The \texttt{exch2} package is an extension to the original cubed  The \texttt{exch2} package extends the original cubed sphere topology
20  sphere topological configuration that allows more flexible domain  configuration to allow more flexible domain decomposition and
21  decomposition and parallelization.  Cube faces (also called  parallelization.  Cube faces (also called subdomains) may be divided
22  subdomains) may be divided into any number of tiles that divide evenly  into any number of tiles that divide evenly into the grid point
23  into the grid point dimensions of the subdomain.  Furthermore, the  dimensions of the subdomain.  Furthermore, the tiles can run on
24  individual tiles may be run on separate processors in different  separate processors individually or in groups, which provides for
25  combinations, and whether exchanges between particular tiles occur  manual compile-time load balancing across a relatively arbitrary
26  between different processors is determined at runtime.  This  number of processors. \\
 flexibility provides for manual compile-time load balancing across a  
 relatively arbitrary number of processors. \\  
27    
28  The exchange parameters are declared in  The exchange parameters are declared in
29  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
30  and assigned in  and assigned in
31  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
32  validity of the cube topology depends on the \texttt{SIZE.h} file as  validity of the cube topology depends on the \file{SIZE.h} file as
33  detailed below.  Both files are generated by Matlab scripts in  detailed below.  The default files provided in the release configure a
34  \texttt{utils/exch2/matlab-topology-generator}; see Section  cubed sphere topology of six tiles, one per subdomain, each with
35  \ref{sec:topogen} for details on creating alternate topologies.  The  32$\times$32 grid points, with all tiles running on a single processor.  Both
36  default files provided in the release configure a cubed sphere  files are generated by Matlab scripts in
37  topology of six tiles, one per subdomain, each with 32$\times$32 grid  \file{utils/exch2/matlab-topology-generator}; see Section
38  points, all running on a single processor.  Pregenerated examples of  \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
39  these files with alternate topologies are provided under  for details on creating alternate topologies.  Pregenerated examples
40  \texttt{utils/exch2/code-mods} along with the appropriate  of these files with alternate topologies are provided under
41  \texttt{SIZE.h} file for single-processor execution.  \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
42    file for single-processor execution.
43    
44  \subsection{Invoking exch2}  \subsection{Invoking exch2}
45    
46  To use exch2 with the cubed sphere, the following conditions must be  To use exch2 with the cubed sphere, the following conditions must be
47  met: \\  met: \\
48    
49  $\bullet$ The exch2 package is included when \texttt{genmake2} is run.  The  $\bullet$ The exch2 package is included when \file{genmake2} is run.
50    easiest way to do this is to add the line \texttt{exch2} to the    The easiest way to do this is to add the line \code{exch2} to the
51    \texttt{profile.conf} file -- see Section \ref{sect:buildingCode}    \file{profile.conf} file -- see Section
52    for general details. \\    \ref{sect:buildingCode} \sectiontitle{Building the code} for general
53      details. \\
54  $\bullet$ An example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and  
55    \texttt{w2\_e2setup.F} must reside in a directory containing code  $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
56    linked when \texttt{genmake2} runs.  The safest place to put these    \file{w2\_e2setup.F} must reside in a directory containing files
57    is the directory indicated in the \texttt{-mods=DIR} command line    symbolically linked by the \file{genmake2} script.  The safest place to
58    modifier (typically \texttt{../code}), or the build directory.  The    put these is the directory indicated in the \code{-mods=DIR} command
59    default versions of these files reside in \texttt{pkg/exch2} and are    line modifier (typically \file{../code}), or the build directory.
60    linked automatically if no other versions exist elsewhere in the    The default versions of these files reside in \file{pkg/exch2} and
61    link path, but they should be left untouched to avoid breaking    are linked automatically if no other versions exist elsewhere in the
62      build path, but they should be left untouched to avoid breaking
63    configurations other than the one you intend to modify.\\    configurations other than the one you intend to modify.\\
64    
65  $\bullet$ Files containing grid parameters, named  $\bullet$ Files containing grid parameters, named
66    \texttt{tile}???\texttt{.mitgrid} where ??? is \texttt{001} through    \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain),
67    \texttt{006} (one per subdomain), must be in the working directory    must be in the working directory when the MITgcm executable is run.
68    when the MITgcm executable is run.  These files are provided in the    These files are provided in the example experiments for cubed sphere
69    example experiments for cubed sphere configurations with    configurations with 32$\times$32 cube sides
70    32$\times$32 cube sides and are non-trivial to generate -- please    -- please contact MITgcm support if you want to generate
71    contact MITgcm support if you want to generate files for other    files for other configurations. \\
72    configurations. \\  
73    $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must
74  $\bullet$ As always when compiling MITgcm, the file \texttt{SIZE.h}    be placed where \file{genmake2} will find it.  In particular for
75    must be placed where \texttt{genmake2} will find it.  In particular    exch2, the domain decomposition specified in \file{SIZE.h} must
76    for the exch2, the domain decompositin specified in \texttt{SIZE.h}    correspond with the particular configuration's topology specified in
77    must correspond with the particular configuration's topology    \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain
78    specified in \texttt{W2\_EXCH2\_TOPOLOGY.h} and    decomposition issues particular to exch2 are addressed in Section
79    \texttt{w2\_e2setup.F}.  Domain decomposition issues particular to    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
80    exch2 are addressed in Section \ref{sec:topogen}: ``Generating    and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and Multiprocessing}; a more
81    Topology Files for exch2''; a more general background on the subject    general background on the subject relevant to MITgcm is presented in
82    relvant to MITgcm is presented in Section    Section \ref{sect:specifying_a_decomposition}
83    \ref{sect:specifying_a_decomposition}: ``Specifying a    \sectiontitle{Specifying a decomposition}.\\
   decomposition''.\\  
84    
85  As of the time of writing the following examples use exch2 and may be  At the time of this writing the following examples use exch2 and may
86  used for guidance:  be used for guidance:
87    
88  \begin{verbatim}  \begin{verbatim}
89  verification/adjust_nlfs.cs-32x32x1  verification/adjust_nlfs.cs-32x32x1
# Line 101  verification/hs94.cs-32x32x5 Line 100  verification/hs94.cs-32x32x5
100  \label{sec:topogen}  \label{sec:topogen}
101    
102  Alternate cubed sphere topologies may be created using the Matlab  Alternate cubed sphere topologies may be created using the Matlab
103  scripts in \texttt{utils/exch2/matlab-topology-generator}. Running the  scripts in \file{utils/exch2/matlab-topology-generator}. Running the
104  m-file \texttt{driver} from the Matlab prompt (without passing any  m-file
105  function parameters) generates exch2 topology files  \filelink{driver.m}{utils-exch2-matlab-topology-generator_driver.m}
106  \texttt{W2\_EXCH2\_TOPOLOGY.h} and \texttt{w2\_e2setup.F} in the  from the Matlab prompt (there are no parameters to pass) generates
107  working directory and displays via Matlab a figure of the topology.  exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and
108  The other m-files in the directory are subroutines of \texttt{driver}  \file{w2\_e2setup.F} in the working directory and displays a figure of
109  and should not be run except for development purposes. \\  the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:12tile},
110    and \ref{fig:24tile} are examples of the generated diagrams.  The other
111    m-files in the directory are
112    subroutines called from \file{driver.m} and should not be run ``bare'' except
113    for development purposes. \\
114    
115  The parameters that determine the dimensions and topology of the  The parameters that determine the dimensions and topology of the
116  generated configuration are nr, nb, ng, tnx and tny, and all are  generated configuration are \code{nr}, \code{nb}, \code{ng},
117  assigned early in the script.  \code{tnx} and \code{tny}, and all are assigned early in the script. \\
118    
119  The first three determine the size of the subdomains (cube faces) and  The first three determine the height and width of the subdomains and
120  hence the size of the overall domain.  Each one determines the number  hence the size of the overall domain.  Each one determines the number
121  of grid points, and therefore the resolution, along the subdomain  of grid points, and therefore the resolution, along the subdomain
122  sides in a ``great circle'' around each axis of the cube.  At the time  sides in a ``great circle'' around each the three spatial axes of the cube.  At the time
123  of this writing MITgcm requires these three parameters to be equal,  of this writing MITgcm requires these three parameters to be equal,
124  but they provide for future releases of MITgcm to accomodate different  but they provide for future releases  to accomodate different
125  resolutions around the axes to allow (for example) greater resolution  resolutions around the axes to allow subdomains with differing resolutions.\\
126  around the equator.\\  
127    The parameters \code{tnx} and \code{tny} determine the width and height of
128  The parameters tnx and tny determine the dimensions of the tiles into  the tiles into which the subdomains are decomposed, and must evenly
129  which the subdomains are decomposed, and must evenly divide the  divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
130  integer assigned to nr, nb and ng.  The result is a rectangular tiling  The result is a rectangular tiling of the subdomain.  Figure
131  of the subdomain.  Figure \ref{fig:24tile} shows one possible topology  \ref{fig:24tile} shows one possible topology for a twenty-four-tile
132  for a twenty-four tile cube, and figure \ref{fig:12tile} shows one for  cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
 twelve tiles. \\  
133    
134  \begin{figure}  \begin{figure}
135  \begin{center}  \begin{center}
# Line 135  twelve tiles. \\ Line 137  twelve tiles. \\
137    \includegraphics{part6/s24t_16x16.ps}    \includegraphics{part6/s24t_16x16.ps}
138   }   }
139  \end{center}  \end{center}
140  \caption{Plot of cubed sphere topology with a 32$\times$32 grid and  
141  twenty-four tiles (tnx=16, tny=16)  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
142  } \label{fig:24tile}  divided into six 32$\times$32 subdomains, each of which is divided
143    into four tiles of width \code{tnx=16} and height \code{tny=16} for a
144    total of twenty-four tiles.  The colored borders of the subdomains
145    represent the parameters \code{nr} (red), \code{nb} (blue), and
146    \code{ng} (green).  } \label{fig:24tile}
147  \end{figure}  \end{figure}
148    
149  \begin{figure}  \begin{figure}
# Line 146  twenty-four tiles (tnx=16, tny=16) Line 152  twenty-four tiles (tnx=16, tny=16)
152    \includegraphics{part6/s12t_16x32.ps}    \includegraphics{part6/s12t_16x32.ps}
153   }   }
154  \end{center}  \end{center}
155  \caption{Plot of cubed sphere topology with a 32$\times$32 grid and  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
156  twelve tiles (tnx=16, tny=32)  divided into six 32$\times$32 subdomains of two tiles each
157     (\code{tnx=16, tny=32}).
158  } \label{fig:12tile}  } \label{fig:12tile}
159  \end{figure}  \end{figure}
160    
161    \begin{figure}
162    \begin{center}
163     \resizebox{4in}{!}{
164      \includegraphics{part6/s6t_32x32.ps}
165     }
166    \end{center}
167    \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
168    divided into six 32$\times$32 subdomains with one tile each
169    (\code{tnx=32, tny=32}).  This is the default configuration.
170      }
171    \label{fig:6tile}
172    \end{figure}
173    
174    
175  Tiles can be selected from the topology to be omitted from being  Tiles can be selected from the topology to be omitted from being
176  allocated memory and processors.  This kind of tuning is useful in  allocated memory and processors.  This tuning is useful in ocean
177  ocean modeling for omitting tiles that fall entirely on land.  The  modeling for omitting tiles that fall entirely on land.  The tiles
178  tiles omitted are specified in the file \texttt{blanklist.txt} by  omitted are specified in the file
179  their tile number in the topology, separated by a newline. \\  \filelink{blanklist.txt}{utils-exch2-matlab-topology-generator_blanklist.txt}
180    by their tile number in the topology, separated by a newline. \\
181    
182    
183    
184    
185    \subsection{exch2, SIZE.h, and Multiprocessing}
186    \label{sec:exch2mpi}
187    
188    Once the topology configuration files are created, the Fortran
189    \code{PARAMETER}s in \file{SIZE.h} must be configured to match.
190    Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying
191    a decomposition} provides a general description of domain
192    decomposition within MITgcm and its relation to \file{SIZE.h}. The
193    current section specifies constraints that the exch2 package
194    imposes and describes how to enable parallel execution with
195    MPI. \\
196    
197    As in the general case, the parameters \varlink{sNx}{sNx} and
198    \varlink{sNy}{sNy} define the size of the individual tiles, and so
199    must be assigned the same respective values as \code{tnx} and
200    \code{tny} in \file{driver.m}.\\
201    
202    The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy}
203    have no special bearing on exch2 and may be assigned as in the general
204    case. The same holds for \varlink{Nr}{Nr}, the number of vertical
205    levels in the model.\\
206    
207    The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},
208    \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of
209    tiles and how they are distributed on processors.  When using exch2,
210    the tiles are stored in the $x$ dimension, and so
211    \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as
212    configured by exch2 cannot be split up accross processors without
213    regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\
214    
215    The number of tiles MITgcm allocates and how they are distributed
216    between processors depends on \varlink{nPx}{nPx} and
217    \varlink{nSx}{nSx}.  \varlink{nSx}{nSx} is the number of tiles per
218    processor and \varlink{nPx}{nPx} is the number of processors.  The total
219    number of tiles in the topology minus those listed in
220    \file{blanklist.txt} must equal \code{nSx*nPx}.  Note that in order to
221    obtain maximum usage from a given number of processors in some cases,
222    this restriction might entail sharing a processor with a tile that would
223    otherwise be excluded. \\
224    
225    The following is an example of \file{SIZE.h} for the twelve-tile
226    configuration illustrated in figure \ref{fig:12tile} running on
227    one processor: \\
228    
229    \begin{verbatim}
230          PARAMETER (
231         &           sNx =  16,
232         &           sNy =  32,
233         &           OLx =   2,
234         &           OLy =   2,
235         &           nSx =  12,
236         &           nSy =   1,
237         &           nPx =   1,
238         &           nPy =   1,
239         &           Nx  = sNx*nSx*nPx,
240         &           Ny  = sNy*nSy*nPy,
241         &           Nr  =   5)
242    \end{verbatim}
243    
244    The following is an example for the twenty-four-tile topology in
245    figure \ref{fig:24tile} running on six processors:
246    
247    \begin{verbatim}
248          PARAMETER (
249         &           sNx =  16,
250         &           sNy =  16,
251         &           OLx =   2,
252         &           OLy =   2,
253         &           nSx =   4,
254         &           nSy =   1,
255         &           nPx =   6,
256         &           nPy =   1,
257         &           Nx  = sNx*nSx*nPx,
258         &           Ny  = sNy*nSy*nPy,
259         &           Nr  =   5)
260    \end{verbatim}
261    
262    
263    
264    
265    
266  \subsection{Key Variables}  \subsection{Key Variables}
267    
268  The descriptions of the variables are divided up into scalars,  The descriptions of the variables are divided up into scalars,
269  one-dimensional arrays indexed to the tile number, and two and three  one-dimensional arrays indexed to the tile number, and two and
270  dimensional arrays indexed to tile number and neighboring tile.  This  three-dimensional arrays indexed to tile number and neighboring tile.
271  division actually reflects the functionality of these variables: the  This division reflects the functionality of these variables: The
272  scalars are common to every part of the topology, the tile-indexed  scalars are common to every part of the topology, the tile-indexed
273  arrays to individual tiles, and the arrays indexed to tile and  arrays to individual tiles, and the arrays indexed by tile and
274  neighbor to relationships between tiles and their neighbors.  neighbor to relationships between tiles and their neighbors. \\
275    
276  \subsubsection{Scalars}  \subsubsection{Scalars}
277    
278  The number of tiles in a particular topology is set with the parameter  The number of tiles in a particular topology is set with the parameter
279  \texttt{NTILES}, and the maximum number of neighbors of any tiles by  \code{NTILES}, and the maximum number of neighbors of any tiles by
280  \texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the  \code{MAX\_NEIGHBOURS}.  These parameters are used for defining the
281  size of the various one and two dimensional arrays that store tile  size of the various one and two dimensional arrays that store tile
282  parameters indexed to the tile number.\\  parameters indexed to the tile number and are assigned in the files
283    generated by \file{driver.m}.\\
284    
285  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
286  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
287  of tiles in the x and y global indices.  For example, the default  of tiles in the $x$ and $y$ global indices.  For example, the default
288  setup of six tiles has \texttt{exch2\_domain\_nxt=6} and  setup of six tiles (Fig. \ref{fig:6tile}) has
289  \texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in  \code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}.  A
290  gridpoints) tiles, four (2x2) per subdomain, will have  topology of twenty-four square tiles, four per subdomain (as in figure
291  \texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.  \ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and
292  Note that these parameters express the tile layout to allow global  \code{exch2\_domain\_nyt=2}.  Note that these parameters express the
293  data files that are tile-layout-neutral and have no bearing on the  tile layout in order to allow global data files that are tile-layout-neutral.
294  internal storage of the arrays.  The tiles are internally stored in a  They have no bearing on the internal storage of the arrays.  The tiles
295  range from \texttt{1,bi} (in the x axis) and y-axis variable  are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the
296  \texttt{bj} is generally ignored within the package.  $x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to
297    equal \code{1} throughout the package. \\
298  \subsubsection{Arrays Indexed to Tile Number}  
299    \subsubsection{Arrays indexed to tile number}
300  The following arrays are of size \texttt{NTILES}, are indexed to the  
301  tile number, and the indices are omitted in their descriptions.  The following arrays are of length \code{NTILES} and are indexed to
302    the tile number, which is indicated in the diagrams with the notation
303    \textsf{t}$n$.  The indices are omitted in the descriptions. \\
304    
305  The arrays \varlink{exch2\_tnx}{exch2_tnx} and  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
306  \varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each  \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of
307  tile.  At present for each tile \texttt{exch2\_tnx=sNx} and  each tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
308  \texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future  \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in
309  releases of MITgcm are to allow varying tile sizes.  Section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
310    Multiprocessing}.  Future releases of MITgcm may allow varying tile
311  The location of the tiles' Cartesian origin within a subdomain are  sizes. \\
312  determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and  
313  \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to  The arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
314  relate the location of the edges of the tiles to each other.  As an  \varlink{exch2\_tbasey}{exch2_tbasey} determine the tiles'
315  example, in the default six-tile topology (the degenerate case) each  Cartesian origin within a subdomain  
316  index in these arrays are set to 0.  The twenty-four, 32x32 cube face  and locate the edges of different tiles relative to each other.  As
317  case discussed above will have values of 0 or 16, depending on the  an example, in the default six-tile topology (Fig. \ref{fig:6tile})
318  quadrant the tile falls within the subdomain.  The array  each index in these arrays is set to \code{0} since a tile occupies
319  \varlink{exch2\_myFace}{exch2_myFace} contains the number of the  its entire subdomain.  The twenty-four-tile case discussed above will
320  cubeface/subdomain of each tile, numbered 1-6 in the case of the  have values of \code{0} or \code{16}, depending on the quadrant of the
321  standard cube topology.  tile within the subdomain.  The elements of the arrays
322    \varlink{exch2\_txglobalo}{exch2_txglobalo} and
 The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and  
323  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
324  \varlink{exch2\_tbasex}{exch2_tbasex} and  \varlink{exch2\_tbasex}{exch2_tbasex} and
325  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tile edges within the
326  global address space, similar to that used by global files.  global address space, similar to that used by global output and input
327    files. \\
328    
329    The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of
330    the subdomain of each tile, in a range \code{(1:6)} in the case of the
331    standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
332    figures \ref{fig:12tile} and
333    \ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
334    contains a count of the neighboring tiles each tile has, and sets
335    the bounds for looping over neighboring tiles.
336    \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
337    tile, and is used in interprocess communication.  \\
338    
339    
340  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
341  \varlink{exch2\_isEedge}{exch2_isEedge},  \varlink{exch2\_isEedge}{exch2_isEedge},
342  \varlink{exch2\_isSedge}{exch2_isSedge}, and  \varlink{exch2\_isSedge}{exch2_isSedge}, and
343  \varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed  \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the
344  tile lies on the edge of a subdomain, 0 if not.  The values are used  indexed tile lies on the edge of its subdomain, \code{0} if
345  within the topology generator to determine the orientation of  not.  The values are used within the topology generator to determine
346  neighboring tiles and to indicate whether a tile lies on the corner of  the orientation of neighboring tiles, and to indicate whether a tile
347  a subdomain.  The latter case indicates special exchange and numerical  lies on the corner of a subdomain.  The latter case requires special
348  handling for the singularities at the eight corners of the cube.  exchange and numerical handling for the singularities at the eight
349  \varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of  corners of the cube. \\
350  how many neighboring tiles each tile has, and is used for setting  
 bounds for looping over neighboring tiles.  
 \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each  
 tile, and is used in interprocess communication.  
351    
352  \subsubsection{Arrays Indexed to Tile Number and Neighbor}  \subsubsection{Arrays Indexed to Tile Number and Neighbor}
353    
354  The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$  The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and
355  \texttt{NTILES} and describe the orientations between the the tiles.  \code{NTILES} and describe the orientations between the the tiles. \\
356    
357  The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for  The array \code{exch2\_neighbourId(a,T)} holds the tile number
358  each of the $n$ neighboring tiles.  The neighbor tiles are indexed  \code{Tn} for each of the tile number \code{T}'s neighboring tiles
359  \texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north  \code{a}.  The neighbor tiles are indexed
360  then south edges, and then top to bottom on the east and west edges.  \code{(1:exch2\_nNeighbours(T))} in the order right to left on the
361  Maybe throw in a fig here, eh?  north then south edges, and then top to bottom on the east then west
362    edges.  \\
363  The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c  
364  in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.   The \code{exch2\_opposingSend\_record(a,T)} array holds the
365  In other words,  index \code{b} of the element in \texttt{exch2\_neighbourId(b,Tn)}
366    that holds the tile number \code{T}, given
367    \code{Tn=exch2\_neighborId(a,T)}.  In other words,
368  \begin{verbatim}  \begin{verbatim}
369     exch2_neighbourId( exch2_opposingSend_record(a,T),     exch2_neighbourId( exch2_opposingSend_record(a,T),
370                        exch2_neighbourId(a,T) ) = T                        exch2_neighbourId(a,T) ) = T
371  \end{verbatim}  \end{verbatim}
372  and this provides a back-reference from the neighbor tiles.  This provides a back-reference from the neighbor tiles. \\
373    
374  The arrays \varlink{exch2\_pi}{exch2_pi},  The arrays \varlink{exch2\_pi}{exch2_pi} and
375  \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},  \varlink{exch2\_pj}{exch2_pj} specify the transformations of indices
376    in exchanges between the neighboring tiles.  These transformations are
377    necessary in exchanges between subdomains because a horizontal dimension
378    in one subdomain
379    may map to other horizonal dimension in an adjacent subdomain, and
380    may also have its indexing reversed. This swapping arises from the
381    ``folding'' of two-dimensional arrays into a three-dimensional
382    cube. \\
383    
384    The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)}
385    are the neighbor ID \code{N} and the tile number \code{T} as explained
386    above, plus a vector of length \code{2} containing transformation
387    factors \code{t}.  The first element of the transformation vector
388    holds the factor to multiply the index in the same dimension, and the
389    second element holds the the same for the orthogonal dimension.  To
390    clarify, \code{exch2\_pi(1,N,T)} holds the mapping of the $x$ axis
391    index of tile \code{T} to the $x$ axis of tile \code{T}'s neighbor
392    \code{N}, and \code{exch2\_pi(2,N,T)} holds the mapping of \code{T}'s
393    $x$ index to the neighbor \code{N}'s $y$ index. \\
394    
395    One of the two elements of \code{exch2\_pi} or \code{exch2\_pj} for a
396    given tile \code{T} and neighbor \code{N} will be \code{0}, reflecting
397    the fact that the two axes are orthogonal.  The other element will be
398    \code{1} or \code{-1}, depending on whether the axes are indexed in
399    the same or opposite directions.  For example, the transform vector of
400    the arrays for all tile neighbors on the same subdomain will be
401    \code{(1,0)}, since all tiles on the same subdomain are oriented
402    identically.  An axis that corresponds to the orthogonal dimension
403    with the same index direction in a particular tile-neighbor
404    orientation will have \code{(0,1)}.  Those with the opposite index
405    direction will have \code{(0,-1)} in order to reverse the ordering. \\
406    
407    The arrays \varlink{exch2\_oi}{exch2_oi},
408  \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and  \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
409  \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in  \varlink{exch2\_oj\_f}{exch2_oj_f} are indexed to tile number and
410  exchanges between the neighboring tiles.  The dimensions of  neighbor and specify the relative offset within the subdomain of the
411  \texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the  array index of a variable going from a neighboring tile \code{N} to a
412  neighbor ID \textit{N} and the tile number \textit{T} as explained  local tile \code{T}.  Consider \code{T=1} in the six-tile topology
413  above, plus the transformation vector {\em t }, of length two.  The  (Fig. \ref{fig:6tile}), where
 first element of the transformation vector indicates the factor by  
 which variables representing the same vector component of a tile will  
 be multiplied, and the second element indicates the transform to the  
 variable in the other direction.  As an example,  
 \texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a  
 vector variable in tile \texttt{T} to the i-component of tile  
 \texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold  
 the component of neighbor \texttt{N}'s j-component.  
   
 Under the current cube topology, one of the two elements of  
 \texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}  
 and neighbor \texttt{N} will be 0, reflecting the fact that the vector  
 components are orthogonal.  The other element will be 1 or -1,  
 depending on whether the components are indexed in the same or  
 opposite directions.  For example, the transform dimension of the  
 arrays for all tile neighbors on the same subdomain will be [1,0],  
 since all tiles on the same subdomain are oriented identically.  
 Vectors that correspond to the orthogonal dimension with the same  
 index direction will have [0,1], whereas those in the opposite index  
 direction will have [0,-1].  
414    
415    \begin{verbatim}
416           exch2_oi(1,1)=33
417           exch2_oi(2,1)=0
418           exch2_oi(3,1)=32
419           exch2_oi(4,1)=-32
420    \end{verbatim}
421    
422    The simplest case is \code{exch2\_oi(2,1)}, the southern neighbor,
423    which is \code{Tn=6}.  The axes of \code{T} and \code{Tn} have the
424    same orientation and their $x$ axes have the same origin, and so an
425    exchange between the two requires no changes to the $x$ index.  For
426    the western neighbor (\code{Tn=5}), \code{code\_oi(3,1)=32} since the
427    \code{x=0} vector on \code{T} corresponds to the \code{y=32} vector on
428    \code{Tn}.  The eastern edge of \code{T} shows the reverse case
429    (\code{exch2\_oi(4,1)=-32)}), where \code{x=32} on \code{T} exchanges
430    with \code{x=0} on \code{Tn=2}. \\
431    
432     The most interesting case, where \code{exch2\_oi(1,1)=33} and
433    \code{Tn=3}, involves a reversal of indices.  As in every case, the
434    offset \code{exch2\_oi} is added to the original $x$ index of \code{T}
435    multiplied by the transformation factor \code{exch2\_pi(t,N,T)}.  Here
436    \code{exch2\_pi(1,1,1)=0} since the $x$ axis of \code{T} is orthogonal
437    to the $x$ axis of \code{Tn}.  \code{exch2\_pi(2,1,1)=-1} since the
438    $x$ axis of \code{T} corresponds to the $y$ axis of \code{Tn}, but the
439    index is reversed.  The result is that the index of the northern edge
440    of \code{T}, which runs \code{(1:32)}, is transformed to
441    \code{(-1:-32)}. \code{exch2\_oi(1,1)} is then added to this range to
442    get back \code{(32:1)} -- the index of the $y$ axis of \code{Tn}
443    relative to \code{T}.  This transformation may seem overly convoluted
444    for the six-tile case, but it is necessary to provide a general
445    solution for various topologies. \\
446    
447    
448    
449    Finally, \varlink{exch2\_itlo\_c}{exch2_itlo_c},
450    \varlink{exch2\_ithi\_c}{exch2_ithi_c},
451    \varlink{exch2\_jtlo\_c}{exch2_jtlo_c} and
452    \varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index
453    bounds of the edge segment of the neighbor tile \code{N}'s subdomain
454    that gets exchanged with the local tile \code{T}.  To take the example
455    of tile \code{T=2} in the twelve-tile topology
456    (Fig. \ref{fig:12tile}): \\
457    
 {\footnotesize  
458  \begin{verbatim}  \begin{verbatim}
459  C      exch2_pi          :: X index row of target to source permutation         exch2_itlo_c(4,2)=17
460  C                        :: matrix for each neighbour entry.                     exch2_ithi_c(4,2)=17
461  C      exch2_pj          :: Y index row of target to source permutation         exch2_jtlo_c(4,2)=0
462  C                        :: matrix for each neighbour entry.                     exch2_jthi_c(4,2)=33
 C      exch2_oi          :: X index element of target to source  
 C                        :: offset vector for cell-centered quantities    
 C                        :: of each neighbor entry.                      
 C      exch2_oj          :: Y index element of target to source  
 C                        :: offset vector for cell-centered quantities    
 C                        :: of each neighbor entry.                      
 C      exch2_oi_f        :: X index element of target to source  
 C                        :: offset vector for face quantities            
 C                        :: of each neighbor entry.                      
 C      exch2_oj_f        :: Y index element of target to source  
 C                        :: offset vector for face quantities            
 C                        :: of each neighbor entry.                      
463  \end{verbatim}  \end{verbatim}
464  }  
465    Here \code{N=4}, indicating the western neighbor, which is
466    \code{Tn=1}.  \code{Tn} resides on the same subdomain as \code{T}, so
467    the tiles have the same orientation and the same $x$ and $y$ axes.
468    The $x$ axis is orthogonal to the western edge and the tile is 16
469    points wide, so \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c}
470    indicate the column beyond \code{Tn}'s eastern edge, in that tile's
471    halo region. Since the border of the tiles extends through the entire
472    height of the subdomain, the $y$ axis bounds \code{exch2\_jtlo\_c} to
473    \code{exch2\_jthi\_c} cover the height of \code{(1:32)}, plus 1 in
474    either direction to cover part of the halo. \\
475    
476    For the north edge of the same tile \code{T=2} where \code{N=1} and
477    the neighbor tile is \code{Tn=5}:
478    
479    \begin{verbatim}
480           exch2_itlo_c(1,2)=0
481           exch2_ithi_c(1,2)=0
482           exch2_jtlo_c(1,2)=0
483           exch2_jthi_c(1,2)=17
484    \end{verbatim}
485    
486    \code{T}'s northern edge is parallel to the $x$ axis, but since
487    \code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis, \code{T}'s
488    northern edge exchanges with \code{Tn}'s western edge.  The western
489    edge of the tiles corresponds to the lower bound of the $x$ axis, so
490    \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} are \code{0}, in the
491    western halo region of \code{Tn}. The range of
492    \code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the
493    width of \code{T}'s northern edge, expanded by one into the halo. \\
494    
 \subsection{Key Routines}  
495    
496    \subsection{Key Routines}
497    
498    Most of the subroutines particular to exch2 handle the exchanges
499    themselves and are of the same format as those described in
500    \ref{sect:cube_sphere_communication} \sectiontitle{Cube sphere
501    communication}.  Like the original routines, they are written as
502    templates which the local Makefile converts from \code{RX} into
503    \code{RL} and \code{RS} forms. \\
504    
505    The interfaces with the core model subroutines are
506    \code{EXCH\_UV\_XY\_RX}, \code{EXCH\_UV\_XYZ\_RX} and
507    \code{EXCH\_XY\_RX}.  They override the standard exchange routines
508    when \code{genmake2} is run with \code{exch2} option.  They in turn
509    call the local exch2 subroutines \code{EXCH2\_UV\_XY\_RX} and
510    \code{EXCH2\_UV\_XYZ\_RX} for two and three-dimensional vector
511    quantities, and \code{EXCH2\_XY\_RX} and \code{EXCH2\_XYZ\_RX} for two
512    and three-dimensional scalar quantities.  These subroutines set the
513    dimensions of the area to be exchanged, call \code{EXCH2\_RX1\_CUBE}
514    for scalars and \code{EXCH2\_RX2\_CUBE} for vectors, and then handle
515    the singularities at the cube corners. \\
516    
517    The separate scalar and vector forms of \code{EXCH2\_RX1\_CUBE} and
518    \code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subroutine
519    needs to pass both the $u$ and $v$ components of the physical vectors.
520    This swapping arises from the topological folding discussed above, where the
521    $x$ and $y$ axes get swapped in some cases, and is not an
522    issue with the scalar case. These subroutines call
523    \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of
524    the work using the variables discussed above. \\
525    
 \subsection{References}  
Legend:
Removed from v.1.10  
changed lines
  Added in v.1.19
  ViewVC Help
Powered by ViewVC 1.1.22