/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.10 by afe, Mon Mar 15 20:11:56 2004 UTC revision 1.17 by afe, Fri Mar 19 21:25:45 2004 UTC
# Line 16  Line 16 
16    
17  \subsection{Introduction}  \subsection{Introduction}
18    
19  The \texttt{exch2} package is an extension to the original cubed  The \texttt{exch2} package extends the original cubed sphere topology
20  sphere topological configuration that allows more flexible domain  configuration to allow more flexible domain decomposition and
21  decomposition and parallelization.  Cube faces (also called  parallelization.  Cube faces (also called subdomains) may be divided
22  subdomains) may be divided into any number of tiles that divide evenly  into any number of tiles that divide evenly into the grid point
23  into the grid point dimensions of the subdomain.  Furthermore, the  dimensions of the subdomain.  Furthermore, the tiles can run on
24  individual tiles may be run on separate processors in different  separate processors individually or in groups, which provides for
25  combinations, and whether exchanges between particular tiles occur  manual compile-time load balancing across a relatively arbitrary
26  between different processors is determined at runtime.  This  number of processors. \\
 flexibility provides for manual compile-time load balancing across a  
 relatively arbitrary number of processors. \\  
27    
28  The exchange parameters are declared in  The exchange parameters are declared in
29  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
30  and assigned in  and assigned in
31  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
32  validity of the cube topology depends on the \texttt{SIZE.h} file as  validity of the cube topology depends on the \file{SIZE.h} file as
33  detailed below.  Both files are generated by Matlab scripts in  detailed below.  The default files provided in the release configure a
34  \texttt{utils/exch2/matlab-topology-generator}; see Section  cubed sphere topology of six tiles, one per subdomain, each with
35  \ref{sec:topogen} for details on creating alternate topologies.  The  32$\times$32 grid points, all running on a single processor.  Both
36  default files provided in the release configure a cubed sphere  files are generated by Matlab scripts in
37  topology of six tiles, one per subdomain, each with 32$\times$32 grid  \file{utils/exch2/matlab-topology-generator}; see Section
38  points, all running on a single processor.  Pregenerated examples of  \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
39  these files with alternate topologies are provided under  for details on creating alternate topologies.  Pregenerated examples
40  \texttt{utils/exch2/code-mods} along with the appropriate  of these files with alternate topologies are provided under
41  \texttt{SIZE.h} file for single-processor execution.  \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
42    file for single-processor execution.
43    
44  \subsection{Invoking exch2}  \subsection{Invoking exch2}
45    
46  To use exch2 with the cubed sphere, the following conditions must be  To use exch2 with the cubed sphere, the following conditions must be
47  met: \\  met: \\
48    
49  $\bullet$ The exch2 package is included when \texttt{genmake2} is run.  The  $\bullet$ The exch2 package is included when \file{genmake2} is run.
50    easiest way to do this is to add the line \texttt{exch2} to the    The easiest way to do this is to add the line \code{exch2} to the
51    \texttt{profile.conf} file -- see Section \ref{sect:buildingCode}    \file{profile.conf} file -- see Section
52    for general details. \\    \ref{sect:buildingCode} \sectiontitle{Building the code} for general
53      details. \\
54  $\bullet$ An example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and  
55    \texttt{w2\_e2setup.F} must reside in a directory containing code  $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
56    linked when \texttt{genmake2} runs.  The safest place to put these    \file{w2\_e2setup.F} must reside in a directory containing files
57    is the directory indicated in the \texttt{-mods=DIR} command line    symbolically linked when \file{genmake2} runs.  The safest place to
58    modifier (typically \texttt{../code}), or the build directory.  The    put these is the directory indicated in the \code{-mods=DIR} command
59    default versions of these files reside in \texttt{pkg/exch2} and are    line modifier (typically \file{../code}), or the build directory.
60    linked automatically if no other versions exist elsewhere in the    The default versions of these files reside in \file{pkg/exch2} and
61    link path, but they should be left untouched to avoid breaking    are linked automatically if no other versions exist elsewhere in the
62      build path, but they should be left untouched to avoid breaking
63    configurations other than the one you intend to modify.\\    configurations other than the one you intend to modify.\\
64    
65  $\bullet$ Files containing grid parameters, named  $\bullet$ Files containing grid parameters, named
66    \texttt{tile}???\texttt{.mitgrid} where ??? is \texttt{001} through    \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain),
67    \texttt{006} (one per subdomain), must be in the working directory    must be in the working directory when the MITgcm executable is run.
68    when the MITgcm executable is run.  These files are provided in the    These files are provided in the example experiments for cubed sphere
69    example experiments for cubed sphere configurations with    configurations with 32$\times$32 cube sides and are non-trivial to
70    32$\times$32 cube sides and are non-trivial to generate -- please    generate -- please contact MITgcm support if you want to generate
71    contact MITgcm support if you want to generate files for other    files for other configurations. \\
72    configurations. \\  
73    $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must
74  $\bullet$ As always when compiling MITgcm, the file \texttt{SIZE.h}    be placed where \file{genmake2} will find it.  In particular for
75    must be placed where \texttt{genmake2} will find it.  In particular    exch2, the domain decomposition specified in \file{SIZE.h} must
76    for the exch2, the domain decompositin specified in \texttt{SIZE.h}    correspond with the particular configuration's topology specified in
77    must correspond with the particular configuration's topology    \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain
78    specified in \texttt{W2\_EXCH2\_TOPOLOGY.h} and    decomposition issues particular to exch2 are addressed in Section
79    \texttt{w2\_e2setup.F}.  Domain decomposition issues particular to    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
80    exch2 are addressed in Section \ref{sec:topogen}: ``Generating    and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and MPI}; a more
81    Topology Files for exch2''; a more general background on the subject    general background on the subject relevant to MITgcm is presented in
82    relvant to MITgcm is presented in Section    Section \ref{sect:specifying_a_decomposition}
83    \ref{sect:specifying_a_decomposition}: ``Specifying a    \sectiontitle{Specifying a decomposition}.\\
   decomposition''.\\  
84    
85  As of the time of writing the following examples use exch2 and may be  At the time of this writing the following examples use exch2 and may
86  used for guidance:  be used for guidance:
87    
88  \begin{verbatim}  \begin{verbatim}
89  verification/adjust_nlfs.cs-32x32x1  verification/adjust_nlfs.cs-32x32x1
# Line 101  verification/hs94.cs-32x32x5 Line 100  verification/hs94.cs-32x32x5
100  \label{sec:topogen}  \label{sec:topogen}
101    
102  Alternate cubed sphere topologies may be created using the Matlab  Alternate cubed sphere topologies may be created using the Matlab
103  scripts in \texttt{utils/exch2/matlab-topology-generator}. Running the  scripts in \file{utils/exch2/matlab-topology-generator}. Running the
104  m-file \texttt{driver} from the Matlab prompt (without passing any  m-file
105  function parameters) generates exch2 topology files  \filelink{driver.m}{utils-exch2-matlab-topology-generator_driver.m}
106  \texttt{W2\_EXCH2\_TOPOLOGY.h} and \texttt{w2\_e2setup.F} in the  from the Matlab prompt (there are no parameters to pass) generates
107  working directory and displays via Matlab a figure of the topology.  exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and
108  The other m-files in the directory are subroutines of \texttt{driver}  \file{w2\_e2setup.F} in the working directory and displays a figure of
109  and should not be run except for development purposes. \\  the topology via Matlab.  The other m-files in the directory are
110    subroutines of \file{driver.m} and should not be run ``bare'' except
111    for development purposes. \\
112    
113  The parameters that determine the dimensions and topology of the  The parameters that determine the dimensions and topology of the
114  generated configuration are nr, nb, ng, tnx and tny, and all are  generated configuration are \code{nr}, \code{nb}, \code{ng},
115  assigned early in the script.  \code{tnx} and \code{tny}, and all are assigned early in the script. \\
116    
117  The first three determine the size of the subdomains (cube faces) and  The first three determine the size of the subdomains and
118  hence the size of the overall domain.  Each one determines the number  hence the size of the overall domain.  Each one determines the number
119  of grid points, and therefore the resolution, along the subdomain  of grid points, and therefore the resolution, along the subdomain
120  sides in a ``great circle'' around each axis of the cube.  At the time  sides in a ``great circle'' around an axis of the cube.  At the time
121  of this writing MITgcm requires these three parameters to be equal,  of this writing MITgcm requires these three parameters to be equal,
122  but they provide for future releases of MITgcm to accomodate different  but they provide for future releases  to accomodate different
123  resolutions around the axes to allow (for example) greater resolution  resolutions around the axes to allow (for example) greater resolution
124  around the equator.\\  around the equator.\\
125    
126  The parameters tnx and tny determine the dimensions of the tiles into  The parameters \code{tnx} and \code{tny} determine the dimensions of
127  which the subdomains are decomposed, and must evenly divide the  the tiles into which the subdomains are decomposed, and must evenly
128  integer assigned to nr, nb and ng.  The result is a rectangular tiling  divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
129  of the subdomain.  Figure \ref{fig:24tile} shows one possible topology  The result is a rectangular tiling of the subdomain.  Figure
130  for a twenty-four tile cube, and figure \ref{fig:12tile} shows one for  \ref{fig:24tile} shows one possible topology for a twenty-four-tile
131  twelve tiles. \\  cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
132    
133  \begin{figure}  \begin{figure}
134  \begin{center}  \begin{center}
# Line 135  twelve tiles. \\ Line 136  twelve tiles. \\
136    \includegraphics{part6/s24t_16x16.ps}    \includegraphics{part6/s24t_16x16.ps}
137   }   }
138  \end{center}  \end{center}
139  \caption{Plot of cubed sphere topology with a 32$\times$32 grid and  
140  twenty-four tiles (tnx=16, tny=16)  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
141  } \label{fig:24tile}  divided into six 32$\times$32 subdomains, each of which is divided
142    into four tiles (\code{tnx=16, tny=16}) for a total of twenty-four
143    tiles.  } \label{fig:24tile}
144  \end{figure}  \end{figure}
145    
146  \begin{figure}  \begin{figure}
# Line 146  twenty-four tiles (tnx=16, tny=16) Line 149  twenty-four tiles (tnx=16, tny=16)
149    \includegraphics{part6/s12t_16x32.ps}    \includegraphics{part6/s12t_16x32.ps}
150   }   }
151  \end{center}  \end{center}
152  \caption{Plot of cubed sphere topology with a 32$\times$32 grid and  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
153  twelve tiles (tnx=16, tny=32)  divided into six 32$\times$32 subdomains of two tiles each
154     (\code{tnx=16, tny=32}).
155  } \label{fig:12tile}  } \label{fig:12tile}
156  \end{figure}  \end{figure}
157    
158    \begin{figure}
159    \begin{center}
160     \resizebox{4in}{!}{
161      \includegraphics{part6/s6t_32x32.ps}
162     }
163    \end{center}
164    \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
165    divided into six 32$\times$32 subdomains with one tile each
166    (\code{tnx=32, tny=32}).  This is the default configuration.
167      }
168    \label{fig:6tile}
169    \end{figure}
170    
171    
172  Tiles can be selected from the topology to be omitted from being  Tiles can be selected from the topology to be omitted from being
173  allocated memory and processors.  This kind of tuning is useful in  allocated memory and processors.  This tuning is useful in ocean
174  ocean modeling for omitting tiles that fall entirely on land.  The  modeling for omitting tiles that fall entirely on land.  The tiles
175  tiles omitted are specified in the file \texttt{blanklist.txt} by  omitted are specified in the file
176  their tile number in the topology, separated by a newline. \\  \filelink{blanklist.txt}{utils-exch2-matlab-topology-generator_blanklist.txt}
177    by their tile number in the topology, separated by a newline. \\
178    
179    
180    
181    
182    \subsection{exch2, SIZE.h, and multiprocessing}
183    \label{sec:exch2mpi}
184    
185    Once the topology configuration files are created, the Fortran
186    \code{PARAMETER}s in \file{SIZE.h} must be configured to match.
187    Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying
188    a decomposition} provides a general description of domain
189    decomposition within MITgcm and its relation to \file{SIZE.h}. The
190    current section specifies certain constraints the exch2 package
191    imposes as well as describes how to enable parallel execution with
192    MPI. \\
193    
194    As in the general case, the parameters \varlink{sNx}{sNx} and
195    \varlink{sNy}{sNy} define the size of the individual tiles, and so
196    must be assigned the same respective values as \code{tnx} and
197    \code{tny} in \file{driver.m}.\\
198    
199    The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy}
200    have no special bearing on exch2 and may be assigned as in the general
201    case. The same holds for \varlink{Nr}{Nr}, the number of vertical
202    levels in the model.\\
203    
204    The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},
205    \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of
206    tiles and how they are distributed on processors.  When using exch2,
207    the tiles are stored in a single dimension, and so
208    \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as
209    configured by exch2 cannot be split up accross processors without
210    regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\
211    
212    The number of tiles MITgcm allocates and how they are distributed
213    between processors depends on \varlink{nPx}{nPx} and
214    \varlink{nSx}{nSx}.  \varlink{nSx}{nSx} is the number of tiles per
215    processor and \varlink{nPx}{nPx} the number of processors.  The total
216    number of tiles in the topology minus those listed in
217    \file{blanklist.txt} must equal \code{nSx*nPx}. \\
218    
219    The following is an example of \file{SIZE.h} for the twelve-tile
220    configuration illustrated in figure \ref{fig:12tile} running on
221    one processor: \\
222    
223    \begin{verbatim}
224          PARAMETER (
225         &           sNx =  16,
226         &           sNy =  32,
227         &           OLx =   2,
228         &           OLy =   2,
229         &           nSx =  12,
230         &           nSy =   1,
231         &           nPx =   1,
232         &           nPy =   1,
233         &           Nx  = sNx*nSx*nPx,
234         &           Ny  = sNy*nSy*nPy,
235         &           Nr  =   5)
236    \end{verbatim}
237    
238    The following is an example for the twenty-four-tile topology in
239    figure \ref{fig:24tile} running on six processors:
240    
241    \begin{verbatim}
242          PARAMETER (
243         &           sNx =  16,
244         &           sNy =  16,
245         &           OLx =   2,
246         &           OLy =   2,
247         &           nSx =   4,
248         &           nSy =   1,
249         &           nPx =   6,
250         &           nPy =   1,
251         &           Nx  = sNx*nSx*nPx,
252         &           Ny  = sNy*nSy*nPy,
253         &           Nr  =   5)
254    \end{verbatim}
255    
256    
257    
258    
259    
260  \subsection{Key Variables}  \subsection{Key Variables}
261    
262  The descriptions of the variables are divided up into scalars,  The descriptions of the variables are divided up into scalars,
263  one-dimensional arrays indexed to the tile number, and two and three  one-dimensional arrays indexed to the tile number, and two and
264  dimensional arrays indexed to tile number and neighboring tile.  This  three-dimensional arrays indexed to tile number and neighboring tile.
265  division actually reflects the functionality of these variables: the  This division reflects the functionality of these variables: The
266  scalars are common to every part of the topology, the tile-indexed  scalars are common to every part of the topology, the tile-indexed
267  arrays to individual tiles, and the arrays indexed to tile and  arrays to individual tiles, and the arrays indexed by tile and
268  neighbor to relationships between tiles and their neighbors.  neighbor to relationships between tiles and their neighbors. \\
269    
270  \subsubsection{Scalars}  \subsubsection{Scalars}
271    
272  The number of tiles in a particular topology is set with the parameter  The number of tiles in a particular topology is set with the parameter
273  \texttt{NTILES}, and the maximum number of neighbors of any tiles by  \code{NTILES}, and the maximum number of neighbors of any tiles by
274  \texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the  \code{MAX\_NEIGHBOURS}.  These parameters are used for defining the
275  size of the various one and two dimensional arrays that store tile  size of the various one and two dimensional arrays that store tile
276  parameters indexed to the tile number.\\  parameters indexed to the tile number and are assigned in the files
277    generated by \file{driver.m}.\\
278    
279  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
280  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
281  of tiles in the x and y global indices.  For example, the default  of tiles in the $x$ and $y$ global indices.  For example, the default
282  setup of six tiles has \texttt{exch2\_domain\_nxt=6} and  setup of six tiles (Fig. \ref{fig:6tile}) has
283  \texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in  \code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}.  A
284  gridpoints) tiles, four (2x2) per subdomain, will have  topology of twenty-four square tiles, four per subdomain (as in figure
285  \texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.  \ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and
286  Note that these parameters express the tile layout to allow global  \code{exch2\_domain\_nyt=2}.  Note that these parameters express the
287  data files that are tile-layout-neutral and have no bearing on the  tile layout to allow global data files that are tile-layout-neutral
288  internal storage of the arrays.  The tiles are internally stored in a  and have no bearing on the internal storage of the arrays.  The tiles
289  range from \texttt{1,bi} (in the x axis) and y-axis variable  are stored internally in a range from \code{(1:\varlink{bi}{bi})} the
290  \texttt{bj} is generally ignored within the package.  $x$ axis, and the $y$ axis variable \varlink{bj}{bj} generally is
291    ignored within the package. \\
292    
293  \subsubsection{Arrays Indexed to Tile Number}  \subsubsection{Arrays Indexed to Tile Number}
294    
295  The following arrays are of size \texttt{NTILES}, are indexed to the  The following arrays are of length \code{NTILES} and are indexed to
296  tile number, and the indices are omitted in their descriptions.  the tile number, which is indicated in the diagrams with the notation
297    \textsf{t}$n$.  The indices are omitted in the descriptions. \\
298    
299  The arrays \varlink{exch2\_tnx}{exch2_tnx} and  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
300  \varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each  \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of
301  tile.  At present for each tile \texttt{exch2\_tnx=sNx} and  each tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
302  \texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future  \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in
303  releases of MITgcm are to allow varying tile sizes.  section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
304    multiprocessing}.  Future releases of MITgcm are to allow varying tile
305    sizes. \\
306    
307  The location of the tiles' Cartesian origin within a subdomain are  The location of the tiles' Cartesian origin within a subdomain are
308  determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and  determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
309  \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to  \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
310  relate the location of the edges of the tiles to each other.  As an  relate the location of the edges of different tiles to each other.  As
311  example, in the default six-tile topology (the degenerate case) each  an example, in the default six-tile topology (Fig. \ref{fig:6tile})
312  index in these arrays are set to 0.  The twenty-four, 32x32 cube face  each index in these arrays is set to \code{0} since a tile occupies
313  case discussed above will have values of 0 or 16, depending on the  its entire subdomain.  The twenty-four-tile case discussed above will
314  quadrant the tile falls within the subdomain.  The array  have values of \code{0} or \code{16}, depending on the quadrant the
315  \varlink{exch2\_myFace}{exch2_myFace} contains the number of the  tile falls within the subdomain.  The elements of the arrays
316  cubeface/subdomain of each tile, numbered 1-6 in the case of the  \varlink{exch2\_txglobalo}{exch2_txglobalo} and
 standard cube topology.  
   
 The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and  
317  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to  \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
318  \varlink{exch2\_tbasex}{exch2_tbasex} and  \varlink{exch2\_tbasex}{exch2_tbasex} and
319  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
320  global address space, similar to that used by global files.  global address space, similar to that used by global output and input
321    files. \\
322    
323    The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of
324    the subdomain of each tile, in a range \code{(1:6)} in the case of the
325    standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
326    figures \ref{fig:12tile} and
327    \ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
328    contains a count of the neighboring tiles each tile has, and is used
329    for setting bounds for looping over neighboring tiles.
330    \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
331    tile, and is used in interprocess communication.  \\
332    
333    
334  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
335  \varlink{exch2\_isEedge}{exch2_isEedge},  \varlink{exch2\_isEedge}{exch2_isEedge},
336  \varlink{exch2\_isSedge}{exch2_isSedge}, and  \varlink{exch2\_isSedge}{exch2_isSedge}, and
337  \varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed  \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the
338  tile lies on the edge of a subdomain, 0 if not.  The values are used  indexed tile lies on the respective edge of a subdomain, \code{0} if
339  within the topology generator to determine the orientation of  not.  The values are used within the topology generator to determine
340  neighboring tiles and to indicate whether a tile lies on the corner of  the orientation of neighboring tiles, and to indicate whether a tile
341  a subdomain.  The latter case indicates special exchange and numerical  lies on the corner of a subdomain.  The latter case requires special
342  handling for the singularities at the eight corners of the cube.  exchange and numerical handling for the singularities at the eight
343  \varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of  corners of the cube. \\
344  how many neighboring tiles each tile has, and is used for setting  
 bounds for looping over neighboring tiles.  
 \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each  
 tile, and is used in interprocess communication.  
345    
346  \subsubsection{Arrays Indexed to Tile Number and Neighbor}  \subsubsection{Arrays Indexed to Tile Number and Neighbor}
347    
348  The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$  The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and
349  \texttt{NTILES} and describe the orientations between the the tiles.  \code{NTILES} and describe the orientations between the the tiles. \\
350    
351  The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for  The array \code{exch2\_neighbourId(a,T)} holds the tile number
352  each of the $n$ neighboring tiles.  The neighbor tiles are indexed  \code{Tn} for each of the tile number \code{T}'s neighboring tiles
353  \texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north  \code{a}.  The neighbor tiles are indexed
354  then south edges, and then top to bottom on the east and west edges.  \code{(1:exch2\_nNeighbours(T))} in the order right to left on the
355  Maybe throw in a fig here, eh?  north then south edges, and then top to bottom on the east then west
356    edges.  \\
357  The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c  
358  in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.   The \code{exch2\_opposingSend\_record(a,T)} array holds the
359  In other words,  index \code{b} of the element in \texttt{exch2\_neighbourId(b,Tn)}
360    that holds the tile number \code{T}, given
361    \code{Tn=exch2\_neighborId(a,T)}.  In other words,
362  \begin{verbatim}  \begin{verbatim}
363     exch2_neighbourId( exch2_opposingSend_record(a,T),     exch2_neighbourId( exch2_opposingSend_record(a,T),
364                        exch2_neighbourId(a,T) ) = T                        exch2_neighbourId(a,T) ) = T
365  \end{verbatim}  \end{verbatim}
366  and this provides a back-reference from the neighbor tiles.  This provides a back-reference from the neighbor tiles. \\
367    
368  The arrays \varlink{exch2\_pi}{exch2_pi},  The arrays \varlink{exch2\_pi}{exch2_pi} and
369  \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},  \varlink{exch2\_pj}{exch2_pj} specify the transformations of indices
370    in exchanges between the neighboring tiles.  These transformations are
371    necessary in exchanges between subdomains because the array index in
372    one dimension may map to the other index in an adjacent subdomain, and
373    may be have its indexing reversed. This swapping arises from the
374    ``folding'' of two-dimensional arrays into a three-dimensional
375    cube. \\
376    
377    The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)}
378    are the neighbor ID \code{N} and the tile number \code{T} as explained
379    above, plus a vector of length \code{2} containing transformation
380    factors \code{t}.  The first element of the transformation vector
381    holds the factor to multiply the index in the same axis, and the
382    second element holds the the same for the orthogonal index.  To
383    clarify, \code{exch2\_pi(1,N,T)} holds the mapping of the $x$ axis
384    index of tile \code{T} to the $x$ axis of tile \code{T}'s neighbor
385    \code{N}, and \code{exch2\_pi(2,N,T)} holds the mapping of \code{T}'s
386    $x$ index to the neighbor \code{N}'s $y$ index. \\
387    
388    One of the two elements of \code{exch2\_pi} or \code{exch2\_pj} for a
389    given tile \code{T} and neighbor \code{N} will be \code{0}, reflecting
390    the fact that the two axes are orthogonal.  The other element will be
391    \code{1} or \code{-1}, depending on whether the axes are indexed in
392    the same or opposite directions.  For example, the transform vector of
393    the arrays for all tile neighbors on the same subdomain will be
394    \code{(1,0)}, since all tiles on the same subdomain are oriented
395    identically.  An axis that corresponds to the orthogonal dimension
396    with the same index direction in a particular tile-neighbor
397    orientation will have \code{(0,1)}.  Those in the opposite index
398    direction will have \code{(0,-1)} in order to reverse the ordering. \\
399    
400    The arrays \varlink{exch2\_oi}{exch2_oi},
401  \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and  \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
402  \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in  \varlink{exch2\_oj\_f}{exch2_oj_f} are indexed to tile number and
403  exchanges between the neighboring tiles.  The dimensions of  neighbor and specify the relative offset within the subdomain of the
404  \texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the  array index of a variable going from a neighboring tile \code{N} to a
405  neighbor ID \textit{N} and the tile number \textit{T} as explained  local tile \code{T}.  Consider \code{T=1} in the six-tile topology
406  above, plus the transformation vector {\em t }, of length two.  The  (Fig. \ref{fig:6tile}), where
407  first element of the transformation vector indicates the factor by  
408  which variables representing the same vector component of a tile will  \begin{verbatim}
409  be multiplied, and the second element indicates the transform to the         exch2_oi(1,1)=33
410  variable in the other direction.  As an example,         exch2_oi(2,1)=0
411  \texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a         exch2_oi(3,1)=32
412  vector variable in tile \texttt{T} to the i-component of tile         exch2_oi(4,1)=-32
413  \texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold  \end{verbatim}
414  the component of neighbor \texttt{N}'s j-component.  
415    The simplest case is \code{exch2\_oi(2,1)}, the southern neighbor,
416  Under the current cube topology, one of the two elements of  which is \code{Tn=6}.  The axes of \code{T} and \code{Tn} have the
417  \texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}  same orientation and their $x$ axes have the same origin, and so an
418  and neighbor \texttt{N} will be 0, reflecting the fact that the vector  exchange between the two requires no changes to the $x$ index.  For
419  components are orthogonal.  The other element will be 1 or -1,  the western neighbor (\code{Tn=5}), \code{code\_oi(3,1)=32} since the
420  depending on whether the components are indexed in the same or  \code{x=0} vector on \code{T} corresponds to the \code{y=32} vector on
421  opposite directions.  For example, the transform dimension of the  \code{Tn}.  The eastern edge of \code{T} shows the reverse case
422  arrays for all tile neighbors on the same subdomain will be [1,0],  (\code{exch2\_oi(4,1)=-32)}), where \code{x=32} on \code{T} exchanges
423  since all tiles on the same subdomain are oriented identically.  with \code{x=0} on \code{Tn=2}. \\
424  Vectors that correspond to the orthogonal dimension with the same  
425  index direction will have [0,1], whereas those in the opposite index   The most interesting case, where \code{exch2\_oi(1,1)=33} and
426  direction will have [0,-1].  \code{Tn=3}, involves a reversal of indices.  As in every case, the
427    offset \code{exch2\_oi} is added to the original $x$ index of \code{T}
428    multiplied by the transformation factor \code{exch2\_pi(t,N,T)}.  Here
429    \code{exch2\_pi(1,1,1)=0} since the $x$ axis of \code{T} is orthogonal
430    to the $x$ axis of \code{Tn}.  \code{exch2\_pi(2,1,1)=-1} since the
431    $x$ axis of \code{T} corresponds to the $y$ axis of \code{Tn}, but the
432    index is reversed.  The result is that the index of the northern edge
433    of \code{T}, which runs \code{(1:32)}, is transformed to
434    \code{(-1:-32)}. \code{exch2\_oi(1,1)} is then added to this range to
435    get back \code{(32:1)} -- the index of the $y$ axis of \code{Tn}
436    relative to \code{T}.  This transformation may seem overly convoluted
437    for the six-tile case, but it is necessary to provide a general
438    solution for various topologies. \\
439    
440    
441    
442    Finally, \varlink{exch2\_itlo\_c}{exch2_itlo_c},
443    \varlink{exch2\_ithi\_c}{exch2_ithi_c},
444    \varlink{exch2\_jtlo\_c}{exch2_jtlo_c} and
445    \varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index
446    bounds of the edge segment of the neighbor tile \code{N}'s subdomain
447    that gets exchanged with the local tile \code{T}.  To take the example
448    of tile \code{T=2} in the twelve-tile topology
449    (Fig. \ref{fig:12tile}): \\
450    
 {\footnotesize  
451  \begin{verbatim}  \begin{verbatim}
452  C      exch2_pi          :: X index row of target to source permutation         exch2_itlo_c(4,2)=17
453  C                        :: matrix for each neighbour entry.                     exch2_ithi_c(4,2)=17
454  C      exch2_pj          :: Y index row of target to source permutation         exch2_jtlo_c(4,2)=0
455  C                        :: matrix for each neighbour entry.                     exch2_jthi_c(4,2)=33
 C      exch2_oi          :: X index element of target to source  
 C                        :: offset vector for cell-centered quantities    
 C                        :: of each neighbor entry.                      
 C      exch2_oj          :: Y index element of target to source  
 C                        :: offset vector for cell-centered quantities    
 C                        :: of each neighbor entry.                      
 C      exch2_oi_f        :: X index element of target to source  
 C                        :: offset vector for face quantities            
 C                        :: of each neighbor entry.                      
 C      exch2_oj_f        :: Y index element of target to source  
 C                        :: offset vector for face quantities            
 C                        :: of each neighbor entry.                      
456  \end{verbatim}  \end{verbatim}
457  }  
458    Here \code{N=4}, indicating the western neighbor, which is
459    \code{Tn=1}.  \code{Tn} resides on the same subdomain as \code{T}, so
460    the tiles have the same orientation and the same $x$ and $y$ axes.
461    The $x$ axis is orthogonal to the western edge and the tile is 16
462    points wide, so \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c}
463    indicate the column beyond \code{Tn}'s eastern edge, in that tile's
464    halo region. Since the border of the tiles extends through the entire
465    height of the subdomain, the $y$ axis bounds \code{exch2\_jtlo\_c} to
466    \code{exch2\_jthi\_c} cover the height of \code{(1:32)}, plus 1 in
467    either direction to cover part of the halo. \\
468    
469    For the north edge of the same tile \code{T=2} where \code{N=1} and
470    the neighbor tile is \code{Tn=5}:
471    
472    \begin{verbatim}
473           exch2_itlo_c(1,2)=0
474           exch2_ithi_c(1,2)=0
475           exch2_jtlo_c(1,2)=0
476           exch2_jthi_c(1,2)=17
477    \end{verbatim}
478    
479    \code{T}'s northern edge is parallel to the $x$ axis, but since
480    \code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis, \code{T}'s
481    northern edge exchanges with \code{Tn}'s western edge.  The western
482    edge of the tiles corresponds to the lower bound of the $x$ axis, so
483    \code{exch2\_itlo\_c} \code{exch2\_ithi\_c} are \code{0}. The range of
484    \code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the
485    width of \code{T}'s northern edge, plus the halo. \\
486    
 \subsection{Key Routines}  
487    
488    \subsection{Key Routines}
489    
490    Most of the subroutines particular to exch2 handle the exchanges
491    themselves and are of the same format as those described in
492    \ref{sect:cube_sphere_communication} \sectiontitle{Cube sphere
493    communication}.  Like the original routines, they are written as
494    templates which the local Makefile converts from RX into RL and RS
495    forms. \\
496    
497    The interfaces with the core model subroutines are
498    \code{EXCH\_UV\_XY\_RX}, \code{EXCH\_UV\_XYZ\_RX} and
499    \code{EXCH\_XY\_RX}.  They override the standard exchange routines
500    when \code{genmake2} is run with \code{exch2} option.  They in turn
501    call the local exch2 subroutines \code{EXCH2\_UV\_XY\_RX} and
502    \code{EXCH2\_UV\_XYZ\_RX} for two and three-dimensional vector
503    quantities, and \code{EXCH2\_XY\_RX} and \code{EXCH2\_XYZ\_RX} for two
504    and three-dimensional scalar quantities.  These subroutines set the
505    dimensions of the area to be exchanged, call \code{EXCH2\_RX1\_CUBE}
506    for scalars and \code{EXCH2\_RX2\_CUBE} for vectors, and then handle
507    the singularities at the cube corners. \\
508    
509    The separate scalar and vector forms of \code{EXCH2\_RX1\_CUBE} and
510    \code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subrouine
511    needs to pass both the $u$ and $v$ components of the phsical vectors.
512    This arises from the topological folding discussed above, where the
513    $x$ and $y$ axes get swapped in some cases.  This swapping is not an
514    issue with the scalar version. These subroutines call
515    \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of
516    the work using the variables discussed above. \\
517    
 \subsection{References}  
Legend:
Removed from v.1.10  
changed lines
  Added in v.1.17
  ViewVC Help
Powered by ViewVC 1.1.22