/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.4 by afe, Thu Jan 29 17:55:35 2004 UTC revision 1.12 by afe, Tue Mar 16 21:52:15 2004 UTC
# Line 10  Line 10 
10  %%    o automatically inserted at \section{Reference}  %%    o automatically inserted at \section{Reference}
11    
12    
13  \section{exch2: Extended Cubed Sphere Exchange}  \section{exch2: Extended Cubed Sphere \mbox{Topology}}
14  \label{sec:exch2}  \label{sec:exch2}
15    
16    
17  \subsection{Introduction}  \subsection{Introduction}
18    
19  The exch2 package is an extension to the original cubed sphere exchanges  The \texttt{exch2} package extends the original cubed
20  to allow more flexible domain decomposition and parallelization.  Cube faces  sphere topology configuration to allow more flexible domain
21  (subdomains) may be divided into whatever number of tiles that divide evenly  decomposition and parallelization.  Cube faces (also called
22  into the grid point dimensions of the subdomain.  Furthermore, the individual  subdomains) may be divided into any number of tiles that divide evenly
23  tiles may be run on separate processors in different combinations,  into the grid point dimensions of the subdomain.  Furthermore, the
24  and whether exchanges between particular tiles occur between different  individual tiles may be run on separate processors in different
25  processors is determined at runtime.  combinations, and whether exchanges between particular tiles occur
26    between different processors is determined at runtime.  This
27  The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and  flexibility provides for manual compile-time load balancing across a
28  assigned in {\em w2\_e2setup.F}, both in the  relatively arbitrary number of processors. \\
29  {\em pkg/exch2} directory.  The validity of the cube topology depends  
30  on the {\em SIZE.h} file as detailed below.  Both files are generated by  The exchange parameters are declared in
31  Matlab scripts and  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
32  should not be edited.  The default files provided in the release set up  and assigned in
33  a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
34  points, running on a single processor.    validity of the cube topology depends on the \file{SIZE.h} file as
35    detailed below.  The default files provided in the release configure a
36    cubed sphere topology of six tiles, one per subdomain, each with
37    32$\times$32 grid points, all running on a single processor.  Both
38    files are generated by Matlab scripts in
39    \file{utils/exch2/matlab-topology-generator}; see Section
40    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
41    for details on creating alternate topologies.  Pregenerated examples
42    of these files with alternate topologies are provided under
43    \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
44    file for single-processor execution.
45    
46    \subsection{Invoking exch2}
47    
48    To use exch2 with the cubed sphere, the following conditions must be
49    met: \\
50    
51    $\bullet$ The exch2 package is included when \file{genmake2} is run.
52      The easiest way to do this is to add the line \code{exch2} to the
53      \file{profile.conf} file -- see Section
54      \ref{sect:buildingCode} \sectiontitle{Building the code} for general
55      details. \\
56    
57    $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
58      \file{w2\_e2setup.F} must reside in a directory containing code
59      linked when \file{genmake2} runs.  The safest place to put these
60      is the directory indicated in the \code{-mods=DIR} command line
61      modifier (typically \file{../code}), or the build directory.  The
62      default versions of these files reside in \file{pkg/exch2} and are
63      linked automatically if no other versions exist elsewhere in the
64      link path, but they should be left untouched to avoid breaking
65      configurations other than the one you intend to modify.\\
66    
67    $\bullet$ Files containing grid parameters, named
68      \file{tile00$n$.mitgrid} where $n$=[1,6] (one per subdomain), must
69      be in the working directory when the MITgcm executable is run.
70      These files are provided in the example experiments for cubed sphere
71      configurations with 32$\times$32 cube sides and are non-trivial to
72      generate -- please contact MITgcm support if you want to generate
73      files for other configurations. \\
74    
75    $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must
76      be placed where \file{genmake2} will find it.  In particular for the
77      exch2, the domain decomposition specified in \file{SIZE.h} must
78      correspond with the particular configuration's topology specified in
79      \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain
80      decomposition issues particular to exch2 are addressed in Section
81      \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
82      and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and MPI}; a more
83      general background on the subject relevant to MITgcm is presented in
84      Section \ref{sect:specifying_a_decomposition}
85      \sectiontitle{Specifying a decomposition}.\\
86    
87    As of the time of writing the following examples use exch2 and may be
88    used for guidance:
89    
90    \begin{verbatim}
91    verification/adjust_nlfs.cs-32x32x1
92    verification/adjustment.cs-32x32x1
93    verification/aim.5l_cs
94    verification/global_ocean.cs32x15
95    verification/hs94.cs-32x32x5
96    \end{verbatim}
97    
98    
99    
100    
101    \subsection{Generating Topology Files for exch2}
102    \label{sec:topogen}
103    
104    Alternate cubed sphere topologies may be created using the Matlab
105    scripts in \file{utils/exch2/matlab-topology-generator}. Running the
106    m-file
107    \filelink{driver.m}{utils-exch2-matlab-topology-generator_driver.m}
108    from the Matlab prompt (there are no parameters to pass) generates
109    exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and
110    \file{w2\_e2setup.F} in the working directory and displays a figure of
111    the topology via Matlab.  The other m-files in the directory are
112    subroutines of \file{driver.m} and should not be run ``bare'' except
113    for development purposes. \\
114    
115    The parameters that determine the dimensions and topology of the
116    generated configuration are \code{nr}, \code{nb}, \code{ng},
117    \code{tnx} and \code{tny}, and all are assigned early in the script. \\
118    
119    The first three determine the size of the subdomains and
120    hence the size of the overall domain.  Each one determines the number
121    of grid points, and therefore the resolution, along the subdomain
122    sides in a ``great circle'' around each axis of the cube.  At the time
123    of this writing MITgcm requires these three parameters to be equal,
124    but they provide for future releases  to accomodate different
125    resolutions around the axes to allow (for example) greater resolution
126    around the equator.\\
127    
128    The parameters \code{tnx} and \code{tny} determine the dimensions of
129    the tiles into which the subdomains are decomposed, and must evenly
130    divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
131    The result is a rectangular tiling of the subdomain.  Figure
132    \ref{fig:24tile} shows one possible topology for a twenty-four tile
133    cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
134    
135    \begin{figure}
136    \begin{center}
137     \resizebox{4in}{!}{
138      \includegraphics{part6/s24t_16x16.ps}
139     }
140    \end{center}
141    
142    \caption{Plot of cubed sphere topology with a 32$\times$192 domain
143    divided into six 32$\times$32 subdomains, each of which is divided into four tiles
144    (\code{tnx=16, tny=16}) for a total of twenty-four tiles.
145    } \label{fig:24tile}
146    \end{figure}
147    
148    \begin{figure}
149    \begin{center}
150     \resizebox{4in}{!}{
151      \includegraphics{part6/s12t_16x32.ps}
152     }
153    \end{center}
154    \caption{Plot of cubed sphere topology with a 32$\times$192 domain
155    divided into six 32$\times$32 subdomains of two tiles each
156     (\code{tnx=16, tny=32}).
157    } \label{fig:12tile}
158    \end{figure}
159    
160    Tiles can be selected from the topology to be omitted from being
161    allocated memory and processors.  This tuning is useful in ocean
162    modeling for omitting tiles that fall entirely on land.  The tiles
163    omitted are specified in the file
164    \filelink{blanklist.txt}{utils-exch2-matlab-topology-generator_blanklist.txt}
165    by their tile number in the topology, separated by a newline. \\
166    
167    
168    
169    
170    \subsection{exch2, SIZE.h, and multiprocessing}
171    \label{sec:exch2mpi}
172    
173    Once the topology configuration files are created, the Fortran
174    parameters in \file{SIZE.h} must be configured to match.  Section
175    \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying a
176    decomposition} provides a general description of domain decomposition
177    within MITgcm and its relation to \file{SIZE.h}. The current section
178    specifies certain constraints the exch2 package imposes as well as
179    describes how to enable parallel execution with MPI. \\
180    
181    As in the general case, the parameters \varlink{sNx}{sNx} and
182    \varlink{sNy}{sNy} define the size of the individual tiles, and so
183    must be assigned the same respective values as \code{tnx} and
184    \code{tny} in \file{driver.m}.\\
185    
186    The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy}
187    have no special bearing on exch2 and may be assigned as in the general
188    case. The same holds for \varlink{Nr}{Nr}, the number of vertical
189    levels in the model.\\
190    
191    The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},
192    \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of
193    tiles and how they are distributed on processors.  When using exch2,
194    the tiles are stored in single dimension, and so
195    \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as
196    configured by exch2 cannot be split up accross processors without
197    regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\
198    
199    The number of tiles MITgcm allocates and how they are distributed
200    between processors depends on \varlink{nPx}{nPx} and
201    \varlink{nSx}{nSx}.  \varlink{nSx}{nSx} is the number of tiles per
202    processor and \varlink{nPx}{nPx} the number of processors.  The total
203    number of tiles in the topology minus those listed in
204    \file{blanklist.txt} must equal \code{nSx*nPx}. \\
205    
206    The following is an example of \file{SIZE.h} for the twelve-tile
207    configuration illustrated in figure \ref{fig:12tile} running on
208    one processor: \\
209    
210    \begin{verbatim}
211          PARAMETER (
212         &           sNx =  16,
213         &           sNy =  32,
214         &           OLx =   2,
215         &           OLy =   2,
216         &           nSx =  12,
217         &           nSy =   1,
218         &           nPx =   1,
219         &           nPy =   1,
220         &           Nx  = sNx*nSx*nPx,
221         &           Ny  = sNy*nSy*nPy,
222         &           Nr  =   5)
223    \end{verbatim}
224    
225    The following is an example for the twentyfour-tile topology in figure
226    \ref{fig:24tile} running on six processors:
227    
228    \begin{verbatim}
229          PARAMETER (
230         &           sNx =  16,
231         &           sNy =  16,
232         &           OLx =   2,
233         &           OLy =   2,
234         &           nSx =   4,
235         &           nSy =   1,
236         &           nPx =   6,
237         &           nPy =   1,
238         &           Nx  = sNx*nSx*nPx,
239         &           Ny  = sNy*nSy*nPy,
240         &           Nr  =   5)
241    \end{verbatim}
242    
243    
244    
245    
246    
247  \subsection{Key Variables}  \subsection{Key Variables}
248    
249  The descriptions of the variables are divided up into scalars,  The descriptions of the variables are divided up into scalars,
250  one-dimensional arrays indexed to the tile number, and two-dimensional  one-dimensional arrays indexed to the tile number, and two and three
251  arrays indexed to tile number and neighboring tile.  This division  dimensional arrays indexed to tile number and neighboring tile.  This
252  actually reflects  the functionality of these variables, not just the  division reflects the functionality of these variables: The
253  whim of some FORTRAN enthusiast.  scalars are common to every part of the topology, the tile-indexed
254    arrays to individual tiles, and the arrays indexed by tile and
255    neighbor to relationships between tiles and their neighbors. \\
256    
257  \subsubsection{Scalars}  \subsubsection{Scalars}
258    
259  The number of tiles in a particular topology is set with the parameter  The number of tiles in a particular topology is set with the parameter
260  {\em NTILES}, and the maximum number of neighbors of any tiles by  \code{NTILES}, and the maximum number of neighbors of any tiles by
261  {\em MAX\_NEIGHBOURS}.  These parameters are used for defining the size of  \code{MAX\_NEIGHBOURS}.  These parameters are used for defining the
262  the various one and two dimensional arrays that store tile parameters  size of the various one and two dimensional arrays that store tile
263  indexed to the tile number.  parameters indexed to the tile number and are assigned in the files
264    generated by \file{driver.m}.\\
265  The scalar parameters {\em exch2\_domain\_nxt} and  
266  {\em exch2\_domain\_nyt} express the number of tiles in the x and y global  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
267  indices.  For example, the default setup of six tiles has  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
268  {\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}.  A topology of  of tiles in the $x$ and $y$ global indices.  For example, the default
269  twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will  setup of six tiles has \code{exch2\_domain\_nxt=6} and
270  have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}.  Note  \code{exch2\_domain\_nyt=1}.  A topology of twenty-four square tiles,
271  that these parameters express the tile layout to allow global data files that  four per subdomain (as in figure \ref{fig:24tile}), will have
272  are tile-layout-neutral and have no bearing on the internal storage of the  \code{exch2\_domain\_nxt=12} and \code{exch2\_domain\_nyt=2}.  Note
273  arrays.  The tiles are internally stored in a range from {\em 1,bi} (in the  that these parameters express the tile layout to allow global data
274  x axis) and y-axis variable {\em bj} is generally ignored within the package.  files that are tile-layout-neutral and have no bearing on the internal
275    storage of the arrays.  The tiles are internally stored in a range
276  \subsubsection{One-Dimensional Arrays}  from [1,\varlink{bi}{bi}] the $x$ axis and $y$ axis variable
277    \varlink{bj}{bj} is generally ignored within the package. \\
278  The following arrays are indexed to the tile number, and the indices are  
279  omitted in their descriptions.  \subsubsection{Arrays Indexed to Tile Number}
280    
281  The arrays {\em exch2\_tnx} and {\em exch2\_tny}  The following arrays are of size \code{NTILES}, are indexed to the
282  express the x and y dimensions of each tile.  At present for each tile  tile number, and the indices are omitted in their descriptions. \\
283  {\em exch2\_tnx = sNx}  
284  and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}.  Future releases of  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
285  MITgcm are to allow varying tile sizes.  \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of
286    each tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
287  The location of the tiles' Cartesian origin within a subdomain are determined  \texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in
288  by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}.  These  section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
289    multiprocessing}.  Future releases of MITgcm are to allow varying tile
290  \subsubsection{Two-Dimensional Arrays}  sizes. \\
291    
292    The location of the tiles' Cartesian origin within a subdomain are
293    determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
294    \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
295    relate the location of the edges of different tiles to each other.  As
296    an example, in the default six-tile topology ??  each index in these
297    arrays are set to \code{0}.  The twentyfour-tile case discussed above
298    will have values of \code{0} or \code{16}, depending on the quadrant
299    the tile falls within the subdomain.  The array
300    \varlink{exch2\_myFace}{exch2_myFace} contains the number of the
301    subdomain of each tile, numbered \code{(1:6)} in the case of the
302    standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
303    figures \ref{fig:12tile}) and \ref{fig:24tile}). \\
304    
305    The elements of the arrays \varlink{exch2\_txglobalo}{exch2_txglobalo}
306    and \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
307    \varlink{exch2\_tbasex}{exch2_tbasex} and
308    \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
309    global address space, similar to that used by global files. \\
310    
311    The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
312    \varlink{exch2\_isEedge}{exch2_isEedge},
313    \varlink{exch2\_isSedge}{exch2_isSedge}, and
314    \varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the
315    indexed tile lies on the edge of a subdomain, \code{0} if not.  The
316    values are used within the topology generator to determine the
317    orientation of neighboring tiles, and to indicate whether a tile lies
318    on the corner of a subdomain.  The latter case requires special
319    exchange and numerical handling for the singularities at the eight
320    corners of the cube.  \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
321    contains a count of how many neighboring tiles each tile has, and is
322    used for setting bounds for looping over neighboring tiles.
323    \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
324    tile, and is used in interprocess communication.  \\
325    
326    \subsubsection{Arrays Indexed to Tile Number and Neighbor}
327    
328    The following arrays are all of size
329    \code{MAX\_NEIGHBOURS}$\times$\code{NTILES} and describe the
330    orientations between the the tiles. \\
331    
332    The array \code{exch2\_neighbourId(a,T)} holds the tile number
333    \code{Tn} for each of the tile number \code{T}'s neighboring tiles
334    \code{a}.  The neighbor tiles are indexed \code{(1:MAX\_NEIGHBOURS)}
335    in the order right to left on the north then south edges, and then top
336    to bottom on the east and west edges.  Maybe throw in a fig here, eh?
337    \\
338    
339    The \code{exch2\_opposingSend\_record(a,T)} array holds the index
340    \code{b} in \texttt{exch2\_neighbourId(b,Tn)} that holds the tile
341    number \code{T}.  In other words,
342    \begin{verbatim}
343       exch2_neighbourId( exch2_opposingSend_record(a,T),
344                          exch2_neighbourId(a,T) ) = T
345    \end{verbatim}
346    This provides a back-reference from the neighbor tiles. \\
347    
348    The arrays \varlink{exch2\_pi}{exch2_pi},
349    \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
350    \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
351    \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
352    exchanges between the neighboring tiles.  The dimensions of
353    \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)} are the neighbor
354    ID \code{N} and the tile number \code{T} as explained above, plus a
355    vector of length 2 containing transformation factors \code{t}.  The
356    first element of the transformation vector indicates the factor
357    \code{t} by which variables representing the same vector component of
358    a tile \code{T} will be multiplied in exchanges with neighbor
359    \code{N}, and the second element indicates the transform to the
360    variable in the other direction.  As an example,
361    \code{exch2\_pi(1,N,T)} holds the transform of the $i$ component of a
362    vector variable in tile \code{T} to the $i$ component of tile
363    \code{T}'s neighbor \code{N}, and \code{exch2\_pi(2,N,T)} hold the
364    component of neighbor \code{N}'s $j$ component. \\
365    
366    Under the current cube topology, one of the two elements of
367    \code{exch2\_pi} or \code{exch2\_pj} for a given tile \code{T} and
368    neighbor \code{N} will be \code{0}, reflecting the fact that the two
369    vector components are orthogonal.  The other element will be 1 or -1,
370    depending on whether the components are indexed in the same or
371    opposite directions.  For example, the transform vector of the arrays
372    for all tile neighbors on the same subdomain will be \code{(1,0)},
373    since all tiles on the same subdomain are oriented identically.  A
374    vector direction that corresponds to the orthogonal dimension with the
375    same index direction in a particular tile-neighbor orientation will
376    have \code{(0,1)}, whereas those in the opposite index direction will
377    have \code{(0,-1)}.  This needs some diagrams.
378    
 //  
379    
380    {\footnotesize
381  \begin{verbatim}  \begin{verbatim}
 C      NTILES            :: Number of tiles in this topology  
 C      MAX_NEIGHBOURS    :: Maximum number of neighbours any tile has.  
 C      exch2_domain_nxt  :: Total domain length in tiles.  
 C      exch2_domain_nyt  :: Maximum domain height in tiles.  
 C      exch2_tnx         :: Size in X for each tile.                    
 C      exch2_tny         :: Size in Y for each tile.                    
 C      exch2_tbasex      :: Tile offset in X within its sub-domain (cube face)  
 C      exch2_tbasey      :: Tile offset in Y within its sub-domain (cube face)  
 C      exch2_tglobalxlo  :: Tile base X index within global index space.  
 C      exch2_tglobalylo  :: Tile base Y index within global index space.  
 C      exch2_isWedge     :: 0 if West not at domain edge, 1 if it is.    
 C      exch2_isNedge     :: 0 if North not at domain edge, 1 if it is.    
 C      exch2_isEedge     :: 0 if East not at domain edge, 1 if it is.    
 C      exch2_isSedge     :: 0 if South not at domain edge, 1 if it is.    
 C      exch2_myFace      :: Cube face number used for I/O.                
 C      exch2_nNeighbours :: Tile neighbour entries count.                
 C      exch2_tProc       :: Rank of process owning tile                  
 C                        :: (filled at run time).                        
 C      exch2_neighbourId :: Tile number for each neighbour entry.          
 C      exch2_opposingSend_record :: Record for entry in target tile send  
 C                                :: list that has this tile and face      
 C                                :: as its target.                        
382  C      exch2_pi          :: X index row of target to source permutation  C      exch2_pi          :: X index row of target to source permutation
383  C                        :: matrix for each neighbour entry.              C                        :: matrix for each neighbour entry.            
384  C      exch2_pj          :: Y index row of target to source permutation  C      exch2_pj          :: Y index row of target to source permutation
# Line 119  C      exch2_oj_f        :: Y index elem Line 396  C      exch2_oj_f        :: Y index elem
396  C                        :: offset vector for face quantities            C                        :: offset vector for face quantities          
397  C                        :: of each neighbor entry.                      C                        :: of each neighbor entry.                    
398  \end{verbatim}  \end{verbatim}
399    }
400    
401    
402    
Legend:
Removed from v.1.4  
changed lines
  Added in v.1.12
  ViewVC Help
Powered by ViewVC 1.1.22