/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.7 by afe, Wed Feb 11 20:48:14 2004 UTC revision 1.11 by afe, Mon Mar 15 22:39:28 2004 UTC
# Line 10  Line 10 
10  %%    o automatically inserted at \section{Reference}  %%    o automatically inserted at \section{Reference}
11    
12    
13  \section{exch2: Extended Cubed Sphere Exchange}  \section{exch2: Extended Cubed Sphere \mbox{Topology}}
14  \label{sec:exch2}  \label{sec:exch2}
15    
16    
17  \subsection{Introduction}  \subsection{Introduction}
18    
19  The exch2 package is an extension to the original cubed sphere exchanges  The \texttt{exch2} package is an extension to the original cubed
20  to allow more flexible domain decomposition and parallelization.  Cube faces  sphere topological configuration that allows more flexible domain
21  (subdomains) may be divided into whatever number of tiles that divide evenly  decomposition and parallelization.  Cube faces (also called
22  into the grid point dimensions of the subdomain.  Furthermore, the individual  subdomains) may be divided into any number of tiles that divide evenly
23  tiles may be run on separate processors in different combinations,  into the grid point dimensions of the subdomain.  Furthermore, the
24  and whether exchanges between particular tiles occur between different  individual tiles may be run on separate processors in different
25  processors is determined at runtime.  combinations, and whether exchanges between particular tiles occur
26    between different processors is determined at runtime.  This
27  The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and  flexibility provides for manual compile-time load balancing across a
28  assigned in {\em w2\_e2setup.F}, both in the  relatively arbitrary number of processors. \\
29  {\em pkg/exch2} directory.  The validity of the cube topology depends  
30  on the {\em SIZE.h} file as detailed below.  Both files are generated by  The exchange parameters are declared in
31  Matlab scripts and  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
32  should not be edited.  The default files provided in the release set up  and assigned in
33  a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
34  points, running on a single processor.    validity of the cube topology depends on the \file{SIZE.h} file as
35    detailed below.  Both files are generated by Matlab scripts in
36    \file{utils/exch2/matlab-topology-generator}; see Section
37    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
38    for details on creating alternate topologies.  The default files
39    provided in the release configure a cubed sphere topology of six
40    tiles, one per subdomain, each with 32$\times$32 grid points, all
41    running on a single processor.  Pregenerated examples of these files
42    with alternate topologies are provided under
43    \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
44    file for single-processor execution.
45    
46    \subsection{Invoking exch2}
47    
48    To use exch2 with the cubed sphere, the following conditions must be
49    met: \\
50    
51    $\bullet$ The exch2 package is included when \file{genmake2} is run.
52      The easiest way to do this is to add the line \code{exch2} to the
53      \file{profile.conf} file -- see Section
54      \ref{sect:buildingCode}\sectiontitle{Building the code} for general
55      details. \\
56    
57    $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
58      \file{w2\_e2setup.F} must reside in a directory containing code
59      linked when \file{genmake2} runs.  The safest place to put these
60      is the directory indicated in the \code{-mods=DIR} command line
61      modifier (typically \file{../code}), or the build directory.  The
62      default versions of these files reside in \file{pkg/exch2} and are
63      linked automatically if no other versions exist elsewhere in the
64      link path, but they should be left untouched to avoid breaking
65      configurations other than the one you intend to modify.\\
66    
67    $\bullet$ Files containing grid parameters, named
68      \file{tile???.mitgrid} where \file{???} is \file{001} through
69      \file{006} (one per subdomain), must be in the working directory
70      when the MITgcm executable is run.  These files are provided in the
71      example experiments for cubed sphere configurations with
72      32$\times$32 cube sides and are non-trivial to generate -- please
73      contact MITgcm support if you want to generate files for other
74      configurations. \\
75    
76    $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h}
77      must be placed where \file{genmake2} will find it.  In particular
78      for the exch2, the domain decomposition specified in \file{SIZE.h}
79      must correspond with the particular configuration's topology
80      specified in \file{W2\_EXCH2\_TOPOLOGY.h} and
81      \file{w2\_e2setup.F}.  Domain decomposition issues particular to
82      exch2 are addressed in Section \ref{sec:topogen} \sectiontitle{Generating
83      Topology Files for exch2}; a more general background on the subject
84      relvant to MITgcm is presented in Section
85      \ref{sect:specifying_a_decomposition}\sectiontitle{Specifying a
86      decomposition}.\\
87    
88  \subsection{Key Variables}  As of the time of writing the following examples use exch2 and may be
89    used for guidance:
90    
91  The descriptions of the variables are divided up into scalars,  \begin{verbatim}
92  one-dimensional arrays indexed to the tile number, and two and three  verification/adjust_nlfs.cs-32x32x1
93  dimensional  verification/adjustment.cs-32x32x1
94  arrays indexed to tile number and neighboring tile.  This division  verification/aim.5l_cs
95  actually reflects  the functionality of these variables: the scalars  verification/global_ocean.cs32x15
96  are common to every part of the topology, the tile-indexed arrays to  verification/hs94.cs-32x32x5
97  individual tiles, and the arrays indexed to tile and neighbor to  \end{verbatim}
 relationships between tiles and their neighbors.  
98    
 \subsubsection{Scalars}  
99    
 The number of tiles in a particular topology is set with the parameter  
 {\em NTILES}, and the maximum number of neighbors of any tiles by  
 {\em MAX\_NEIGHBOURS}.  These parameters are used for defining the size of  
 the various one and two dimensional arrays that store tile parameters  
 indexed to the tile number.  
   
 The scalar parameters {\em exch2\_domain\_nxt} and  
 {\em exch2\_domain\_nyt} express the number of tiles in the x and y global  
 indices.  For example, the default setup of six tiles has  
 {\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}.  A topology of  
 twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will  
 have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}.  Note  
 that these parameters express the tile layout to allow global data files that  
 are tile-layout-neutral and have no bearing on the internal storage of the  
 arrays.  The tiles are internally stored in a range from {\em 1,bi} (in the  
 x axis) and y-axis variable {\em bj} is generally ignored within the package.  
100    
 \subsubsection{Arrays Indexed to Tile Number}  
101    
102  The following arrays are of size {\em NTILES}, are indexed to the tile number,  \subsection{Generating Topology Files for exch2}
103  and the indices are omitted in their descriptions.  \label{sec:topogen}
104    
105    Alternate cubed sphere topologies may be created using the Matlab
106    scripts in \file{utils/exch2/matlab-topology-generator}. Running the
107    m-file \file{driver.m} from the Matlab prompt (there are no parameters
108    to pass) generates exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h}
109    and \file{w2\_e2setup.F} in the working directory and displays a
110    figure of the topology via Matlab.  The other m-files in the directory
111    are subroutines of \file{driver.m} and should not be run except for
112    development purposes. \\
113    
114    The parameters that determine the dimensions and topology of the
115    generated configuration are \code{nr}, \code{nb}, \code{ng},
116    \code{tnx} and \code{tny}, and all are assigned early in the script.
117    
118    The first three determine the size of the subdomains (cube faces) and
119    hence the size of the overall domain.  Each one determines the number
120    of grid points, and therefore the resolution, along the subdomain
121    sides in a ``great circle'' around each axis of the cube.  At the time
122    of this writing MITgcm requires these three parameters to be equal,
123    but they provide for future releases of MITgcm to accomodate different
124    resolutions around the axes to allow (for example) greater resolution
125    around the equator.\\
126    
127    The parameters \code{tnx} and \code{tny} determine the dimensions of
128    the tiles into which the subdomains are decomposed, and must evenly
129    divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
130    The result is a rectangular tiling of the subdomain.  Figure
131    \ref{fig:24tile} shows one possible topology for a twenty-four tile
132    cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
133    
134    \begin{figure}
135    \begin{center}
136     \resizebox{4in}{!}{
137      \includegraphics{part6/s24t_16x16.ps}
138     }
139    \end{center}
140    \caption{Plot of cubed sphere topology with a 32$\times$32 grid and
141    twenty-four tiles (\code{tnx=16, tny=16})
142    } \label{fig:24tile}
143    \end{figure}
144    
145    \begin{figure}
146    \begin{center}
147     \resizebox{4in}{!}{
148      \includegraphics{part6/s12t_16x32.ps}
149     }
150    \end{center}
151    \caption{Plot of cubed sphere topology with a 32$\times$32 grid and
152    twelve tiles (\code{tnx=16, tny=32})
153    } \label{fig:12tile}
154    \end{figure}
155    
156    Tiles can be selected from the topology to be omitted from being
157    allocated memory and processors.  This kind otuning is useful in
158    ocean modeling for omitting tiles that fall entirely on land.  The
159    tiles omitted are specified in the file \file{blanklist.txt} by
160    their tile number in the topology, separated by a newline. \\
161    
162    
 The arrays {\em exch2\_tnx} and {\em exch2\_tny}  
 express the x and y dimensions of each tile.  At present for each tile  
 {\em exch2\_tnx = sNx}  
 and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}.  Future releases of  
 MITgcm are to allow varying tile sizes.  
   
 The location of the tiles' Cartesian origin within a subdomain are determined  
 by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}.  These variables  
 are used to relate the location of the edges of the tiles to each other.  As  
 an example, in the default six-tile topology (the degenerate case)  
 each index in these arrays are  
 set to 0.  The twenty-four, 32x32 cube face case discussed above will have  
 values of 0 or 16, depending on the quadrant the tile falls within the  
 subdomain.  {\em exch2\_myFace} contains the number of the  
 cubeface/subdomain of each tile, numbered 1-6 in the case of the standard  
 cube topology.    
   
 The arrays {\em exch2\_txglobalo} and {\em exch2\_txglobalo} are similar to  
 {\em exch2\_tbasex} and {\em exch2\_tbasey}, but locate the tiles within  
 the global address space, similar to that used by global files.    
   
 The arrays {\em exch2\_isWedge}, {\em exch2\_isEedge}, {\em exch2\_isSedge},  
 and {\em exch2\_isNedge} are set to 1 if the indexed tile lies on the edge  
 of a subdomain, 0 if not.  The values are used within the topology generator  
 to determine the orientation of neighboring tiles and to indicate whether  
 a tile lies on the corner of a subdomain.  The latter case indicates  
 special exchange and numerical handling for the singularities at the eight  
 corners of the cube.  {\em exch2\_isNedge} contains a count of how many  
 neighboring tiles each tile has, and is used for setting bounds for looping  
 over neighboring tiles.  {\em exch2\_tProc} holds the process rank of each tile,  
 and is used in interprocess communication.  
163    
 \subsubsection{Arrays Indexed to Tile Number and Neighbor}  
164    
 The following arrays are all of size {\em MAX\_NEIGHBOURS}x{\em NTILES} and  
 describe the orientations between the the tiles.  
165    
 The array {\em exch2\_neighbourId(a,T)} holds the tile number $T_{n}$ for each tile  
 {\em T}'s neighbor tile {\em a}.  The neighbor tiles are indexed {\em 1,MAX\_NEIGHBOURS }  
 in the order right to left on the north then south edges, and then top to bottom on the east  
 and west edges. maybe throw in a fig here, eh?    
   
 {\em exch2\_opposingSend\_record(a,T)} holds  
 the index c in {\em exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.  
 In other words,  
   
 \begin{verbatim}    
 exch2_neighbourId( exch2_opposingSend_record(a,T),  
                            exch2_neighbourId(a,T) ) = T  
 \end{verbatim}  
166    
167  % {\em exch2\_neighbourId(exch2\_opposingSend\_record(a,T),exch2\_neighbourId(a,T))=T}.  \subsection{Key Variables}
 % alternate version  
168    
169  This is to provide a backreference from the neighbor tiles.  The descriptions of the variables are divided up into scalars,
170    one-dimensional arrays indexed to the tile number, and two and three
171    dimensional arrays indexed to tile number and neighboring tile.  This
172    division actually reflects the functionality of these variables: the
173    scalars are common to every part of the topology, the tile-indexed
174    arrays to individual tiles, and the arrays indexed to tile and
175    neighbor to relationships between tiles and their neighbors.
176    
177  The arrays {\em exch2\_pi }, {\em exch2\_pj }, {\em exch2\_oi },  \subsubsection{Scalars}
 {\em exch2\_oj }, {\em exch2\_oi\_f }, and {\em exch2\_oj\_f }  specify  
 the transformations in exchanges between the neighboring tiles.  The dimensions    
 of {\em exch2\_pi(t,N,T) } and {\em exch2\_pj(t,N,T) } are the neighbor ID  
 { \em N } and the tile number {\em T } as explained above, plus the transformation  
 vector {\em t }, of length two.  The first element of the transformation vector indicates  
 the factor by which variables representing the same vector component  of a tile  
 will be multiplied, and the second element indicates the transform to the  
 variable in the other direction.  As an example, {\em exch2\_pi(1,N,T) } holds the  
 transform of the i-component of a vector variable in tile {\em T } to the i-component of  
 tile  {\em T }'s neighbor  {\em N }, and {\em exch2\_pi(2,N,T) } hold the component  
 of neighbor  {\em N }'s j-component.  
   
 Under the current cube topology, one of the two elements of {\em exch2\_pi } or {\em exch2\_pj }  
 for a given tile   {\em T } and  neighbor  {\em N } will be 0, reflecting the fact that  
 the vector components are orthogonal.  The other element will be 1 or -1, depending on whether  
 the components are indexed in the same or opposite directions.  For example, the transform dimension  
 of the arrays for all tile neighbors on the same subdomain will be {\em [1 , 0] }, since all tiles on  
 the same subdomain are oriented identically.  Vectors that correspond to the orthogonal dimension with the  
 same index direction will have {\em [0 , 1] }, whereas those in the opposite index direction will have  
 {\em [0 , -1] }.  
178    
179    The number of tiles in a particular topology is set with the parameter
180    \texttt{NTILES}, and the maximum number of neighbors of any tiles by
181    \texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the
182    size of the various one and two dimensional arrays that store tile
183    parameters indexed to the tile number.\\
184    
185    The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
186    and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
187    of tiles in the x and y global indices.  For example, the default
188    setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
189    \texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in
190    gridpoints) tiles, four (2x2) per subdomain, will have
191    \texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
192    Note that these parameters express the tile layout to allow global
193    data files that are tile-layout-neutral and have no bearing on the
194    internal storage of the arrays.  The tiles are internally stored in a
195    range from \texttt{1,bi} (in the x axis) and y-axis variable
196    \texttt{bj} is generally ignored within the package.
197    
198    \subsubsection{Arrays Indexed to Tile Number}
199    
200    The following arrays are of size \texttt{NTILES}, are indexed to the
201    tile number, and the indices are omitted in their descriptions.
202    
203  //  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
204    \varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
205    tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
206    \texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future
207    releases of MITgcm are to allow varying tile sizes.
208    
209    The location of the tiles' Cartesian origin within a subdomain are
210    determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
211    \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
212    relate the location of the edges of the tiles to each other.  As an
213    example, in the default six-tile topology (the degenerate case) each
214    index in these arrays are set to 0.  The twenty-four, 32x32 cube face
215    case discussed above will have values of 0 or 16, depending on the
216    quadrant the tile falls within the subdomain.  The array
217    \varlink{exch2\_myFace}{exch2_myFace} contains the number of the
218    cubeface/subdomain of each tile, numbered 1-6 in the case of the
219    standard cube topology.
220    
221    The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
222    \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
223    \varlink{exch2\_tbasex}{exch2_tbasex} and
224    \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
225    global address space, similar to that used by global files.
226    
227    The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
228    \varlink{exch2\_isEedge}{exch2_isEedge},
229    \varlink{exch2\_isSedge}{exch2_isSedge}, and
230    \varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
231    tile lies on the edge of a subdomain, 0 if not.  The values are used
232    within the topology generator to determine the orientation of
233    neighboring tiles and to indicate whether a tile lies on the corner of
234    a subdomain.  The latter case indicates special exchange and numerical
235    handling for the singularities at the eight corners of the cube.
236    \varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
237    how many neighboring tiles each tile has, and is used for setting
238    bounds for looping over neighboring tiles.
239    \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
240    tile, and is used in interprocess communication.
241    
242    \subsubsection{Arrays Indexed to Tile Number and Neighbor}
243    
244    The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
245    \texttt{NTILES} and describe the orientations between the the tiles.
246    
247    The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
248    each of the $n$ neighboring tiles.  The neighbor tiles are indexed
249    \texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
250    then south edges, and then top to bottom on the east and west edges.
251    Maybe throw in a fig here, eh?
252    
253    The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
254    in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
255    In other words,
256  \begin{verbatim}  \begin{verbatim}
257       exch2_neighbourId( exch2_opposingSend_record(a,T),
258                          exch2_neighbourId(a,T) ) = T
259    \end{verbatim}
260    and this provides a back-reference from the neighbor tiles.
261    
262    The arrays \varlink{exch2\_pi}{exch2_pi},
263    \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
264    \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
265    \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
266    exchanges between the neighboring tiles.  The dimensions of
267    \texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
268    neighbor ID \textit{N} and the tile number \textit{T} as explained
269    above, plus the transformation vector {\em t }, of length two.  The
270    first element of the transformation vector indicates the factor by
271    which variables representing the same vector component of a tile will
272    be multiplied, and the second element indicates the transform to the
273    variable in the other direction.  As an example,
274    \texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
275    vector variable in tile \texttt{T} to the i-component of tile
276    \texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
277    the component of neighbor \texttt{N}'s j-component.
278    
279    Under the current cube topology, one of the two elements of
280    \texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
281    and neighbor \texttt{N} will be 0, reflecting the fact that the vector
282    components are orthogonal.  The other element will be 1 or -1,
283    depending on whether the components are indexed in the same or
284    opposite directions.  For example, the transform dimension of the
285    arrays for all tile neighbors on the same subdomain will be [1,0],
286    since all tiles on the same subdomain are oriented identically.
287    Vectors that correspond to the orthogonal dimension with the same
288    index direction will have [0,1], whereas those in the opposite index
289    direction will have [0,-1].
290    
291    
292    {\footnotesize
293    \begin{verbatim}
294  C      exch2_pi          :: X index row of target to source permutation  C      exch2_pi          :: X index row of target to source permutation
295  C                        :: matrix for each neighbour entry.              C                        :: matrix for each neighbour entry.            
296  C      exch2_pj          :: Y index row of target to source permutation  C      exch2_pj          :: Y index row of target to source permutation
# Line 172  C      exch2_oj_f        :: Y index elem Line 308  C      exch2_oj_f        :: Y index elem
308  C                        :: offset vector for face quantities            C                        :: offset vector for face quantities          
309  C                        :: of each neighbor entry.                      C                        :: of each neighbor entry.                    
310  \end{verbatim}  \end{verbatim}
311    }
312    
313    
314    
Legend:
Removed from v.1.7  
changed lines
  Added in v.1.11
  ViewVC Help
Powered by ViewVC 1.1.22