/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.7 by afe, Wed Feb 11 20:48:14 2004 UTC revision 1.10 by afe, Mon Mar 15 20:11:56 2004 UTC
# Line 10  Line 10 
10  %%    o automatically inserted at \section{Reference}  %%    o automatically inserted at \section{Reference}
11    
12    
13  \section{exch2: Extended Cubed Sphere Exchange}  \section{exch2: Extended Cubed Sphere \mbox{Topology}}
14  \label{sec:exch2}  \label{sec:exch2}
15    
16    
17  \subsection{Introduction}  \subsection{Introduction}
18    
19  The exch2 package is an extension to the original cubed sphere exchanges  The \texttt{exch2} package is an extension to the original cubed
20  to allow more flexible domain decomposition and parallelization.  Cube faces  sphere topological configuration that allows more flexible domain
21  (subdomains) may be divided into whatever number of tiles that divide evenly  decomposition and parallelization.  Cube faces (also called
22  into the grid point dimensions of the subdomain.  Furthermore, the individual  subdomains) may be divided into any number of tiles that divide evenly
23  tiles may be run on separate processors in different combinations,  into the grid point dimensions of the subdomain.  Furthermore, the
24  and whether exchanges between particular tiles occur between different  individual tiles may be run on separate processors in different
25  processors is determined at runtime.  combinations, and whether exchanges between particular tiles occur
26    between different processors is determined at runtime.  This
27  The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and  flexibility provides for manual compile-time load balancing across a
28  assigned in {\em w2\_e2setup.F}, both in the  relatively arbitrary number of processors. \\
29  {\em pkg/exch2} directory.  The validity of the cube topology depends  
30  on the {\em SIZE.h} file as detailed below.  Both files are generated by  The exchange parameters are declared in
31  Matlab scripts and  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
32  should not be edited.  The default files provided in the release set up  and assigned in
33  a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
34  points, running on a single processor.    validity of the cube topology depends on the \texttt{SIZE.h} file as
35    detailed below.  Both files are generated by Matlab scripts in
36    \texttt{utils/exch2/matlab-topology-generator}; see Section
37    \ref{sec:topogen} for details on creating alternate topologies.  The
38    default files provided in the release configure a cubed sphere
39    topology of six tiles, one per subdomain, each with 32$\times$32 grid
40    points, all running on a single processor.  Pregenerated examples of
41    these files with alternate topologies are provided under
42    \texttt{utils/exch2/code-mods} along with the appropriate
43    \texttt{SIZE.h} file for single-processor execution.
44    
45    \subsection{Invoking exch2}
46    
47    To use exch2 with the cubed sphere, the following conditions must be
48    met: \\
49    
50    $\bullet$ The exch2 package is included when \texttt{genmake2} is run.  The
51      easiest way to do this is to add the line \texttt{exch2} to the
52      \texttt{profile.conf} file -- see Section \ref{sect:buildingCode}
53      for general details. \\
54    
55    $\bullet$ An example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and
56      \texttt{w2\_e2setup.F} must reside in a directory containing code
57      linked when \texttt{genmake2} runs.  The safest place to put these
58      is the directory indicated in the \texttt{-mods=DIR} command line
59      modifier (typically \texttt{../code}), or the build directory.  The
60      default versions of these files reside in \texttt{pkg/exch2} and are
61      linked automatically if no other versions exist elsewhere in the
62      link path, but they should be left untouched to avoid breaking
63      configurations other than the one you intend to modify.\\
64    
65    $\bullet$ Files containing grid parameters, named
66      \texttt{tile}???\texttt{.mitgrid} where ??? is \texttt{001} through
67      \texttt{006} (one per subdomain), must be in the working directory
68      when the MITgcm executable is run.  These files are provided in the
69      example experiments for cubed sphere configurations with
70      32$\times$32 cube sides and are non-trivial to generate -- please
71      contact MITgcm support if you want to generate files for other
72      configurations. \\
73    
74    $\bullet$ As always when compiling MITgcm, the file \texttt{SIZE.h}
75      must be placed where \texttt{genmake2} will find it.  In particular
76      for the exch2, the domain decompositin specified in \texttt{SIZE.h}
77      must correspond with the particular configuration's topology
78      specified in \texttt{W2\_EXCH2\_TOPOLOGY.h} and
79      \texttt{w2\_e2setup.F}.  Domain decomposition issues particular to
80      exch2 are addressed in Section \ref{sec:topogen}: ``Generating
81      Topology Files for exch2''; a more general background on the subject
82      relvant to MITgcm is presented in Section
83      \ref{sect:specifying_a_decomposition}: ``Specifying a
84      decomposition''.\\
85    
86  \subsection{Key Variables}  As of the time of writing the following examples use exch2 and may be
87    used for guidance:
88    
89  The descriptions of the variables are divided up into scalars,  \begin{verbatim}
90  one-dimensional arrays indexed to the tile number, and two and three  verification/adjust_nlfs.cs-32x32x1
91  dimensional  verification/adjustment.cs-32x32x1
92  arrays indexed to tile number and neighboring tile.  This division  verification/aim.5l_cs
93  actually reflects  the functionality of these variables: the scalars  verification/global_ocean.cs32x15
94  are common to every part of the topology, the tile-indexed arrays to  verification/hs94.cs-32x32x5
95  individual tiles, and the arrays indexed to tile and neighbor to  \end{verbatim}
 relationships between tiles and their neighbors.  
96    
 \subsubsection{Scalars}  
97    
 The number of tiles in a particular topology is set with the parameter  
 {\em NTILES}, and the maximum number of neighbors of any tiles by  
 {\em MAX\_NEIGHBOURS}.  These parameters are used for defining the size of  
 the various one and two dimensional arrays that store tile parameters  
 indexed to the tile number.  
   
 The scalar parameters {\em exch2\_domain\_nxt} and  
 {\em exch2\_domain\_nyt} express the number of tiles in the x and y global  
 indices.  For example, the default setup of six tiles has  
 {\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}.  A topology of  
 twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will  
 have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}.  Note  
 that these parameters express the tile layout to allow global data files that  
 are tile-layout-neutral and have no bearing on the internal storage of the  
 arrays.  The tiles are internally stored in a range from {\em 1,bi} (in the  
 x axis) and y-axis variable {\em bj} is generally ignored within the package.  
98    
 \subsubsection{Arrays Indexed to Tile Number}  
99    
100  The following arrays are of size {\em NTILES}, are indexed to the tile number,  \subsection{Generating Topology Files for exch2}
101  and the indices are omitted in their descriptions.  \label{sec:topogen}
102    
103    Alternate cubed sphere topologies may be created using the Matlab
104    scripts in \texttt{utils/exch2/matlab-topology-generator}. Running the
105    m-file \texttt{driver} from the Matlab prompt (without passing any
106    function parameters) generates exch2 topology files
107    \texttt{W2\_EXCH2\_TOPOLOGY.h} and \texttt{w2\_e2setup.F} in the
108    working directory and displays via Matlab a figure of the topology.
109    The other m-files in the directory are subroutines of \texttt{driver}
110    and should not be run except for development purposes. \\
111    
112    The parameters that determine the dimensions and topology of the
113    generated configuration are nr, nb, ng, tnx and tny, and all are
114    assigned early in the script.
115    
116    The first three determine the size of the subdomains (cube faces) and
117    hence the size of the overall domain.  Each one determines the number
118    of grid points, and therefore the resolution, along the subdomain
119    sides in a ``great circle'' around each axis of the cube.  At the time
120    of this writing MITgcm requires these three parameters to be equal,
121    but they provide for future releases of MITgcm to accomodate different
122    resolutions around the axes to allow (for example) greater resolution
123    around the equator.\\
124    
125    The parameters tnx and tny determine the dimensions of the tiles into
126    which the subdomains are decomposed, and must evenly divide the
127    integer assigned to nr, nb and ng.  The result is a rectangular tiling
128    of the subdomain.  Figure \ref{fig:24tile} shows one possible topology
129    for a twenty-four tile cube, and figure \ref{fig:12tile} shows one for
130    twelve tiles. \\
131    
132    \begin{figure}
133    \begin{center}
134     \resizebox{4in}{!}{
135      \includegraphics{part6/s24t_16x16.ps}
136     }
137    \end{center}
138    \caption{Plot of cubed sphere topology with a 32$\times$32 grid and
139    twenty-four tiles (tnx=16, tny=16)
140    } \label{fig:24tile}
141    \end{figure}
142    
143    \begin{figure}
144    \begin{center}
145     \resizebox{4in}{!}{
146      \includegraphics{part6/s12t_16x32.ps}
147     }
148    \end{center}
149    \caption{Plot of cubed sphere topology with a 32$\times$32 grid and
150    twelve tiles (tnx=16, tny=32)
151    } \label{fig:12tile}
152    \end{figure}
153    
154    Tiles can be selected from the topology to be omitted from being
155    allocated memory and processors.  This kind of tuning is useful in
156    ocean modeling for omitting tiles that fall entirely on land.  The
157    tiles omitted are specified in the file \texttt{blanklist.txt} by
158    their tile number in the topology, separated by a newline. \\
159    
160    
 The arrays {\em exch2\_tnx} and {\em exch2\_tny}  
 express the x and y dimensions of each tile.  At present for each tile  
 {\em exch2\_tnx = sNx}  
 and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}.  Future releases of  
 MITgcm are to allow varying tile sizes.  
   
 The location of the tiles' Cartesian origin within a subdomain are determined  
 by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}.  These variables  
 are used to relate the location of the edges of the tiles to each other.  As  
 an example, in the default six-tile topology (the degenerate case)  
 each index in these arrays are  
 set to 0.  The twenty-four, 32x32 cube face case discussed above will have  
 values of 0 or 16, depending on the quadrant the tile falls within the  
 subdomain.  {\em exch2\_myFace} contains the number of the  
 cubeface/subdomain of each tile, numbered 1-6 in the case of the standard  
 cube topology.    
   
 The arrays {\em exch2\_txglobalo} and {\em exch2\_txglobalo} are similar to  
 {\em exch2\_tbasex} and {\em exch2\_tbasey}, but locate the tiles within  
 the global address space, similar to that used by global files.    
   
 The arrays {\em exch2\_isWedge}, {\em exch2\_isEedge}, {\em exch2\_isSedge},  
 and {\em exch2\_isNedge} are set to 1 if the indexed tile lies on the edge  
 of a subdomain, 0 if not.  The values are used within the topology generator  
 to determine the orientation of neighboring tiles and to indicate whether  
 a tile lies on the corner of a subdomain.  The latter case indicates  
 special exchange and numerical handling for the singularities at the eight  
 corners of the cube.  {\em exch2\_isNedge} contains a count of how many  
 neighboring tiles each tile has, and is used for setting bounds for looping  
 over neighboring tiles.  {\em exch2\_tProc} holds the process rank of each tile,  
 and is used in interprocess communication.  
161    
 \subsubsection{Arrays Indexed to Tile Number and Neighbor}  
162    
 The following arrays are all of size {\em MAX\_NEIGHBOURS}x{\em NTILES} and  
 describe the orientations between the the tiles.  
163    
 The array {\em exch2\_neighbourId(a,T)} holds the tile number $T_{n}$ for each tile  
 {\em T}'s neighbor tile {\em a}.  The neighbor tiles are indexed {\em 1,MAX\_NEIGHBOURS }  
 in the order right to left on the north then south edges, and then top to bottom on the east  
 and west edges. maybe throw in a fig here, eh?    
   
 {\em exch2\_opposingSend\_record(a,T)} holds  
 the index c in {\em exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.  
 In other words,  
   
 \begin{verbatim}    
 exch2_neighbourId( exch2_opposingSend_record(a,T),  
                            exch2_neighbourId(a,T) ) = T  
 \end{verbatim}  
164    
165  % {\em exch2\_neighbourId(exch2\_opposingSend\_record(a,T),exch2\_neighbourId(a,T))=T}.  \subsection{Key Variables}
 % alternate version  
166    
167  This is to provide a backreference from the neighbor tiles.  The descriptions of the variables are divided up into scalars,
168    one-dimensional arrays indexed to the tile number, and two and three
169    dimensional arrays indexed to tile number and neighboring tile.  This
170    division actually reflects the functionality of these variables: the
171    scalars are common to every part of the topology, the tile-indexed
172    arrays to individual tiles, and the arrays indexed to tile and
173    neighbor to relationships between tiles and their neighbors.
174    
175  The arrays {\em exch2\_pi }, {\em exch2\_pj }, {\em exch2\_oi },  \subsubsection{Scalars}
 {\em exch2\_oj }, {\em exch2\_oi\_f }, and {\em exch2\_oj\_f }  specify  
 the transformations in exchanges between the neighboring tiles.  The dimensions    
 of {\em exch2\_pi(t,N,T) } and {\em exch2\_pj(t,N,T) } are the neighbor ID  
 { \em N } and the tile number {\em T } as explained above, plus the transformation  
 vector {\em t }, of length two.  The first element of the transformation vector indicates  
 the factor by which variables representing the same vector component  of a tile  
 will be multiplied, and the second element indicates the transform to the  
 variable in the other direction.  As an example, {\em exch2\_pi(1,N,T) } holds the  
 transform of the i-component of a vector variable in tile {\em T } to the i-component of  
 tile  {\em T }'s neighbor  {\em N }, and {\em exch2\_pi(2,N,T) } hold the component  
 of neighbor  {\em N }'s j-component.  
   
 Under the current cube topology, one of the two elements of {\em exch2\_pi } or {\em exch2\_pj }  
 for a given tile   {\em T } and  neighbor  {\em N } will be 0, reflecting the fact that  
 the vector components are orthogonal.  The other element will be 1 or -1, depending on whether  
 the components are indexed in the same or opposite directions.  For example, the transform dimension  
 of the arrays for all tile neighbors on the same subdomain will be {\em [1 , 0] }, since all tiles on  
 the same subdomain are oriented identically.  Vectors that correspond to the orthogonal dimension with the  
 same index direction will have {\em [0 , 1] }, whereas those in the opposite index direction will have  
 {\em [0 , -1] }.  
176    
177    The number of tiles in a particular topology is set with the parameter
178    \texttt{NTILES}, and the maximum number of neighbors of any tiles by
179    \texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the
180    size of the various one and two dimensional arrays that store tile
181    parameters indexed to the tile number.\\
182    
183    The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
184    and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
185    of tiles in the x and y global indices.  For example, the default
186    setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
187    \texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in
188    gridpoints) tiles, four (2x2) per subdomain, will have
189    \texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
190    Note that these parameters express the tile layout to allow global
191    data files that are tile-layout-neutral and have no bearing on the
192    internal storage of the arrays.  The tiles are internally stored in a
193    range from \texttt{1,bi} (in the x axis) and y-axis variable
194    \texttt{bj} is generally ignored within the package.
195    
196    \subsubsection{Arrays Indexed to Tile Number}
197    
198    The following arrays are of size \texttt{NTILES}, are indexed to the
199    tile number, and the indices are omitted in their descriptions.
200    
201  //  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
202    \varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
203    tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
204    \texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future
205    releases of MITgcm are to allow varying tile sizes.
206    
207    The location of the tiles' Cartesian origin within a subdomain are
208    determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
209    \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
210    relate the location of the edges of the tiles to each other.  As an
211    example, in the default six-tile topology (the degenerate case) each
212    index in these arrays are set to 0.  The twenty-four, 32x32 cube face
213    case discussed above will have values of 0 or 16, depending on the
214    quadrant the tile falls within the subdomain.  The array
215    \varlink{exch2\_myFace}{exch2_myFace} contains the number of the
216    cubeface/subdomain of each tile, numbered 1-6 in the case of the
217    standard cube topology.
218    
219    The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
220    \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
221    \varlink{exch2\_tbasex}{exch2_tbasex} and
222    \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
223    global address space, similar to that used by global files.
224    
225    The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
226    \varlink{exch2\_isEedge}{exch2_isEedge},
227    \varlink{exch2\_isSedge}{exch2_isSedge}, and
228    \varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
229    tile lies on the edge of a subdomain, 0 if not.  The values are used
230    within the topology generator to determine the orientation of
231    neighboring tiles and to indicate whether a tile lies on the corner of
232    a subdomain.  The latter case indicates special exchange and numerical
233    handling for the singularities at the eight corners of the cube.
234    \varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
235    how many neighboring tiles each tile has, and is used for setting
236    bounds for looping over neighboring tiles.
237    \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
238    tile, and is used in interprocess communication.
239    
240    \subsubsection{Arrays Indexed to Tile Number and Neighbor}
241    
242    The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
243    \texttt{NTILES} and describe the orientations between the the tiles.
244    
245    The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
246    each of the $n$ neighboring tiles.  The neighbor tiles are indexed
247    \texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
248    then south edges, and then top to bottom on the east and west edges.
249    Maybe throw in a fig here, eh?
250    
251    The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
252    in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
253    In other words,
254  \begin{verbatim}  \begin{verbatim}
255       exch2_neighbourId( exch2_opposingSend_record(a,T),
256                          exch2_neighbourId(a,T) ) = T
257    \end{verbatim}
258    and this provides a back-reference from the neighbor tiles.
259    
260    The arrays \varlink{exch2\_pi}{exch2_pi},
261    \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
262    \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
263    \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
264    exchanges between the neighboring tiles.  The dimensions of
265    \texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
266    neighbor ID \textit{N} and the tile number \textit{T} as explained
267    above, plus the transformation vector {\em t }, of length two.  The
268    first element of the transformation vector indicates the factor by
269    which variables representing the same vector component of a tile will
270    be multiplied, and the second element indicates the transform to the
271    variable in the other direction.  As an example,
272    \texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
273    vector variable in tile \texttt{T} to the i-component of tile
274    \texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
275    the component of neighbor \texttt{N}'s j-component.
276    
277    Under the current cube topology, one of the two elements of
278    \texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
279    and neighbor \texttt{N} will be 0, reflecting the fact that the vector
280    components are orthogonal.  The other element will be 1 or -1,
281    depending on whether the components are indexed in the same or
282    opposite directions.  For example, the transform dimension of the
283    arrays for all tile neighbors on the same subdomain will be [1,0],
284    since all tiles on the same subdomain are oriented identically.
285    Vectors that correspond to the orthogonal dimension with the same
286    index direction will have [0,1], whereas those in the opposite index
287    direction will have [0,-1].
288    
289    
290    {\footnotesize
291    \begin{verbatim}
292  C      exch2_pi          :: X index row of target to source permutation  C      exch2_pi          :: X index row of target to source permutation
293  C                        :: matrix for each neighbour entry.              C                        :: matrix for each neighbour entry.            
294  C      exch2_pj          :: Y index row of target to source permutation  C      exch2_pj          :: Y index row of target to source permutation
# Line 172  C      exch2_oj_f        :: Y index elem Line 306  C      exch2_oj_f        :: Y index elem
306  C                        :: offset vector for face quantities            C                        :: offset vector for face quantities          
307  C                        :: of each neighbor entry.                      C                        :: of each neighbor entry.                    
308  \end{verbatim}  \end{verbatim}
309    }
310    
311    
312    
Legend:
Removed from v.1.7  
changed lines
  Added in v.1.10
  ViewVC Help
Powered by ViewVC 1.1.22