/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.6 by afe, Tue Feb 3 19:43:38 2004 UTC revision 1.11 by afe, Mon Mar 15 22:39:28 2004 UTC
# Line 10  Line 10 
10  %%    o automatically inserted at \section{Reference}  %%    o automatically inserted at \section{Reference}
11    
12    
13  \section{exch2: Extended Cubed Sphere Exchange}  \section{exch2: Extended Cubed Sphere \mbox{Topology}}
14  \label{sec:exch2}  \label{sec:exch2}
15    
16    
17  \subsection{Introduction}  \subsection{Introduction}
18    
19  The exch2 package is an extension to the original cubed sphere exchanges  The \texttt{exch2} package is an extension to the original cubed
20  to allow more flexible domain decomposition and parallelization.  Cube faces  sphere topological configuration that allows more flexible domain
21  (subdomains) may be divided into whatever number of tiles that divide evenly  decomposition and parallelization.  Cube faces (also called
22  into the grid point dimensions of the subdomain.  Furthermore, the individual  subdomains) may be divided into any number of tiles that divide evenly
23  tiles may be run on separate processors in different combinations,  into the grid point dimensions of the subdomain.  Furthermore, the
24  and whether exchanges between particular tiles occur between different  individual tiles may be run on separate processors in different
25  processors is determined at runtime.  combinations, and whether exchanges between particular tiles occur
26    between different processors is determined at runtime.  This
27  The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and  flexibility provides for manual compile-time load balancing across a
28  assigned in {\em w2\_e2setup.F}, both in the  relatively arbitrary number of processors. \\
29  {\em pkg/exch2} directory.  The validity of the cube topology depends  
30  on the {\em SIZE.h} file as detailed below.  Both files are generated by  The exchange parameters are declared in
31  Matlab scripts and  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
32  should not be edited.  The default files provided in the release set up  and assigned in
33  a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
34  points, running on a single processor.    validity of the cube topology depends on the \file{SIZE.h} file as
35    detailed below.  Both files are generated by Matlab scripts in
36    \file{utils/exch2/matlab-topology-generator}; see Section
37    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
38    for details on creating alternate topologies.  The default files
39    provided in the release configure a cubed sphere topology of six
40    tiles, one per subdomain, each with 32$\times$32 grid points, all
41    running on a single processor.  Pregenerated examples of these files
42    with alternate topologies are provided under
43    \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
44    file for single-processor execution.
45    
46    \subsection{Invoking exch2}
47    
48    To use exch2 with the cubed sphere, the following conditions must be
49    met: \\
50    
51    $\bullet$ The exch2 package is included when \file{genmake2} is run.
52      The easiest way to do this is to add the line \code{exch2} to the
53      \file{profile.conf} file -- see Section
54      \ref{sect:buildingCode}\sectiontitle{Building the code} for general
55      details. \\
56    
57    $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
58      \file{w2\_e2setup.F} must reside in a directory containing code
59      linked when \file{genmake2} runs.  The safest place to put these
60      is the directory indicated in the \code{-mods=DIR} command line
61      modifier (typically \file{../code}), or the build directory.  The
62      default versions of these files reside in \file{pkg/exch2} and are
63      linked automatically if no other versions exist elsewhere in the
64      link path, but they should be left untouched to avoid breaking
65      configurations other than the one you intend to modify.\\
66    
67    $\bullet$ Files containing grid parameters, named
68      \file{tile???.mitgrid} where \file{???} is \file{001} through
69      \file{006} (one per subdomain), must be in the working directory
70      when the MITgcm executable is run.  These files are provided in the
71      example experiments for cubed sphere configurations with
72      32$\times$32 cube sides and are non-trivial to generate -- please
73      contact MITgcm support if you want to generate files for other
74      configurations. \\
75    
76    $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h}
77      must be placed where \file{genmake2} will find it.  In particular
78      for the exch2, the domain decomposition specified in \file{SIZE.h}
79      must correspond with the particular configuration's topology
80      specified in \file{W2\_EXCH2\_TOPOLOGY.h} and
81      \file{w2\_e2setup.F}.  Domain decomposition issues particular to
82      exch2 are addressed in Section \ref{sec:topogen} \sectiontitle{Generating
83      Topology Files for exch2}; a more general background on the subject
84      relvant to MITgcm is presented in Section
85      \ref{sect:specifying_a_decomposition}\sectiontitle{Specifying a
86      decomposition}.\\
87    
88    As of the time of writing the following examples use exch2 and may be
89    used for guidance:
90    
91    \begin{verbatim}
92    verification/adjust_nlfs.cs-32x32x1
93    verification/adjustment.cs-32x32x1
94    verification/aim.5l_cs
95    verification/global_ocean.cs32x15
96    verification/hs94.cs-32x32x5
97    \end{verbatim}
98    
99    
100    
101    
102    \subsection{Generating Topology Files for exch2}
103    \label{sec:topogen}
104    
105    Alternate cubed sphere topologies may be created using the Matlab
106    scripts in \file{utils/exch2/matlab-topology-generator}. Running the
107    m-file \file{driver.m} from the Matlab prompt (there are no parameters
108    to pass) generates exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h}
109    and \file{w2\_e2setup.F} in the working directory and displays a
110    figure of the topology via Matlab.  The other m-files in the directory
111    are subroutines of \file{driver.m} and should not be run except for
112    development purposes. \\
113    
114    The parameters that determine the dimensions and topology of the
115    generated configuration are \code{nr}, \code{nb}, \code{ng},
116    \code{tnx} and \code{tny}, and all are assigned early in the script.
117    
118    The first three determine the size of the subdomains (cube faces) and
119    hence the size of the overall domain.  Each one determines the number
120    of grid points, and therefore the resolution, along the subdomain
121    sides in a ``great circle'' around each axis of the cube.  At the time
122    of this writing MITgcm requires these three parameters to be equal,
123    but they provide for future releases of MITgcm to accomodate different
124    resolutions around the axes to allow (for example) greater resolution
125    around the equator.\\
126    
127    The parameters \code{tnx} and \code{tny} determine the dimensions of
128    the tiles into which the subdomains are decomposed, and must evenly
129    divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
130    The result is a rectangular tiling of the subdomain.  Figure
131    \ref{fig:24tile} shows one possible topology for a twenty-four tile
132    cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
133    
134    \begin{figure}
135    \begin{center}
136     \resizebox{4in}{!}{
137      \includegraphics{part6/s24t_16x16.ps}
138     }
139    \end{center}
140    \caption{Plot of cubed sphere topology with a 32$\times$32 grid and
141    twenty-four tiles (\code{tnx=16, tny=16})
142    } \label{fig:24tile}
143    \end{figure}
144    
145    \begin{figure}
146    \begin{center}
147     \resizebox{4in}{!}{
148      \includegraphics{part6/s12t_16x32.ps}
149     }
150    \end{center}
151    \caption{Plot of cubed sphere topology with a 32$\times$32 grid and
152    twelve tiles (\code{tnx=16, tny=32})
153    } \label{fig:12tile}
154    \end{figure}
155    
156    Tiles can be selected from the topology to be omitted from being
157    allocated memory and processors.  This kind otuning is useful in
158    ocean modeling for omitting tiles that fall entirely on land.  The
159    tiles omitted are specified in the file \file{blanklist.txt} by
160    their tile number in the topology, separated by a newline. \\
161    
162    
163    
164    
165    
166    
167  \subsection{Key Variables}  \subsection{Key Variables}
168    
169  The descriptions of the variables are divided up into scalars,  The descriptions of the variables are divided up into scalars,
170  one-dimensional arrays indexed to the tile number, and two and three  one-dimensional arrays indexed to the tile number, and two and three
171  dimensional  dimensional arrays indexed to tile number and neighboring tile.  This
172  arrays indexed to tile number and neighboring tile.  This division  division actually reflects the functionality of these variables: the
173  actually reflects  the functionality of these variables: the scalars  scalars are common to every part of the topology, the tile-indexed
174  are common to every part of the topology, the tile-indexed arrays to  arrays to individual tiles, and the arrays indexed to tile and
175  individual tiles, and the arrays indexed to tile and neighbor to  neighbor to relationships between tiles and their neighbors.
 relationships between tiles and their neighbors.  
176    
177  \subsubsection{Scalars}  \subsubsection{Scalars}
178    
179  The number of tiles in a particular topology is set with the parameter  The number of tiles in a particular topology is set with the parameter
180  {\em NTILES}, and the maximum number of neighbors of any tiles by  \texttt{NTILES}, and the maximum number of neighbors of any tiles by
181  {\em MAX\_NEIGHBOURS}.  These parameters are used for defining the size of  \texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the
182  the various one and two dimensional arrays that store tile parameters  size of the various one and two dimensional arrays that store tile
183  indexed to the tile number.  parameters indexed to the tile number.\\
184    
185  The scalar parameters {\em exch2\_domain\_nxt} and  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
186  {\em exch2\_domain\_nyt} express the number of tiles in the x and y global  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
187  indices.  For example, the default setup of six tiles has  of tiles in the x and y global indices.  For example, the default
188  {\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}.  A topology of  setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
189  twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will  \texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in
190  have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}.  Note  gridpoints) tiles, four (2x2) per subdomain, will have
191  that these parameters express the tile layout to allow global data files that  \texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
192  are tile-layout-neutral and have no bearing on the internal storage of the  Note that these parameters express the tile layout to allow global
193  arrays.  The tiles are internally stored in a range from {\em 1,bi} (in the  data files that are tile-layout-neutral and have no bearing on the
194  x axis) and y-axis variable {\em bj} is generally ignored within the package.  internal storage of the arrays.  The tiles are internally stored in a
195    range from \texttt{1,bi} (in the x axis) and y-axis variable
196    \texttt{bj} is generally ignored within the package.
197    
198  \subsubsection{Arrays Indexed to Tile Number}  \subsubsection{Arrays Indexed to Tile Number}
199    
200  The following arrays are of size {\em NTILES}, are indexed to the tile number,  The following arrays are of size \texttt{NTILES}, are indexed to the
201  and the indices are omitted in their descriptions.  tile number, and the indices are omitted in their descriptions.
202    
203  The arrays {\em exch2\_tnx} and {\em exch2\_tny}  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
204  express the x and y dimensions of each tile.  At present for each tile  \varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
205  {\em exch2\_tnx = sNx}  tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
206  and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}.  Future releases of  \texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future
207  MITgcm are to allow varying tile sizes.  releases of MITgcm are to allow varying tile sizes.
208    
209  The location of the tiles' Cartesian origin within a subdomain are determined  The location of the tiles' Cartesian origin within a subdomain are
210  by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}.  These variables  determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
211  are used to relate the location of the edges of the tiles to each other.  As  \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
212  an example, in the default six-tile topology (the degenerate case)  relate the location of the edges of the tiles to each other.  As an
213  each index in these arrays are  example, in the default six-tile topology (the degenerate case) each
214  set to 0.  The twenty-four, 32x32 cube face case discussed above will have  index in these arrays are set to 0.  The twenty-four, 32x32 cube face
215  values of 0 or 16, depending on the quadrant the tile falls within the  case discussed above will have values of 0 or 16, depending on the
216  subdomain.  {\em exch2\_myFace} contains the number of the  quadrant the tile falls within the subdomain.  The array
217  cubeface/subdomain of each tile, numbered 1-6 in the case of the standard  \varlink{exch2\_myFace}{exch2_myFace} contains the number of the
218  cube topology.    cubeface/subdomain of each tile, numbered 1-6 in the case of the
219    standard cube topology.
220  The arrays {\em exch2\_txglobalo} and {\em exch2\_txglobalo} are similar to  
221  {\em exch2\_tbasex} and {\em exch2\_tbasey}, but locate the tiles within  The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
222  the global address space, similar to that used by global files.    \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
223    \varlink{exch2\_tbasex}{exch2_tbasex} and
224  The arrays {\em exch2\_isWedge}, {\em exch2\_isEedge}, {\em exch2\_isSedge},  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
225  and {\em exch2\_isNedge} are set to 1 if the indexed tile lies on the edge  global address space, similar to that used by global files.
226  of a subdomain, 0 if not.  The values are used within the topology generator  
227  to determine the orientation of neighboring tiles and to indicate whether  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
228  a tile lies on the corner of a subdomain.  The latter case indicates  \varlink{exch2\_isEedge}{exch2_isEedge},
229  special exchange and numerical handling for the singularities at the eight  \varlink{exch2\_isSedge}{exch2_isSedge}, and
230  corners of the cube.  {\em exch2\_isNedge} contains a count of how many  \varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
231  neighboring tiles each tile has, and is used for setting bounds for looping  tile lies on the edge of a subdomain, 0 if not.  The values are used
232  over neighboring tiles.  {\em exch2\_tProc} holds the process rank of each tile,  within the topology generator to determine the orientation of
233  and is used in interprocess communication.  neighboring tiles and to indicate whether a tile lies on the corner of
234    a subdomain.  The latter case indicates special exchange and numerical
235    handling for the singularities at the eight corners of the cube.
236    \varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
237    how many neighboring tiles each tile has, and is used for setting
238    bounds for looping over neighboring tiles.
239    \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
240    tile, and is used in interprocess communication.
241    
242  \subsubsection{Arrays Indexed to Tile Number and Neighbor}  \subsubsection{Arrays Indexed to Tile Number and Neighbor}
243    
244  The following arrays are all of size {\em MAX\_NEIGHBOURS}x{\em NTILES} and  The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
245  describe the orientations between the the tiles.  \texttt{NTILES} and describe the orientations between the the tiles.
246    
247  The array {\em exch2\_neighbourId(a,T)} holds the tile number $T_{n}$ for each tile  The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
248  {\em T}'s neighbor tile {\em a}, and {\em exch2\_opposingSend\_record(a,T)} holds  each of the $n$ neighboring tiles.  The neighbor tiles are indexed
249  the index c in {\em exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.  \texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
250  In other words,  then south edges, and then top to bottom on the east and west edges.
251    Maybe throw in a fig here, eh?
252  \begin{verbatim}    
253  exch2_neighbourId( exch2_opposingSend_record(a,T), exch2_neighbourId(a,T) ) = T  The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
254    in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
255    In other words,
256    \begin{verbatim}
257       exch2_neighbourId( exch2_opposingSend_record(a,T),
258                          exch2_neighbourId(a,T) ) = T
259  \end{verbatim}  \end{verbatim}
260    and this provides a back-reference from the neighbor tiles.
261    
262  % {\em exch2\_neighbourId(exch2\_opposingSend\_record(a,T),exch2\_neighbourId(a,T))=T}.  The arrays \varlink{exch2\_pi}{exch2_pi},
263  % alternate version  \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
264    \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
265  This is to provide a backreference from the neighbor tiles.  \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
266    exchanges between the neighboring tiles.  The dimensions of
267    \texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
268    neighbor ID \textit{N} and the tile number \textit{T} as explained
269    above, plus the transformation vector {\em t }, of length two.  The
270    first element of the transformation vector indicates the factor by
271    which variables representing the same vector component of a tile will
272    be multiplied, and the second element indicates the transform to the
273    variable in the other direction.  As an example,
274    \texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
275    vector variable in tile \texttt{T} to the i-component of tile
276    \texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
277    the component of neighbor \texttt{N}'s j-component.
278    
279    Under the current cube topology, one of the two elements of
280    \texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
281    and neighbor \texttt{N} will be 0, reflecting the fact that the vector
282    components are orthogonal.  The other element will be 1 or -1,
283    depending on whether the components are indexed in the same or
284    opposite directions.  For example, the transform dimension of the
285    arrays for all tile neighbors on the same subdomain will be [1,0],
286    since all tiles on the same subdomain are oriented identically.
287    Vectors that correspond to the orthogonal dimension with the same
288    index direction will have [0,1], whereas those in the opposite index
289    direction will have [0,-1].
290    
291    
292  //  {\footnotesize
   
293  \begin{verbatim}  \begin{verbatim}
   
   
   
294  C      exch2_pi          :: X index row of target to source permutation  C      exch2_pi          :: X index row of target to source permutation
295  C                        :: matrix for each neighbour entry.              C                        :: matrix for each neighbour entry.            
296  C      exch2_pj          :: Y index row of target to source permutation  C      exch2_pj          :: Y index row of target to source permutation
# Line 143  C      exch2_oj_f        :: Y index elem Line 308  C      exch2_oj_f        :: Y index elem
308  C                        :: offset vector for face quantities            C                        :: offset vector for face quantities          
309  C                        :: of each neighbor entry.                      C                        :: of each neighbor entry.                    
310  \end{verbatim}  \end{verbatim}
311    }
312    
313    
314    
Legend:
Removed from v.1.6  
changed lines
  Added in v.1.11
  ViewVC Help
Powered by ViewVC 1.1.22