/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.7 by afe, Wed Feb 11 20:48:14 2004 UTC revision 1.8 by edhill, Tue Feb 17 21:58:56 2004 UTC
# Line 10  Line 10 
10  %%    o automatically inserted at \section{Reference}  %%    o automatically inserted at \section{Reference}
11    
12    
13  \section{exch2: Extended Cubed Sphere Exchange}  \section{Extended Cubed Sphere Exchange}
14  \label{sec:exch2}  \label{sec:exch2}
15    
16    
17  \subsection{Introduction}  \subsection{Introduction}
18    
19  The exch2 package is an extension to the original cubed sphere exchanges  The \texttt{exch2} package is an extension to the original cubed
20  to allow more flexible domain decomposition and parallelization.  Cube faces  sphere exchanges to allow more flexible domain decomposition and
21  (subdomains) may be divided into whatever number of tiles that divide evenly  parallelization.  Cube faces (subdomains) may be divided into whatever
22  into the grid point dimensions of the subdomain.  Furthermore, the individual  number of tiles that divide evenly into the grid point dimensions of
23  tiles may be run on separate processors in different combinations,  the subdomain.  Furthermore, the individual tiles may be run on
24  and whether exchanges between particular tiles occur between different  separate processors in different combinations, and whether exchanges
25  processors is determined at runtime.  between particular tiles occur between different processors is
26    determined at runtime.
27  The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and  
28  assigned in {\em w2\_e2setup.F}, both in the  The exchange parameters are declared in
29  {\em pkg/exch2} directory.  The validity of the cube topology depends  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
30  on the {\em SIZE.h} file as detailed below.  Both files are generated by  and assigned in
31  Matlab scripts and  \filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}, both in
32  should not be edited.  The default files provided in the release set up  the \texttt{pkg/exch2} directory.  The validity of the cube topology
33  a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid  depends on the \texttt{SIZE.h} file as detailed below.  Both files are
34  points, running on a single processor.    generated by Matlab scripts and should not be edited.  The default
35    files provided in the release set up a cube sphere arrangement of six
36    tiles, one per subdomain, each with 32x32 grid points, running on a
37    single processor.
38    
39  \subsection{Key Variables}  \subsection{Key Variables}
40    
41  The descriptions of the variables are divided up into scalars,  The descriptions of the variables are divided up into scalars,
42  one-dimensional arrays indexed to the tile number, and two and three  one-dimensional arrays indexed to the tile number, and two and three
43  dimensional  dimensional arrays indexed to tile number and neighboring tile.  This
44  arrays indexed to tile number and neighboring tile.  This division  division actually reflects the functionality of these variables: the
45  actually reflects  the functionality of these variables: the scalars  scalars are common to every part of the topology, the tile-indexed
46  are common to every part of the topology, the tile-indexed arrays to  arrays to individual tiles, and the arrays indexed to tile and
47  individual tiles, and the arrays indexed to tile and neighbor to  neighbor to relationships between tiles and their neighbors.
 relationships between tiles and their neighbors.  
48    
49  \subsubsection{Scalars}  \subsubsection{Scalars}
50    
51  The number of tiles in a particular topology is set with the parameter  The number of tiles in a particular topology is set with the parameter
52  {\em NTILES}, and the maximum number of neighbors of any tiles by  \texttt{NTILES}, and the maximum number of neighbors of any tiles by
53  {\em MAX\_NEIGHBOURS}.  These parameters are used for defining the size of  \texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the
54  the various one and two dimensional arrays that store tile parameters  size of the various one and two dimensional arrays that store tile
55  indexed to the tile number.  parameters indexed to the tile number.
56    
57  The scalar parameters {\em exch2\_domain\_nxt} and  The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
58  {\em exch2\_domain\_nyt} express the number of tiles in the x and y global  and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
59  indices.  For example, the default setup of six tiles has  of tiles in the x and y global indices.  For example, the default
60  {\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}.  A topology of  setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
61  twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will  \texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in
62  have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}.  Note  gridpoints) tiles, four (2x2) per subdomain, will have
63  that these parameters express the tile layout to allow global data files that  \texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
64  are tile-layout-neutral and have no bearing on the internal storage of the  Note that these parameters express the tile layout to allow global
65  arrays.  The tiles are internally stored in a range from {\em 1,bi} (in the  data files that are tile-layout-neutral and have no bearing on the
66  x axis) and y-axis variable {\em bj} is generally ignored within the package.  internal storage of the arrays.  The tiles are internally stored in a
67    range from \texttt{1,bi} (in the x axis) and y-axis variable
68    \texttt{bj} is generally ignored within the package.
69    
70  \subsubsection{Arrays Indexed to Tile Number}  \subsubsection{Arrays Indexed to Tile Number}
71    
72  The following arrays are of size {\em NTILES}, are indexed to the tile number,  The following arrays are of size \texttt{NTILES}, are indexed to the
73  and the indices are omitted in their descriptions.  tile number, and the indices are omitted in their descriptions.
74    
75  The arrays {\em exch2\_tnx} and {\em exch2\_tny}  The arrays \varlink{exch2\_tnx}{exch2_tnx} and
76  express the x and y dimensions of each tile.  At present for each tile  \varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
77  {\em exch2\_tnx = sNx}  tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
78  and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}.  Future releases of  \texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future
79  MITgcm are to allow varying tile sizes.  releases of MITgcm are to allow varying tile sizes.
80    
81  The location of the tiles' Cartesian origin within a subdomain are determined  The location of the tiles' Cartesian origin within a subdomain are
82  by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}.  These variables  determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
83  are used to relate the location of the edges of the tiles to each other.  As  \varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
84  an example, in the default six-tile topology (the degenerate case)  relate the location of the edges of the tiles to each other.  As an
85  each index in these arrays are  example, in the default six-tile topology (the degenerate case) each
86  set to 0.  The twenty-four, 32x32 cube face case discussed above will have  index in these arrays are set to 0.  The twenty-four, 32x32 cube face
87  values of 0 or 16, depending on the quadrant the tile falls within the  case discussed above will have values of 0 or 16, depending on the
88  subdomain.  {\em exch2\_myFace} contains the number of the  quadrant the tile falls within the subdomain.  The array
89  cubeface/subdomain of each tile, numbered 1-6 in the case of the standard  \varlink{exch2\_myFace}{exch2_myFace} contains the number of the
90  cube topology.    cubeface/subdomain of each tile, numbered 1-6 in the case of the
91    standard cube topology.
92  The arrays {\em exch2\_txglobalo} and {\em exch2\_txglobalo} are similar to  
93  {\em exch2\_tbasex} and {\em exch2\_tbasey}, but locate the tiles within  The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
94  the global address space, similar to that used by global files.    \varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
95    \varlink{exch2\_tbasex}{exch2_tbasex} and
96  The arrays {\em exch2\_isWedge}, {\em exch2\_isEedge}, {\em exch2\_isSedge},  \varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
97  and {\em exch2\_isNedge} are set to 1 if the indexed tile lies on the edge  global address space, similar to that used by global files.
98  of a subdomain, 0 if not.  The values are used within the topology generator  
99  to determine the orientation of neighboring tiles and to indicate whether  The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
100  a tile lies on the corner of a subdomain.  The latter case indicates  \varlink{exch2\_isEedge}{exch2_isEedge},
101  special exchange and numerical handling for the singularities at the eight  \varlink{exch2\_isSedge}{exch2_isSedge}, and
102  corners of the cube.  {\em exch2\_isNedge} contains a count of how many  \varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
103  neighboring tiles each tile has, and is used for setting bounds for looping  tile lies on the edge of a subdomain, 0 if not.  The values are used
104  over neighboring tiles.  {\em exch2\_tProc} holds the process rank of each tile,  within the topology generator to determine the orientation of
105  and is used in interprocess communication.  neighboring tiles and to indicate whether a tile lies on the corner of
106    a subdomain.  The latter case indicates special exchange and numerical
107    handling for the singularities at the eight corners of the cube.
108    \varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
109    how many neighboring tiles each tile has, and is used for setting
110    bounds for looping over neighboring tiles.
111    \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
112    tile, and is used in interprocess communication.
113    
114  \subsubsection{Arrays Indexed to Tile Number and Neighbor}  \subsubsection{Arrays Indexed to Tile Number and Neighbor}
115    
116  The following arrays are all of size {\em MAX\_NEIGHBOURS}x{\em NTILES} and  The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
117  describe the orientations between the the tiles.  \texttt{NTILES} and describe the orientations between the the tiles.
118    
119  The array {\em exch2\_neighbourId(a,T)} holds the tile number $T_{n}$ for each tile  The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
120  {\em T}'s neighbor tile {\em a}.  The neighbor tiles are indexed {\em 1,MAX\_NEIGHBOURS }  each of the $n$ neighboring tiles.  The neighbor tiles are indexed
121  in the order right to left on the north then south edges, and then top to bottom on the east  \texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
122  and west edges. maybe throw in a fig here, eh?    then south edges, and then top to bottom on the east and west edges.
123    Maybe throw in a fig here, eh?
124  {\em exch2\_opposingSend\_record(a,T)} holds  
125  the index c in {\em exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.  The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
126  In other words,  in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
127    In other words,
128  \begin{verbatim}    \begin{verbatim}
129  exch2_neighbourId( exch2_opposingSend_record(a,T),     exch2_neighbourId( exch2_opposingSend_record(a,T),
130                             exch2_neighbourId(a,T) ) = T                        exch2_neighbourId(a,T) ) = T
131  \end{verbatim}  \end{verbatim}
132    and this provides a back-reference from the neighbor tiles.
133    
134  % {\em exch2\_neighbourId(exch2\_opposingSend\_record(a,T),exch2\_neighbourId(a,T))=T}.  The arrays \varlink{exch2\_pi}{exch2_pi},
135  % alternate version  \varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
136    \varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
137  This is to provide a backreference from the neighbor tiles.  \varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
138    exchanges between the neighboring tiles.  The dimensions of
139  The arrays {\em exch2\_pi }, {\em exch2\_pj }, {\em exch2\_oi },  \texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
140  {\em exch2\_oj }, {\em exch2\_oi\_f }, and {\em exch2\_oj\_f }  specify  neighbor ID \textit{N} and the tile number \textit{T} as explained
141  the transformations in exchanges between the neighboring tiles.  The dimensions    above, plus the transformation vector {\em t }, of length two.  The
142  of {\em exch2\_pi(t,N,T) } and {\em exch2\_pj(t,N,T) } are the neighbor ID  first element of the transformation vector indicates the factor by
143  { \em N } and the tile number {\em T } as explained above, plus the transformation  which variables representing the same vector component of a tile will
144  vector {\em t }, of length two.  The first element of the transformation vector indicates  be multiplied, and the second element indicates the transform to the
145  the factor by which variables representing the same vector component  of a tile  variable in the other direction.  As an example,
146  will be multiplied, and the second element indicates the transform to the  \texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
147  variable in the other direction.  As an example, {\em exch2\_pi(1,N,T) } holds the  vector variable in tile \texttt{T} to the i-component of tile
148  transform of the i-component of a vector variable in tile {\em T } to the i-component of  \texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
149  tile  {\em T }'s neighbor  {\em N }, and {\em exch2\_pi(2,N,T) } hold the component  the component of neighbor \texttt{N}'s j-component.
150  of neighbor  {\em N }'s j-component.  
151    Under the current cube topology, one of the two elements of
152  Under the current cube topology, one of the two elements of {\em exch2\_pi } or {\em exch2\_pj }  \texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
153  for a given tile   {\em T } and  neighbor  {\em N } will be 0, reflecting the fact that  and neighbor \texttt{N} will be 0, reflecting the fact that the vector
154  the vector components are orthogonal.  The other element will be 1 or -1, depending on whether  components are orthogonal.  The other element will be 1 or -1,
155  the components are indexed in the same or opposite directions.  For example, the transform dimension  depending on whether the components are indexed in the same or
156  of the arrays for all tile neighbors on the same subdomain will be {\em [1 , 0] }, since all tiles on  opposite directions.  For example, the transform dimension of the
157  the same subdomain are oriented identically.  Vectors that correspond to the orthogonal dimension with the  arrays for all tile neighbors on the same subdomain will be [1,0],
158  same index direction will have {\em [0 , 1] }, whereas those in the opposite index direction will have  since all tiles on the same subdomain are oriented identically.
159  {\em [0 , -1] }.  Vectors that correspond to the orthogonal dimension with the same
160    index direction will have [0,1], whereas those in the opposite index
161    direction will have [0,-1].
162    
163    
164  //  {\footnotesize
   
165  \begin{verbatim}  \begin{verbatim}
   
   
   
166  C      exch2_pi          :: X index row of target to source permutation  C      exch2_pi          :: X index row of target to source permutation
167  C                        :: matrix for each neighbour entry.              C                        :: matrix for each neighbour entry.            
168  C      exch2_pj          :: Y index row of target to source permutation  C      exch2_pj          :: Y index row of target to source permutation
# Line 172  C      exch2_oj_f        :: Y index elem Line 180  C      exch2_oj_f        :: Y index elem
180  C                        :: offset vector for face quantities            C                        :: offset vector for face quantities          
181  C                        :: of each neighbor entry.                      C                        :: of each neighbor entry.                    
182  \end{verbatim}  \end{verbatim}
183    }
184    
185    
186    
Legend:
Removed from v.1.7  
changed lines
  Added in v.1.8
  ViewVC Help
Powered by ViewVC 1.1.22