s_phys_pkgs/text/exch2.tex

% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.8 2004/02/17 21:58:56 edhill Exp $
% $Name:  $

%%  * Introduction
%%    o what it does, citations (refs go into mitgcm_manual.bib, 
%%      preferably in alphabetic order)
%%    o Equations 
%%  * Key subroutines and parameters
%%  * Reference material (auto generated from Protex and structured comments)
%%    o automatically inserted at \section{Reference} 


\section{Extended Cubed Sphere Exchange}
\label{sec:exch2}


\subsection{Introduction}

The \texttt{exch2} package is an extension to the original cubed
sphere topological configuration that allows more flexible domain
decomposition and parallelization.  Cube faces (also called
subdomains) may be divided into any number of tiles that divide evenly
into the grid point dimensions of the subdomain.  Furthermore, the
individual tiles may be run on separate processors in different
combinations, and whether exchanges between particular tiles occur
between different processors is determined at runtime.  This
flexibility provides for manual load balancing across a relatively
arbitrary number of processors.

The exchange parameters are declared in
\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
and assigned in
\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
validity of the cube topology depends on the \texttt{SIZE.h} file as
detailed below.  Both files are generated by Matlab scripts in ??
check these in already! and should not be edited.  The default files
provided in the release configure a cubed sphere arrangement of six
tiles, one per subdomain, each with 32$\times$32 grid points, all
running on a single processor.  Pregenerated examples of these files 
with alternate topologies are provided in ??.

\subsection{Invoking exch2}

To use exch2 with the cubed sphere, the following conditions must be met:

- the exch2 package is included when \texttt{genmake2} is run.  The
  easiest way to do this is to add the line \texttt{exch2} to the
  \texttt{profile.conf} file -- see Section \ref{sect:buildingCode}
  for general details. \\

- an example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and
  \texttt{w2\_e2setup.F} must reside in a directory containing code
  linked when \texttt{genmake2} runs.  The safest place to put these
  is the directory indicated in the \texttt{-mods=DIR} command line
  modifier (typically \texttt{../code}), or the build directory.  The
  default versions of these files reside in \texttt{pkg/exch2}, but
  they should be left untouched to avoid breaking configurations other
  than the one you intend to modify.\\

- files containing grid parameters, named
  \texttt{tile}xxx\texttt{.mitgrid} where xxx is \texttt{001} through
  \texttt{006}, must be in the working directory when the MITgcm
  executable is run.  These files are provided in the example
  experiments for cubed sphere configurations with 32$\times$32 cube
  sides and are non-trivial to generate -- please contact MITgcm
  support if you want to generate files for other configurations.
  This is lame. ?? \\

As of the time of writing the following examples use exch2 and may be
used for guidance:

\begin{verbatim}
verification/adjust_nlfs.cs-32x32x1
verification/adjustment.cs-32x32x1 
verification/aim.5l_cs
verification/global_ocean.cs32x15
verification/hs94.cs-32x32x5
\end{verbatim}


\subsection{Generating Topology Files}

\subsection{Key Variables}

The descriptions of the variables are divided up into scalars,
one-dimensional arrays indexed to the tile number, and two and three
dimensional arrays indexed to tile number and neighboring tile.  This
division actually reflects the functionality of these variables: the
scalars are common to every part of the topology, the tile-indexed
arrays to individual tiles, and the arrays indexed to tile and
neighbor to relationships between tiles and their neighbors.

\subsubsection{Scalars}

The number of tiles in a particular topology is set with the parameter
\texttt{NTILES}, and the maximum number of neighbors of any tiles by
\texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the
size of the various one and two dimensional arrays that store tile
parameters indexed to the tile number.\\

The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
of tiles in the x and y global indices.  For example, the default
setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
\texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in
gridpoints) tiles, four (2x2) per subdomain, will have
\texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
Note that these parameters express the tile layout to allow global
data files that are tile-layout-neutral and have no bearing on the
internal storage of the arrays.  The tiles are internally stored in a
range from \texttt{1,bi} (in the x axis) and y-axis variable
\texttt{bj} is generally ignored within the package.

\subsubsection{Arrays Indexed to Tile Number}

The following arrays are of size \texttt{NTILES}, are indexed to the
tile number, and the indices are omitted in their descriptions.

The arrays \varlink{exch2\_tnx}{exch2_tnx} and
\varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
\texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future
releases of MITgcm are to allow varying tile sizes.

The location of the tiles' Cartesian origin within a subdomain are
determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
\varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
relate the location of the edges of the tiles to each other.  As an
example, in the default six-tile topology (the degenerate case) each
index in these arrays are set to 0.  The twenty-four, 32x32 cube face
case discussed above will have values of 0 or 16, depending on the
quadrant the tile falls within the subdomain.  The array 
\varlink{exch2\_myFace}{exch2_myFace} contains the number of the
cubeface/subdomain of each tile, numbered 1-6 in the case of the
standard cube topology.

The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
\varlink{exch2\_tbasex}{exch2_tbasex} and
\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
global address space, similar to that used by global files.

The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
\varlink{exch2\_isEedge}{exch2_isEedge},
\varlink{exch2\_isSedge}{exch2_isSedge}, and
\varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
tile lies on the edge of a subdomain, 0 if not.  The values are used
within the topology generator to determine the orientation of
neighboring tiles and to indicate whether a tile lies on the corner of
a subdomain.  The latter case indicates special exchange and numerical
handling for the singularities at the eight corners of the cube.
\varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
how many neighboring tiles each tile has, and is used for setting
bounds for looping over neighboring tiles.
\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
tile, and is used in interprocess communication.

\subsubsection{Arrays Indexed to Tile Number and Neighbor}

The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
\texttt{NTILES} and describe the orientations between the the tiles.

The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
each of the $n$ neighboring tiles.  The neighbor tiles are indexed
\texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
then south edges, and then top to bottom on the east and west edges.
Maybe throw in a fig here, eh?

The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
In other words, 
\begin{verbatim}
   exch2_neighbourId( exch2_opposingSend_record(a,T),
                      exch2_neighbourId(a,T) ) = T
\end{verbatim}
and this provides a back-reference from the neighbor tiles.

The arrays \varlink{exch2\_pi}{exch2_pi},
\varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
\varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
exchanges between the neighboring tiles.  The dimensions of
\texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
neighbor ID \textit{N} and the tile number \textit{T} as explained
above, plus the transformation vector {\em t }, of length two.  The
first element of the transformation vector indicates the factor by
which variables representing the same vector component of a tile will
be multiplied, and the second element indicates the transform to the
variable in the other direction.  As an example,
\texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
vector variable in tile \texttt{T} to the i-component of tile
\texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
the component of neighbor \texttt{N}'s j-component.

Under the current cube topology, one of the two elements of
\texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
and neighbor \texttt{N} will be 0, reflecting the fact that the vector
components are orthogonal.  The other element will be 1 or -1,
depending on whether the components are indexed in the same or
opposite directions.  For example, the transform dimension of the
arrays for all tile neighbors on the same subdomain will be [1,0],
since all tiles on the same subdomain are oriented identically.
Vectors that correspond to the orthogonal dimension with the same
index direction will have [0,1], whereas those in the opposite index
direction will have [0,-1].


{\footnotesize
\begin{verbatim}
C      exch2_pi          :: X index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_pj          :: Y index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_oi          :: X index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oj          :: Y index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oi_f        :: X index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
C      exch2_oj_f        :: Y index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
\end{verbatim}
}


\subsection{Key Routines}


\subsection{References}
1	afe	1.9	% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.8 2004/02/17 21:58:56 edhill Exp $
2	afe	1.1	% $Name: $
3
4			%% * Introduction
5			%% o what it does, citations (refs go into mitgcm_manual.bib,
6			%% preferably in alphabetic order)
7			%% o Equations
8			%% * Key subroutines and parameters
9			%% * Reference material (auto generated from Protex and structured comments)
10			%% o automatically inserted at \section{Reference}
11
12
13	edhill	1.8	\section{Extended Cubed Sphere Exchange}
14	afe	1.3	\label{sec:exch2}
15
16	afe	1.1
17			\subsection{Introduction}
18	afe	1.2
19	edhill	1.8	The \texttt{exch2} package is an extension to the original cubed
20	afe	1.9	sphere topological configuration that allows more flexible domain
21			decomposition and parallelization. Cube faces (also called
22			subdomains) may be divided into any number of tiles that divide evenly
23			into the grid point dimensions of the subdomain. Furthermore, the
24			individual tiles may be run on separate processors in different
25			combinations, and whether exchanges between particular tiles occur
26			between different processors is determined at runtime. This
27			flexibility provides for manual load balancing across a relatively
28			arbitrary number of processors.
29	edhill	1.8
30			The exchange parameters are declared in
31			\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
32			and assigned in
33	afe	1.9	\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
34			validity of the cube topology depends on the \texttt{SIZE.h} file as
35			detailed below. Both files are generated by Matlab scripts in ??
36			check these in already! and should not be edited. The default files
37			provided in the release configure a cubed sphere arrangement of six
38			tiles, one per subdomain, each with 32$\times$32 grid points, all
39			running on a single processor. Pregenerated examples of these files
40			with alternate topologies are provided in ??.
41
42			\subsection{Invoking exch2}
43
44			To use exch2 with the cubed sphere, the following conditions must be met:
45
46			- the exch2 package is included when \texttt{genmake2} is run. The
47			easiest way to do this is to add the line \texttt{exch2} to the
48			\texttt{profile.conf} file -- see Section \ref{sect:buildingCode}
49			for general details. \\
50
51			- an example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and
52			\texttt{w2\_e2setup.F} must reside in a directory containing code
53			linked when \texttt{genmake2} runs. The safest place to put these
54			is the directory indicated in the \texttt{-mods=DIR} command line
55			modifier (typically \texttt{../code}), or the build directory. The
56			default versions of these files reside in \texttt{pkg/exch2}, but
57			they should be left untouched to avoid breaking configurations other
58			than the one you intend to modify.\\
59
60			- files containing grid parameters, named
61			\texttt{tile}xxx\texttt{.mitgrid} where xxx is \texttt{001} through
62			\texttt{006}, must be in the working directory when the MITgcm
63			executable is run. These files are provided in the example
64			experiments for cubed sphere configurations with 32$\times$32 cube
65			sides and are non-trivial to generate -- please contact MITgcm
66			support if you want to generate files for other configurations.
67			This is lame. ?? \\
68
69			As of the time of writing the following examples use exch2 and may be
70			used for guidance:
71
72			\begin{verbatim}
73			verification/adjust_nlfs.cs-32x32x1
74			verification/adjustment.cs-32x32x1
75			verification/aim.5l_cs
76			verification/global_ocean.cs32x15
77			verification/hs94.cs-32x32x5
78			\end{verbatim}
79
80
81
82
83			\subsection{Generating Topology Files}
84	afe	1.4
85			\subsection{Key Variables}
86
87			The descriptions of the variables are divided up into scalars,
88	edhill	1.8	one-dimensional arrays indexed to the tile number, and two and three
89			dimensional arrays indexed to tile number and neighboring tile. This
90			division actually reflects the functionality of these variables: the
91			scalars are common to every part of the topology, the tile-indexed
92			arrays to individual tiles, and the arrays indexed to tile and
93			neighbor to relationships between tiles and their neighbors.
94	afe	1.4
95			\subsubsection{Scalars}
96
97			The number of tiles in a particular topology is set with the parameter
98	edhill	1.8	\texttt{NTILES}, and the maximum number of neighbors of any tiles by
99			\texttt{MAX\_NEIGHBOURS}. These parameters are used for defining the
100			size of the various one and two dimensional arrays that store tile
101	afe	1.9	parameters indexed to the tile number.\\
102	edhill	1.8
103			The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
104			and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
105			of tiles in the x and y global indices. For example, the default
106			setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
107			\texttt{exch2\_domain\_nyt=1}. A topology of twenty-four square (in
108			gridpoints) tiles, four (2x2) per subdomain, will have
109			\texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
110			Note that these parameters express the tile layout to allow global
111			data files that are tile-layout-neutral and have no bearing on the
112			internal storage of the arrays. The tiles are internally stored in a
113			range from \texttt{1,bi} (in the x axis) and y-axis variable
114			\texttt{bj} is generally ignored within the package.
115	afe	1.4
116	afe	1.6	\subsubsection{Arrays Indexed to Tile Number}
117	afe	1.4
118	edhill	1.8	The following arrays are of size \texttt{NTILES}, are indexed to the
119			tile number, and the indices are omitted in their descriptions.
120	afe	1.4
121	edhill	1.8	The arrays \varlink{exch2\_tnx}{exch2_tnx} and
122			\varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
123			tile. At present for each tile \texttt{exch2\_tnx=sNx} and
124			\texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}. Future
125			releases of MITgcm are to allow varying tile sizes.
126
127			The location of the tiles' Cartesian origin within a subdomain are
128			determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
129			\varlink{exch2\_tbasey}{exch2_tbasey}. These variables are used to
130			relate the location of the edges of the tiles to each other. As an
131			example, in the default six-tile topology (the degenerate case) each
132			index in these arrays are set to 0. The twenty-four, 32x32 cube face
133			case discussed above will have values of 0 or 16, depending on the
134			quadrant the tile falls within the subdomain. The array
135			\varlink{exch2\_myFace}{exch2_myFace} contains the number of the
136			cubeface/subdomain of each tile, numbered 1-6 in the case of the
137			standard cube topology.
138
139			The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
140			\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
141			\varlink{exch2\_tbasex}{exch2_tbasex} and
142			\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
143			global address space, similar to that used by global files.
144
145			The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
146			\varlink{exch2\_isEedge}{exch2_isEedge},
147			\varlink{exch2\_isSedge}{exch2_isSedge}, and
148			\varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
149			tile lies on the edge of a subdomain, 0 if not. The values are used
150			within the topology generator to determine the orientation of
151			neighboring tiles and to indicate whether a tile lies on the corner of
152			a subdomain. The latter case indicates special exchange and numerical
153			handling for the singularities at the eight corners of the cube.
154			\varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
155			how many neighboring tiles each tile has, and is used for setting
156			bounds for looping over neighboring tiles.
157			\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
158			tile, and is used in interprocess communication.
159	afe	1.4
160	afe	1.6	\subsubsection{Arrays Indexed to Tile Number and Neighbor}
161	afe	1.4
162	edhill	1.8	The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
163			\texttt{NTILES} and describe the orientations between the the tiles.
164	afe	1.5
165	edhill	1.8	The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
166			each of the $n$ neighboring tiles. The neighbor tiles are indexed
167			\texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
168			then south edges, and then top to bottom on the east and west edges.
169			Maybe throw in a fig here, eh?
170
171			The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
172			in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
173			In other words,
174			\begin{verbatim}
175			exch2_neighbourId( exch2_opposingSend_record(a,T),
176			exch2_neighbourId(a,T) ) = T
177	afe	1.5	\end{verbatim}
178	edhill	1.8	and this provides a back-reference from the neighbor tiles.
179	afe	1.5
180	edhill	1.8	The arrays \varlink{exch2\_pi}{exch2_pi},
181			\varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
182			\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
183			\varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
184			exchanges between the neighboring tiles. The dimensions of
185			\texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
186			neighbor ID \textit{N} and the tile number \textit{T} as explained
187			above, plus the transformation vector {\em t }, of length two. The
188			first element of the transformation vector indicates the factor by
189			which variables representing the same vector component of a tile will
190			be multiplied, and the second element indicates the transform to the
191			variable in the other direction. As an example,
192			\texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
193			vector variable in tile \texttt{T} to the i-component of tile
194			\texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
195			the component of neighbor \texttt{N}'s j-component.
196
197			Under the current cube topology, one of the two elements of
198			\texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
199			and neighbor \texttt{N} will be 0, reflecting the fact that the vector
200			components are orthogonal. The other element will be 1 or -1,
201			depending on whether the components are indexed in the same or
202			opposite directions. For example, the transform dimension of the
203			arrays for all tile neighbors on the same subdomain will be [1,0],
204			since all tiles on the same subdomain are oriented identically.
205			Vectors that correspond to the orthogonal dimension with the same
206			index direction will have [0,1], whereas those in the opposite index
207			direction will have [0,-1].
208	afe	1.5
209	afe	1.4
210	edhill	1.8	{\footnotesize
211	afe	1.4	\begin{verbatim}
212			C exch2_pi :: X index row of target to source permutation
213			C :: matrix for each neighbour entry.
214			C exch2_pj :: Y index row of target to source permutation
215			C :: matrix for each neighbour entry.
216			C exch2_oi :: X index element of target to source
217			C :: offset vector for cell-centered quantities
218			C :: of each neighbor entry.
219			C exch2_oj :: Y index element of target to source
220			C :: offset vector for cell-centered quantities
221			C :: of each neighbor entry.
222			C exch2_oi_f :: X index element of target to source
223			C :: offset vector for face quantities
224			C :: of each neighbor entry.
225			C exch2_oj_f :: Y index element of target to source
226			C :: offset vector for face quantities
227			C :: of each neighbor entry.
228			\end{verbatim}
229	edhill	1.8	}
230	afe	1.1
231
232
233			\subsection{Key Routines}
234
235
236
237			\subsection{References}