s_phys_pkgs/text/exch2.tex

% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.6 2004/02/03 19:43:38 afe Exp $
% $Name:  $

%%  * Introduction
%%    o what it does, citations (refs go into mitgcm_manual.bib, 
%%      preferably in alphabetic order)
%%    o Equations 
%%  * Key subroutines and parameters
%%  * Reference material (auto generated from Protex and structured comments)
%%    o automatically inserted at \section{Reference} 


\section{exch2: Extended Cubed Sphere Exchange}
\label{sec:exch2}


\subsection{Introduction}

The exch2 package is an extension to the original cubed sphere exchanges
to allow more flexible domain decomposition and parallelization.  Cube faces
(subdomains) may be divided into whatever number of tiles that divide evenly
into the grid point dimensions of the subdomain.  Furthermore, the individual
tiles may be run on separate processors in different combinations,
and whether exchanges between particular tiles occur between different
processors is determined at runtime.

The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and 
assigned in {\em w2\_e2setup.F}, both in the 
{\em pkg/exch2} directory.  The validity of the cube topology depends
on the {\em SIZE.h} file as detailed below.  Both files are generated by 
Matlab scripts and
should not be edited.  The default files provided in the release set up
a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid
points, running on a single processor.  

\subsection{Key Variables}

The descriptions of the variables are divided up into scalars,
one-dimensional arrays indexed to the tile number, and two and three 
dimensional
arrays indexed to tile number and neighboring tile.  This division
actually reflects  the functionality of these variables: the scalars
are common to every part of the topology, the tile-indexed arrays to 
individual tiles, and the arrays indexed to tile and neighbor to 
relationships between tiles and their neighbors.

\subsubsection{Scalars}

The number of tiles in a particular topology is set with the parameter
{\em NTILES}, and the maximum number of neighbors of any tiles by 
{\em MAX\_NEIGHBOURS}.  These parameters are used for defining the size of
the various one and two dimensional arrays that store tile parameters
indexed to the tile number.

The scalar parameters {\em exch2\_domain\_nxt} and 
{\em exch2\_domain\_nyt} express the number of tiles in the x and y global
indices.  For example, the default setup of six tiles has 
{\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}.  A topology of
twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will
have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}.  Note 
that these parameters express the tile layout to allow global data files that
are tile-layout-neutral and have no bearing on the internal storage of the
arrays.  The tiles are internally stored in a range from {\em 1,bi} (in the
x axis) and y-axis variable {\em bj} is generally ignored within the package.

\subsubsection{Arrays Indexed to Tile Number}

The following arrays are of size {\em NTILES}, are indexed to the tile number, 
and the indices are omitted in their descriptions.

The arrays {\em exch2\_tnx} and {\em exch2\_tny} 
express the x and y dimensions of each tile.  At present for each tile
{\em exch2\_tnx = sNx} 
and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}.  Future releases of 
MITgcm are to allow varying tile sizes.

The location of the tiles' Cartesian origin within a subdomain are determined 
by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}.  These variables
are used to relate the location of the edges of the tiles to each other.  As 
an example, in the default six-tile topology (the degenerate case) 
each index in these arrays are 
set to 0.  The twenty-four, 32x32 cube face case discussed above will have
values of 0 or 16, depending on the quadrant the tile falls within the 
subdomain.  {\em exch2\_myFace} contains the number of the 
cubeface/subdomain of each tile, numbered 1-6 in the case of the standard
cube topology.  

The arrays {\em exch2\_txglobalo} and {\em exch2\_txglobalo} are similar to
{\em exch2\_tbasex} and {\em exch2\_tbasey}, but locate the tiles within
the global address space, similar to that used by global files.  

The arrays {\em exch2\_isWedge}, {\em exch2\_isEedge}, {\em exch2\_isSedge}, 
and {\em exch2\_isNedge} are set to 1 if the indexed tile lies on the edge
of a subdomain, 0 if not.  The values are used within the topology generator
to determine the orientation of neighboring tiles and to indicate whether 
a tile lies on the corner of a subdomain.  The latter case indicates 
special exchange and numerical handling for the singularities at the eight 
corners of the cube.  {\em exch2\_isNedge} contains a count of how many
neighboring tiles each tile has, and is used for setting bounds for looping
over neighboring tiles.  {\em exch2\_tProc} holds the process rank of each tile,
and is used in interprocess communication.

\subsubsection{Arrays Indexed to Tile Number and Neighbor}

The following arrays are all of size {\em MAX\_NEIGHBOURS}x{\em NTILES} and
describe the orientations between the the tiles.

The array {\em exch2\_neighbourId(a,T)} holds the tile number $T_{n}$ for each tile 
{\em T}'s neighbor tile {\em a}.  The neighbor tiles are indexed {\em 1,MAX\_NEIGHBOURS }
in the order right to left on the north then south edges, and then top to bottom on the east 
and west edges. maybe throw in a fig here, eh?  

{\em exch2\_opposingSend\_record(a,T)} holds 
the index c in {\em exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
In other words,

\begin{verbatim}   
exch2_neighbourId( exch2_opposingSend_record(a,T), 
                           exch2_neighbourId(a,T) ) = T
\end{verbatim}

% {\em exch2\_neighbourId(exch2\_opposingSend\_record(a,T),exch2\_neighbourId(a,T))=T}.
% alternate version

This is to provide a backreference from the neighbor tiles.

The arrays {\em exch2\_pi }, {\em exch2\_pj }, {\em exch2\_oi }, 
{\em exch2\_oj }, {\em exch2\_oi\_f }, and {\em exch2\_oj\_f }  specify 
the transformations in exchanges between the neighboring tiles.  The dimensions  
of {\em exch2\_pi(t,N,T) } and {\em exch2\_pj(t,N,T) } are the neighbor ID 
{ \em N } and the tile number {\em T } as explained above, plus the transformation
vector {\em t }, of length two.  The first element of the transformation vector indicates
the factor by which variables representing the same vector component  of a tile 
will be multiplied, and the second element indicates the transform to the 
variable in the other direction.  As an example, {\em exch2\_pi(1,N,T) } holds the 
transform of the i-component of a vector variable in tile {\em T } to the i-component of 
tile  {\em T }'s neighbor  {\em N }, and {\em exch2\_pi(2,N,T) } hold the component 
of neighbor  {\em N }'s j-component.

Under the current cube topology, one of the two elements of {\em exch2\_pi } or {\em exch2\_pj }
for a given tile   {\em T } and  neighbor  {\em N } will be 0, reflecting the fact that
the vector components are orthogonal.  The other element will be 1 or -1, depending on whether
the components are indexed in the same or opposite directions.  For example, the transform dimension
of the arrays for all tile neighbors on the same subdomain will be {\em [1 , 0] }, since all tiles on 
the same subdomain are oriented identically.  Vectors that correspond to the orthogonal dimension with the 
same index direction will have {\em [0 , 1] }, whereas those in the opposite index direction will have
{\em [0 , -1] }.


//

\begin{verbatim}


C      exch2_pi          :: X index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_pj          :: Y index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_oi          :: X index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oj          :: Y index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oi_f        :: X index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
C      exch2_oj_f        :: Y index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
\end{verbatim}


\subsection{Key Routines}


\subsection{References}
1	afe	1.7	% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.6 2004/02/03 19:43:38 afe Exp $
2	afe	1.1	% $Name: $
3
4			%% * Introduction
5			%% o what it does, citations (refs go into mitgcm_manual.bib,
6			%% preferably in alphabetic order)
7			%% o Equations
8			%% * Key subroutines and parameters
9			%% * Reference material (auto generated from Protex and structured comments)
10			%% o automatically inserted at \section{Reference}
11
12
13			\section{exch2: Extended Cubed Sphere Exchange}
14	afe	1.3	\label{sec:exch2}
15
16	afe	1.1
17			\subsection{Introduction}
18	afe	1.2
19			The exch2 package is an extension to the original cubed sphere exchanges
20			to allow more flexible domain decomposition and parallelization. Cube faces
21	afe	1.4	(subdomains) may be divided into whatever number of tiles that divide evenly
22	afe	1.2	into the grid point dimensions of the subdomain. Furthermore, the individual
23	afe	1.4	tiles may be run on separate processors in different combinations,
24			and whether exchanges between particular tiles occur between different
25			processors is determined at runtime.
26
27			The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and
28			assigned in {\em w2\_e2setup.F}, both in the
29			{\em pkg/exch2} directory. The validity of the cube topology depends
30			on the {\em SIZE.h} file as detailed below. Both files are generated by
31			Matlab scripts and
32			should not be edited. The default files provided in the release set up
33			a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid
34			points, running on a single processor.
35
36			\subsection{Key Variables}
37
38			The descriptions of the variables are divided up into scalars,
39	afe	1.6	one-dimensional arrays indexed to the tile number, and two and three
40			dimensional
41	afe	1.4	arrays indexed to tile number and neighboring tile. This division
42	afe	1.6	actually reflects the functionality of these variables: the scalars
43			are common to every part of the topology, the tile-indexed arrays to
44			individual tiles, and the arrays indexed to tile and neighbor to
45			relationships between tiles and their neighbors.
46	afe	1.4
47			\subsubsection{Scalars}
48
49			The number of tiles in a particular topology is set with the parameter
50			{\em NTILES}, and the maximum number of neighbors of any tiles by
51			{\em MAX\_NEIGHBOURS}. These parameters are used for defining the size of
52			the various one and two dimensional arrays that store tile parameters
53			indexed to the tile number.
54
55			The scalar parameters {\em exch2\_domain\_nxt} and
56			{\em exch2\_domain\_nyt} express the number of tiles in the x and y global
57			indices. For example, the default setup of six tiles has
58			{\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}. A topology of
59			twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will
60			have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}. Note
61			that these parameters express the tile layout to allow global data files that
62			are tile-layout-neutral and have no bearing on the internal storage of the
63			arrays. The tiles are internally stored in a range from {\em 1,bi} (in the
64			x axis) and y-axis variable {\em bj} is generally ignored within the package.
65
66	afe	1.6	\subsubsection{Arrays Indexed to Tile Number}
67	afe	1.4
68	afe	1.5	The following arrays are of size {\em NTILES}, are indexed to the tile number,
69			and the indices are omitted in their descriptions.
70	afe	1.4
71			The arrays {\em exch2\_tnx} and {\em exch2\_tny}
72			express the x and y dimensions of each tile. At present for each tile
73			{\em exch2\_tnx = sNx}
74			and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}. Future releases of
75			MITgcm are to allow varying tile sizes.
76
77			The location of the tiles' Cartesian origin within a subdomain are determined
78	afe	1.5	by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}. These variables
79			are used to relate the location of the edges of the tiles to each other. As
80			an example, in the default six-tile topology (the degenerate case)
81			each index in these arrays are
82			set to 0. The twenty-four, 32x32 cube face case discussed above will have
83			values of 0 or 16, depending on the quadrant the tile falls within the
84			subdomain. {\em exch2\_myFace} contains the number of the
85			cubeface/subdomain of each tile, numbered 1-6 in the case of the standard
86			cube topology.
87
88			The arrays {\em exch2\_txglobalo} and {\em exch2\_txglobalo} are similar to
89			{\em exch2\_tbasex} and {\em exch2\_tbasey}, but locate the tiles within
90			the global address space, similar to that used by global files.
91
92			The arrays {\em exch2\_isWedge}, {\em exch2\_isEedge}, {\em exch2\_isSedge},
93			and {\em exch2\_isNedge} are set to 1 if the indexed tile lies on the edge
94			of a subdomain, 0 if not. The values are used within the topology generator
95			to determine the orientation of neighboring tiles and to indicate whether
96			a tile lies on the corner of a subdomain. The latter case indicates
97			special exchange and numerical handling for the singularities at the eight
98			corners of the cube. {\em exch2\_isNedge} contains a count of how many
99			neighboring tiles each tile has, and is used for setting bounds for looping
100			over neighboring tiles. {\em exch2\_tProc} holds the process rank of each tile,
101			and is used in interprocess communication.
102	afe	1.4
103	afe	1.6	\subsubsection{Arrays Indexed to Tile Number and Neighbor}
104	afe	1.4
105	afe	1.5	The following arrays are all of size {\em MAX\_NEIGHBOURS}x{\em NTILES} and
106			describe the orientations between the the tiles.
107
108			The array {\em exch2\_neighbourId(a,T)} holds the tile number $T_{n}$ for each tile
109	afe	1.7	{\em T}'s neighbor tile {\em a}. The neighbor tiles are indexed {\em 1,MAX\_NEIGHBOURS }
110			in the order right to left on the north then south edges, and then top to bottom on the east
111			and west edges. maybe throw in a fig here, eh?
112
113			{\em exch2\_opposingSend\_record(a,T)} holds
114	afe	1.5	the index c in {\em exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
115			In other words,
116
117			\begin{verbatim}
118	afe	1.7	exch2_neighbourId( exch2_opposingSend_record(a,T),
119			exch2_neighbourId(a,T) ) = T
120	afe	1.5	\end{verbatim}
121
122	afe	1.6	% {\em exch2\_neighbourId(exch2\_opposingSend\_record(a,T),exch2\_neighbourId(a,T))=T}.
123			% alternate version
124
125	afe	1.5	This is to provide a backreference from the neighbor tiles.
126	afe	1.7
127			The arrays {\em exch2\_pi }, {\em exch2\_pj }, {\em exch2\_oi },
128			{\em exch2\_oj }, {\em exch2\_oi\_f }, and {\em exch2\_oj\_f } specify
129			the transformations in exchanges between the neighboring tiles. The dimensions
130			of {\em exch2\_pi(t,N,T) } and {\em exch2\_pj(t,N,T) } are the neighbor ID
131			{ \em N } and the tile number {\em T } as explained above, plus the transformation
132			vector {\em t }, of length two. The first element of the transformation vector indicates
133			the factor by which variables representing the same vector component of a tile
134			will be multiplied, and the second element indicates the transform to the
135			variable in the other direction. As an example, {\em exch2\_pi(1,N,T) } holds the
136			transform of the i-component of a vector variable in tile {\em T } to the i-component of
137			tile {\em T }'s neighbor {\em N }, and {\em exch2\_pi(2,N,T) } hold the component
138			of neighbor {\em N }'s j-component.
139
140			Under the current cube topology, one of the two elements of {\em exch2\_pi } or {\em exch2\_pj }
141			for a given tile {\em T } and neighbor {\em N } will be 0, reflecting the fact that
142			the vector components are orthogonal. The other element will be 1 or -1, depending on whether
143			the components are indexed in the same or opposite directions. For example, the transform dimension
144			of the arrays for all tile neighbors on the same subdomain will be {\em [1 , 0] }, since all tiles on
145			the same subdomain are oriented identically. Vectors that correspond to the orthogonal dimension with the
146			same index direction will have {\em [0 , 1] }, whereas those in the opposite index direction will have
147			{\em [0 , -1] }.
148
149
150	afe	1.5
151	afe	1.4
152			//
153
154			\begin{verbatim}
155	afe	1.5
156
157	afe	1.6
158	afe	1.4	C exch2_pi :: X index row of target to source permutation
159			C :: matrix for each neighbour entry.
160			C exch2_pj :: Y index row of target to source permutation
161			C :: matrix for each neighbour entry.
162			C exch2_oi :: X index element of target to source
163			C :: offset vector for cell-centered quantities
164			C :: of each neighbor entry.
165			C exch2_oj :: Y index element of target to source
166			C :: offset vector for cell-centered quantities
167			C :: of each neighbor entry.
168			C exch2_oi_f :: X index element of target to source
169			C :: offset vector for face quantities
170			C :: of each neighbor entry.
171			C exch2_oj_f :: Y index element of target to source
172			C :: offset vector for face quantities
173			C :: of each neighbor entry.
174			\end{verbatim}
175	afe	1.1
176
177
178
179			\subsection{Key Routines}
180
181
182
183			\subsection{References}