s_phys_pkgs/text/exch2.tex

% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.9 2004/03/12 20:58:19 afe Exp $
% $Name:  $

%%  * Introduction
%%    o what it does, citations (refs go into mitgcm_manual.bib, 
%%      preferably in alphabetic order)
%%    o Equations 
%%  * Key subroutines and parameters
%%  * Reference material (auto generated from Protex and structured comments)
%%    o automatically inserted at \section{Reference} 


\section{exch2: Extended Cubed Sphere \mbox{Topology}}
\label{sec:exch2}


\subsection{Introduction}

The \texttt{exch2} package is an extension to the original cubed
sphere topological configuration that allows more flexible domain
decomposition and parallelization.  Cube faces (also called
subdomains) may be divided into any number of tiles that divide evenly
into the grid point dimensions of the subdomain.  Furthermore, the
individual tiles may be run on separate processors in different
combinations, and whether exchanges between particular tiles occur
between different processors is determined at runtime.  This
flexibility provides for manual compile-time load balancing across a
relatively arbitrary number of processors. \\

The exchange parameters are declared in
\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
and assigned in
\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
validity of the cube topology depends on the \texttt{SIZE.h} file as
detailed below.  Both files are generated by Matlab scripts in
\texttt{utils/exch2/matlab-topology-generator}; see Section
\ref{sec:topogen} for details on creating alternate topologies.  The
default files provided in the release configure a cubed sphere
topology of six tiles, one per subdomain, each with 32$\times$32 grid
points, all running on a single processor.  Pregenerated examples of
these files with alternate topologies are provided under
\texttt{utils/exch2/code-mods} along with the appropriate
\texttt{SIZE.h} file for single-processor execution.

\subsection{Invoking exch2}

To use exch2 with the cubed sphere, the following conditions must be
met: \\

$\bullet$ The exch2 package is included when \texttt{genmake2} is run.  The
  easiest way to do this is to add the line \texttt{exch2} to the
  \texttt{profile.conf} file -- see Section \ref{sect:buildingCode}
  for general details. \\

$\bullet$ An example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and
  \texttt{w2\_e2setup.F} must reside in a directory containing code
  linked when \texttt{genmake2} runs.  The safest place to put these
  is the directory indicated in the \texttt{-mods=DIR} command line
  modifier (typically \texttt{../code}), or the build directory.  The
  default versions of these files reside in \texttt{pkg/exch2} and are
  linked automatically if no other versions exist elsewhere in the
  link path, but they should be left untouched to avoid breaking
  configurations other than the one you intend to modify.\\

$\bullet$ Files containing grid parameters, named
  \texttt{tile}???\texttt{.mitgrid} where ??? is \texttt{001} through
  \texttt{006} (one per subdomain), must be in the working directory
  when the MITgcm executable is run.  These files are provided in the
  example experiments for cubed sphere configurations with
  32$\times$32 cube sides and are non-trivial to generate -- please
  contact MITgcm support if you want to generate files for other
  configurations. \\

$\bullet$ As always when compiling MITgcm, the file \texttt{SIZE.h}
  must be placed where \texttt{genmake2} will find it.  In particular
  for the exch2, the domain decompositin specified in \texttt{SIZE.h}
  must correspond with the particular configuration's topology
  specified in \texttt{W2\_EXCH2\_TOPOLOGY.h} and
  \texttt{w2\_e2setup.F}.  Domain decomposition issues particular to
  exch2 are addressed in Section \ref{sec:topogen}: ``Generating
  Topology Files for exch2''; a more general background on the subject
  relvant to MITgcm is presented in Section
  \ref{sect:specifying_a_decomposition}: ``Specifying a
  decomposition''.\\

As of the time of writing the following examples use exch2 and may be
used for guidance:

\begin{verbatim}
verification/adjust_nlfs.cs-32x32x1
verification/adjustment.cs-32x32x1 
verification/aim.5l_cs
verification/global_ocean.cs32x15
verification/hs94.cs-32x32x5
\end{verbatim}


\subsection{Generating Topology Files for exch2}
\label{sec:topogen}

Alternate cubed sphere topologies may be created using the Matlab
scripts in \texttt{utils/exch2/matlab-topology-generator}. Running the
m-file \texttt{driver} from the Matlab prompt (without passing any
function parameters) generates exch2 topology files
\texttt{W2\_EXCH2\_TOPOLOGY.h} and \texttt{w2\_e2setup.F} in the
working directory and displays via Matlab a figure of the topology.
The other m-files in the directory are subroutines of \texttt{driver}
and should not be run except for development purposes. \\

The parameters that determine the dimensions and topology of the
generated configuration are nr, nb, ng, tnx and tny, and all are
assigned early in the script.

The first three determine the size of the subdomains (cube faces) and
hence the size of the overall domain.  Each one determines the number
of grid points, and therefore the resolution, along the subdomain
sides in a ``great circle'' around each axis of the cube.  At the time
of this writing MITgcm requires these three parameters to be equal,
but they provide for future releases of MITgcm to accomodate different
resolutions around the axes to allow (for example) greater resolution
around the equator.\\

The parameters tnx and tny determine the dimensions of the tiles into
which the subdomains are decomposed, and must evenly divide the
integer assigned to nr, nb and ng.  The result is a rectangular tiling
of the subdomain.  Figure \ref{fig:24tile} shows one possible topology
for a twenty-four tile cube, and figure \ref{fig:12tile} shows one for
twelve tiles. \\

\begin{figure}
\begin{center}
 \resizebox{4in}{!}{
  \includegraphics{part6/s24t_16x16.ps}
 }
\end{center} 
\caption{Plot of cubed sphere topology with a 32$\times$32 grid and
twenty-four tiles (tnx=16, tny=16)
} \label{fig:24tile}
\end{figure}

\begin{figure}
\begin{center}
 \resizebox{4in}{!}{
  \includegraphics{part6/s12t_16x32.ps}
 }
\end{center} 
\caption{Plot of cubed sphere topology with a 32$\times$32 grid and
twelve tiles (tnx=16, tny=32)
} \label{fig:12tile}
\end{figure}

Tiles can be selected from the topology to be omitted from being
allocated memory and processors.  This kind of tuning is useful in
ocean modeling for omitting tiles that fall entirely on land.  The
tiles omitted are specified in the file \texttt{blanklist.txt} by
their tile number in the topology, separated by a newline. \\


\subsection{Key Variables}

The descriptions of the variables are divided up into scalars,
one-dimensional arrays indexed to the tile number, and two and three
dimensional arrays indexed to tile number and neighboring tile.  This
division actually reflects the functionality of these variables: the
scalars are common to every part of the topology, the tile-indexed
arrays to individual tiles, and the arrays indexed to tile and
neighbor to relationships between tiles and their neighbors.

\subsubsection{Scalars}

The number of tiles in a particular topology is set with the parameter
\texttt{NTILES}, and the maximum number of neighbors of any tiles by
\texttt{MAX\_NEIGHBOURS}.  These parameters are used for defining the
size of the various one and two dimensional arrays that store tile
parameters indexed to the tile number.\\

The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
of tiles in the x and y global indices.  For example, the default
setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
\texttt{exch2\_domain\_nyt=1}.  A topology of twenty-four square (in
gridpoints) tiles, four (2x2) per subdomain, will have
\texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
Note that these parameters express the tile layout to allow global
data files that are tile-layout-neutral and have no bearing on the
internal storage of the arrays.  The tiles are internally stored in a
range from \texttt{1,bi} (in the x axis) and y-axis variable
\texttt{bj} is generally ignored within the package.

\subsubsection{Arrays Indexed to Tile Number}

The following arrays are of size \texttt{NTILES}, are indexed to the
tile number, and the indices are omitted in their descriptions.

The arrays \varlink{exch2\_tnx}{exch2_tnx} and
\varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
tile.  At present for each tile \texttt{exch2\_tnx=sNx} and
\texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}.  Future
releases of MITgcm are to allow varying tile sizes.

The location of the tiles' Cartesian origin within a subdomain are
determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
\varlink{exch2\_tbasey}{exch2_tbasey}.  These variables are used to
relate the location of the edges of the tiles to each other.  As an
example, in the default six-tile topology (the degenerate case) each
index in these arrays are set to 0.  The twenty-four, 32x32 cube face
case discussed above will have values of 0 or 16, depending on the
quadrant the tile falls within the subdomain.  The array 
\varlink{exch2\_myFace}{exch2_myFace} contains the number of the
cubeface/subdomain of each tile, numbered 1-6 in the case of the
standard cube topology.

The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
\varlink{exch2\_tbasex}{exch2_tbasex} and
\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
global address space, similar to that used by global files.

The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
\varlink{exch2\_isEedge}{exch2_isEedge},
\varlink{exch2\_isSedge}{exch2_isSedge}, and
\varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
tile lies on the edge of a subdomain, 0 if not.  The values are used
within the topology generator to determine the orientation of
neighboring tiles and to indicate whether a tile lies on the corner of
a subdomain.  The latter case indicates special exchange and numerical
handling for the singularities at the eight corners of the cube.
\varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
how many neighboring tiles each tile has, and is used for setting
bounds for looping over neighboring tiles.
\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
tile, and is used in interprocess communication.

\subsubsection{Arrays Indexed to Tile Number and Neighbor}

The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
\texttt{NTILES} and describe the orientations between the the tiles.

The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
each of the $n$ neighboring tiles.  The neighbor tiles are indexed
\texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
then south edges, and then top to bottom on the east and west edges.
Maybe throw in a fig here, eh?

The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
In other words, 
\begin{verbatim}
   exch2_neighbourId( exch2_opposingSend_record(a,T),
                      exch2_neighbourId(a,T) ) = T
\end{verbatim}
and this provides a back-reference from the neighbor tiles.

The arrays \varlink{exch2\_pi}{exch2_pi},
\varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
\varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
exchanges between the neighboring tiles.  The dimensions of
\texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
neighbor ID \textit{N} and the tile number \textit{T} as explained
above, plus the transformation vector {\em t }, of length two.  The
first element of the transformation vector indicates the factor by
which variables representing the same vector component of a tile will
be multiplied, and the second element indicates the transform to the
variable in the other direction.  As an example,
\texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
vector variable in tile \texttt{T} to the i-component of tile
\texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
the component of neighbor \texttt{N}'s j-component.

Under the current cube topology, one of the two elements of
\texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
and neighbor \texttt{N} will be 0, reflecting the fact that the vector
components are orthogonal.  The other element will be 1 or -1,
depending on whether the components are indexed in the same or
opposite directions.  For example, the transform dimension of the
arrays for all tile neighbors on the same subdomain will be [1,0],
since all tiles on the same subdomain are oriented identically.
Vectors that correspond to the orthogonal dimension with the same
index direction will have [0,1], whereas those in the opposite index
direction will have [0,-1].


{\footnotesize
\begin{verbatim}
C      exch2_pi          :: X index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_pj          :: Y index row of target to source permutation 
C                        :: matrix for each neighbour entry.            
C      exch2_oi          :: X index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oj          :: Y index element of target to source 
C                        :: offset vector for cell-centered quantities  
C                        :: of each neighbor entry.                     
C      exch2_oi_f        :: X index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
C      exch2_oj_f        :: Y index element of target to source 
C                        :: offset vector for face quantities           
C                        :: of each neighbor entry.                     
\end{verbatim}
}


\subsection{Key Routines}


\subsection{References}
1	afe	1.10	% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.9 2004/03/12 20:58:19 afe Exp $
2	afe	1.1	% $Name: $
3
4			%% * Introduction
5			%% o what it does, citations (refs go into mitgcm_manual.bib,
6			%% preferably in alphabetic order)
7			%% o Equations
8			%% * Key subroutines and parameters
9			%% * Reference material (auto generated from Protex and structured comments)
10			%% o automatically inserted at \section{Reference}
11
12
13	afe	1.10	\section{exch2: Extended Cubed Sphere \mbox{Topology}}
14	afe	1.3	\label{sec:exch2}
15
16	afe	1.1
17			\subsection{Introduction}
18	afe	1.2
19	edhill	1.8	The \texttt{exch2} package is an extension to the original cubed
20	afe	1.9	sphere topological configuration that allows more flexible domain
21			decomposition and parallelization. Cube faces (also called
22			subdomains) may be divided into any number of tiles that divide evenly
23			into the grid point dimensions of the subdomain. Furthermore, the
24			individual tiles may be run on separate processors in different
25			combinations, and whether exchanges between particular tiles occur
26			between different processors is determined at runtime. This
27	afe	1.10	flexibility provides for manual compile-time load balancing across a
28			relatively arbitrary number of processors. \\
29	edhill	1.8
30			The exchange parameters are declared in
31			\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
32			and assigned in
33	afe	1.9	\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The
34			validity of the cube topology depends on the \texttt{SIZE.h} file as
35	afe	1.10	detailed below. Both files are generated by Matlab scripts in
36			\texttt{utils/exch2/matlab-topology-generator}; see Section
37			\ref{sec:topogen} for details on creating alternate topologies. The
38			default files provided in the release configure a cubed sphere
39			topology of six tiles, one per subdomain, each with 32$\times$32 grid
40			points, all running on a single processor. Pregenerated examples of
41			these files with alternate topologies are provided under
42			\texttt{utils/exch2/code-mods} along with the appropriate
43			\texttt{SIZE.h} file for single-processor execution.
44	afe	1.9
45			\subsection{Invoking exch2}
46
47	afe	1.10	To use exch2 with the cubed sphere, the following conditions must be
48			met: \\
49	afe	1.9
50	afe	1.10	$\bullet$ The exch2 package is included when \texttt{genmake2} is run. The
51	afe	1.9	easiest way to do this is to add the line \texttt{exch2} to the
52			\texttt{profile.conf} file -- see Section \ref{sect:buildingCode}
53			for general details. \\
54
55	afe	1.10	$\bullet$ An example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and
56	afe	1.9	\texttt{w2\_e2setup.F} must reside in a directory containing code
57			linked when \texttt{genmake2} runs. The safest place to put these
58			is the directory indicated in the \texttt{-mods=DIR} command line
59			modifier (typically \texttt{../code}), or the build directory. The
60	afe	1.10	default versions of these files reside in \texttt{pkg/exch2} and are
61			linked automatically if no other versions exist elsewhere in the
62			link path, but they should be left untouched to avoid breaking
63			configurations other than the one you intend to modify.\\
64
65			$\bullet$ Files containing grid parameters, named
66			\texttt{tile}???\texttt{.mitgrid} where ??? is \texttt{001} through
67			\texttt{006} (one per subdomain), must be in the working directory
68			when the MITgcm executable is run. These files are provided in the
69			example experiments for cubed sphere configurations with
70			32$\times$32 cube sides and are non-trivial to generate -- please
71			contact MITgcm support if you want to generate files for other
72			configurations. \\
73
74			$\bullet$ As always when compiling MITgcm, the file \texttt{SIZE.h}
75			must be placed where \texttt{genmake2} will find it. In particular
76			for the exch2, the domain decompositin specified in \texttt{SIZE.h}
77			must correspond with the particular configuration's topology
78			specified in \texttt{W2\_EXCH2\_TOPOLOGY.h} and
79			\texttt{w2\_e2setup.F}. Domain decomposition issues particular to
80			exch2 are addressed in Section \ref{sec:topogen}: ``Generating
81			Topology Files for exch2''; a more general background on the subject
82			relvant to MITgcm is presented in Section
83			\ref{sect:specifying_a_decomposition}: ``Specifying a
84			decomposition''.\\
85	afe	1.9
86			As of the time of writing the following examples use exch2 and may be
87			used for guidance:
88
89			\begin{verbatim}
90			verification/adjust_nlfs.cs-32x32x1
91			verification/adjustment.cs-32x32x1
92			verification/aim.5l_cs
93			verification/global_ocean.cs32x15
94			verification/hs94.cs-32x32x5
95			\end{verbatim}
96
97
98
99
100	afe	1.10	\subsection{Generating Topology Files for exch2}
101			\label{sec:topogen}
102
103			Alternate cubed sphere topologies may be created using the Matlab
104			scripts in \texttt{utils/exch2/matlab-topology-generator}. Running the
105			m-file \texttt{driver} from the Matlab prompt (without passing any
106			function parameters) generates exch2 topology files
107			\texttt{W2\_EXCH2\_TOPOLOGY.h} and \texttt{w2\_e2setup.F} in the
108			working directory and displays via Matlab a figure of the topology.
109			The other m-files in the directory are subroutines of \texttt{driver}
110			and should not be run except for development purposes. \\
111
112			The parameters that determine the dimensions and topology of the
113			generated configuration are nr, nb, ng, tnx and tny, and all are
114			assigned early in the script.
115
116			The first three determine the size of the subdomains (cube faces) and
117			hence the size of the overall domain. Each one determines the number
118			of grid points, and therefore the resolution, along the subdomain
119			sides in a ``great circle'' around each axis of the cube. At the time
120			of this writing MITgcm requires these three parameters to be equal,
121			but they provide for future releases of MITgcm to accomodate different
122			resolutions around the axes to allow (for example) greater resolution
123			around the equator.\\
124
125			The parameters tnx and tny determine the dimensions of the tiles into
126			which the subdomains are decomposed, and must evenly divide the
127			integer assigned to nr, nb and ng. The result is a rectangular tiling
128			of the subdomain. Figure \ref{fig:24tile} shows one possible topology
129			for a twenty-four tile cube, and figure \ref{fig:12tile} shows one for
130			twelve tiles. \\
131
132			\begin{figure}
133			\begin{center}
134			\resizebox{4in}{!}{
135			\includegraphics{part6/s24t_16x16.ps}
136			}
137			\end{center}
138			\caption{Plot of cubed sphere topology with a 32$\times$32 grid and
139			twenty-four tiles (tnx=16, tny=16)
140			} \label{fig:24tile}
141			\end{figure}
142
143			\begin{figure}
144			\begin{center}
145			\resizebox{4in}{!}{
146			\includegraphics{part6/s12t_16x32.ps}
147			}
148			\end{center}
149			\caption{Plot of cubed sphere topology with a 32$\times$32 grid and
150			twelve tiles (tnx=16, tny=32)
151			} \label{fig:12tile}
152			\end{figure}
153
154			Tiles can be selected from the topology to be omitted from being
155			allocated memory and processors. This kind of tuning is useful in
156			ocean modeling for omitting tiles that fall entirely on land. The
157			tiles omitted are specified in the file \texttt{blanklist.txt} by
158			their tile number in the topology, separated by a newline. \\
159
160
161
162
163
164	afe	1.4
165			\subsection{Key Variables}
166
167			The descriptions of the variables are divided up into scalars,
168	edhill	1.8	one-dimensional arrays indexed to the tile number, and two and three
169			dimensional arrays indexed to tile number and neighboring tile. This
170			division actually reflects the functionality of these variables: the
171			scalars are common to every part of the topology, the tile-indexed
172			arrays to individual tiles, and the arrays indexed to tile and
173			neighbor to relationships between tiles and their neighbors.
174	afe	1.4
175			\subsubsection{Scalars}
176
177			The number of tiles in a particular topology is set with the parameter
178	edhill	1.8	\texttt{NTILES}, and the maximum number of neighbors of any tiles by
179			\texttt{MAX\_NEIGHBOURS}. These parameters are used for defining the
180			size of the various one and two dimensional arrays that store tile
181	afe	1.9	parameters indexed to the tile number.\\
182	edhill	1.8
183			The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt}
184			and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number
185			of tiles in the x and y global indices. For example, the default
186			setup of six tiles has \texttt{exch2\_domain\_nxt=6} and
187			\texttt{exch2\_domain\_nyt=1}. A topology of twenty-four square (in
188			gridpoints) tiles, four (2x2) per subdomain, will have
189			\texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}.
190			Note that these parameters express the tile layout to allow global
191			data files that are tile-layout-neutral and have no bearing on the
192			internal storage of the arrays. The tiles are internally stored in a
193			range from \texttt{1,bi} (in the x axis) and y-axis variable
194			\texttt{bj} is generally ignored within the package.
195	afe	1.4
196	afe	1.6	\subsubsection{Arrays Indexed to Tile Number}
197	afe	1.4
198	edhill	1.8	The following arrays are of size \texttt{NTILES}, are indexed to the
199			tile number, and the indices are omitted in their descriptions.
200	afe	1.4
201	edhill	1.8	The arrays \varlink{exch2\_tnx}{exch2_tnx} and
202			\varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each
203			tile. At present for each tile \texttt{exch2\_tnx=sNx} and
204			\texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}. Future
205			releases of MITgcm are to allow varying tile sizes.
206
207			The location of the tiles' Cartesian origin within a subdomain are
208			determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and
209			\varlink{exch2\_tbasey}{exch2_tbasey}. These variables are used to
210			relate the location of the edges of the tiles to each other. As an
211			example, in the default six-tile topology (the degenerate case) each
212			index in these arrays are set to 0. The twenty-four, 32x32 cube face
213			case discussed above will have values of 0 or 16, depending on the
214			quadrant the tile falls within the subdomain. The array
215			\varlink{exch2\_myFace}{exch2_myFace} contains the number of the
216			cubeface/subdomain of each tile, numbered 1-6 in the case of the
217			standard cube topology.
218
219			The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and
220			\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to
221			\varlink{exch2\_tbasex}{exch2_tbasex} and
222			\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the
223			global address space, similar to that used by global files.
224
225			The arrays \varlink{exch2\_isWedge}{exch2_isWedge},
226			\varlink{exch2\_isEedge}{exch2_isEedge},
227			\varlink{exch2\_isSedge}{exch2_isSedge}, and
228			\varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed
229			tile lies on the edge of a subdomain, 0 if not. The values are used
230			within the topology generator to determine the orientation of
231			neighboring tiles and to indicate whether a tile lies on the corner of
232			a subdomain. The latter case indicates special exchange and numerical
233			handling for the singularities at the eight corners of the cube.
234			\varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of
235			how many neighboring tiles each tile has, and is used for setting
236			bounds for looping over neighboring tiles.
237			\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
238			tile, and is used in interprocess communication.
239	afe	1.4
240	afe	1.6	\subsubsection{Arrays Indexed to Tile Number and Neighbor}
241	afe	1.4
242	edhill	1.8	The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$
243			\texttt{NTILES} and describe the orientations between the the tiles.
244	afe	1.5
245	edhill	1.8	The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for
246			each of the $n$ neighboring tiles. The neighbor tiles are indexed
247			\texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north
248			then south edges, and then top to bottom on the east and west edges.
249			Maybe throw in a fig here, eh?
250
251			The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c
252			in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T.
253			In other words,
254			\begin{verbatim}
255			exch2_neighbourId( exch2_opposingSend_record(a,T),
256			exch2_neighbourId(a,T) ) = T
257	afe	1.5	\end{verbatim}
258	edhill	1.8	and this provides a back-reference from the neighbor tiles.
259	afe	1.5
260	edhill	1.8	The arrays \varlink{exch2\_pi}{exch2_pi},
261			\varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi},
262			\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and
263			\varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in
264			exchanges between the neighboring tiles. The dimensions of
265			\texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the
266			neighbor ID \textit{N} and the tile number \textit{T} as explained
267			above, plus the transformation vector {\em t }, of length two. The
268			first element of the transformation vector indicates the factor by
269			which variables representing the same vector component of a tile will
270			be multiplied, and the second element indicates the transform to the
271			variable in the other direction. As an example,
272			\texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a
273			vector variable in tile \texttt{T} to the i-component of tile
274			\texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold
275			the component of neighbor \texttt{N}'s j-component.
276
277			Under the current cube topology, one of the two elements of
278			\texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T}
279			and neighbor \texttt{N} will be 0, reflecting the fact that the vector
280			components are orthogonal. The other element will be 1 or -1,
281			depending on whether the components are indexed in the same or
282			opposite directions. For example, the transform dimension of the
283			arrays for all tile neighbors on the same subdomain will be [1,0],
284			since all tiles on the same subdomain are oriented identically.
285			Vectors that correspond to the orthogonal dimension with the same
286			index direction will have [0,1], whereas those in the opposite index
287			direction will have [0,-1].
288	afe	1.5
289	afe	1.4
290	edhill	1.8	{\footnotesize
291	afe	1.4	\begin{verbatim}
292			C exch2_pi :: X index row of target to source permutation
293			C :: matrix for each neighbour entry.
294			C exch2_pj :: Y index row of target to source permutation
295			C :: matrix for each neighbour entry.
296			C exch2_oi :: X index element of target to source
297			C :: offset vector for cell-centered quantities
298			C :: of each neighbor entry.
299			C exch2_oj :: Y index element of target to source
300			C :: offset vector for cell-centered quantities
301			C :: of each neighbor entry.
302			C exch2_oi_f :: X index element of target to source
303			C :: offset vector for face quantities
304			C :: of each neighbor entry.
305			C exch2_oj_f :: Y index element of target to source
306			C :: offset vector for face quantities
307			C :: of each neighbor entry.
308			\end{verbatim}
309	edhill	1.8	}
310	afe	1.1
311
312
313			\subsection{Key Routines}
314
315
316
317			\subsection{References}