10 |
%% o automatically inserted at \section{Reference} |
%% o automatically inserted at \section{Reference} |
11 |
|
|
12 |
|
|
13 |
\section{exch2: Extended Cubed Sphere \mbox{Topology}} |
\subsection{exch2: Extended Cubed Sphere \mbox{Topology}} |
14 |
\label{sec:exch2} |
\label{sec:exch2} |
|
\label{sec:pkg:exch2} |
|
|
\begin{rawhtml} |
|
|
<!-- CMIREDIR:package_exch2: --> |
|
|
\end{rawhtml} |
|
15 |
|
|
16 |
|
|
17 |
\subsection{Introduction} |
\subsubsection{Introduction} |
18 |
|
|
19 |
The \texttt{exch2} package extends the original cubed sphere topology |
The \texttt{exch2} package extends the original cubed sphere topology |
20 |
configuration to allow more flexible domain decomposition and |
configuration to allow more flexible domain decomposition and |
23 |
dimensions of the subdomain. Furthermore, the tiles can run on |
dimensions of the subdomain. Furthermore, the tiles can run on |
24 |
separate processors individually or in groups, which provides for |
separate processors individually or in groups, which provides for |
25 |
manual compile-time load balancing across a relatively arbitrary |
manual compile-time load balancing across a relatively arbitrary |
26 |
number of processors. \\ |
number of processors. |
27 |
|
|
28 |
The exchange parameters are declared in |
The exchange parameters are declared in |
29 |
\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} |
\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} |
41 |
\file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h} |
\file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h} |
42 |
file for single-processor execution. |
file for single-processor execution. |
43 |
|
|
44 |
\subsection{Invoking exch2} |
\subsubsection{Invoking exch2} |
45 |
|
|
46 |
To use exch2 with the cubed sphere, the following conditions must be |
To use exch2 with the cubed sphere, the following conditions must be |
47 |
met: |
met: |
49 |
\begin{itemize} |
\begin{itemize} |
50 |
\item The exch2 package is included when \file{genmake2} is run. The |
\item The exch2 package is included when \file{genmake2} is run. The |
51 |
easiest way to do this is to add the line \code{exch2} to the |
easiest way to do this is to add the line \code{exch2} to the |
52 |
\file{profile.conf} file -- see Section \ref{sect:buildingCode} |
\file{packages.conf} file -- see Section \ref{sect:buildingCode} |
53 |
\sectiontitle{Building the code} for general details. |
\sectiontitle{Building the code} for general |
54 |
|
details. |
55 |
|
|
56 |
\item An example of \file{W2\_EXCH2\_TOPOLOGY.h} and |
\item An example of \file{W2\_EXCH2\_TOPOLOGY.h} and |
57 |
\file{w2\_e2setup.F} must reside in a directory containing files |
\file{w2\_e2setup.F} must reside in a directory containing files |
58 |
symbolically linked by the \file{genmake2} script. The safest place |
symbolically linked by the \file{genmake2} script. The safest place |
63 |
exist elsewhere in the build path, but they should be left untouched |
exist elsewhere in the build path, but they should be left untouched |
64 |
to avoid breaking configurations other than the one you intend to |
to avoid breaking configurations other than the one you intend to |
65 |
modify. |
modify. |
66 |
|
|
67 |
\item Files containing grid parameters, named \file{tile00$n$.mitgrid} |
\item Files containing grid parameters, named \file{tile00$n$.mitgrid} |
68 |
where $n$=\code{(1:6)} (one per subdomain), must be in the working |
where $n$=\code{(1:6)} (one per subdomain), must be in the working |
69 |
directory when the MITgcm executable is run. These files are |
directory when the MITgcm executable is run. These files are |
70 |
provided in the example experiments for cubed sphere configurations |
provided in the example experiments for cubed sphere configurations |
71 |
with 32$\times$32 cube sides -- please contact |
with 32$\times$32 cube sides -- please contact MITgcm support if you |
72 |
\begin{rawhtml} |
want to generate files for other configurations. |
73 |
<A href="mailto:mitgcm-support@dev.mitgcm.org"> |
|
|
\end{rawhtml} |
|
|
\begin{verbatim} |
|
|
MITgcm-support@mitgcm.org |
|
|
\end{verbatim} |
|
|
\begin{rawhtml} </A> \end{rawhtml} |
|
|
if you want to generate files for other configurations. |
|
|
|
|
74 |
\item As always when compiling MITgcm, the file \file{SIZE.h} must be |
\item As always when compiling MITgcm, the file \file{SIZE.h} must be |
75 |
placed where \file{genmake2} will find it. In particular for exch2, |
placed where \file{genmake2} will find it. In particular for exch2, |
76 |
the domain decomposition specified in \file{SIZE.h} must correspond |
the domain decomposition specified in \file{SIZE.h} must correspond |
81 |
and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and |
and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and |
82 |
Multiprocessing}; a more general background on the subject |
Multiprocessing}; a more general background on the subject |
83 |
relevant to MITgcm is presented in Section |
relevant to MITgcm is presented in Section |
84 |
\ref{sect:specifying_a_decomposition} \sectiontitle{Specifying a |
\ref{sect:specifying_a_decomposition} |
85 |
decomposition}. |
\sectiontitle{Specifying a decomposition}. |
86 |
\end{itemize} |
\end{itemize} |
87 |
|
|
|
|
|
|
|
|
88 |
At the time of this writing the following examples use exch2 and may |
At the time of this writing the following examples use exch2 and may |
89 |
be used for guidance: |
be used for guidance: |
90 |
|
|
99 |
|
|
100 |
|
|
101 |
|
|
102 |
\subsection{Generating Topology Files for exch2} |
\subsubsection{Generating Topology Files for exch2} |
103 |
\label{sec:topogen} |
\label{sec:topogen} |
104 |
|
|
105 |
Alternate cubed sphere topologies may be created using the Matlab |
Alternate cubed sphere topologies may be created using the Matlab |
109 |
from the Matlab prompt (there are no parameters to pass) generates |
from the Matlab prompt (there are no parameters to pass) generates |
110 |
exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and |
exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and |
111 |
\file{w2\_e2setup.F} in the working directory and displays a figure of |
\file{w2\_e2setup.F} in the working directory and displays a figure of |
112 |
the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:12tile}, |
the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:18tile}, |
113 |
and \ref{fig:24tile} are examples of the generated diagrams. The other |
and \ref{fig:48tile} are examples of the generated diagrams. The other |
114 |
m-files in the directory are |
m-files in the directory are |
115 |
subroutines called from \file{driver.m} and should not be run ``bare'' except |
subroutines called from \file{driver.m} and should not be run ``bare'' except |
116 |
for development purposes. \\ |
for development purposes. \\ |
131 |
the tiles into which the subdomains are decomposed, and must evenly |
the tiles into which the subdomains are decomposed, and must evenly |
132 |
divide the integer assigned to \code{nr}, \code{nb} and \code{ng}. |
divide the integer assigned to \code{nr}, \code{nb} and \code{ng}. |
133 |
The result is a rectangular tiling of the subdomain. Figure |
The result is a rectangular tiling of the subdomain. Figure |
134 |
\ref{fig:24tile} shows one possible topology for a twenty-four-tile |
\ref{fig:48tile} shows one possible topology for a twenty-four-tile |
135 |
cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\ |
cube, and figure \ref{fig:6tile} shows one for six tiles. \\ |
136 |
|
|
137 |
\begin{figure} |
\begin{figure} |
138 |
\begin{center} |
\begin{center} |
139 |
\resizebox{4in}{!}{ |
\resizebox{6in}{!}{ |
140 |
\includegraphics{part6/s24t_16x16.ps} |
% \includegraphics{s_phys_pkgs/figs/s24t_16x16.ps} |
141 |
|
\includegraphics{s_phys_pkgs/figs/adjust_cs.ps} |
142 |
} |
} |
143 |
\end{center} |
\end{center} |
144 |
|
|
145 |
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
146 |
divided into six 32$\times$32 subdomains, each of which is divided |
divided into six 32$\times$32 subdomains, each of which is divided |
147 |
into four tiles of width \code{tnx=16} and height \code{tny=16} for a |
into eight tiles of width \code{tnx=16} and height \code{tny=8} for a |
148 |
total of twenty-four tiles. The colored borders of the subdomains |
total of forty-eight tiles. The colored borders of the subdomains |
149 |
represent the parameters \code{nr} (red), \code{nb} (blue), and |
represent the parameters \code{nr} (red), \code{ng} (green), and |
150 |
\code{ng} (green). } \label{fig:24tile} |
\code{nb} (blue). |
151 |
|
This tiling is used in the example |
152 |
|
verification/adjustment.cs-32x32x1/ |
153 |
|
with the option (blanklist.txt) to remove the land-only 4 tiles |
154 |
|
(11,12,13,14) which are filled in red on the plot. |
155 |
|
} \label{fig:48tile} |
156 |
\end{figure} |
\end{figure} |
157 |
|
|
158 |
\begin{figure} |
\begin{figure} |
159 |
\begin{center} |
\begin{center} |
160 |
\resizebox{4in}{!}{ |
\resizebox{6in}{!}{ |
161 |
\includegraphics{part6/s12t_16x32.ps} |
% \includegraphics{s_phys_pkgs/figs/s12t_16x32.ps} |
162 |
|
\includegraphics{s_phys_pkgs/figs/polarcap.ps} |
163 |
} |
} |
164 |
\end{center} |
\end{center} |
165 |
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
\caption{Plot of a non-square cubed sphere topology with |
166 |
divided into six 32$\times$32 subdomains of two tiles each |
6 subdomains of different size (nr=90,ng=360,nb=90), |
167 |
(\code{tnx=16, tny=32}). |
divided into one to four tiles each |
168 |
} \label{fig:12tile} |
(\code{tnx=90, tny=90}), resulting in a total of 18 tiles. |
169 |
|
} \label{fig:18tile} |
170 |
\end{figure} |
\end{figure} |
171 |
|
|
172 |
\begin{figure} |
\begin{figure} |
173 |
\begin{center} |
\begin{center} |
174 |
\resizebox{4in}{!}{ |
\resizebox{4in}{!}{ |
175 |
\includegraphics{part6/s6t_32x32.ps} |
% \includegraphics{s_phys_pkgs/figs/s6t_32x32.ps} |
176 |
|
\includegraphics{s_phys_pkgs/figs/s6t_32x32.ps} |
177 |
} |
} |
178 |
\end{center} |
\end{center} |
179 |
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
194 |
|
|
195 |
|
|
196 |
|
|
197 |
\subsection{exch2, SIZE.h, and Multiprocessing} |
\subsubsection{exch2, SIZE.h, and Multiprocessing} |
198 |
\label{sec:exch2mpi} |
\label{sec:exch2mpi} |
199 |
|
|
200 |
Once the topology configuration files are created, the Fortran |
Once the topology configuration files are created, the Fortran |
201 |
\code{PARAMETER}s in \file{SIZE.h} must be configured to match. |
\code{PARAMETER}s in \file{SIZE.h} must be configured to match. |
202 |
Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying |
Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying |
203 |
a decomposition} provides a general description of domain |
a decomposition} provides a general description of domain |
204 |
decomposition within MITgcm and its relation to \file{SIZE.h}. The |
decomposition within MITgcm and its relation to \file{SIZE.h}. The |
205 |
current section specifies constraints that the exch2 package |
current section specifies constraints that the exch2 package imposes |
206 |
imposes and describes how to enable parallel execution with |
and describes how to enable parallel execution with MPI. |
|
MPI. \\ |
|
207 |
|
|
208 |
As in the general case, the parameters \varlink{sNx}{sNx} and |
As in the general case, the parameters \varlink{sNx}{sNx} and |
209 |
\varlink{sNy}{sNy} define the size of the individual tiles, and so |
\varlink{sNy}{sNy} define the size of the individual tiles, and so |
210 |
must be assigned the same respective values as \code{tnx} and |
must be assigned the same respective values as \code{tnx} and |
211 |
\code{tny} in \file{driver.m}.\\ |
\code{tny} in \file{driver.m}. |
212 |
|
|
213 |
The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy} |
The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy} |
214 |
have no special bearing on exch2 and may be assigned as in the general |
have no special bearing on exch2 and may be assigned as in the general |
215 |
case. The same holds for \varlink{Nr}{Nr}, the number of vertical |
case. The same holds for \varlink{Nr}{Nr}, the number of vertical |
216 |
levels in the model.\\ |
levels in the model. |
217 |
|
|
218 |
The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy}, |
The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy}, |
219 |
\varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of |
\varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of |
221 |
the tiles are stored in the $x$ dimension, and so |
the tiles are stored in the $x$ dimension, and so |
222 |
\code{\varlink{nSy}{nSy}=1} in all cases. Since the tiles as |
\code{\varlink{nSy}{nSy}=1} in all cases. Since the tiles as |
223 |
configured by exch2 cannot be split up accross processors without |
configured by exch2 cannot be split up accross processors without |
224 |
regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\ |
regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. |
225 |
|
|
226 |
The number of tiles MITgcm allocates and how they are distributed |
The number of tiles MITgcm allocates and how they are distributed |
227 |
between processors depends on \varlink{nPx}{nPx} and |
between processors depends on \varlink{nPx}{nPx} and |
228 |
\varlink{nSx}{nSx}. \varlink{nSx}{nSx} is the number of tiles per |
\varlink{nSx}{nSx}. \varlink{nSx}{nSx} is the number of tiles per |
229 |
processor and \varlink{nPx}{nPx} is the number of processors. The total |
processor and \varlink{nPx}{nPx} is the number of processors. The |
230 |
number of tiles in the topology minus those listed in |
total number of tiles in the topology minus those listed in |
231 |
\file{blanklist.txt} must equal \code{nSx*nPx}. Note that in order to |
\file{blanklist.txt} must equal \code{nSx*nPx}. Note that in order to |
232 |
obtain maximum usage from a given number of processors in some cases, |
obtain maximum usage from a given number of processors in some cases, |
233 |
this restriction might entail sharing a processor with a tile that would |
this restriction might entail sharing a processor with a tile that |
234 |
otherwise be excluded. \\ |
would otherwise be excluded because it is topographically outside of |
235 |
|
the domain and therefore in \file{blanklist.txt}. For example, |
236 |
The following is an example of \file{SIZE.h} for the twelve-tile |
suppose you have five processors and a domain decomposition of |
237 |
configuration illustrated in figure \ref{fig:12tile} running on |
thirty-six tiles that allows you to exclude seven tiles. To evenly |
238 |
one processor: \\ |
distribute the remaining twenty-nine tiles among five processors, you |
239 |
|
would have to run one ``dummy'' tile to make an even six tiles per |
240 |
|
processor. Such dummy tiles are \emph{not} listed in |
241 |
|
\file{blanklist.txt}. |
242 |
|
|
243 |
|
The following is an example of \file{SIZE.h} for the six-tile |
244 |
|
configuration illustrated in figure \ref{fig:6tile} |
245 |
|
running on one processor: |
246 |
|
|
247 |
\begin{verbatim} |
\begin{verbatim} |
248 |
PARAMETER ( |
PARAMETER ( |
249 |
& sNx = 16, |
& sNx = 32, |
250 |
& sNy = 32, |
& sNy = 32, |
251 |
& OLx = 2, |
& OLx = 2, |
252 |
& OLy = 2, |
& OLy = 2, |
253 |
& nSx = 12, |
& nSx = 6, |
254 |
& nSy = 1, |
& nSy = 1, |
255 |
& nPx = 1, |
& nPx = 1, |
256 |
& nPy = 1, |
& nPy = 1, |
259 |
& Nr = 5) |
& Nr = 5) |
260 |
\end{verbatim} |
\end{verbatim} |
261 |
|
|
262 |
The following is an example for the twenty-four-tile topology in |
The following is an example for the forty-eight-tile topology in |
263 |
figure \ref{fig:24tile} running on six processors: |
figure \ref{fig:48tile} running on six processors: |
264 |
|
|
265 |
\begin{verbatim} |
\begin{verbatim} |
266 |
PARAMETER ( |
PARAMETER ( |
267 |
& sNx = 16, |
& sNx = 16, |
268 |
& sNy = 16, |
& sNy = 8, |
269 |
& OLx = 2, |
& OLx = 2, |
270 |
& OLy = 2, |
& OLy = 2, |
271 |
& nSx = 4, |
& nSx = 8, |
272 |
& nSy = 1, |
& nSy = 1, |
273 |
& nPx = 6, |
& nPx = 6, |
274 |
& nPy = 1, |
& nPy = 1, |
278 |
\end{verbatim} |
\end{verbatim} |
279 |
|
|
280 |
|
|
281 |
|
\subsubsection{Key Variables} |
|
|
|
|
|
|
|
\subsection{Key Variables} |
|
282 |
|
|
283 |
The descriptions of the variables are divided up into scalars, |
The descriptions of the variables are divided up into scalars, |
284 |
one-dimensional arrays indexed to the tile number, and two and |
one-dimensional arrays indexed to the tile number, and two and |
288 |
arrays to individual tiles, and the arrays indexed by tile and |
arrays to individual tiles, and the arrays indexed by tile and |
289 |
neighbor to relationships between tiles and their neighbors. \\ |
neighbor to relationships between tiles and their neighbors. \\ |
290 |
|
|
291 |
\subsubsection{Scalars} |
Scalars: |
292 |
|
|
293 |
The number of tiles in a particular topology is set with the parameter |
The number of tiles in a particular topology is set with the parameter |
294 |
\code{NTILES}, and the maximum number of neighbors of any tiles by |
\code{NTILES}, and the maximum number of neighbors of any tiles by |
302 |
of tiles in the $x$ and $y$ global indices. For example, the default |
of tiles in the $x$ and $y$ global indices. For example, the default |
303 |
setup of six tiles (Fig. \ref{fig:6tile}) has |
setup of six tiles (Fig. \ref{fig:6tile}) has |
304 |
\code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}. A |
\code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}. A |
305 |
topology of twenty-four square tiles, four per subdomain (as in figure |
topology of forty-eight tiles, eight per subdomain (as in figure |
306 |
\ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and |
\ref{fig:48tile}), will have \code{exch2\_domain\_nxt=12} and |
307 |
\code{exch2\_domain\_nyt=2}. Note that these parameters express the |
\code{exch2\_domain\_nyt=4}. Note that these parameters express the |
308 |
tile layout in order to allow global data files that are tile-layout-neutral. |
tile layout in order to allow global data files that are tile-layout-neutral. |
309 |
They have no bearing on the internal storage of the arrays. The tiles |
They have no bearing on the internal storage of the arrays. The tiles |
310 |
are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the |
are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the |
311 |
$x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to |
$x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to |
312 |
equal \code{1} throughout the package. \\ |
equal \code{1} throughout the package. \\ |
313 |
|
|
314 |
\subsubsection{Arrays indexed to tile number} |
Arrays indexed to tile number: |
315 |
|
|
316 |
The following arrays are of length \code{NTILES} and are indexed to |
The following arrays are of length \code{NTILES} and are indexed to |
317 |
the tile number, which is indicated in the diagrams with the notation |
the tile number, which is indicated in the diagrams with the notation |
318 |
\code{tn}. The indices are omitted in the descriptions. \\ |
\textsf{t}$n$. The indices are omitted in the descriptions. \\ |
319 |
|
|
320 |
The arrays \varlink{exch2\_tnx}{exch2_tnx} and |
The arrays \varlink{exch2\_tnx}{exch2_tnx} and |
321 |
\varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of |
\varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of |
343 |
|
|
344 |
The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of |
The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of |
345 |
the subdomain of each tile, in a range \code{(1:6)} in the case of the |
the subdomain of each tile, in a range \code{(1:6)} in the case of the |
346 |
standard cube topology and indicated by \textbf{\textsf{fn}} in |
standard cube topology and indicated by \textbf{\textsf{f}}$n$ in |
347 |
figures \ref{fig:12tile} and \ref{fig:24tile}. The |
figures \ref{fig:6tile} and |
348 |
\varlink{exch2\_nNeighbours}{exch2_nNeighbours} variable contains a |
\ref{fig:48tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours} |
349 |
count of the neighboring tiles each tile has, and sets the bounds for |
contains a count of the neighboring tiles each tile has, and sets |
350 |
looping over neighboring tiles. And |
the bounds for looping over neighboring tiles. |
351 |
\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each |
\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each |
352 |
tile, and is used in interprocess communication. \\ |
tile, and is used in interprocess communication. \\ |
353 |
|
|
364 |
corners of the cube. \\ |
corners of the cube. \\ |
365 |
|
|
366 |
|
|
367 |
\subsubsection{Arrays Indexed to Tile Number and Neighbor} |
Arrays Indexed to Tile Number and Neighbor: |
368 |
|
|
369 |
The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and |
The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and |
370 |
\code{NTILES} and describe the orientations between the the tiles. \\ |
\code{NTILES} and describe the orientations between the the tiles. \\ |
467 |
\varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index |
\varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index |
468 |
bounds of the edge segment of the neighbor tile \code{N}'s subdomain |
bounds of the edge segment of the neighbor tile \code{N}'s subdomain |
469 |
that gets exchanged with the local tile \code{T}. To take the example |
that gets exchanged with the local tile \code{T}. To take the example |
470 |
of tile \code{T=2} in the twelve-tile topology |
of tile \code{T=2} in the forty-eight-tile topology |
471 |
(Fig. \ref{fig:12tile}): \\ |
(Fig. \ref{fig:48tile}): \\ |
472 |
|
|
473 |
\begin{verbatim} |
\begin{verbatim} |
474 |
exch2_itlo_c(4,2)=17 |
exch2_itlo_c(4,2)=17 |
508 |
width of \code{T}'s northern edge, expanded by one into the halo. \\ |
width of \code{T}'s northern edge, expanded by one into the halo. \\ |
509 |
|
|
510 |
|
|
511 |
\subsection{Key Routines} |
\subsubsection{Key Routines} |
512 |
|
|
513 |
Most of the subroutines particular to exch2 handle the exchanges |
Most of the subroutines particular to exch2 handle the exchanges |
514 |
themselves and are of the same format as those described in |
themselves and are of the same format as those described in |
538 |
\code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of |
\code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of |
539 |
the work using the variables discussed above. \\ |
the work using the variables discussed above. \\ |
540 |
|
|
541 |
|
\subsubsection{Experiments and tutorials that use exch2} |
542 |
|
\label{sec:pkg:exch2:experiments} |
543 |
|
|
544 |
|
\begin{itemize} |
545 |
|
\item{Held Suarez tutorial, in tutorial\_held\_suarez\_cs verification directory, |
546 |
|
described in section \ref{sect:eg-hs} } |
547 |
|
\end{itemize} |