1 |
jmc |
1.29 |
% $Header: /u/gcmpack/manual/s_phys_pkgs/text/exch2.tex,v 1.28 2010/08/27 13:15:37 jmc Exp $ |
2 |
afe |
1.1 |
% $Name: $ |
3 |
|
|
|
4 |
|
|
%% * Introduction |
5 |
|
|
%% o what it does, citations (refs go into mitgcm_manual.bib, |
6 |
|
|
%% preferably in alphabetic order) |
7 |
|
|
%% o Equations |
8 |
|
|
%% * Key subroutines and parameters |
9 |
|
|
%% * Reference material (auto generated from Protex and structured comments) |
10 |
|
|
%% o automatically inserted at \section{Reference} |
11 |
|
|
|
12 |
|
|
|
13 |
molod |
1.24 |
\subsection{exch2: Extended Cubed Sphere \mbox{Topology}} |
14 |
afe |
1.3 |
\label{sec:exch2} |
15 |
|
|
|
16 |
afe |
1.1 |
|
17 |
molod |
1.24 |
\subsubsection{Introduction} |
18 |
afe |
1.2 |
|
19 |
afe |
1.17 |
The \texttt{exch2} package extends the original cubed sphere topology |
20 |
|
|
configuration to allow more flexible domain decomposition and |
21 |
|
|
parallelization. Cube faces (also called subdomains) may be divided |
22 |
|
|
into any number of tiles that divide evenly into the grid point |
23 |
|
|
dimensions of the subdomain. Furthermore, the tiles can run on |
24 |
|
|
separate processors individually or in groups, which provides for |
25 |
|
|
manual compile-time load balancing across a relatively arbitrary |
26 |
edhill |
1.25 |
number of processors. |
27 |
edhill |
1.8 |
|
28 |
|
|
The exchange parameters are declared in |
29 |
|
|
\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} |
30 |
|
|
and assigned in |
31 |
afe |
1.9 |
\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The |
32 |
afe |
1.11 |
validity of the cube topology depends on the \file{SIZE.h} file as |
33 |
afe |
1.12 |
detailed below. The default files provided in the release configure a |
34 |
|
|
cubed sphere topology of six tiles, one per subdomain, each with |
35 |
afe |
1.18 |
32$\times$32 grid points, with all tiles running on a single processor. Both |
36 |
afe |
1.12 |
files are generated by Matlab scripts in |
37 |
afe |
1.11 |
\file{utils/exch2/matlab-topology-generator}; see Section |
38 |
|
|
\ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2} |
39 |
afe |
1.12 |
for details on creating alternate topologies. Pregenerated examples |
40 |
|
|
of these files with alternate topologies are provided under |
41 |
afe |
1.11 |
\file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h} |
42 |
|
|
file for single-processor execution. |
43 |
afe |
1.9 |
|
44 |
molod |
1.24 |
\subsubsection{Invoking exch2} |
45 |
afe |
1.9 |
|
46 |
afe |
1.10 |
To use exch2 with the cubed sphere, the following conditions must be |
47 |
edhill |
1.25 |
met: |
48 |
afe |
1.9 |
|
49 |
edhill |
1.25 |
\begin{itemize} |
50 |
|
|
\item The exch2 package is included when \file{genmake2} is run. The |
51 |
|
|
easiest way to do this is to add the line \code{exch2} to the |
52 |
jmc |
1.29 |
\file{packages.conf} file -- see Section \ref{sec:buildingCode} |
53 |
edhill |
1.25 |
\sectiontitle{Building the code} for general |
54 |
|
|
details. |
55 |
afe |
1.23 |
|
56 |
edhill |
1.25 |
\item An example of \file{W2\_EXCH2\_TOPOLOGY.h} and |
57 |
afe |
1.17 |
\file{w2\_e2setup.F} must reside in a directory containing files |
58 |
edhill |
1.25 |
symbolically linked by the \file{genmake2} script. The safest place |
59 |
|
|
to put these is the directory indicated in the \code{-mods=DIR} |
60 |
|
|
command line modifier (typically \file{../code}), or the build |
61 |
|
|
directory. The default versions of these files reside in |
62 |
|
|
\file{pkg/exch2} and are linked automatically if no other versions |
63 |
|
|
exist elsewhere in the build path, but they should be left untouched |
64 |
|
|
to avoid breaking configurations other than the one you intend to |
65 |
|
|
modify. |
66 |
|
|
|
67 |
|
|
\item Files containing grid parameters, named \file{tile00$n$.mitgrid} |
68 |
|
|
where $n$=\code{(1:6)} (one per subdomain), must be in the working |
69 |
|
|
directory when the MITgcm executable is run. These files are |
70 |
|
|
provided in the example experiments for cubed sphere configurations |
71 |
|
|
with 32$\times$32 cube sides -- please contact MITgcm support if you |
72 |
|
|
want to generate files for other configurations. |
73 |
|
|
|
74 |
|
|
\item As always when compiling MITgcm, the file \file{SIZE.h} must be |
75 |
|
|
placed where \file{genmake2} will find it. In particular for exch2, |
76 |
|
|
the domain decomposition specified in \file{SIZE.h} must correspond |
77 |
|
|
with the particular configuration's topology specified in |
78 |
afe |
1.12 |
\file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}. Domain |
79 |
|
|
decomposition issues particular to exch2 are addressed in Section |
80 |
|
|
\ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2} |
81 |
edhill |
1.25 |
and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and |
82 |
|
|
Multiprocessing}; a more general background on the subject |
83 |
|
|
relevant to MITgcm is presented in Section |
84 |
jmc |
1.29 |
\ref{sec:specifying_a_decomposition} |
85 |
edhill |
1.25 |
\sectiontitle{Specifying a decomposition}. |
86 |
|
|
\end{itemize} |
87 |
afe |
1.9 |
|
88 |
afe |
1.17 |
At the time of this writing the following examples use exch2 and may |
89 |
|
|
be used for guidance: |
90 |
afe |
1.9 |
|
91 |
|
|
\begin{verbatim} |
92 |
|
|
verification/adjust_nlfs.cs-32x32x1 |
93 |
|
|
verification/adjustment.cs-32x32x1 |
94 |
|
|
verification/aim.5l_cs |
95 |
|
|
verification/global_ocean.cs32x15 |
96 |
|
|
verification/hs94.cs-32x32x5 |
97 |
|
|
\end{verbatim} |
98 |
|
|
|
99 |
|
|
|
100 |
|
|
|
101 |
|
|
|
102 |
molod |
1.24 |
\subsubsection{Generating Topology Files for exch2} |
103 |
afe |
1.10 |
\label{sec:topogen} |
104 |
|
|
|
105 |
|
|
Alternate cubed sphere topologies may be created using the Matlab |
106 |
afe |
1.11 |
scripts in \file{utils/exch2/matlab-topology-generator}. Running the |
107 |
afe |
1.12 |
m-file |
108 |
|
|
\filelink{driver.m}{utils-exch2-matlab-topology-generator_driver.m} |
109 |
|
|
from the Matlab prompt (there are no parameters to pass) generates |
110 |
|
|
exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and |
111 |
|
|
\file{w2\_e2setup.F} in the working directory and displays a figure of |
112 |
jmc |
1.27 |
the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:18tile}, |
113 |
|
|
and \ref{fig:48tile} are examples of the generated diagrams. The other |
114 |
afe |
1.19 |
m-files in the directory are |
115 |
|
|
subroutines called from \file{driver.m} and should not be run ``bare'' except |
116 |
afe |
1.12 |
for development purposes. \\ |
117 |
afe |
1.10 |
|
118 |
|
|
The parameters that determine the dimensions and topology of the |
119 |
afe |
1.11 |
generated configuration are \code{nr}, \code{nb}, \code{ng}, |
120 |
afe |
1.12 |
\code{tnx} and \code{tny}, and all are assigned early in the script. \\ |
121 |
afe |
1.10 |
|
122 |
afe |
1.19 |
The first three determine the height and width of the subdomains and |
123 |
afe |
1.10 |
hence the size of the overall domain. Each one determines the number |
124 |
|
|
of grid points, and therefore the resolution, along the subdomain |
125 |
afe |
1.18 |
sides in a ``great circle'' around each the three spatial axes of the cube. At the time |
126 |
afe |
1.10 |
of this writing MITgcm requires these three parameters to be equal, |
127 |
afe |
1.12 |
but they provide for future releases to accomodate different |
128 |
afe |
1.19 |
resolutions around the axes to allow subdomains with differing resolutions.\\ |
129 |
afe |
1.10 |
|
130 |
afe |
1.18 |
The parameters \code{tnx} and \code{tny} determine the width and height of |
131 |
afe |
1.11 |
the tiles into which the subdomains are decomposed, and must evenly |
132 |
|
|
divide the integer assigned to \code{nr}, \code{nb} and \code{ng}. |
133 |
|
|
The result is a rectangular tiling of the subdomain. Figure |
134 |
jmc |
1.27 |
\ref{fig:48tile} shows one possible topology for a twenty-four-tile |
135 |
|
|
cube, and figure \ref{fig:6tile} shows one for six tiles. \\ |
136 |
afe |
1.10 |
|
137 |
|
|
\begin{figure} |
138 |
|
|
\begin{center} |
139 |
jmc |
1.27 |
\resizebox{6in}{!}{ |
140 |
jmc |
1.28 |
% \includegraphics{s_phys_pkgs/figs/s24t_16x16.ps} |
141 |
|
|
\includegraphics{s_phys_pkgs/figs/adjust_cs.ps} |
142 |
afe |
1.10 |
} |
143 |
|
|
\end{center} |
144 |
afe |
1.12 |
|
145 |
afe |
1.13 |
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
146 |
afe |
1.17 |
divided into six 32$\times$32 subdomains, each of which is divided |
147 |
jmc |
1.27 |
into eight tiles of width \code{tnx=16} and height \code{tny=8} for a |
148 |
|
|
total of forty-eight tiles. The colored borders of the subdomains |
149 |
|
|
represent the parameters \code{nr} (red), \code{ng} (green), and |
150 |
|
|
\code{nb} (blue). |
151 |
|
|
This tiling is used in the example |
152 |
|
|
verification/adjustment.cs-32x32x1/ |
153 |
|
|
with the option (blanklist.txt) to remove the land-only 4 tiles |
154 |
|
|
(11,12,13,14) which are filled in red on the plot. |
155 |
|
|
} \label{fig:48tile} |
156 |
afe |
1.10 |
\end{figure} |
157 |
|
|
|
158 |
|
|
\begin{figure} |
159 |
|
|
\begin{center} |
160 |
jmc |
1.27 |
\resizebox{6in}{!}{ |
161 |
jmc |
1.28 |
% \includegraphics{s_phys_pkgs/figs/s12t_16x32.ps} |
162 |
|
|
\includegraphics{s_phys_pkgs/figs/polarcap.ps} |
163 |
afe |
1.10 |
} |
164 |
|
|
\end{center} |
165 |
jmc |
1.27 |
\caption{Plot of a non-square cubed sphere topology with |
166 |
|
|
6 subdomains of different size (nr=90,ng=360,nb=90), |
167 |
|
|
divided into one to four tiles each |
168 |
|
|
(\code{tnx=90, tny=90}), resulting in a total of 18 tiles. |
169 |
|
|
} \label{fig:18tile} |
170 |
afe |
1.10 |
\end{figure} |
171 |
|
|
|
172 |
afe |
1.13 |
\begin{figure} |
173 |
|
|
\begin{center} |
174 |
|
|
\resizebox{4in}{!}{ |
175 |
jmc |
1.28 |
% \includegraphics{s_phys_pkgs/figs/s6t_32x32.ps} |
176 |
|
|
\includegraphics{s_phys_pkgs/figs/s6t_32x32.ps} |
177 |
afe |
1.13 |
} |
178 |
|
|
\end{center} |
179 |
|
|
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
180 |
|
|
divided into six 32$\times$32 subdomains with one tile each |
181 |
|
|
(\code{tnx=32, tny=32}). This is the default configuration. |
182 |
|
|
} |
183 |
|
|
\label{fig:6tile} |
184 |
|
|
\end{figure} |
185 |
|
|
|
186 |
|
|
|
187 |
afe |
1.10 |
Tiles can be selected from the topology to be omitted from being |
188 |
afe |
1.12 |
allocated memory and processors. This tuning is useful in ocean |
189 |
|
|
modeling for omitting tiles that fall entirely on land. The tiles |
190 |
|
|
omitted are specified in the file |
191 |
|
|
\filelink{blanklist.txt}{utils-exch2-matlab-topology-generator_blanklist.txt} |
192 |
|
|
by their tile number in the topology, separated by a newline. \\ |
193 |
|
|
|
194 |
afe |
1.10 |
|
195 |
|
|
|
196 |
|
|
|
197 |
molod |
1.24 |
\subsubsection{exch2, SIZE.h, and Multiprocessing} |
198 |
afe |
1.12 |
\label{sec:exch2mpi} |
199 |
|
|
|
200 |
|
|
Once the topology configuration files are created, the Fortran |
201 |
afe |
1.13 |
\code{PARAMETER}s in \file{SIZE.h} must be configured to match. |
202 |
jmc |
1.29 |
Section \ref{sec:specifying_a_decomposition} \sectiontitle{Specifying |
203 |
edhill |
1.25 |
a decomposition} provides a general description of domain |
204 |
afe |
1.13 |
decomposition within MITgcm and its relation to \file{SIZE.h}. The |
205 |
edhill |
1.25 |
current section specifies constraints that the exch2 package imposes |
206 |
|
|
and describes how to enable parallel execution with MPI. |
207 |
afe |
1.12 |
|
208 |
|
|
As in the general case, the parameters \varlink{sNx}{sNx} and |
209 |
|
|
\varlink{sNy}{sNy} define the size of the individual tiles, and so |
210 |
|
|
must be assigned the same respective values as \code{tnx} and |
211 |
edhill |
1.25 |
\code{tny} in \file{driver.m}. |
212 |
afe |
1.12 |
|
213 |
|
|
The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy} |
214 |
|
|
have no special bearing on exch2 and may be assigned as in the general |
215 |
edhill |
1.25 |
case. The same holds for \varlink{Nr}{Nr}, the number of vertical |
216 |
|
|
levels in the model. |
217 |
afe |
1.12 |
|
218 |
|
|
The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy}, |
219 |
|
|
\varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of |
220 |
|
|
tiles and how they are distributed on processors. When using exch2, |
221 |
afe |
1.19 |
the tiles are stored in the $x$ dimension, and so |
222 |
afe |
1.12 |
\code{\varlink{nSy}{nSy}=1} in all cases. Since the tiles as |
223 |
|
|
configured by exch2 cannot be split up accross processors without |
224 |
edhill |
1.25 |
regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. |
225 |
afe |
1.12 |
|
226 |
|
|
The number of tiles MITgcm allocates and how they are distributed |
227 |
|
|
between processors depends on \varlink{nPx}{nPx} and |
228 |
|
|
\varlink{nSx}{nSx}. \varlink{nSx}{nSx} is the number of tiles per |
229 |
afe |
1.23 |
processor and \varlink{nPx}{nPx} is the number of processors. The |
230 |
|
|
total number of tiles in the topology minus those listed in |
231 |
|
|
\file{blanklist.txt} must equal \code{nSx*nPx}. Note that in order to |
232 |
afe |
1.19 |
obtain maximum usage from a given number of processors in some cases, |
233 |
afe |
1.23 |
this restriction might entail sharing a processor with a tile that |
234 |
|
|
would otherwise be excluded because it is topographically outside of |
235 |
|
|
the domain and therefore in \file{blanklist.txt}. For example, |
236 |
|
|
suppose you have five processors and a domain decomposition of |
237 |
|
|
thirty-six tiles that allows you to exclude seven tiles. To evenly |
238 |
|
|
distribute the remaining twenty-nine tiles among five processors, you |
239 |
|
|
would have to run one ``dummy'' tile to make an even six tiles per |
240 |
|
|
processor. Such dummy tiles are \emph{not} listed in |
241 |
edhill |
1.25 |
\file{blanklist.txt}. |
242 |
afe |
1.12 |
|
243 |
jmc |
1.27 |
The following is an example of \file{SIZE.h} for the six-tile |
244 |
|
|
configuration illustrated in figure \ref{fig:6tile} |
245 |
|
|
running on one processor: |
246 |
afe |
1.12 |
|
247 |
|
|
\begin{verbatim} |
248 |
|
|
PARAMETER ( |
249 |
jmc |
1.27 |
& sNx = 32, |
250 |
afe |
1.12 |
& sNy = 32, |
251 |
|
|
& OLx = 2, |
252 |
|
|
& OLy = 2, |
253 |
jmc |
1.27 |
& nSx = 6, |
254 |
afe |
1.12 |
& nSy = 1, |
255 |
|
|
& nPx = 1, |
256 |
|
|
& nPy = 1, |
257 |
|
|
& Nx = sNx*nSx*nPx, |
258 |
|
|
& Ny = sNy*nSy*nPy, |
259 |
|
|
& Nr = 5) |
260 |
|
|
\end{verbatim} |
261 |
|
|
|
262 |
jmc |
1.27 |
The following is an example for the forty-eight-tile topology in |
263 |
|
|
figure \ref{fig:48tile} running on six processors: |
264 |
afe |
1.12 |
|
265 |
|
|
\begin{verbatim} |
266 |
|
|
PARAMETER ( |
267 |
|
|
& sNx = 16, |
268 |
jmc |
1.27 |
& sNy = 8, |
269 |
afe |
1.12 |
& OLx = 2, |
270 |
|
|
& OLy = 2, |
271 |
jmc |
1.27 |
& nSx = 8, |
272 |
afe |
1.12 |
& nSy = 1, |
273 |
|
|
& nPx = 6, |
274 |
|
|
& nPy = 1, |
275 |
|
|
& Nx = sNx*nSx*nPx, |
276 |
|
|
& Ny = sNy*nSy*nPy, |
277 |
|
|
& Nr = 5) |
278 |
|
|
\end{verbatim} |
279 |
|
|
|
280 |
|
|
|
281 |
molod |
1.24 |
\subsubsection{Key Variables} |
282 |
afe |
1.4 |
|
283 |
|
|
The descriptions of the variables are divided up into scalars, |
284 |
afe |
1.17 |
one-dimensional arrays indexed to the tile number, and two and |
285 |
|
|
three-dimensional arrays indexed to tile number and neighboring tile. |
286 |
|
|
This division reflects the functionality of these variables: The |
287 |
edhill |
1.8 |
scalars are common to every part of the topology, the tile-indexed |
288 |
afe |
1.12 |
arrays to individual tiles, and the arrays indexed by tile and |
289 |
|
|
neighbor to relationships between tiles and their neighbors. \\ |
290 |
afe |
1.4 |
|
291 |
molod |
1.24 |
Scalars: |
292 |
afe |
1.4 |
|
293 |
|
|
The number of tiles in a particular topology is set with the parameter |
294 |
afe |
1.12 |
\code{NTILES}, and the maximum number of neighbors of any tiles by |
295 |
|
|
\code{MAX\_NEIGHBOURS}. These parameters are used for defining the |
296 |
edhill |
1.8 |
size of the various one and two dimensional arrays that store tile |
297 |
afe |
1.12 |
parameters indexed to the tile number and are assigned in the files |
298 |
|
|
generated by \file{driver.m}.\\ |
299 |
edhill |
1.8 |
|
300 |
|
|
The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt} |
301 |
|
|
and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number |
302 |
afe |
1.12 |
of tiles in the $x$ and $y$ global indices. For example, the default |
303 |
afe |
1.15 |
setup of six tiles (Fig. \ref{fig:6tile}) has |
304 |
|
|
\code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}. A |
305 |
jmc |
1.27 |
topology of forty-eight tiles, eight per subdomain (as in figure |
306 |
|
|
\ref{fig:48tile}), will have \code{exch2\_domain\_nxt=12} and |
307 |
|
|
\code{exch2\_domain\_nyt=4}. Note that these parameters express the |
308 |
afe |
1.19 |
tile layout in order to allow global data files that are tile-layout-neutral. |
309 |
|
|
They have no bearing on the internal storage of the arrays. The tiles |
310 |
|
|
are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the |
311 |
afe |
1.18 |
$x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to |
312 |
|
|
equal \code{1} throughout the package. \\ |
313 |
afe |
1.4 |
|
314 |
molod |
1.24 |
Arrays indexed to tile number: |
315 |
afe |
1.4 |
|
316 |
afe |
1.17 |
The following arrays are of length \code{NTILES} and are indexed to |
317 |
|
|
the tile number, which is indicated in the diagrams with the notation |
318 |
afe |
1.23 |
\textsf{t}$n$. The indices are omitted in the descriptions. \\ |
319 |
afe |
1.4 |
|
320 |
edhill |
1.8 |
The arrays \varlink{exch2\_tnx}{exch2_tnx} and |
321 |
afe |
1.12 |
\varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of |
322 |
|
|
each tile. At present for each tile \texttt{exch2\_tnx=sNx} and |
323 |
|
|
\texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in |
324 |
afe |
1.19 |
Section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and |
325 |
|
|
Multiprocessing}. Future releases of MITgcm may allow varying tile |
326 |
afe |
1.12 |
sizes. \\ |
327 |
edhill |
1.8 |
|
328 |
afe |
1.19 |
The arrays \varlink{exch2\_tbasex}{exch2_tbasex} and |
329 |
|
|
\varlink{exch2\_tbasey}{exch2_tbasey} determine the tiles' |
330 |
|
|
Cartesian origin within a subdomain |
331 |
|
|
and locate the edges of different tiles relative to each other. As |
332 |
afe |
1.13 |
an example, in the default six-tile topology (Fig. \ref{fig:6tile}) |
333 |
|
|
each index in these arrays is set to \code{0} since a tile occupies |
334 |
afe |
1.17 |
its entire subdomain. The twenty-four-tile case discussed above will |
335 |
afe |
1.19 |
have values of \code{0} or \code{16}, depending on the quadrant of the |
336 |
|
|
tile within the subdomain. The elements of the arrays |
337 |
afe |
1.13 |
\varlink{exch2\_txglobalo}{exch2_txglobalo} and |
338 |
|
|
\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to |
339 |
edhill |
1.8 |
\varlink{exch2\_tbasex}{exch2_tbasex} and |
340 |
afe |
1.19 |
\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tile edges within the |
341 |
afe |
1.17 |
global address space, similar to that used by global output and input |
342 |
|
|
files. \\ |
343 |
edhill |
1.8 |
|
344 |
afe |
1.13 |
The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of |
345 |
|
|
the subdomain of each tile, in a range \code{(1:6)} in the case of the |
346 |
afe |
1.23 |
standard cube topology and indicated by \textbf{\textsf{f}}$n$ in |
347 |
jmc |
1.27 |
figures \ref{fig:6tile} and |
348 |
|
|
\ref{fig:48tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours} |
349 |
afe |
1.23 |
contains a count of the neighboring tiles each tile has, and sets |
350 |
|
|
the bounds for looping over neighboring tiles. |
351 |
afe |
1.13 |
\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each |
352 |
|
|
tile, and is used in interprocess communication. \\ |
353 |
|
|
|
354 |
|
|
|
355 |
edhill |
1.8 |
The arrays \varlink{exch2\_isWedge}{exch2_isWedge}, |
356 |
|
|
\varlink{exch2\_isEedge}{exch2_isEedge}, |
357 |
|
|
\varlink{exch2\_isSedge}{exch2_isSedge}, and |
358 |
afe |
1.12 |
\varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the |
359 |
afe |
1.19 |
indexed tile lies on the edge of its subdomain, \code{0} if |
360 |
afe |
1.15 |
not. The values are used within the topology generator to determine |
361 |
|
|
the orientation of neighboring tiles, and to indicate whether a tile |
362 |
|
|
lies on the corner of a subdomain. The latter case requires special |
363 |
afe |
1.12 |
exchange and numerical handling for the singularities at the eight |
364 |
afe |
1.13 |
corners of the cube. \\ |
365 |
|
|
|
366 |
afe |
1.4 |
|
367 |
molod |
1.24 |
Arrays Indexed to Tile Number and Neighbor: |
368 |
afe |
1.4 |
|
369 |
afe |
1.17 |
The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and |
370 |
|
|
\code{NTILES} and describe the orientations between the the tiles. \\ |
371 |
afe |
1.12 |
|
372 |
|
|
The array \code{exch2\_neighbourId(a,T)} holds the tile number |
373 |
|
|
\code{Tn} for each of the tile number \code{T}'s neighboring tiles |
374 |
afe |
1.15 |
\code{a}. The neighbor tiles are indexed |
375 |
afe |
1.17 |
\code{(1:exch2\_nNeighbours(T))} in the order right to left on the |
376 |
|
|
north then south edges, and then top to bottom on the east then west |
377 |
|
|
edges. \\ |
378 |
afe |
1.15 |
|
379 |
afe |
1.17 |
The \code{exch2\_opposingSend\_record(a,T)} array holds the |
380 |
afe |
1.15 |
index \code{b} of the element in \texttt{exch2\_neighbourId(b,Tn)} |
381 |
|
|
that holds the tile number \code{T}, given |
382 |
|
|
\code{Tn=exch2\_neighborId(a,T)}. In other words, |
383 |
edhill |
1.8 |
\begin{verbatim} |
384 |
|
|
exch2_neighbourId( exch2_opposingSend_record(a,T), |
385 |
|
|
exch2_neighbourId(a,T) ) = T |
386 |
afe |
1.5 |
\end{verbatim} |
387 |
afe |
1.12 |
This provides a back-reference from the neighbor tiles. \\ |
388 |
afe |
1.5 |
|
389 |
afe |
1.13 |
The arrays \varlink{exch2\_pi}{exch2_pi} and |
390 |
afe |
1.15 |
\varlink{exch2\_pj}{exch2_pj} specify the transformations of indices |
391 |
afe |
1.13 |
in exchanges between the neighboring tiles. These transformations are |
392 |
afe |
1.19 |
necessary in exchanges between subdomains because a horizontal dimension |
393 |
|
|
in one subdomain |
394 |
|
|
may map to other horizonal dimension in an adjacent subdomain, and |
395 |
|
|
may also have its indexing reversed. This swapping arises from the |
396 |
afe |
1.17 |
``folding'' of two-dimensional arrays into a three-dimensional |
397 |
|
|
cube. \\ |
398 |
afe |
1.13 |
|
399 |
|
|
The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)} |
400 |
|
|
are the neighbor ID \code{N} and the tile number \code{T} as explained |
401 |
afe |
1.15 |
above, plus a vector of length \code{2} containing transformation |
402 |
|
|
factors \code{t}. The first element of the transformation vector |
403 |
afe |
1.19 |
holds the factor to multiply the index in the same dimension, and the |
404 |
|
|
second element holds the the same for the orthogonal dimension. To |
405 |
afe |
1.15 |
clarify, \code{exch2\_pi(1,N,T)} holds the mapping of the $x$ axis |
406 |
|
|
index of tile \code{T} to the $x$ axis of tile \code{T}'s neighbor |
407 |
|
|
\code{N}, and \code{exch2\_pi(2,N,T)} holds the mapping of \code{T}'s |
408 |
|
|
$x$ index to the neighbor \code{N}'s $y$ index. \\ |
409 |
afe |
1.12 |
|
410 |
afe |
1.15 |
One of the two elements of \code{exch2\_pi} or \code{exch2\_pj} for a |
411 |
|
|
given tile \code{T} and neighbor \code{N} will be \code{0}, reflecting |
412 |
|
|
the fact that the two axes are orthogonal. The other element will be |
413 |
|
|
\code{1} or \code{-1}, depending on whether the axes are indexed in |
414 |
|
|
the same or opposite directions. For example, the transform vector of |
415 |
|
|
the arrays for all tile neighbors on the same subdomain will be |
416 |
afe |
1.13 |
\code{(1,0)}, since all tiles on the same subdomain are oriented |
417 |
afe |
1.15 |
identically. An axis that corresponds to the orthogonal dimension |
418 |
|
|
with the same index direction in a particular tile-neighbor |
419 |
afe |
1.19 |
orientation will have \code{(0,1)}. Those with the opposite index |
420 |
afe |
1.15 |
direction will have \code{(0,-1)} in order to reverse the ordering. \\ |
421 |
afe |
1.13 |
|
422 |
afe |
1.14 |
The arrays \varlink{exch2\_oi}{exch2_oi}, |
423 |
|
|
\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and |
424 |
|
|
\varlink{exch2\_oj\_f}{exch2_oj_f} are indexed to tile number and |
425 |
|
|
neighbor and specify the relative offset within the subdomain of the |
426 |
afe |
1.17 |
array index of a variable going from a neighboring tile \code{N} to a |
427 |
|
|
local tile \code{T}. Consider \code{T=1} in the six-tile topology |
428 |
afe |
1.16 |
(Fig. \ref{fig:6tile}), where |
429 |
|
|
|
430 |
|
|
\begin{verbatim} |
431 |
|
|
exch2_oi(1,1)=33 |
432 |
|
|
exch2_oi(2,1)=0 |
433 |
|
|
exch2_oi(3,1)=32 |
434 |
|
|
exch2_oi(4,1)=-32 |
435 |
|
|
\end{verbatim} |
436 |
|
|
|
437 |
|
|
The simplest case is \code{exch2\_oi(2,1)}, the southern neighbor, |
438 |
|
|
which is \code{Tn=6}. The axes of \code{T} and \code{Tn} have the |
439 |
|
|
same orientation and their $x$ axes have the same origin, and so an |
440 |
|
|
exchange between the two requires no changes to the $x$ index. For |
441 |
|
|
the western neighbor (\code{Tn=5}), \code{code\_oi(3,1)=32} since the |
442 |
|
|
\code{x=0} vector on \code{T} corresponds to the \code{y=32} vector on |
443 |
|
|
\code{Tn}. The eastern edge of \code{T} shows the reverse case |
444 |
afe |
1.17 |
(\code{exch2\_oi(4,1)=-32)}), where \code{x=32} on \code{T} exchanges |
445 |
|
|
with \code{x=0} on \code{Tn=2}. \\ |
446 |
|
|
|
447 |
|
|
The most interesting case, where \code{exch2\_oi(1,1)=33} and |
448 |
|
|
\code{Tn=3}, involves a reversal of indices. As in every case, the |
449 |
|
|
offset \code{exch2\_oi} is added to the original $x$ index of \code{T} |
450 |
|
|
multiplied by the transformation factor \code{exch2\_pi(t,N,T)}. Here |
451 |
|
|
\code{exch2\_pi(1,1,1)=0} since the $x$ axis of \code{T} is orthogonal |
452 |
|
|
to the $x$ axis of \code{Tn}. \code{exch2\_pi(2,1,1)=-1} since the |
453 |
|
|
$x$ axis of \code{T} corresponds to the $y$ axis of \code{Tn}, but the |
454 |
|
|
index is reversed. The result is that the index of the northern edge |
455 |
|
|
of \code{T}, which runs \code{(1:32)}, is transformed to |
456 |
afe |
1.16 |
\code{(-1:-32)}. \code{exch2\_oi(1,1)} is then added to this range to |
457 |
afe |
1.17 |
get back \code{(32:1)} -- the index of the $y$ axis of \code{Tn} |
458 |
|
|
relative to \code{T}. This transformation may seem overly convoluted |
459 |
|
|
for the six-tile case, but it is necessary to provide a general |
460 |
|
|
solution for various topologies. \\ |
461 |
afe |
1.16 |
|
462 |
|
|
|
463 |
afe |
1.14 |
|
464 |
|
|
Finally, \varlink{exch2\_itlo\_c}{exch2_itlo_c}, |
465 |
|
|
\varlink{exch2\_ithi\_c}{exch2_ithi_c}, |
466 |
|
|
\varlink{exch2\_jtlo\_c}{exch2_jtlo_c} and |
467 |
|
|
\varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index |
468 |
|
|
bounds of the edge segment of the neighbor tile \code{N}'s subdomain |
469 |
|
|
that gets exchanged with the local tile \code{T}. To take the example |
470 |
jmc |
1.27 |
of tile \code{T=2} in the forty-eight-tile topology |
471 |
|
|
(Fig. \ref{fig:48tile}): \\ |
472 |
afe |
1.14 |
|
473 |
|
|
\begin{verbatim} |
474 |
|
|
exch2_itlo_c(4,2)=17 |
475 |
|
|
exch2_ithi_c(4,2)=17 |
476 |
|
|
exch2_jtlo_c(4,2)=0 |
477 |
|
|
exch2_jthi_c(4,2)=33 |
478 |
|
|
\end{verbatim} |
479 |
|
|
|
480 |
afe |
1.17 |
Here \code{N=4}, indicating the western neighbor, which is |
481 |
|
|
\code{Tn=1}. \code{Tn} resides on the same subdomain as \code{T}, so |
482 |
|
|
the tiles have the same orientation and the same $x$ and $y$ axes. |
483 |
|
|
The $x$ axis is orthogonal to the western edge and the tile is 16 |
484 |
|
|
points wide, so \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} |
485 |
|
|
indicate the column beyond \code{Tn}'s eastern edge, in that tile's |
486 |
|
|
halo region. Since the border of the tiles extends through the entire |
487 |
afe |
1.14 |
height of the subdomain, the $y$ axis bounds \code{exch2\_jtlo\_c} to |
488 |
afe |
1.17 |
\code{exch2\_jthi\_c} cover the height of \code{(1:32)}, plus 1 in |
489 |
|
|
either direction to cover part of the halo. \\ |
490 |
afe |
1.14 |
|
491 |
|
|
For the north edge of the same tile \code{T=2} where \code{N=1} and |
492 |
|
|
the neighbor tile is \code{Tn=5}: |
493 |
|
|
|
494 |
|
|
\begin{verbatim} |
495 |
|
|
exch2_itlo_c(1,2)=0 |
496 |
|
|
exch2_ithi_c(1,2)=0 |
497 |
|
|
exch2_jtlo_c(1,2)=0 |
498 |
|
|
exch2_jthi_c(1,2)=17 |
499 |
|
|
\end{verbatim} |
500 |
|
|
|
501 |
|
|
\code{T}'s northern edge is parallel to the $x$ axis, but since |
502 |
afe |
1.17 |
\code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis, \code{T}'s |
503 |
|
|
northern edge exchanges with \code{Tn}'s western edge. The western |
504 |
|
|
edge of the tiles corresponds to the lower bound of the $x$ axis, so |
505 |
afe |
1.19 |
\code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} are \code{0}, in the |
506 |
|
|
western halo region of \code{Tn}. The range of |
507 |
afe |
1.17 |
\code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the |
508 |
afe |
1.19 |
width of \code{T}'s northern edge, expanded by one into the halo. \\ |
509 |
afe |
1.14 |
|
510 |
|
|
|
511 |
molod |
1.24 |
\subsubsection{Key Routines} |
512 |
afe |
1.1 |
|
513 |
afe |
1.16 |
Most of the subroutines particular to exch2 handle the exchanges |
514 |
|
|
themselves and are of the same format as those described in |
515 |
jmc |
1.29 |
\ref{sec:cube_sphere_communication} \sectiontitle{Cube sphere |
516 |
afe |
1.16 |
communication}. Like the original routines, they are written as |
517 |
afe |
1.19 |
templates which the local Makefile converts from \code{RX} into |
518 |
|
|
\code{RL} and \code{RS} forms. \\ |
519 |
afe |
1.16 |
|
520 |
|
|
The interfaces with the core model subroutines are |
521 |
afe |
1.17 |
\code{EXCH\_UV\_XY\_RX}, \code{EXCH\_UV\_XYZ\_RX} and |
522 |
|
|
\code{EXCH\_XY\_RX}. They override the standard exchange routines |
523 |
|
|
when \code{genmake2} is run with \code{exch2} option. They in turn |
524 |
|
|
call the local exch2 subroutines \code{EXCH2\_UV\_XY\_RX} and |
525 |
|
|
\code{EXCH2\_UV\_XYZ\_RX} for two and three-dimensional vector |
526 |
|
|
quantities, and \code{EXCH2\_XY\_RX} and \code{EXCH2\_XYZ\_RX} for two |
527 |
|
|
and three-dimensional scalar quantities. These subroutines set the |
528 |
|
|
dimensions of the area to be exchanged, call \code{EXCH2\_RX1\_CUBE} |
529 |
|
|
for scalars and \code{EXCH2\_RX2\_CUBE} for vectors, and then handle |
530 |
|
|
the singularities at the cube corners. \\ |
531 |
afe |
1.16 |
|
532 |
|
|
The separate scalar and vector forms of \code{EXCH2\_RX1\_CUBE} and |
533 |
afe |
1.19 |
\code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subroutine |
534 |
|
|
needs to pass both the $u$ and $v$ components of the physical vectors. |
535 |
|
|
This swapping arises from the topological folding discussed above, where the |
536 |
|
|
$x$ and $y$ axes get swapped in some cases, and is not an |
537 |
|
|
issue with the scalar case. These subroutines call |
538 |
afe |
1.17 |
\code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of |
539 |
|
|
the work using the variables discussed above. \\ |
540 |
afe |
1.1 |
|
541 |
molod |
1.26 |
\subsubsection{Experiments and tutorials that use exch2} |
542 |
|
|
\label{sec:pkg:exch2:experiments} |
543 |
|
|
|
544 |
|
|
\begin{itemize} |
545 |
|
|
\item{Held Suarez tutorial, in tutorial\_held\_suarez\_cs verification directory, |
546 |
jmc |
1.29 |
described in section \ref{sec:eg-hs} } |
547 |
molod |
1.26 |
\end{itemize} |