| 10 |
%% o automatically inserted at \section{Reference} |
%% o automatically inserted at \section{Reference} |
| 11 |
|
|
| 12 |
|
|
| 13 |
\section{Extended Cubed Sphere Exchange} |
\section{exch2: Extended Cubed Sphere \mbox{Topology}} |
| 14 |
\label{sec:exch2} |
\label{sec:exch2} |
| 15 |
|
|
| 16 |
|
|
| 17 |
\subsection{Introduction} |
\subsection{Introduction} |
| 18 |
|
|
| 19 |
The \texttt{exch2} package is an extension to the original cubed |
The \texttt{exch2} package extends the original cubed sphere topology |
| 20 |
sphere topological configuration that allows more flexible domain |
configuration to allow more flexible domain decomposition and |
| 21 |
decomposition and parallelization. Cube faces (also called |
parallelization. Cube faces (also called subdomains) may be divided |
| 22 |
subdomains) may be divided into any number of tiles that divide evenly |
into any number of tiles that divide evenly into the grid point |
| 23 |
into the grid point dimensions of the subdomain. Furthermore, the |
dimensions of the subdomain. Furthermore, the tiles can run on |
| 24 |
individual tiles may be run on separate processors in different |
separate processors individually or in groups, which provides for |
| 25 |
combinations, and whether exchanges between particular tiles occur |
manual compile-time load balancing across a relatively arbitrary |
| 26 |
between different processors is determined at runtime. This |
number of processors. \\ |
|
flexibility provides for manual load balancing across a relatively |
|
|
arbitrary number of processors. |
|
| 27 |
|
|
| 28 |
The exchange parameters are declared in |
The exchange parameters are declared in |
| 29 |
\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} |
\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} |
| 30 |
and assigned in |
and assigned in |
| 31 |
\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The |
\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The |
| 32 |
validity of the cube topology depends on the \texttt{SIZE.h} file as |
validity of the cube topology depends on the \file{SIZE.h} file as |
| 33 |
detailed below. Both files are generated by Matlab scripts in ?? |
detailed below. The default files provided in the release configure a |
| 34 |
check these in already! and should not be edited. The default files |
cubed sphere topology of six tiles, one per subdomain, each with |
| 35 |
provided in the release configure a cubed sphere arrangement of six |
32$\times$32 grid points, with all tiles running on a single processor. Both |
| 36 |
tiles, one per subdomain, each with 32$\times$32 grid points, all |
files are generated by Matlab scripts in |
| 37 |
running on a single processor. Pregenerated examples of these files |
\file{utils/exch2/matlab-topology-generator}; see Section |
| 38 |
with alternate topologies are provided in ??. |
\ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2} |
| 39 |
|
for details on creating alternate topologies. Pregenerated examples |
| 40 |
|
of these files with alternate topologies are provided under |
| 41 |
|
\file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h} |
| 42 |
|
file for single-processor execution. |
| 43 |
|
|
| 44 |
\subsection{Invoking exch2} |
\subsection{Invoking exch2} |
| 45 |
|
|
| 46 |
To use exch2 with the cubed sphere, the following conditions must be met: |
To use exch2 with the cubed sphere, the following conditions must be |
| 47 |
|
met: |
| 48 |
|
|
| 49 |
|
\begin{itemize} |
| 50 |
|
\item The exch2 package is included when \file{genmake2} is run. The |
| 51 |
|
easiest way to do this is to add the line \code{exch2} to the |
| 52 |
|
\file{profile.conf} file -- see Section \ref{sect:buildingCode} |
| 53 |
|
\sectiontitle{Building the code} for general details. |
| 54 |
|
|
| 55 |
|
\item An example of \file{W2\_EXCH2\_TOPOLOGY.h} and |
| 56 |
|
\file{w2\_e2setup.F} must reside in a directory containing files |
| 57 |
|
symbolically linked by the \file{genmake2} script. The safest place |
| 58 |
|
to put these is the directory indicated in the \code{-mods=DIR} |
| 59 |
|
command line modifier (typically \file{../code}), or the build |
| 60 |
|
directory. The default versions of these files reside in |
| 61 |
|
\file{pkg/exch2} and are linked automatically if no other versions |
| 62 |
|
exist elsewhere in the build path, but they should be left untouched |
| 63 |
|
to avoid breaking configurations other than the one you intend to |
| 64 |
|
modify. |
| 65 |
|
|
| 66 |
|
\item Files containing grid parameters, named \file{tile00$n$.mitgrid} |
| 67 |
|
where $n$=\code{(1:6)} (one per subdomain), must be in the working |
| 68 |
|
directory when the MITgcm executable is run. These files are |
| 69 |
|
provided in the example experiments for cubed sphere configurations |
| 70 |
|
with 32$\times$32 cube sides -- please contact |
| 71 |
|
\begin{rawhtml} |
| 72 |
|
<A href=''mailto:mitgcm-support@dev.mitgcm.org"> |
| 73 |
|
\end{rawhtml} |
| 74 |
|
\begin{verbatim} |
| 75 |
|
MITgcm-support@mitgcm.org |
| 76 |
|
\end{verbatim} |
| 77 |
|
\begin{rawhtml} </A> \end{rawhtml} |
| 78 |
|
if you want to generate files for other configurations. |
| 79 |
|
|
| 80 |
|
\item As always when compiling MITgcm, the file \file{SIZE.h} must be |
| 81 |
|
placed where \file{genmake2} will find it. In particular for exch2, |
| 82 |
|
the domain decomposition specified in \file{SIZE.h} must correspond |
| 83 |
|
with the particular configuration's topology specified in |
| 84 |
|
\file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}. Domain |
| 85 |
|
decomposition issues particular to exch2 are addressed in Section |
| 86 |
|
\ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2} |
| 87 |
|
and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and |
| 88 |
|
Multiprocessing}; a more general background on the subject |
| 89 |
|
relevant to MITgcm is presented in Section |
| 90 |
|
\ref{sect:specifying_a_decomposition} \sectiontitle{Specifying a |
| 91 |
|
decomposition}. |
| 92 |
|
\end{itemize} |
| 93 |
|
|
|
- the exch2 package is included when \texttt{genmake2} is run. The |
|
|
easiest way to do this is to add the line \texttt{exch2} to the |
|
|
\texttt{profile.conf} file -- see Section \ref{sect:buildingCode} |
|
|
for general details. \\ |
|
|
|
|
|
- an example of \texttt{W2\_EXCH2\_TOPOLOGY.h} and |
|
|
\texttt{w2\_e2setup.F} must reside in a directory containing code |
|
|
linked when \texttt{genmake2} runs. The safest place to put these |
|
|
is the directory indicated in the \texttt{-mods=DIR} command line |
|
|
modifier (typically \texttt{../code}), or the build directory. The |
|
|
default versions of these files reside in \texttt{pkg/exch2}, but |
|
|
they should be left untouched to avoid breaking configurations other |
|
|
than the one you intend to modify.\\ |
|
|
|
|
|
- files containing grid parameters, named |
|
|
\texttt{tile}xxx\texttt{.mitgrid} where xxx is \texttt{001} through |
|
|
\texttt{006}, must be in the working directory when the MITgcm |
|
|
executable is run. These files are provided in the example |
|
|
experiments for cubed sphere configurations with 32$\times$32 cube |
|
|
sides and are non-trivial to generate -- please contact MITgcm |
|
|
support if you want to generate files for other configurations. |
|
|
This is lame. ?? \\ |
|
| 94 |
|
|
| 95 |
As of the time of writing the following examples use exch2 and may be |
|
| 96 |
used for guidance: |
At the time of this writing the following examples use exch2 and may |
| 97 |
|
be used for guidance: |
| 98 |
|
|
| 99 |
\begin{verbatim} |
\begin{verbatim} |
| 100 |
verification/adjust_nlfs.cs-32x32x1 |
verification/adjust_nlfs.cs-32x32x1 |
| 107 |
|
|
| 108 |
|
|
| 109 |
|
|
| 110 |
\subsection{Generating Topology Files} |
\subsection{Generating Topology Files for exch2} |
| 111 |
|
\label{sec:topogen} |
| 112 |
|
|
| 113 |
|
Alternate cubed sphere topologies may be created using the Matlab |
| 114 |
|
scripts in \file{utils/exch2/matlab-topology-generator}. Running the |
| 115 |
|
m-file |
| 116 |
|
\filelink{driver.m}{utils-exch2-matlab-topology-generator_driver.m} |
| 117 |
|
from the Matlab prompt (there are no parameters to pass) generates |
| 118 |
|
exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and |
| 119 |
|
\file{w2\_e2setup.F} in the working directory and displays a figure of |
| 120 |
|
the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:12tile}, |
| 121 |
|
and \ref{fig:24tile} are examples of the generated diagrams. The other |
| 122 |
|
m-files in the directory are |
| 123 |
|
subroutines called from \file{driver.m} and should not be run ``bare'' except |
| 124 |
|
for development purposes. \\ |
| 125 |
|
|
| 126 |
|
The parameters that determine the dimensions and topology of the |
| 127 |
|
generated configuration are \code{nr}, \code{nb}, \code{ng}, |
| 128 |
|
\code{tnx} and \code{tny}, and all are assigned early in the script. \\ |
| 129 |
|
|
| 130 |
|
The first three determine the height and width of the subdomains and |
| 131 |
|
hence the size of the overall domain. Each one determines the number |
| 132 |
|
of grid points, and therefore the resolution, along the subdomain |
| 133 |
|
sides in a ``great circle'' around each the three spatial axes of the cube. At the time |
| 134 |
|
of this writing MITgcm requires these three parameters to be equal, |
| 135 |
|
but they provide for future releases to accomodate different |
| 136 |
|
resolutions around the axes to allow subdomains with differing resolutions.\\ |
| 137 |
|
|
| 138 |
|
The parameters \code{tnx} and \code{tny} determine the width and height of |
| 139 |
|
the tiles into which the subdomains are decomposed, and must evenly |
| 140 |
|
divide the integer assigned to \code{nr}, \code{nb} and \code{ng}. |
| 141 |
|
The result is a rectangular tiling of the subdomain. Figure |
| 142 |
|
\ref{fig:24tile} shows one possible topology for a twenty-four-tile |
| 143 |
|
cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\ |
| 144 |
|
|
| 145 |
|
\begin{figure} |
| 146 |
|
\begin{center} |
| 147 |
|
\resizebox{4in}{!}{ |
| 148 |
|
\includegraphics{part6/s24t_16x16.ps} |
| 149 |
|
} |
| 150 |
|
\end{center} |
| 151 |
|
|
| 152 |
|
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
| 153 |
|
divided into six 32$\times$32 subdomains, each of which is divided |
| 154 |
|
into four tiles of width \code{tnx=16} and height \code{tny=16} for a |
| 155 |
|
total of twenty-four tiles. The colored borders of the subdomains |
| 156 |
|
represent the parameters \code{nr} (red), \code{nb} (blue), and |
| 157 |
|
\code{ng} (green). } \label{fig:24tile} |
| 158 |
|
\end{figure} |
| 159 |
|
|
| 160 |
|
\begin{figure} |
| 161 |
|
\begin{center} |
| 162 |
|
\resizebox{4in}{!}{ |
| 163 |
|
\includegraphics{part6/s12t_16x32.ps} |
| 164 |
|
} |
| 165 |
|
\end{center} |
| 166 |
|
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
| 167 |
|
divided into six 32$\times$32 subdomains of two tiles each |
| 168 |
|
(\code{tnx=16, tny=32}). |
| 169 |
|
} \label{fig:12tile} |
| 170 |
|
\end{figure} |
| 171 |
|
|
| 172 |
|
\begin{figure} |
| 173 |
|
\begin{center} |
| 174 |
|
\resizebox{4in}{!}{ |
| 175 |
|
\includegraphics{part6/s6t_32x32.ps} |
| 176 |
|
} |
| 177 |
|
\end{center} |
| 178 |
|
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
| 179 |
|
divided into six 32$\times$32 subdomains with one tile each |
| 180 |
|
(\code{tnx=32, tny=32}). This is the default configuration. |
| 181 |
|
} |
| 182 |
|
\label{fig:6tile} |
| 183 |
|
\end{figure} |
| 184 |
|
|
| 185 |
|
|
| 186 |
|
Tiles can be selected from the topology to be omitted from being |
| 187 |
|
allocated memory and processors. This tuning is useful in ocean |
| 188 |
|
modeling for omitting tiles that fall entirely on land. The tiles |
| 189 |
|
omitted are specified in the file |
| 190 |
|
\filelink{blanklist.txt}{utils-exch2-matlab-topology-generator_blanklist.txt} |
| 191 |
|
by their tile number in the topology, separated by a newline. \\ |
| 192 |
|
|
| 193 |
|
|
| 194 |
|
|
| 195 |
|
|
| 196 |
|
\subsection{exch2, SIZE.h, and Multiprocessing} |
| 197 |
|
\label{sec:exch2mpi} |
| 198 |
|
|
| 199 |
|
Once the topology configuration files are created, the Fortran |
| 200 |
|
\code{PARAMETER}s in \file{SIZE.h} must be configured to match. |
| 201 |
|
Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying |
| 202 |
|
a decomposition} provides a general description of domain |
| 203 |
|
decomposition within MITgcm and its relation to \file{SIZE.h}. The |
| 204 |
|
current section specifies constraints that the exch2 package |
| 205 |
|
imposes and describes how to enable parallel execution with |
| 206 |
|
MPI. \\ |
| 207 |
|
|
| 208 |
|
As in the general case, the parameters \varlink{sNx}{sNx} and |
| 209 |
|
\varlink{sNy}{sNy} define the size of the individual tiles, and so |
| 210 |
|
must be assigned the same respective values as \code{tnx} and |
| 211 |
|
\code{tny} in \file{driver.m}.\\ |
| 212 |
|
|
| 213 |
|
The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy} |
| 214 |
|
have no special bearing on exch2 and may be assigned as in the general |
| 215 |
|
case. The same holds for \varlink{Nr}{Nr}, the number of vertical |
| 216 |
|
levels in the model.\\ |
| 217 |
|
|
| 218 |
|
The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy}, |
| 219 |
|
\varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of |
| 220 |
|
tiles and how they are distributed on processors. When using exch2, |
| 221 |
|
the tiles are stored in the $x$ dimension, and so |
| 222 |
|
\code{\varlink{nSy}{nSy}=1} in all cases. Since the tiles as |
| 223 |
|
configured by exch2 cannot be split up accross processors without |
| 224 |
|
regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\ |
| 225 |
|
|
| 226 |
|
The number of tiles MITgcm allocates and how they are distributed |
| 227 |
|
between processors depends on \varlink{nPx}{nPx} and |
| 228 |
|
\varlink{nSx}{nSx}. \varlink{nSx}{nSx} is the number of tiles per |
| 229 |
|
processor and \varlink{nPx}{nPx} is the number of processors. The total |
| 230 |
|
number of tiles in the topology minus those listed in |
| 231 |
|
\file{blanklist.txt} must equal \code{nSx*nPx}. Note that in order to |
| 232 |
|
obtain maximum usage from a given number of processors in some cases, |
| 233 |
|
this restriction might entail sharing a processor with a tile that would |
| 234 |
|
otherwise be excluded. \\ |
| 235 |
|
|
| 236 |
|
The following is an example of \file{SIZE.h} for the twelve-tile |
| 237 |
|
configuration illustrated in figure \ref{fig:12tile} running on |
| 238 |
|
one processor: \\ |
| 239 |
|
|
| 240 |
|
\begin{verbatim} |
| 241 |
|
PARAMETER ( |
| 242 |
|
& sNx = 16, |
| 243 |
|
& sNy = 32, |
| 244 |
|
& OLx = 2, |
| 245 |
|
& OLy = 2, |
| 246 |
|
& nSx = 12, |
| 247 |
|
& nSy = 1, |
| 248 |
|
& nPx = 1, |
| 249 |
|
& nPy = 1, |
| 250 |
|
& Nx = sNx*nSx*nPx, |
| 251 |
|
& Ny = sNy*nSy*nPy, |
| 252 |
|
& Nr = 5) |
| 253 |
|
\end{verbatim} |
| 254 |
|
|
| 255 |
|
The following is an example for the twenty-four-tile topology in |
| 256 |
|
figure \ref{fig:24tile} running on six processors: |
| 257 |
|
|
| 258 |
|
\begin{verbatim} |
| 259 |
|
PARAMETER ( |
| 260 |
|
& sNx = 16, |
| 261 |
|
& sNy = 16, |
| 262 |
|
& OLx = 2, |
| 263 |
|
& OLy = 2, |
| 264 |
|
& nSx = 4, |
| 265 |
|
& nSy = 1, |
| 266 |
|
& nPx = 6, |
| 267 |
|
& nPy = 1, |
| 268 |
|
& Nx = sNx*nSx*nPx, |
| 269 |
|
& Ny = sNy*nSy*nPy, |
| 270 |
|
& Nr = 5) |
| 271 |
|
\end{verbatim} |
| 272 |
|
|
| 273 |
|
|
| 274 |
|
|
| 275 |
|
|
| 276 |
|
|
| 277 |
\subsection{Key Variables} |
\subsection{Key Variables} |
| 278 |
|
|
| 279 |
The descriptions of the variables are divided up into scalars, |
The descriptions of the variables are divided up into scalars, |
| 280 |
one-dimensional arrays indexed to the tile number, and two and three |
one-dimensional arrays indexed to the tile number, and two and |
| 281 |
dimensional arrays indexed to tile number and neighboring tile. This |
three-dimensional arrays indexed to tile number and neighboring tile. |
| 282 |
division actually reflects the functionality of these variables: the |
This division reflects the functionality of these variables: The |
| 283 |
scalars are common to every part of the topology, the tile-indexed |
scalars are common to every part of the topology, the tile-indexed |
| 284 |
arrays to individual tiles, and the arrays indexed to tile and |
arrays to individual tiles, and the arrays indexed by tile and |
| 285 |
neighbor to relationships between tiles and their neighbors. |
neighbor to relationships between tiles and their neighbors. \\ |
| 286 |
|
|
| 287 |
\subsubsection{Scalars} |
\subsubsection{Scalars} |
| 288 |
|
|
| 289 |
The number of tiles in a particular topology is set with the parameter |
The number of tiles in a particular topology is set with the parameter |
| 290 |
\texttt{NTILES}, and the maximum number of neighbors of any tiles by |
\code{NTILES}, and the maximum number of neighbors of any tiles by |
| 291 |
\texttt{MAX\_NEIGHBOURS}. These parameters are used for defining the |
\code{MAX\_NEIGHBOURS}. These parameters are used for defining the |
| 292 |
size of the various one and two dimensional arrays that store tile |
size of the various one and two dimensional arrays that store tile |
| 293 |
parameters indexed to the tile number.\\ |
parameters indexed to the tile number and are assigned in the files |
| 294 |
|
generated by \file{driver.m}.\\ |
| 295 |
|
|
| 296 |
The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt} |
The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt} |
| 297 |
and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number |
and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number |
| 298 |
of tiles in the x and y global indices. For example, the default |
of tiles in the $x$ and $y$ global indices. For example, the default |
| 299 |
setup of six tiles has \texttt{exch2\_domain\_nxt=6} and |
setup of six tiles (Fig. \ref{fig:6tile}) has |
| 300 |
\texttt{exch2\_domain\_nyt=1}. A topology of twenty-four square (in |
\code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}. A |
| 301 |
gridpoints) tiles, four (2x2) per subdomain, will have |
topology of twenty-four square tiles, four per subdomain (as in figure |
| 302 |
\texttt{exch2\_domain\_nxt=12} and \texttt{exch2\_domain\_nyt=2}. |
\ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and |
| 303 |
Note that these parameters express the tile layout to allow global |
\code{exch2\_domain\_nyt=2}. Note that these parameters express the |
| 304 |
data files that are tile-layout-neutral and have no bearing on the |
tile layout in order to allow global data files that are tile-layout-neutral. |
| 305 |
internal storage of the arrays. The tiles are internally stored in a |
They have no bearing on the internal storage of the arrays. The tiles |
| 306 |
range from \texttt{1,bi} (in the x axis) and y-axis variable |
are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the |
| 307 |
\texttt{bj} is generally ignored within the package. |
$x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to |
| 308 |
|
equal \code{1} throughout the package. \\ |
| 309 |
\subsubsection{Arrays Indexed to Tile Number} |
|
| 310 |
|
\subsubsection{Arrays indexed to tile number} |
| 311 |
The following arrays are of size \texttt{NTILES}, are indexed to the |
|
| 312 |
tile number, and the indices are omitted in their descriptions. |
The following arrays are of length \code{NTILES} and are indexed to |
| 313 |
|
the tile number, which is indicated in the diagrams with the notation |
| 314 |
|
\code{tn}. The indices are omitted in the descriptions. \\ |
| 315 |
|
|
| 316 |
The arrays \varlink{exch2\_tnx}{exch2_tnx} and |
The arrays \varlink{exch2\_tnx}{exch2_tnx} and |
| 317 |
\varlink{exch2\_tny}{exch2_tny} express the x and y dimensions of each |
\varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of |
| 318 |
tile. At present for each tile \texttt{exch2\_tnx=sNx} and |
each tile. At present for each tile \texttt{exch2\_tnx=sNx} and |
| 319 |
\texttt{exch2\_tny=sNy}, as assigned in \texttt{SIZE.h}. Future |
\texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in |
| 320 |
releases of MITgcm are to allow varying tile sizes. |
Section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and |
| 321 |
|
Multiprocessing}. Future releases of MITgcm may allow varying tile |
| 322 |
The location of the tiles' Cartesian origin within a subdomain are |
sizes. \\ |
| 323 |
determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and |
|
| 324 |
\varlink{exch2\_tbasey}{exch2_tbasey}. These variables are used to |
The arrays \varlink{exch2\_tbasex}{exch2_tbasex} and |
| 325 |
relate the location of the edges of the tiles to each other. As an |
\varlink{exch2\_tbasey}{exch2_tbasey} determine the tiles' |
| 326 |
example, in the default six-tile topology (the degenerate case) each |
Cartesian origin within a subdomain |
| 327 |
index in these arrays are set to 0. The twenty-four, 32x32 cube face |
and locate the edges of different tiles relative to each other. As |
| 328 |
case discussed above will have values of 0 or 16, depending on the |
an example, in the default six-tile topology (Fig. \ref{fig:6tile}) |
| 329 |
quadrant the tile falls within the subdomain. The array |
each index in these arrays is set to \code{0} since a tile occupies |
| 330 |
\varlink{exch2\_myFace}{exch2_myFace} contains the number of the |
its entire subdomain. The twenty-four-tile case discussed above will |
| 331 |
cubeface/subdomain of each tile, numbered 1-6 in the case of the |
have values of \code{0} or \code{16}, depending on the quadrant of the |
| 332 |
standard cube topology. |
tile within the subdomain. The elements of the arrays |
| 333 |
|
\varlink{exch2\_txglobalo}{exch2_txglobalo} and |
|
The arrays \varlink{exch2\_txglobalo}{exch2_txglobalo} and |
|
| 334 |
\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to |
\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to |
| 335 |
\varlink{exch2\_tbasex}{exch2_tbasex} and |
\varlink{exch2\_tbasex}{exch2_tbasex} and |
| 336 |
\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the |
\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tile edges within the |
| 337 |
global address space, similar to that used by global files. |
global address space, similar to that used by global output and input |
| 338 |
|
files. \\ |
| 339 |
|
|
| 340 |
|
The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of |
| 341 |
|
the subdomain of each tile, in a range \code{(1:6)} in the case of the |
| 342 |
|
standard cube topology and indicated by \textbf{\textsf{fn}} in |
| 343 |
|
figures \ref{fig:12tile} and \ref{fig:24tile}. The |
| 344 |
|
\varlink{exch2\_nNeighbours}{exch2_nNeighbours} variable contains a |
| 345 |
|
count of the neighboring tiles each tile has, and sets the bounds for |
| 346 |
|
looping over neighboring tiles. And |
| 347 |
|
\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each |
| 348 |
|
tile, and is used in interprocess communication. \\ |
| 349 |
|
|
| 350 |
|
|
| 351 |
The arrays \varlink{exch2\_isWedge}{exch2_isWedge}, |
The arrays \varlink{exch2\_isWedge}{exch2_isWedge}, |
| 352 |
\varlink{exch2\_isEedge}{exch2_isEedge}, |
\varlink{exch2\_isEedge}{exch2_isEedge}, |
| 353 |
\varlink{exch2\_isSedge}{exch2_isSedge}, and |
\varlink{exch2\_isSedge}{exch2_isSedge}, and |
| 354 |
\varlink{exch2\_isNedge}{exch2_isNedge} are set to 1 if the indexed |
\varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the |
| 355 |
tile lies on the edge of a subdomain, 0 if not. The values are used |
indexed tile lies on the edge of its subdomain, \code{0} if |
| 356 |
within the topology generator to determine the orientation of |
not. The values are used within the topology generator to determine |
| 357 |
neighboring tiles and to indicate whether a tile lies on the corner of |
the orientation of neighboring tiles, and to indicate whether a tile |
| 358 |
a subdomain. The latter case indicates special exchange and numerical |
lies on the corner of a subdomain. The latter case requires special |
| 359 |
handling for the singularities at the eight corners of the cube. |
exchange and numerical handling for the singularities at the eight |
| 360 |
\varlink{exch2\_nNeighbours}{exch2_nNeighbours} contains a count of |
corners of the cube. \\ |
| 361 |
how many neighboring tiles each tile has, and is used for setting |
|
|
bounds for looping over neighboring tiles. |
|
|
\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each |
|
|
tile, and is used in interprocess communication. |
|
| 362 |
|
|
| 363 |
\subsubsection{Arrays Indexed to Tile Number and Neighbor} |
\subsubsection{Arrays Indexed to Tile Number and Neighbor} |
| 364 |
|
|
| 365 |
The following arrays are all of size \texttt{MAX\_NEIGHBOURS} $\times$ |
The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and |
| 366 |
\texttt{NTILES} and describe the orientations between the the tiles. |
\code{NTILES} and describe the orientations between the the tiles. \\ |
| 367 |
|
|
| 368 |
The array \texttt{exch2\_neighbourId(a,T)} holds the tile number for |
The array \code{exch2\_neighbourId(a,T)} holds the tile number |
| 369 |
each of the $n$ neighboring tiles. The neighbor tiles are indexed |
\code{Tn} for each of the tile number \code{T}'s neighboring tiles |
| 370 |
\texttt{(1,MAX\_NEIGHBOURS} in the order right to left on the north |
\code{a}. The neighbor tiles are indexed |
| 371 |
then south edges, and then top to bottom on the east and west edges. |
\code{(1:exch2\_nNeighbours(T))} in the order right to left on the |
| 372 |
Maybe throw in a fig here, eh? |
north then south edges, and then top to bottom on the east then west |
| 373 |
|
edges. \\ |
| 374 |
The \texttt{exch2\_opposingSend\_record(a,T)} array holds the index c |
|
| 375 |
in \texttt{exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T. |
The \code{exch2\_opposingSend\_record(a,T)} array holds the |
| 376 |
In other words, |
index \code{b} of the element in \texttt{exch2\_neighbourId(b,Tn)} |
| 377 |
|
that holds the tile number \code{T}, given |
| 378 |
|
\code{Tn=exch2\_neighborId(a,T)}. In other words, |
| 379 |
\begin{verbatim} |
\begin{verbatim} |
| 380 |
exch2_neighbourId( exch2_opposingSend_record(a,T), |
exch2_neighbourId( exch2_opposingSend_record(a,T), |
| 381 |
exch2_neighbourId(a,T) ) = T |
exch2_neighbourId(a,T) ) = T |
| 382 |
\end{verbatim} |
\end{verbatim} |
| 383 |
and this provides a back-reference from the neighbor tiles. |
This provides a back-reference from the neighbor tiles. \\ |
| 384 |
|
|
| 385 |
|
The arrays \varlink{exch2\_pi}{exch2_pi} and |
| 386 |
|
\varlink{exch2\_pj}{exch2_pj} specify the transformations of indices |
| 387 |
|
in exchanges between the neighboring tiles. These transformations are |
| 388 |
|
necessary in exchanges between subdomains because a horizontal dimension |
| 389 |
|
in one subdomain |
| 390 |
|
may map to other horizonal dimension in an adjacent subdomain, and |
| 391 |
|
may also have its indexing reversed. This swapping arises from the |
| 392 |
|
``folding'' of two-dimensional arrays into a three-dimensional |
| 393 |
|
cube. \\ |
| 394 |
|
|
| 395 |
|
The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)} |
| 396 |
|
are the neighbor ID \code{N} and the tile number \code{T} as explained |
| 397 |
|
above, plus a vector of length \code{2} containing transformation |
| 398 |
|
factors \code{t}. The first element of the transformation vector |
| 399 |
|
holds the factor to multiply the index in the same dimension, and the |
| 400 |
|
second element holds the the same for the orthogonal dimension. To |
| 401 |
|
clarify, \code{exch2\_pi(1,N,T)} holds the mapping of the $x$ axis |
| 402 |
|
index of tile \code{T} to the $x$ axis of tile \code{T}'s neighbor |
| 403 |
|
\code{N}, and \code{exch2\_pi(2,N,T)} holds the mapping of \code{T}'s |
| 404 |
|
$x$ index to the neighbor \code{N}'s $y$ index. \\ |
| 405 |
|
|
| 406 |
|
One of the two elements of \code{exch2\_pi} or \code{exch2\_pj} for a |
| 407 |
|
given tile \code{T} and neighbor \code{N} will be \code{0}, reflecting |
| 408 |
|
the fact that the two axes are orthogonal. The other element will be |
| 409 |
|
\code{1} or \code{-1}, depending on whether the axes are indexed in |
| 410 |
|
the same or opposite directions. For example, the transform vector of |
| 411 |
|
the arrays for all tile neighbors on the same subdomain will be |
| 412 |
|
\code{(1,0)}, since all tiles on the same subdomain are oriented |
| 413 |
|
identically. An axis that corresponds to the orthogonal dimension |
| 414 |
|
with the same index direction in a particular tile-neighbor |
| 415 |
|
orientation will have \code{(0,1)}. Those with the opposite index |
| 416 |
|
direction will have \code{(0,-1)} in order to reverse the ordering. \\ |
| 417 |
|
|
| 418 |
The arrays \varlink{exch2\_pi}{exch2_pi}, |
The arrays \varlink{exch2\_oi}{exch2_oi}, |
|
\varlink{exch2\_pj}{exch2_pj}, \varlink{exch2\_oi}{exch2_oi}, |
|
| 419 |
\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and |
\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and |
| 420 |
\varlink{exch2\_oj\_f}{exch2_oj_f} specify the transformations in |
\varlink{exch2\_oj\_f}{exch2_oj_f} are indexed to tile number and |
| 421 |
exchanges between the neighboring tiles. The dimensions of |
neighbor and specify the relative offset within the subdomain of the |
| 422 |
\texttt{exch2\_pi(t,N,T)} and \texttt{exch2\_pj(t,N,T)} are the |
array index of a variable going from a neighboring tile \code{N} to a |
| 423 |
neighbor ID \textit{N} and the tile number \textit{T} as explained |
local tile \code{T}. Consider \code{T=1} in the six-tile topology |
| 424 |
above, plus the transformation vector {\em t }, of length two. The |
(Fig. \ref{fig:6tile}), where |
| 425 |
first element of the transformation vector indicates the factor by |
|
| 426 |
which variables representing the same vector component of a tile will |
\begin{verbatim} |
| 427 |
be multiplied, and the second element indicates the transform to the |
exch2_oi(1,1)=33 |
| 428 |
variable in the other direction. As an example, |
exch2_oi(2,1)=0 |
| 429 |
\texttt{exch2\_pi(1,N,T)} holds the transform of the i-component of a |
exch2_oi(3,1)=32 |
| 430 |
vector variable in tile \texttt{T} to the i-component of tile |
exch2_oi(4,1)=-32 |
|
\texttt{T}'s neighbor \texttt{N}, and \texttt{exch2\_pi(2,N,T)} hold |
|
|
the component of neighbor \texttt{N}'s j-component. |
|
|
|
|
|
Under the current cube topology, one of the two elements of |
|
|
\texttt{exch2\_pi} or \texttt{exch2\_pj} for a given tile \texttt{T} |
|
|
and neighbor \texttt{N} will be 0, reflecting the fact that the vector |
|
|
components are orthogonal. The other element will be 1 or -1, |
|
|
depending on whether the components are indexed in the same or |
|
|
opposite directions. For example, the transform dimension of the |
|
|
arrays for all tile neighbors on the same subdomain will be [1,0], |
|
|
since all tiles on the same subdomain are oriented identically. |
|
|
Vectors that correspond to the orthogonal dimension with the same |
|
|
index direction will have [0,1], whereas those in the opposite index |
|
|
direction will have [0,-1]. |
|
|
|
|
|
|
|
|
{\footnotesize |
|
|
\begin{verbatim} |
|
|
C exch2_pi :: X index row of target to source permutation |
|
|
C :: matrix for each neighbour entry. |
|
|
C exch2_pj :: Y index row of target to source permutation |
|
|
C :: matrix for each neighbour entry. |
|
|
C exch2_oi :: X index element of target to source |
|
|
C :: offset vector for cell-centered quantities |
|
|
C :: of each neighbor entry. |
|
|
C exch2_oj :: Y index element of target to source |
|
|
C :: offset vector for cell-centered quantities |
|
|
C :: of each neighbor entry. |
|
|
C exch2_oi_f :: X index element of target to source |
|
|
C :: offset vector for face quantities |
|
|
C :: of each neighbor entry. |
|
|
C exch2_oj_f :: Y index element of target to source |
|
|
C :: offset vector for face quantities |
|
|
C :: of each neighbor entry. |
|
| 431 |
\end{verbatim} |
\end{verbatim} |
|
} |
|
| 432 |
|
|
| 433 |
|
The simplest case is \code{exch2\_oi(2,1)}, the southern neighbor, |
| 434 |
|
which is \code{Tn=6}. The axes of \code{T} and \code{Tn} have the |
| 435 |
|
same orientation and their $x$ axes have the same origin, and so an |
| 436 |
|
exchange between the two requires no changes to the $x$ index. For |
| 437 |
|
the western neighbor (\code{Tn=5}), \code{code\_oi(3,1)=32} since the |
| 438 |
|
\code{x=0} vector on \code{T} corresponds to the \code{y=32} vector on |
| 439 |
|
\code{Tn}. The eastern edge of \code{T} shows the reverse case |
| 440 |
|
(\code{exch2\_oi(4,1)=-32)}), where \code{x=32} on \code{T} exchanges |
| 441 |
|
with \code{x=0} on \code{Tn=2}. \\ |
| 442 |
|
|
| 443 |
|
The most interesting case, where \code{exch2\_oi(1,1)=33} and |
| 444 |
|
\code{Tn=3}, involves a reversal of indices. As in every case, the |
| 445 |
|
offset \code{exch2\_oi} is added to the original $x$ index of \code{T} |
| 446 |
|
multiplied by the transformation factor \code{exch2\_pi(t,N,T)}. Here |
| 447 |
|
\code{exch2\_pi(1,1,1)=0} since the $x$ axis of \code{T} is orthogonal |
| 448 |
|
to the $x$ axis of \code{Tn}. \code{exch2\_pi(2,1,1)=-1} since the |
| 449 |
|
$x$ axis of \code{T} corresponds to the $y$ axis of \code{Tn}, but the |
| 450 |
|
index is reversed. The result is that the index of the northern edge |
| 451 |
|
of \code{T}, which runs \code{(1:32)}, is transformed to |
| 452 |
|
\code{(-1:-32)}. \code{exch2\_oi(1,1)} is then added to this range to |
| 453 |
|
get back \code{(32:1)} -- the index of the $y$ axis of \code{Tn} |
| 454 |
|
relative to \code{T}. This transformation may seem overly convoluted |
| 455 |
|
for the six-tile case, but it is necessary to provide a general |
| 456 |
|
solution for various topologies. \\ |
| 457 |
|
|
| 458 |
|
|
|
\subsection{Key Routines} |
|
| 459 |
|
|
| 460 |
|
Finally, \varlink{exch2\_itlo\_c}{exch2_itlo_c}, |
| 461 |
|
\varlink{exch2\_ithi\_c}{exch2_ithi_c}, |
| 462 |
|
\varlink{exch2\_jtlo\_c}{exch2_jtlo_c} and |
| 463 |
|
\varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index |
| 464 |
|
bounds of the edge segment of the neighbor tile \code{N}'s subdomain |
| 465 |
|
that gets exchanged with the local tile \code{T}. To take the example |
| 466 |
|
of tile \code{T=2} in the twelve-tile topology |
| 467 |
|
(Fig. \ref{fig:12tile}): \\ |
| 468 |
|
|
| 469 |
|
\begin{verbatim} |
| 470 |
|
exch2_itlo_c(4,2)=17 |
| 471 |
|
exch2_ithi_c(4,2)=17 |
| 472 |
|
exch2_jtlo_c(4,2)=0 |
| 473 |
|
exch2_jthi_c(4,2)=33 |
| 474 |
|
\end{verbatim} |
| 475 |
|
|
| 476 |
|
Here \code{N=4}, indicating the western neighbor, which is |
| 477 |
|
\code{Tn=1}. \code{Tn} resides on the same subdomain as \code{T}, so |
| 478 |
|
the tiles have the same orientation and the same $x$ and $y$ axes. |
| 479 |
|
The $x$ axis is orthogonal to the western edge and the tile is 16 |
| 480 |
|
points wide, so \code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} |
| 481 |
|
indicate the column beyond \code{Tn}'s eastern edge, in that tile's |
| 482 |
|
halo region. Since the border of the tiles extends through the entire |
| 483 |
|
height of the subdomain, the $y$ axis bounds \code{exch2\_jtlo\_c} to |
| 484 |
|
\code{exch2\_jthi\_c} cover the height of \code{(1:32)}, plus 1 in |
| 485 |
|
either direction to cover part of the halo. \\ |
| 486 |
|
|
| 487 |
|
For the north edge of the same tile \code{T=2} where \code{N=1} and |
| 488 |
|
the neighbor tile is \code{Tn=5}: |
| 489 |
|
|
| 490 |
|
\begin{verbatim} |
| 491 |
|
exch2_itlo_c(1,2)=0 |
| 492 |
|
exch2_ithi_c(1,2)=0 |
| 493 |
|
exch2_jtlo_c(1,2)=0 |
| 494 |
|
exch2_jthi_c(1,2)=17 |
| 495 |
|
\end{verbatim} |
| 496 |
|
|
| 497 |
|
\code{T}'s northern edge is parallel to the $x$ axis, but since |
| 498 |
|
\code{Tn}'s $y$ axis corresponds to \code{T}'s $x$ axis, \code{T}'s |
| 499 |
|
northern edge exchanges with \code{Tn}'s western edge. The western |
| 500 |
|
edge of the tiles corresponds to the lower bound of the $x$ axis, so |
| 501 |
|
\code{exch2\_itlo\_c} and \code{exch2\_ithi\_c} are \code{0}, in the |
| 502 |
|
western halo region of \code{Tn}. The range of |
| 503 |
|
\code{exch2\_jtlo\_c} and \code{exch2\_jthi\_c} correspond to the |
| 504 |
|
width of \code{T}'s northern edge, expanded by one into the halo. \\ |
| 505 |
|
|
| 506 |
|
|
| 507 |
|
\subsection{Key Routines} |
| 508 |
|
|
| 509 |
|
Most of the subroutines particular to exch2 handle the exchanges |
| 510 |
|
themselves and are of the same format as those described in |
| 511 |
|
\ref{sect:cube_sphere_communication} \sectiontitle{Cube sphere |
| 512 |
|
communication}. Like the original routines, they are written as |
| 513 |
|
templates which the local Makefile converts from \code{RX} into |
| 514 |
|
\code{RL} and \code{RS} forms. \\ |
| 515 |
|
|
| 516 |
|
The interfaces with the core model subroutines are |
| 517 |
|
\code{EXCH\_UV\_XY\_RX}, \code{EXCH\_UV\_XYZ\_RX} and |
| 518 |
|
\code{EXCH\_XY\_RX}. They override the standard exchange routines |
| 519 |
|
when \code{genmake2} is run with \code{exch2} option. They in turn |
| 520 |
|
call the local exch2 subroutines \code{EXCH2\_UV\_XY\_RX} and |
| 521 |
|
\code{EXCH2\_UV\_XYZ\_RX} for two and three-dimensional vector |
| 522 |
|
quantities, and \code{EXCH2\_XY\_RX} and \code{EXCH2\_XYZ\_RX} for two |
| 523 |
|
and three-dimensional scalar quantities. These subroutines set the |
| 524 |
|
dimensions of the area to be exchanged, call \code{EXCH2\_RX1\_CUBE} |
| 525 |
|
for scalars and \code{EXCH2\_RX2\_CUBE} for vectors, and then handle |
| 526 |
|
the singularities at the cube corners. \\ |
| 527 |
|
|
| 528 |
|
The separate scalar and vector forms of \code{EXCH2\_RX1\_CUBE} and |
| 529 |
|
\code{EXCH2\_RX2\_CUBE} reflect that the vector-handling subroutine |
| 530 |
|
needs to pass both the $u$ and $v$ components of the physical vectors. |
| 531 |
|
This swapping arises from the topological folding discussed above, where the |
| 532 |
|
$x$ and $y$ axes get swapped in some cases, and is not an |
| 533 |
|
issue with the scalar case. These subroutines call |
| 534 |
|
\code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of |
| 535 |
|
the work using the variables discussed above. \\ |
| 536 |
|
|
|
\subsection{References} |
|
Parent Directory
| 
Revision Log
| 
Revision Graph
| 
Patch