verification/global_ocean.90x40x15/README

============================================
Example: "4x4 Global Simulation with Seasonal Forcing"
============================================
(see also similar set-up in: verification/tutorial_global_oce_latlon/)

From verification/global_ocean.90x40x15 dir:

Configure and compile the code:
  cd build
  ../../../tools/genmake2 -mods ../code [-of my_platform_optionFile]
 [make Clean]
  make depend
  make
  cd ..

To run:
  cd run
  ln -s ../input/* .
  ../input/prepare_run
  ln -s ../build/mitgcmuv .
  ./mitgcmuv > output.txt
  cd ..

There is comparison output in the directory:
  results/output.txt

There is comparison output in directory:
  (verification/global_ocean.90x40x1/) results

Comments:
o The input data is real*4.
o The surface fluxes are derived from monthly means of the NCEP climatology;
  - a matlab script is provided that created the surface flux data files from
    the original NCEP data: ncep2global_ocean.m in the diags_matlab directory,
    needs editing to adjust search paths.
o matlab scripts that make a simple diagnostic (barotropic stream function,
  overturning stream functions, averaged hydrography etc.) is provided in
  verification/tutorial_global_oce_latlon/diags_matlab:
  - mit_loadglobal is the toplevel script that run all other scripts
  - mit_globalmovie animates theta, salinity, and 3D-velocity field for
    a layer "iz", if "meanfields=0"

--------------------------------------------
Additional example:
  similar set-up, with the same executable, and using pkg/dwnslp:
to run this 2nd example:
  cd input.dwnslp
  ln -s ../input/* .
  ../input/prepare_run
  ../build/mitgcmuv > output.dwnslp.txt
  cd ..


============================================
Use of "blank tiles" in conjunction with exch2 package:
============================================

This verification experiment also demonstrate the omission of tiles
(or processors) for tiles that are fully land cover and don't need computation.
The relevant config. files to be manipulated are:
* at compile time, in dir. code/: packages.conf, SIZE.h
* at run time, in dir. input/: data.exch2

To enable this feature requires package "exch2"
(see Section 6.2.4 "exch2: Extended Cubed Sphere Topology" of online manual),
i.e. in code/packages.conf add "exch2"

The basic layout of the experiment is Nx*Ny = 90x40.
In a single-processor config. with very small tile sizes (sNx*sNy=10*10)
this can be represented, e.g. via
     &           sNx =  10,
     &           sNy =  10,
     &           OLx =   3,
     &           OLy =   3,
     &           nSx =   9,
     &           nSy =   4,
     &           nPx =   1,
     &           nPy =   1,
i.e. we use nSx*nSy=9*4=36 virtual processors.

An equivalent parallel setup using 9 virtual and 4 real processors
would look like:
     &           sNx =  10,
     &           sNy =  10,
     &           OLx =   3,
     &           OLy =   3,
     &           nSx =   9,
     &           nSy =   1,
     &           nPx =   1,
     &           nPy =   4,

In this layout it turns out that tile number 30 is "empty", i.e. fully land covered.
We wish to remove this tile from our calculation. How to proceed?

1. Find out which tiles to eliminate via a config. that uses all tiles
---
1.1. At compile time:
* add line "exch2" to packages.conf
* configure SIZE.h using your desired individual tile size, e.g. sNx*sNy=10*10, as follows:
     &           sNx =  10,
     &           sNy =  10,
     &           OLx =   3,
     &           OLy =   3,
     &           nSx =   9,
     &           nSy =   1,
     &           nPx =   1,
     &           nPy =   4,
  As described above you are using 4 real processors with 9 virtual tiles per processor.
* compile (don't forget to compile via -mpi or similar)

1.2 At runtime:
* need to reflect your basic layout in data.exch2
  This is simple: since you are not using any non-trivial topology with
  multiple facets (such as cubed-sphere),
  you only need to specify one basic facet (Nx,Ny)=(90,40) layout via following line:
  dimsFacets = 90, 40,
  Then run the moodel, e.g. via:
  mpirun -np 4 ./mitgcmuv

1.3 Diagnose which tile numbers are empty:
* from STDOUT.000[0-3] you can infer which tiles are empty using following grep:
  grep "Empty tile" STDOUT.* | awk '{print "  " $6 ","}' > empty_tiles.txt 
* In this example there's only one empty tile, and it is #30.

2. Configuring with empty tiles removed
---
2.1 At compile time:
   We've determined that 1 out of nSx*nPx*nSy*nPy=36 tiles is empty and can be removed,
   leaving 36-1=35 non-empty tiles. We are free to re-order nSx,nPx,nSy,nPy in any way,
   as long as nSx*nPx*nSy*nPy=35. Here's how it's chosen in the verif. experiment
   (see file SIZE.h_mpi)
     &           sNx =  10,
     &           sNy =  10,
     &           OLx =   3,
     &           OLy =   3,
     &           nSx =   7,
     &           nSy =   1,
     &           nPx =   1,
     &           nPy =   5,
   and for which, as required nSx*nPx*nSy*nPy=35

2.2 At runtime:
We now need to specify in data.exch2 the number of the empty tile to be removed.
This is done as follows (see file data.exch2.mpi):
  blankList  = 30,
If there were more empty tiles, this would be a more extensive index array, e.g.
  blankList = tile1, tile2, tile3, ... 
Now run the model (note that we've selected nPy=5, so in this example we actually
*increase* the number of "real" processors used, despite reducing the number of
total, i.e. "real" plus virtual tiles from 36 to 35):
mpirun -np 5 ./mitgcmuv


============================================
Adjoint set-up example:
============================================

Configure and compile the code:
  cd build
  ../../../tools/genmake2 -mods='../code_ad'
 [make Clean]
  make depend
  make adall
  cd ..
To run the code:
  cd input_ad
  ./prepare_run
  ../build/mitgcmuv_ad > output_adm.txt
  cd ..
1	============================================
2	Example: "4x4 Global Simulation with Seasonal Forcing"
3	============================================
4	(see also similar set-up in: verification/tutorial_global_oce_latlon/)
5
6	From verification/global_ocean.90x40x15 dir:
7
8	Configure and compile the code:
9	cd build
10	../../../tools/genmake2 -mods ../code [-of my_platform_optionFile]
11	[make Clean]
12	make depend
13	make
14	cd ..
15
16	To run:
17	cd run
18	ln -s ../input/* .
19	../input/prepare_run
20	ln -s ../build/mitgcmuv .
21	./mitgcmuv > output.txt
22	cd ..
23
24	There is comparison output in the directory:
25	results/output.txt
26
27	There is comparison output in directory:
28	(verification/global_ocean.90x40x1/) results
29
30	Comments:
31	o The input data is real*4.
32	o The surface fluxes are derived from monthly means of the NCEP climatology;
33	- a matlab script is provided that created the surface flux data files from
34	the original NCEP data: ncep2global_ocean.m in the diags_matlab directory,
35	needs editing to adjust search paths.
36	o matlab scripts that make a simple diagnostic (barotropic stream function,
37	overturning stream functions, averaged hydrography etc.) is provided in
38	verification/tutorial_global_oce_latlon/diags_matlab:
39	- mit_loadglobal is the toplevel script that run all other scripts
40	- mit_globalmovie animates theta, salinity, and 3D-velocity field for
41	a layer "iz", if "meanfields=0"
42
43	--------------------------------------------
44	Additional example:
45	similar set-up, with the same executable, and using pkg/dwnslp:
46	to run this 2nd example:
47	cd input.dwnslp
48	ln -s ../input/* .
49	../input/prepare_run
50	../build/mitgcmuv > output.dwnslp.txt
51	cd ..
52
53
54	============================================
55	Use of "blank tiles" in conjunction with exch2 package:
56	============================================
57
58	This verification experiment also demonstrate the omission of tiles
59	(or processors) for tiles that are fully land cover and don't need computation.
60	The relevant config. files to be manipulated are:
61	* at compile time, in dir. code/: packages.conf, SIZE.h
62	* at run time, in dir. input/: data.exch2
63
64	To enable this feature requires package "exch2"
65	(see Section 6.2.4 "exch2: Extended Cubed Sphere Topology" of online manual),
66	i.e. in code/packages.conf add "exch2"
67
68	The basic layout of the experiment is Nx*Ny = 90x40.
69	In a single-processor config. with very small tile sizes (sNxsNy=1010)
70	this can be represented, e.g. via
71	& sNx = 10,
72	& sNy = 10,
73	& OLx = 3,
74	& OLy = 3,
75	& nSx = 9,
76	& nSy = 4,
77	& nPx = 1,
78	& nPy = 1,
79	i.e. we use nSxnSy=94=36 virtual processors.
80
81	An equivalent parallel setup using 9 virtual and 4 real processors
82	would look like:
83	& sNx = 10,
84	& sNy = 10,
85	& OLx = 3,
86	& OLy = 3,
87	& nSx = 9,
88	& nSy = 1,
89	& nPx = 1,
90	& nPy = 4,
91
92	In this layout it turns out that tile number 30 is "empty", i.e. fully land covered.
93	We wish to remove this tile from our calculation. How to proceed?
94
95	1. Find out which tiles to eliminate via a config. that uses all tiles
96	---
97	1.1. At compile time:
98	* add line "exch2" to packages.conf
99	* configure SIZE.h using your desired individual tile size, e.g. sNxsNy=1010, as follows:
100	& sNx = 10,
101	& sNy = 10,
102	& OLx = 3,
103	& OLy = 3,
104	& nSx = 9,
105	& nSy = 1,
106	& nPx = 1,
107	& nPy = 4,
108	As described above you are using 4 real processors with 9 virtual tiles per processor.
109	* compile (don't forget to compile via -mpi or similar)
110
111	1.2 At runtime:
112	* need to reflect your basic layout in data.exch2
113	This is simple: since you are not using any non-trivial topology with
114	multiple facets (such as cubed-sphere),
115	you only need to specify one basic facet (Nx,Ny)=(90,40) layout via following line:
116	dimsFacets = 90, 40,
117	Then run the moodel, e.g. via:
118	mpirun -np 4 ./mitgcmuv
119
120	1.3 Diagnose which tile numbers are empty:
121	* from STDOUT.000[0-3] you can infer which tiles are empty using following grep:
122	grep "Empty tile" STDOUT.* \| awk '{print " " $6 ","}' > empty_tiles.txt
123	* In this example there's only one empty tile, and it is #30.
124
125	2. Configuring with empty tiles removed
126	---
127	2.1 At compile time:
128	We've determined that 1 out of nSxnPxnSy*nPy=36 tiles is empty and can be removed,
129	leaving 36-1=35 non-empty tiles. We are free to re-order nSx,nPx,nSy,nPy in any way,
130	as long as nSxnPxnSy*nPy=35. Here's how it's chosen in the verif. experiment
131	(see file SIZE.h_mpi)
132	& sNx = 10,
133	& sNy = 10,
134	& OLx = 3,
135	& OLy = 3,
136	& nSx = 7,
137	& nSy = 1,
138	& nPx = 1,
139	& nPy = 5,
140	and for which, as required nSxnPxnSy*nPy=35
141
142	2.2 At runtime:
143	We now need to specify in data.exch2 the number of the empty tile to be removed.
144	This is done as follows (see file data.exch2.mpi):
145	blankList = 30,
146	If there were more empty tiles, this would be a more extensive index array, e.g.
147	blankList = tile1, tile2, tile3, ...
148	Now run the model (note that we've selected nPy=5, so in this example we actually
149	increase the number of "real" processors used, despite reducing the number of
150	total, i.e. "real" plus virtual tiles from 36 to 35):
151	mpirun -np 5 ./mitgcmuv
152
153
154	============================================
155	Adjoint set-up example:
156	============================================
157
158	Configure and compile the code:
159	cd build
160	../../../tools/genmake2 -mods='../code_ad'
161	[make Clean]
162	make depend
163	make adall
164	cd ..
165	To run the code:
166	cd input_ad
167	./prepare_run
168	../build/mitgcmuv_ad > output_adm.txt
169	cd ..