( last edited : October 14th, 2014 -- author : Gael Forget )

---------------------------------------------------------------
A) setup the experiments (incl. llc90 and cs32 core2.cnyf runs) 
---------------------------------------------------------------

0) get MITgcm as explained @ http://mitgcm.org/public/source_code.html
1) the following assumes that you are in directory MITgcm/verification/
   i.e., after checkout of the model repository, do:
      cd MITgcm/verification/
   from here, get the MITgcm_contrib/gael/verification/setup_these_exps.csh
   c-shell setup script e.g. by typing
      cvs co MITgcm_contrib/gael/verification/setup_these_exps.csh
      mv MITgcm_contrib/gael/verification/setup_these_exps.csh .
   Unless you already had something in MITgcm_contrib you may want 
   to remove that directory. Hereafter we wont need it anymore.
2) execute the setup script by typing
      source ./setup_these_exps.csh
   A pre-requisite is that you are logged in to the MITgcm 
   cvs server. If you completed step1, you indeed already are.
   If otherwise, see http://mitgcm.org/public/source_code.html
3) step2 creates a directory global_oce_tmp_download that 
   contains the experiments directories. Move them (incl. 
   global_oce_input_fields) to MITgcm/verification. Once you 
   have done this succesfully you may want to remove the empty
   global_oce_tmp_download directory. We wont need it anymore. 
4) exectute MITgcm with testreport. Two examples from one machine.
   To run the cs32 test on one cpu :
       ./testreport -t global_oce_cs32
   And to run the llc90 test on 24 cpus, using mpi
      ./testreport -of=../tools/build_options/linux_amd64_gfortran -MPI 24 -command 'mpirun -np TR_NPROC ./mitgcmuv' -t 'global_oce_cs32 global_oce_llc90'
   The optfile may be swaped for one that better suits your machine.

Notes: - when using mpi and netcdf, testreport expects two environment variables :
         export MPI_INC_DIR=/opt/local/include
         export NETCDF_ROOT=/opt/local
         or something similar depending on your computing environment
       - whether with testreport or not mpi and netcdf libraries are a pre-requisite
5) upon completion of the experiments, testreport should return something like
    Y Y Y Y>16<16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16  .  . pass  global_oce_cs32
    Y Y Y Y> 7<16 16 12 12 16 16 13 11 13 12 10 11 13 13 11 11  .  . FAIL  global_oce_llc90
    Y Y Y Y>10<16 13 12 12 13 16 14 11 13 13 11 11 13 13 11 11  .  . pass  global_oce_llc90.core2
    Y Y Y Y> 4<13 11  9  9 12 16 11  9 10 12 10 10 11 10  8 10  .  . FAIL  global_oce_llc90.ecco_v4
    Y Y Y Y> 7<10 11 12 11 12 11 13 10 10 12 11 11 12 11 11 10  .  . FAIL  global_oce_llc90.ecmwf
   implying that the main experiment ran as expected in global_oce_llc90/run.

-----------------------------------------------------------------------
B) in case you want to change stuff on your end, or for troubleshooting
-----------------------------------------------------------------------

1) In linking the experiments to one another we rely on 'build/genmake_local' and 
'input*/prepare_run' which may be worth a special mention, at least to new MITgcm users.

  1.1) most importantly : if you remove these files, the exps wont run or even compile.
  1.2) build/genmake_local specifies the use of xmakedepend (since the default 
       cant handle so many files) and in global_oce_cs32 it points to 
       "../../global_oce_llc90/code/" If you move global_oce_llc90 away from 
       global_oce_cs32 you will need to edit that path in genmake_local.
  1.3) in analogy input*/prepare_run are a set of instructions that "testreport"
       will execute before running MITgcm, to link all of the proper inputs. 
       And again the experiments form a self-consistent group, such that
       e.g. global_oce_cs32 will get shared files from global_oce_llc90. If you move
       the directories appart, you will need to edit the paths in input*/prepare_run
  1.4) the logic we followed in linking exps to one another :
           - forward subexperiments : get input.xxx/namelists then run input/prepare_run
           - adjoint subexperiments : get input_ad.xxx/namelists then run corresponding forward 
           subexperiment input.xxx/prepare_run (that itself ultimately runs input/prepare_run)

2) In a couple .h files (that you find in the build directory after compiling)
   small size parameters are used as a default (to save memory of course)
   and they can become limiting in certain cases :
   - you may want to integrate the model beyond a few time steps. You will need to 
     increase nchklev_1/_2/_3 in tamc.h (e.g. use the values in tamc.h_itXX) and recompile.
   - you may want to use pkg/profiles (i.e. process insitu data via data.profiles). 
     You will need to increase NOBSGLOB in profiles.h (e.g. use profiles.h_itXX), and recompile.

3) To run the adjoint subexperiments you need the taf/tamc software/license.

---------------------------------------------------------------
---------------------------------------------------------------