high_res_cube/eddy_flux/data_transfer.txt


Getting the "cube_5" data from NAS (NASA Ames)
==============================================

Due mostly to the latency of long-distance connections, its rather
inefficient to copy large files one at a time using a single "scp"
command [1].  In order to speed-up and automate the process of
transferring a large number of large files, a multi-threaded scp
command wrapper called "mt_scp" was written.  To build mt_scp, issue
the following commands

   export CVSROOT=':pserver:cvsanon@mitgcm.org:/u/gcmpack'
   cvs login
     <== "cvsanon"
   cvs co development/edhill/mt_scp
   cd development/edhill/mt_scp/
   ./configure 
   make

at which point the directory should contain an "mt_scp" executable.
Copy that executable to a suitable install location or wherever you
need to use it.

To use mt_scp, it is best to first setup so-called "password-less"
(public-key) SSH authentication between the two hosts.  The
configuration of public-key SSH authentication is described on many
helpful on-line sites [2] and takes only a few minutes.

To use mt_scp, some example commands are:

  #  Get a description of all the mt_scp options
  #
  ./mt_scp

  #  Copy all files matching the pattern "Stave.00*" at the 
  #  location lou.nas.nasa.gov:/ecco/menemenl/cube5/output 
  #  using 8 concurrent threads (note how the file pattern
  #  or "glob" has been escaped so that expansion only 
  #  on the "remote" side):
  #
  ./mt_scp -n 8 -p 'Stave\*' lou.nas.nasa.gov /ecco/menemenl/cube5/output


Please send questions about mt_scp to Ed Hill <eh3@mit.edu>.


References
==========

[1] Speedups of 10x--11.5x have been routinely observed:
      http://mitgcm.org/eh3/lou.nas.nasa.gov/
      http://mitgcm.org/eh3/lou.nas.nasa.gov/lou_data_xfer.png

[2] http://www-106.ibm.com/developerworks/library/l-keyc.html
    http://cfm.gs.washington.edu/security/ssh/client-pkauth/

1	edhill	1.1
2			Getting the "cube_5" data from NAS (NASA Ames)
3			==============================================
4
5			Due mostly to the latency of long-distance connections, its rather
6			inefficient to copy large files one at a time using a single "scp"
7			command [1]. In order to speed-up and automate the process of
8			transferring a large number of large files, a multi-threaded scp
9			command wrapper called "mt_scp" was written. To build mt_scp, issue
10			the following commands
11
12			export CVSROOT=':pserver:cvsanon@mitgcm.org:/u/gcmpack'
13			cvs login
14			<== "cvsanon"
15			cvs co development/edhill/mt_scp
16			cd development/edhill/mt_scp/
17			./configure
18			make
19
20			at which point the directory should contain an "mt_scp" executable.
21			Copy that executable to a suitable install location or wherever you
22			need to use it.
23
24			To use mt_scp, it is best to first setup so-called "password-less"
25			(public-key) SSH authentication between the two hosts. The
26			configuration of public-key SSH authentication is described on many
27			helpful on-line sites [2] and takes only a few minutes.
28
29			To use mt_scp, some example commands are:
30
31			# Get a description of all the mt_scp options
32			#
33			./mt_scp
34
35			# Copy all files matching the pattern "Stave.00*" at the
36			# location lou.nas.nasa.gov:/ecco/menemenl/cube5/output
37			# using 8 concurrent threads (note how the file pattern
38			# or "glob" has been escaped so that expansion only
39			# on the "remote" side):
40			#
41			./mt_scp -n 8 -p 'Stave\*' lou.nas.nasa.gov /ecco/menemenl/cube5/output
42
43
44			Please send questions about mt_scp to Ed Hill <eh3@mit.edu>.
45
46
47			References
48			==========
49
50			[1] Speedups of 10x--11.5x have been routinely observed:
51			http://mitgcm.org/eh3/lou.nas.nasa.gov/
52			http://mitgcm.org/eh3/lou.nas.nasa.gov/lou_data_xfer.png
53
54			[2] http://www-106.ibm.com/developerworks/library/l-keyc.html
55			http://cfm.gs.washington.edu/security/ssh/client-pkauth/
56