JIT-DT

JIT-DT (Just-In-Time Data Transfer)

The JIT-DT (Just-In-Time Data Transfer) software was designed for transferring a set of data generated at observatories to a supercomputer site and distribute it to time-critical applications. It consists of watch_and_transfer, kwatcher, and lwacther commands.

watch_and_transfer

It watches a specified directory. When a new file is created under the directory, the file is transferred to the remote computer immediately. When a new directory is created under the watched directory, this directory is also watched directory. All new directories under the watched directories are also watched directories. Using the “inotify” Linux system call, the command does not consume CPU cycles for watching directories. Transferring selective and synchronized files
A file name may consit of generated date and data type. It uses selective and synchronous files transfer. An example is shown below:
region kobe {
pattern = ".*_\\(.*\\)_A08_pawr_\\(.*\\).dat";
date = "$1";
type = "$2";
fname = "kobe_$1_$2.dat";
}
sync {
type = "vr";
type = "ze";
}
The “region” keyword followed by a site name specifies processing rules of a file name generated at the site. This example specifies the rules at “kobe” observatory.
The “pattern” keyword specifies a file name pattern using a regular expression. The “date” and “type” keywords specify which matched patterns are used for date and type, respectively. The definition with “sync” keyword defines which files generated at the same time are transferred.
We assume that files whose names have the same date may not arrive at the same time due to different post processing time. For example, the following two files whose data type is “ze” has been created, but no “vr” files arrive for date “20190509171200”.
kobe_20190509171200_A08_pawr_ze.dat
kobe_20190509171230_A08_pawr_ze.dat
In this case, the watch_and_transfer command waits for the following file.
kobe_20190509171200_A08_pawr_vr.dat

kwatcher and lwatcher

The kwatcher command receives files transferred by the watch_and_transfer command and distributes them to applications via local files. The lwatcher command receives files transferred by the watch_and_transfer command and distributes them to applications via TCP/IP connections. For more detailed, please read the JIT-DT documentation in the distribution.