bash looping over files while tailing logs

I'm running a daemon that process files in a specific dir. I want to process those one by one. so idea is I will copy files 1 by 1 in daemon dir, and tail its log, 2nd file should be copied to daemon dir when log file finishes processing first, i.e, log file stops updating.

how can I achieve that? say

daemon_dir=/var/daemon_dir/
daemon_log_file=/var/daemon/daemon_log

and I have 5 files to process i.e, file{1..5}

# ls -1
file1
file2
file3
file4
file5

so my loop should go as is.

for file in file{1..5}
do
  cp ${file} ${daemon_dir}
  # daemon sees new file arriving in its dir, & starts processing
  # log is tailed
  tail -f ${daemon_log_file}
  # when log file stops updating say for 1minute or specified time, tail quits
done

and loop starts with next file.

Answers 1

  • With zsh instead of bash, you could do:

    zmodload zsh/zselect
    for file in file{1..5}; do
      cp -- $file $daemon_dir || exit
      touch -- $daemon_log_file
      while
        ()(($#)) $daemon_log_file(Nmm-1)
      do
        zselect -t 100
      done
    done
    

    (zselect -t 100 sleeps for 1 second. Like sleep 1 except zselect is builtin. tail -f typically also checks for new input in the file every second (though on Linux some implementations use inotify instead to detect incoming input)).

    Above, we use the mm-1 (modified less than 1 minute ago) glob qualifier to detect when the daemon has finished processing the file.

    If on Linux, you could monitor for the IN_CLOSE_NOWRITE inotify event on the file instead, assuming the daemon opens the input files only once (and is the only one opening it) and takes enough time to process the file so we have time to setup the watch

    for file in file{1..5}; do
      cp -- "$file" "$daemon_dir" || exit
      inotifywait -qqe close_nowrite -- "$daemon_dir/$file" 
    done
    

    For a tail -f-based approach, you could do (with zsh) something like:

    zmodload zsh/system
    tail -f -- $daemon_log_file |
      for file in file{1..5}; do
        cp -- $file $daemon_dir || exit
        while sysread -t 60 -o 1; do
          continue
        done
      done
    

Related Questions