Any chance of some help with a Linux Shell Script please?
Discussion
I haveaprocess that is writing log files to a directory, the log file gets closed every 15 minutes and a new one started so I end up with a heap of files named
host_date_time_index.state
e.g.
burum_20051103-1115_00000.state
burum_20051103-1130_00000.state
burum_20051103-1145_00000.state
burum_20051103-1200_00000.state
each file is marked with SOF and EOF at the beginning and end respectively.
I need a shell script that given some time and date paramters will cat all the files between the specified dates - this I can do.
However, given only a start date and time, when it reaches the (last) most current file, it will tail instead of cat, until it sees the EOF and then move onto the next file tailing again.
Is there any quick way to get this functionallity? I guess the tail needs to pipe into awk or something such that when it sees the EOF is stops the curernt tail and calculates the next file name it should be looking for (based o the 15 minute interval) and starts up again.
What's worse is that these files are not all in the one directory, they are split daily, so if I specify last night 23:00 as the start date, when it reaches the last file in the 200601/25 directory it needs to go to the 200601/26 directory and carry on cating through all the files in there until it reaches the current one, then it needs to switch to tail and progress fro mthere until stopped.
Any bright ideas?
Many thanks
Ex
host_date_time_index.state
e.g.
burum_20051103-1115_00000.state
burum_20051103-1130_00000.state
burum_20051103-1145_00000.state
burum_20051103-1200_00000.state
each file is marked with SOF and EOF at the beginning and end respectively.
I need a shell script that given some time and date paramters will cat all the files between the specified dates - this I can do.
However, given only a start date and time, when it reaches the (last) most current file, it will tail instead of cat, until it sees the EOF and then move onto the next file tailing again.
Is there any quick way to get this functionallity? I guess the tail needs to pipe into awk or something such that when it sees the EOF is stops the curernt tail and calculates the next file name it should be looking for (based o the 15 minute interval) and starts up again.
What's worse is that these files are not all in the one directory, they are split daily, so if I specify last night 23:00 as the start date, when it reaches the last file in the 200601/25 directory it needs to go to the 200601/26 directory and carry on cating through all the files in there until it reaches the current one, then it needs to switch to tail and progress fro mthere until stopped.
Any bright ideas?
Many thanks
Ex
TheExcession said:
Any bright ideas?
Many thanks
Ex
yeah, the shell script needs to be able to tell what the time is and that there are 15 minute intervals based on each 15 minutes - which should be easy enough to do.
you then need the script to say "if the time is 12:37" then i need to tail file xxxxxxxx12-30 rather than cat it.
thanks guys, the problem I'm having at the moment is braking out of the tail command.
I thought maybe by piping tail into say awk, when awk saw the EOF marker it could break out and also stop tail from running....
Not so, just looking at a way to get tail to give up when the other process has gone off and started a new file.
I thought maybe by piping tail into say awk, when awk saw the EOF marker it could break out and also stop tail from running....
Not so, just looking at a way to get tail to give up when the other process has gone off and started a new file.
tinman0 said:
Any bright ideas?
Many thanks
Ex
yeah, the shell script needs to be able to tell what the time is and that there are 15 minute intervals based on each 15 minutes - which should be easy enough to do.
you then need the script to say "if the time is 12:37" then i need to tail file xxxxxxxx12-30 rather than cat it.
That only works if the program generating the logs does things at exact times, and even then you have problems at the changeover times, since the closure of the log file and opening of the new one is not atomic.
Like "Mr. E" says, I'd throw away the "tail" command and do the "tail" in perl;
#!/usr/bin/perl
use File::Tail;
$file=File::Tail->new( $file_to_tail );
while (defined($line=$file->read))
{
print $line;
}
Now you have the logfile lines inside your own code so you can do what you like with it.
In case you haven't guessed, this problme is slightly non-trivial!
#!/usr/bin/perl
use File::Tail;
$file=File::Tail->new( $file_to_tail );
while (defined($line=$file->read))
{
print $line;
}
Now you have the logfile lines inside your own code so you can do what you like with it.
In case you haven't guessed, this problme is slightly non-trivial!
so, should I be looking at spawning a monitoring process?
I set one process up to tail the current file - this will provde the output I need, then, another to periodically look to see if the EOF last line indicator in the monitored file has appeard, if it has ten kill the tailing process and start a new one with the next filename.
Does this seem like the way one would tackle this kind of problem?
I set one process up to tail the current file - this will provde the output I need, then, another to periodically look to see if the EOF last line indicator in the monitored file has appeard, if it has ten kill the tailing process and start a new one with the next filename.
Does this seem like the way one would tackle this kind of problem?
I think you want to do something like this (n.b. not on a unix box, so excuse all the syntax errors and stupid mistakes)
monitor_task()
{
..complete=0
..while [ $complete ]; do
....tail -l 1 $1 | grep "EOF" > /dev/null
....if [ $? ]; then
......find and kill the tail process
......complete=1
....else
......sleep 1m
....fi
..done
}
/main thread stuff/
monitor_task $current_file &
tail -f $(current_file)
/more main thread stuff/
this method isn't nice, but it's all i can think of in shell script.
monitor_task()
{
..complete=0
..while [ $complete ]; do
....tail -l 1 $1 | grep "EOF" > /dev/null
....if [ $? ]; then
......find and kill the tail process
......complete=1
....else
......sleep 1m
....fi
..done
}
/main thread stuff/
monitor_task $current_file &
tail -f $(current_file)
/more main thread stuff/
this method isn't nice, but it's all i can think of in shell script.
The code fragment I posted above would be the basis of this. Instead of the "print $line" you could do something like (very much 'ad hoc' and off the top of my head!);
if ($line == EOF)
{
# End of current file
close $file;
system ("cat $file >> $archivefile"
; # Deal with the file you just finished with (1)
$file = newfile; # newfile is a function you write that returns the name of the next file (2)
open $file || die "No new logfile to read";
}
else
{
print $file $line; # Output the line just read to the current file
}
(1) This is where you decide what to do with the file you've just finished reading - write it to archive, or whatever.
(2) This is where you open the next logfile - you might want to loop and wait for the app to open the next file, or whatever.
What you're doing is just reading lines, looking for EOFs. You don't have to decide whether to terminate the tail or whatever, this app just sucks in lines from successive log files and when it sees an EOF, does things with the file it's just seen and opens the next one.
>> Edited by zumbruk on Friday 27th January 11:44
if ($line == EOF)
{
# End of current file
close $file;
system ("cat $file >> $archivefile"
; # Deal with the file you just finished with (1) $file = newfile; # newfile is a function you write that returns the name of the next file (2)
open $file || die "No new logfile to read";
}
else
{
print $file $line; # Output the line just read to the current file
}
(1) This is where you decide what to do with the file you've just finished reading - write it to archive, or whatever.
(2) This is where you open the next logfile - you might want to loop and wait for the app to open the next file, or whatever.
What you're doing is just reading lines, looking for EOFs. You don't have to decide whether to terminate the tail or whatever, this app just sucks in lines from successive log files and when it sees an EOF, does things with the file it's just seen and opens the next one.
>> Edited by zumbruk on Friday 27th January 11:44
Whoops, just noticed a pretty terminal bug! The line that reads;
print $file $line; # Output the line just read to the current file
Should read something like;
print $outfile $line; # Output the line just read to the current file
"$file" is the name of the log file you're "tailing", so you don't want to output to it!!!
print $file $line; # Output the line just read to the current file
Should read something like;
print $outfile $line; # Output the line just read to the current file
"$file" is the name of the log file you're "tailing", so you don't want to output to it!!!
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff