Sort logs by date field in bash
For GNU sort: sort -k2M -k3n -k4
-k2M
sorts by second column by month (this way "March" comes before "April")-k3n
sorts by third column in numeric mode (so that " 9" comes before "10")-k4
sorts by the fourth column.
See more details in the manual.
How to sort data according to the date in bash?
The relevant field be must rendered suitable for sorting, that is, in the form of YYYY-MM-DD, using a utility such as sed
or awk
. For example, with GNU sed
:
sed -E 's/([0-9]{2})-([0-9]{2})-([0-9]{4})/\3-\2-\1/' employees.txt |
sort -r -t'|' -k5,5 | head -n1 | cut -d'|' -f2
sort by datetime format in bash
Sort is able to deal with month names, thanks to the option M
No need to change ,
into !
. Use the white space as delimiter and just issue:
LC_ALL=en sort -k7nr -k5Mr -k6nr -k2r sample
If you use this as content of the file sample
:
ip=2.3.4.5, setup_time=05:59:30.260 GMT Tue Apr 1 2021, foo=moshe2, bar=haim2
ip=2.3.4.5, setup_time=05:59:30.260 GMT Tue Mar 17 2021, foo=moshe2, bar=haim2
ip=1.2.3.4, setup_time=06:58:38.617 GMT Tue Mar 16 2021, foo=moshe, bar=haim
ip=1.2.3.4, setup_time=06:58:38.617 GMT Tue Feb 28 2021, foo=moshe, bar=haim
ip=2.3.4.5, setup_time=06:50:30.260 GMT Tue Mar 18 2020, foo=moshe2, bar=haim2
ip=2.3.4.5, setup_time=06:50:30.260 GMT Tue Mar 18 2021, foo=moshe2, bar=haim2
you will get this as output:
ip=2.3.4.5, setup_time=05:59:30.260 GMT Tue Apr 1 2021, foo=moshe2, bar=haim2
ip=2.3.4.5, setup_time=06:50:30.260 GMT Tue Mar 18 2021, foo=moshe2, bar=haim2
ip=2.3.4.5, setup_time=05:59:30.260 GMT Tue Mar 17 2021, foo=moshe2, bar=haim2
ip=1.2.3.4, setup_time=06:58:38.617 GMT Tue Mar 16 2021, foo=moshe, bar=haim
ip=1.2.3.4, setup_time=06:58:38.617 GMT Tue Feb 28 2021, foo=moshe, bar=haim
ip=2.3.4.5, setup_time=06:50:30.260 GMT Tue Mar 18 2020, foo=moshe2, bar=haim2
Specifying -k7
means to sort on the seventh field. The r
option reverses the order of sorting to descending. The M
option sorts according the name of the month. The n
option sorts numerically. To sort on the time, just consider the whole second field (beginning with the string setup_time=
) as a fixed length string using -k2
.
LC_ALL=en
in the begin of the command line tells the system to use the English names of the months.
How to sort data based on date field by excluding header
The way do to this is with Command Grouping where you can extract the header from an input stream, print it, and consume the remaining data:
{
IFS= read -r header
echo "$header"
sort ...
} < file.txt
However, sorting dates with that format is tricky. Here's how you have to do it so the output is sorted chronologically. This assumes GNU sort:
$ cat file.txt # I added a couple of extra records
NAME|AGE|COURSE|DATES
v1|31|MC|12 JUL 2019
v2|33|MB|4 JUL 2019
v3|12|GG|13 JUL 2019
v4|21|JJ|7 JUL 2019
11|22|33|1 JUL 2020
aa|bb|cc|10 AUG 2019
$ {
IFS= read -r header
echo "$header"
sort -t'|' -n -s -k4 | sort -M -s -k 2,2 | sort -n -s -k 3,3
} < file.txt
NAME|AGE|COURSE|DATES
v2|33|MB|4 JUL 2019
v4|21|JJ|7 JUL 2019
v1|31|MC|12 JUL 2019
v3|12|GG|13 JUL 2019
aa|bb|cc|10 AUG 2019
11|22|33|1 JUL 2020
That uses the GNU sort "stable" option so you sort first by day, then by month, then by year.
Sorting by Date in Shell
The sort you are using will fail for any date before year 2000
(e.g. 1999
will sort after 2098
). Continuing from your question in the comment, you currently show
sort -n -t":" -k3.9 -k3.4,3.5 -k3
You should use
sort -n -t":" -k3.7 -k3.4,3.5 -k3.1,3.2
Explanation:
Your -t
separates the fields on each colon. (':'
) The -k
KEYDEF where KEYDEF is in the form f[.c][opt]
(that's field.character option
) (you need no separate option
after character
). Your date field is (field 3
):
d d / m m / y y y y
1 2 3 4 5 6 7 8 9 0 -- chars counting from 1 in field 3
So you first sort by -k3.9
(the 9th
character in field 3
) which is the 2-digit year in the 4-digit field. You really want to sort on -k3.7
(which is the start of the 4-digit date)
You next sort by the month (characters 4,5
) which is fine.
Lastly, you sort on -k3
(which fails to limit the characters considered). Just as you have limited the sort on the month to chars 4,5
, you should limit the sort of the days to characters 1,2
.
Putting that together gives you sort -n -t":" -k3.7 -k3.4,3.5 -k3.1,3.2
. Hope that answers your question from the comment.
sort logfile by timestamp on linux command line
Use sort's -k flag:
sort -k1 -r freeswitch.log
That will sort the file, in reverse, by the first key (i.e. freeswitch.log:2011-09-08 12:21:07.282236). If the filename is always the same (freeswitch.log), then it should sort by the date.
How to sort a specific date-time column in bash
perl
works just fine:
#!/usr/bin/env perl
#
# Sort timestamps
use 5.12.10;
use Time::Piece;
my $fmt='%Y-%m-%d_%H:%M:%S';
my @t;
while( <DATA> ){
if( $_ !~ m@\d{4}-\d{1,2}-\d{1,2}_\d{1,2}:\d{2}:\d{2}@ ){
say STDERR "Invalid format $_";
next
}
chop; # Delete newline
push @t, Time::Piece->strptime($_, $fmt);
}
say $_->strftime($fmt) foreach sort @t;
__DATA__
2022-08-25_22:55:01
2022-08-25_20:23:24
2022-08-25_22:53:07
2022-08-25_21:53:30
2022-08-25_20:23:33
2022-08-25_20:22:14
To put that in a one-liner, you could do:
perl -MTime::Piece -lnE 'eval { push @t, Time::Piece->strptime($_,"'$fmt'") } or say "Invalid input: $_" } { say $_->strftime("'$fmt'") foreach sort @t' input-file
Related Topics
Delete All Files Except the Newest 3 in Bash Script
How to Create Opengl Context via Drm (Linux)
Why Are Core Dump Files Generated
Get Link Speed Programmatically
Sort Files Numerically in Bash
Embed Icc Color Profile in PDF
How Is the Address of the Text Section of a Pie Executable Determined in Linux
How to Read Just a Single Character in Shell Script
How to Increase the /Proc/Pid/Cmdline 4096 Byte Limit
Possible Values for 'Uname -M'
Dependency Walker Equivalent for Linux
How to Handle the Linux Socket Revents Pollerr, Pollhup and Pollnval
How to Point a Docker Image to My .M2 Directory for Running Maven in Docker on a MAC