batch process access_logs with single command?

macworks

macworks

Christopher Raymond
I'm looking to send a "batch" of access_log files to webalizer with a single command to have it process them all.

My access_log names are numbered for each day of the month and look something like this:

access_log.1.gz
access_log.2.gz
access_log.3.gz
...
access_log.30.gz


I'm able to get webalizer to process them one at a time, but I'm wonder if I can make my command "batch" through them all based on a pattern.

Here's what I've been trying to do:

./webalizer -c my.conf /var/log/httpd/access_log.*.gz

However it will only match the first file in the directory (access_log.10.gz) and parse that. Then it stops.

Can anyone help me out here?
 
You can glue them all together again using something like this

zcat /var/log/httpd/access_log.*.gz > big_log

webalizer can take input from standard input so you could just pipe the output of zcat to it directly with something like this

zcat /var/log/httpd/access_log.*.gz | ./webalizer -c my.conf
 
Hmm. I get an error that says access_log.1.gz.Z does not exist. Why is it adding the ".Z" to the end?
 
Humm, it works fine here. Are you sure that access_log.1.gz exists where you are telling zcat to find it? If you try zcat file.xyz and file.xyz does not exist it will also try file.xyz.Z which would be the traditional compressed file name.
 
If you're running ksh, try using a "for" loop:

http://www.bolthole.com/solaris/ksh-basics.html

Something like this in your script:

Code: 
for logfile in `ls /var/log/httpd/access_log.*.gz`
do
./webalizer -c my.conf $logfile
done
That's not 100% correct but should be enough to express the idea. If you're running csh, read on the "foreach" construct.

Peace...
 
The problem with that approach is that it will run ./webalizer on each file separately. The zcat approach reassembles the log into a single file. In this case that might be important for instance if you want the weekly stats and they are spread over several files. Now webalizer can keep a cache of already processed data in which case your approach would work with the right extra arguments.
 
Yeah, I know I have the path correct but for some reason, zcat likes to add ".Z" to then end and then errors looking for files that end in ".Z". The man page for zcat even says it will do this. So I'd have to rename all of the files to add a .Z to them, which defeats the purpose.

Other than zcat, is there any other way I can pipe them in using one command -- I don't care if webalizer has to parse them one-by-one, because I'm using a history file anyway.
 
Checks own system. DAMN BSD LOSSAGE!!!

The problem is that zcat is an old program based on compress that has been replaced on almost every sane system by the zcat from the gzip set of tools that works as Hoyle and all that is holy intended. But not on the BSDs noo that would be to progressive. We have an old defective zcat and we are gonna keep it!

Replace zcat with gzcat in teh commands above and it will work out of the box for you. It worked on my machine since I have a newer version of gzip installed in my system local directory that shadowed the broken monstrosity in /usr/bin

P.S. Sorry for the BSD rant. I do feel better now.
 
Thanks! The gzcat worked flawlessly! Piping gzcat's output directly to webalizer DID NOT work, but creating the combined access log DID work.
 
You must log in or register to reply here.
Back
Top