# batch process access_logs with single command?



## macworks (Mar 14, 2006)

I'm looking to send a "batch" of access_log files to webalizer with a single command to have it process them all.

My access_log names are numbered for each day of the month and look something like this:

access_log.1.gz
access_log.2.gz
access_log.3.gz
...
access_log.30.gz


I'm able to get webalizer to process them one at a time, but I'm wonder if I can make my command "batch" through them all based on a pattern.

Here's what I've been trying to do:

./webalizer -c my.conf /var/log/httpd/access_log.*.gz

However it will only match the first file in the directory (access_log.10.gz) and parse that. Then it stops.

Can anyone help me out here?


----------



## lurk (Mar 14, 2006)

You can glue them all together again using something like this

zcat /var/log/httpd/access_log.*.gz > big_log

webalizer can take input from standard input so you could just pipe the output of zcat to it directly with something like this

zcat /var/log/httpd/access_log.*.gz | ./webalizer -c my.conf


----------



## macworks (Mar 14, 2006)

Hmm. I get an error that says access_log.1.gz.Z does not exist. Why is it adding the ".Z" to the end?


----------



## lurk (Mar 14, 2006)

Humm, it works fine here.  Are you sure that access_log.1.gz exists where you are telling zcat to find it?  If you try zcat file.xyz and file.xyz does not exist it will also try file.xyz.Z which would be the traditional compressed file name.


----------



## tomdkat (Mar 14, 2006)

If you're running ksh, try using a "for" loop:

http://www.bolthole.com/solaris/ksh-basics.html

Something like this in your script:


```
for logfile in `ls /var/log/httpd/access_log.*.gz`
do
./webalizer -c my.conf $logfile
done
```
That's not 100% correct but should be enough to express the idea.  If you're running csh, read on the "foreach" construct.

Peace...


----------



## lurk (Mar 14, 2006)

The problem with that approach is that it will run ./webalizer on each file separately.  The zcat approach reassembles the log into a single file.  In  this case that might be important for instance if you want the weekly stats and they are spread over several files.  Now webalizer can keep a cache of already processed data in which case your approach would work with the right extra arguments.


----------



## macworks (Mar 14, 2006)

Yeah, I know I have the path correct but for some reason, zcat likes to add ".Z" to then end and then errors looking for files that end in ".Z". The man page for zcat even says it will do this. So I'd have to rename all of the files to add a .Z to them, which defeats the purpose.

Other than zcat, is there any other way I can pipe them in using one command -- I don't care if webalizer has to parse them one-by-one, because I'm using a history file anyway.


----------



## lurk (Mar 14, 2006)

Checks own system.  DAMN BSD LOSSAGE!!!  

The problem is that zcat is an old program based on compress that has been replaced on almost every sane system by the zcat from the gzip set of tools that works as Hoyle and all that is holy intended.  But not on the BSDs noo that would be to progressive.  We have an old defective zcat and we are gonna keep it!

Replace zcat with *gzcat* in teh commands above and it will work out of the box for you.  It worked on my machine since I have a newer version of gzip installed in my system local directory that shadowed the broken monstrosity in /usr/bin

P.S. Sorry for the BSD rant.  I do feel better now.


----------



## macworks (Mar 15, 2006)

Thanks! The gzcat worked flawlessly! Piping gzcat's output directly to webalizer DID NOT work, but creating the combined access log DID work.


----------

