tar will not pipe | to log file

jyhm

jyhm

Ultra Geek
Hello,

I am tring to back up my files using tar, at the same time piping verbose output to an external log file using this code,..

Code: 
tar -cvzf backup_file.tar.gz /Users/jyhm | tee /Users/jyhm/backup.log

#or

tar -cvzf backup_file.tar.gz /Users/jyhm > /Users/jyhm/backup.log

All I get is an empty document using the above code. However, the code below will output just fine,..

Code: 
ls -al /Users/jyhm | tee /Users/jyhm/list.log
 
The output you're seeing is standard error (file descriptor 2), and you're redirecting standard out (file descriptor 1)

If your shell is bash, then just replace your > with a 2> and you should be good.
 
Fascinating,.. the syntax that you have suggested (2>) works!

The output you're seeing is standard error (file descriptor 2), and you're redirecting standard out (file descriptor 1)
Click to expand...

What is "standard error" and file descriptor 1 & 2

Does that have some correlation to the SYNOPSIS in the man page for tar?

tar [-]{crtux}[befhmopvwzHLPXZ014578] [archive] [blocksize] [-C directory
] [-s replstr ] file1 [file2...]
Click to expand...
 
Okay, I'll grasp feebly at an explanation of the difference.

STDERR is used mostly for displaying status messages related to how the program is functioning. STDOUT is typically used for the actual meaningful output produced by the program. So for example, if you're tarring to a file, you could do away with the -f and > to the .tar file you want to create. Then the status messages from tar will still be sent to STDERR, which by default is mapped to your terminal. STDOUT is also mapped to your terminal by default, and is the default choice for the > operator.

You can also try it with the fgrep command. If you're in a directory with both files and subdirectories, and do something like "fgrep 'printf' *" to find all files with the string "printf" in them, fgrep will spit out matched files to STDOUT, and errors to STDERR. So the files that are matched get printed, as well as the subdirectories, which obviously can't be processed by fgrep. Thus if you want to make fgrep not print the error messages, you just do "fgrep 'printf' * 2>/dev/null" and only STDOUT is still mapped to your terminal.

Did that almost make sense? Ha ha.
 
Thank you scruffy and davidbrit2,

I think what you are saying is that there is a process output and an error output right? If I used the code below, would that seperate the STDOUT and the STDERR into seperate files? I need to log any errors as well.

Code: 
tar -cvzf backup_file.tar.gz /Users/jyhm/Desktop/ > error.log 2> backup.log

How can I generate an error in order to test this code? fgrep only showed errors and not the process.
 
There are any number of file descriptors a process can use (I think it's a couple of dozen by default in OS X). By convention, fd 0 is standard input, fd 1 standard output, fd 2 standard error (not just for errors, but any diagnostic/status messages tend to go there).

By default, the > operator assumes you mean file descriptor 1, by you can redirect any file descriptor you want - you could use
tar (blah blah arguments) 45> file
but the program won't have a file descriptor 45 by default, so it won't change anything...

When you say
command1 | command2

command1's fd 1 gets sent to command2's fd0

An example of the functions of fd 1 and 2:

poppacrow:~/tmp mark$ ls
foo
poppacrow:~/tmp mark$ ls foo bar
ls: bar: No such file or directory
foo
foopoppacrow:~/tmp mark$ ls foo bar > lsout
ls: bar: No such file or directory
poppacrow:~/tmp mark$ cat lsout
foo
poppacrow:~/tmp mark$ ls foo bar 2> lserr
foo
poppacrow:~/tmp mark$ cat lserr
ls: bar: No such file or directory

Did that help?
 
Scrufy

Thank you for clearifiying that fd 2 is what I wish to use for all error and
diagnostic output. What fd0 and fd1 do is still a mystery to me!

Code: 
poppacrow:~/tmp mark$ ls
 foo
 poppacrow:~/tmp mark$ ls foo bar
 ls: bar: No such file or directory
 foo
 foopoppacrow:~/tmp mark$ ls foo bar > lsout
 ls: bar: No such file or directory
 poppacrow:~/tmp mark$ cat lsout
 foo
 poppacrow:~/tmp mark$ ls foo bar 2> lserr
 foo
 poppacrow:~/tmp mark$ cat lserr
 ls: bar: No such file or directory

This is a little beyond my understanding of command line.
I give myself the ranking of simple navigator, rather than power user.
All the command line I know is from http://www.linux.org/lessons/index.html .
Very useful to someone like me though.

It's funny, I asked this same question some time ago in the apple/discussions board, and the people there said that > worked and generated a log file. I think I might post a question about a system problem that did not get much attention in apple/disscusions! :)
 
With comments then:

poppacrow:~/tmp mark$ ls
foo
# I'm in a directory that has one file, called 'foo'.

poppacrow:~/tmp mark$ ls foo bar
# I try to list two files: foo, and one that's not there, called bar

ls: bar: No such file or directory
foo
#the output is an error message about the nonexistent file, and the listing of the real one

poppacrow:~/tmp mark$ ls foo bar > lsout
# now I redirect fd 1 - the standard output - to a file called lsout

ls: bar: No such file or directory
# I see the error message only - where's the regular output?

poppacrow:~/tmp mark$ cat lsout
# let's see what the file that caught fd 1 output says

foo
# There it is - regular output goes to fd 1

poppacrow:~/tmp mark$ ls foo bar 2> lserr
# now I redirect fd2 to a file called lserr

foo
# there's the regular output, where's the error message

poppacrow:~/tmp mark$ cat lserr
# we read the file that caught fd2 output

ls: bar: No such file or directory
# and there's the error message

So, when I try to ls 'foo' and 'bar', I get two outputs: the listing of 'foo', which is ordinary output, written to fd1, and an error message saying that there is no file called 'bar', written to fd2.

fd 0 is the command's standard input - when you pipe a command to tee like in

tar -cvzf backup_file.tar.gz /Users/jyhm | tee /Users/jyhm/backup.log

What you are doing is running two commands - tar and tee - with tar's output being sent to tee's input. In other words, everything tar writes to fd1 gets sent to tee's fd 0.

The reason it didn't work as you expected is that tar writes its diagnostic messages to fd 2, the standard error file descriptor, and that file descriptor is not touched when you redirect fd 1.
 
Thank you scruffy for your patience and your clear explaination. I have used the below code to generate two output files and it worked!

Code: 
ls -al foo Sites > fd1.txt 2> fd2.txt

By the way, would you know a good site/book that gives a good tutorial for OSX command line? It is slightly different than Linux—and I must study further anyway!

How 'bout a good reference for understanding log files,..anyone? ;)
 
You must log in or register to reply here.
Back
Top