There's really no definitive way to do that that will work 100% of the time, because applications open files in different ways.
For example, if you open a file named "TestMe.TIFF" in PhotoShop, "TestMe.TIFF" won't be listed with lsof, nor is it taking up any kind of "file descriptor" open in the system, either -- because PhotoShop opens the file descriptor, reads the information, then closes the file descriptor. Now, the file is "open" in PhotoShop, but only in PhotoShop's memory (so to speak). Even though the file is "open" in PhotoShop, there is no file descriptor associated with that file in the actual operating system.
A better method would maybe be checking memory mappings, but in C and with UNIX, there is no definitive way to check and see if any kind of file is "open" in any kind of program, because each program opens and closes files differently.
The short answer is that there isn't a definitive solution to your problem. There is no easy way to see if a file is "open" in some application.