Use one of the utilities that allows you to monitor the temperatures in your iMac.
Is your iMac showing high temps, which is the reason that the fan is running?
Did you check the small round opening on the back of your iMac (directly above the power plug) That can cause the fans to run if that is not getting air flow.
Which fan is running fast?
The temp monitor utility will also show you the speed of all the fans. It might help to know if only one of the fans is running fast.
Another tip - run your Activity Monitor, and change the dropdown menu to show All Processes, and not just My Processes. Sort the list by %CPU. Are there any processes that are running hard (showing high %CPU?)