I noticed it, but only at home. At work I have a regular desk (fake wood grain paneling). At home I have an antique oak desk (grain is much tighter than it would be on a new oak desk) and this happens once in a while.
I fixed it by using a mouse pad (not to mention that the finish was starting to rub off )