I ended up majoring in Mathematics, but I sure didn't start out liking it.
In high school, by my junior year, I just hated school completely. I cut classes all the time, but still managed to get 4s on all my AP exams (Physics B, Physics C Mechanics, Physics C E&M and Calculus AB). After high school I took two years off before going back to school (as a physics major at the time).
I always loved physics and just thought the math was there to help out. After I took a Classical Differential Geometry course, I was hooked! After two years of straight courses in mathematics (Differential Geometry, Calculus on Manifolds, Lie Geometries, Differentiable Manifolds, Riemannian Geometry, Integration on Manifolds and Clifford Algebras) I realized that I was enjoying the Mathematics way more than the Physics series I was taking at the time (Mechanics, Electromagnetism and Quantum Mechanics). So I switch majors and had Physics as a minor.
But honestly, it didn't become really fun until after Differential Equations. And doing research was the best!
The most fun I ever had in Mathematics was while working for the National Science Foundation's Geometry Center. There were about 30 of us there, and I was the only one not doing computer oriented research (I was doing pure math and couldn't apply the computers to helping me). It was a great environment. They payed for everything (food, lodging, entertainment), we had no hours to speak of (some of us even went camping one week), and I was getting paid (the money was a combination of funding from the National Science Foundation and the Department of Energy). I was lucky to get that position and I know it was because one of my two letters of recommendation at the time was from a Fields Medalist (I had secured letters from three Fields Medalist before taking my extended break, I most likely could only count on one after this long).
I should get back to school at some point, but moving back to San Diego isn't that easy and I'm a little out of practice when it comes to taking courses (though I've thought about sitting in on some here in Minnesota). I know I'm not going to be able to just jump back in where I left off (part way through the Lie Groups and Lie Algebras series), but I do miss it... specially doing research.
When you get past a certain point, it really can be fun. I just don't remember Calculus being that point.