I actually think the problem stems from the earliest grades.
I agree. My mom is an upper high school math teacher (precalc and calc). She constantly runs into students who find math hard, but instead of working at it, the "study for the test". Not that studying is bad, but studying only for the purpose of passing a test and then forgetting the material is. This might work in a subject like history where what you are learning now doesn't depend completely on what you learned 2 months ago like math (I'm not against history). If you don't actually learn addition, when you get to multiply, you're screwed.
The problem is not just with students. My mom used to be a chemical engineer making computer chips in the 80s. She understood the math she's now teaching along with the applications of it and what is next for her students. She has audited her elementary colleagues and found that many don't understand math much beyond what they teach. This creates problems when they teach such that a student can get 100% yet still not understand all that is really going on or be able to apply it to real life and/or science. Then when they get to precalc, they don't understand basic math stuffs, like factoring to find zeros of a polynomial function. Some of my mom's students have expressed an interest in elementary education because "you don't need the hard math". While you don't need DeMoivre's Theorem to teach multiplication, an understanding of "hard math" is, so that when this potential teacher's students get to precalc, they can learn DeMoivre's Theorem, because their 3rd grade teacher had that (and a lot of other stuff) in mind when teaching them multiplication.