I definitely feel this. I had to take a programing course in university and I was easily able to follow along up until the lesson on pointers, whereupon I completely lost the thread and never recovered.
I’ve known a good number of computer scientists over the years, and the general consensus I got from them is that my story is neither unique nor uncommon.