The most challenging but rewarding system that I have learned is the d20 3.X D&D/Pathfinder system. The first version of the d20 3.X system that I played was Pathfinder . While the basic mechanics of the system are simple (roll a d20 and add modifiers), the elegance of the system is how much can be added. The feats and combat options alone create a sophisticated mechanism that allows for unlimited variety in play.
GMing Pathfinder was a challenge, especially when I was running adventures for up to seven players. Reading the Pathfinder rules reminded me of wargaming rules. They were long, complicated, and rules from one section affected rules in another section. Trying to learn and remember enough to run a game was challenging and I am not sure if I ever had mastery of the system. In fact, I had to quit running for a while due to fatigue. However, all that rules-learning worked my brain like a muscle.
I now have a great respect for the d20 3.X system. I appreciate how huge the system is and how many different options are available to those who play it. I appreciate the elegance of how all the rules can work together. My favorite thing about the d20 3.X system has to be the OGL that it was published under. The OGL paved the way for other games such as Mutants and Masterminds, Spycraft, Castles & Crusades, and the OSR retro-clone movement. When Paizo releases Starfinder I know that I will be ready to run some d20 again, attacks of opportunity be damned!