We tend to think this mitzwah is arcane because when you think about it, it doesn't seem to make very much sense. But, I'd like to present an alternative understanding.
The mitzwah of Shiluah HaKen, of sending off the mother bird before taking the eggs from the nest, seems equally arcane. You might at first say, "well, it teaches you to be merciful and considerate of animal's feelings." We already have a mitzwah for that, it's called Tza'ar Ba'alei Hayyim, it is forebidden to cause undue suffering to living things. So what is Shiluah HaKen about, it seems equally arcane.
Let's try and learn about this mitzwah in an unorthodox (in the original sense of the word) way: Let's try and do this mitzwah and see what it teaches us, rather than try to understand it without knowing what it really is. After all isn't that what we committed to when we said, "We will do, and [then] we will hear (read: understand)"?
This week it so happened that I needed to convince a family of pigeons that they couldn't build their nest in my stairwell, well they couldn't keep their nest in my stairwell, they'd already built it, and it was a snug home to two white eggs. Wow, as scary as it may be, I get to do a mitzwah!
I assume it was the father that flew off almost as soon as I started to make some noise to scare the birds. Presumably the mother bird didn't budge. As I tried to cause her to panic and make her fly away she did something amazing. She hid her head and ignored me. No matter how big and scary I was, she didn't try and fight me off, she knew she wouldn't win. But, she wouldn't abandon those eggs, even though her own life was in danger. I begged and pleaded, told her I didn't mean to offend her, tried to drive her off, eventually having to physically push her with a stick (the nest was located in an unreachable position) very gently to avoid hurting her, even so, it took about a half an hour until I managed to push, scare, and irritate her (all without causing her any physical harm) into leaving. I was totally exhausted, very thankful for being able to do a rare mitzwah and blown away by what I had witnessed.
The way the mother bird acted goes against any understanding I had of animal's drive to self-preservation. She knew she couldn't save the eggs, but it was like she didn't want to live if she couldn't. It was an amazing testament to the mothering instinct. I have no desire to ever witness a more direct exhibition of this instinct, what I saw was overpowering itself.
So now, armed with the knowledge of actually having performed the mitzwah (the hard way) what is this mitzwah all about? To me it teaches just how committed the Shechinah is to staying by our side no matter how far or deep we fall into galut. No matter how scary it may be. Even in the darkest places this world has ever sunk to, the Shechinah went down with us into exile. It's heart-rendingly beautiful to know of HaShem's endless love for us.
What can we learn from this about the Parah Adumah? Why was it so complicated to understand what the mitzwah was about? Perhaps because even after they did the mitzwah, they still couldn't begin to understand what it meant.