The rationale I was given for Joseph Smiths philandering was that he practiced spiritual wifery and, if God directs it, it must be okay and Joseph did nothing wrong.
Now here's the problem (and bear with me, but this has to be discussed). The LDS teaching on wifery stems from a misapplication of ancient polygamous interactions. You see, while true that Abraham, David, Solomon, etc., practiced polygamy and concubinage, there was nothing from God (even in Judaic law) permitting either and, when one reads the stories, it looks to me like practicing them leads only to unrest and trouble.
Sarah and Hagar hated one another; Rachel and Leah got into a pregnancy pissing contest; David had nothing but trouble (re: Amnon and Tamar story) and on and on.
But Joseph Smith read these accounts without looking at the lesson. Instead he said: well God didn't stop them so it must be ok!
Poor Joseph. He could've spared himself and his followers alot of heartburn had he not delved into polygamy.
I hope this explains a little insight.
I know what you mean. I went almost 20 years before I realized my parents, well my mom at least, was a racist. I didn't pay enough attention or even know many blacks in the church to know anything about the priesthood issue until later. It never came up when I was younger. I realized I, like you, was learning life lessons outside of my parents that I would have otherwise assummed were lessons they were teaching me. Love thy neighbor was something they taught me but they never specified that it was conditional until after I had friends that were black, gay, or what have you. Then it came out. My mom said "I have nothing against blacks but my kids better not bring one home and say they are marrying one" or "I'm not a racist but" and then say something totally racist. Blew my miind. I mean, she was never really the nicest or loving person but I just didn't know she was racist. And her reason? That god commanded we shouldn't marry outside of our race. I don't understand that one bit. What did it have to do with blacks? They are part of the human race. We dont' criticize dogs or cats or say they are going outside of their "race" when they mate with another that has a different color or body type. So why would being with a black person be such a big deal. Yet in her eyes it was ok if the person was Asian, Hispanic, White, or anything else other than being black. And she made sooo many snide comments. To this day I still cannot understand why she is such a racist, or why anyone would be anymore honestly. I just dont' get it.
Should you address it? I think only you know the answer to that. I've thought about addressing issues with my parents in regards to the Mormon religion but they aren't in a place where they want to hear it. They'll continue making excuses. I remember their faces when I first asked a question and this man, my dad, who taught me to never just believe what I was told told me that I was not to question the religion but just take things on faith. You may want to tread the waters lightly to gauge whether it would be productive to bring it up or not. Otherwise I like your live and let live philosophy. You are treating others the way you want to be treated. And don't feel too bad about your feelings in regards to wanting them to see the truth. IN a way, remember when they share religious thoughts with you they also think they are sharing an important truth with you.