Libertarian Dilema

As a libertarian I typical take the side of an argument that allows more freedom and liberty to those involved. Suppose somebody wants to restrict another person’s ability to, let's say, play soccer because they feel it is unsafe. They may have a valid point and can show loads of reports on broken ankles, head injuries and mangled fingers. Some would be tempted to allow legal restrictions to be put on playing soccer. But what if the activity is only harmful to those who are participating in it, as is the case with soccer? Do I as a non-soccer player have the right to tell others what they should or should not think is safe? I would say no. Whether I choose to play it or not I don't have a right to restrict other's freedom to participate.
The dilemma I have with taking this pro-liberty, pro-freedom stance is that all too frequently a libertarian may appear to be defending the decision of another rather than just their right to have that decision. If I think that kids should have the right to play soccer that does not mean that I would have chosen the same way.
Granted this soccer analogy is not quite to the point. Few have serious concerns about whether or not people should be allowed to play soccer. However, please keep this analogy in mind as I attempt to make my next point.
What if the choice goes against my personal morality? Should I be allowed to restrict someone else’s freedom in that case? Let's take smoking for instance. I can provide reams of evidence that shows how dangerous it is. But should I be allowed to restrict another’s right to hurt themselves? How about harder stuff like narcotics? If somebody wants to toke it up in their home and they don't drive while high why should I have the right to force my moral decisions on them? I don't believe that I do have that right.
I fear that because I take this position I end up alienating people on both sides of the issue. I don't condone the use of narcotics but I believe that the choice should be left up to the user. I don't condone accessing internet porn but I also don't think it's my job to force my morality on others and suggest censoring.
I have given a lot of consideration to the morality of this issue. And I have made peace with my decision. In LDS theology we have a story about the War in Heaven. Lucifer proposed a plan that would force everybody to be obedient and Christ proposed a plan that allowed us to choose whether or not we would obey. The Father then chose Christ's plan. This story always comes to mind when I hear attempts to limit the freedom of others. All too frequently I get little pangs inside me that say, "This sounds similar to Satan's plan".
So in those situations where I chose to allow others to potentially choose differently than I would prefer I am comforted by the fact that I believe restricting their right to chose would be even more immoral.