The only thing that I have to add to what has been said already by most of the posts here, is this:
Again, most likely you are going through the binge/purge cycles that many of us have gone through, and there is likely a deep part of you that actually really wants this life style.
So where is the part of you coming from that is telling you, that you don't want this lifestyle? Since wearing diapers isn't unhealthy on your body, it isn't immoral (or provably depending on if your religious leader is a nut who thinks wearing diapers is evil [example: my old one]), and it doesn't harm anybody, then why do you think that you have such a distaste for diapers, while also really wanting them? If you are anything like me, then the hate is sourced from socialization. Really ask yourself why you hate diapers, and think about if it could be because you have been socially conditioned to dislike them, even though personally, you still like them anyway.
For instance:
1. The fact is, most of everybody gets told, "Diapers are for babies," when they grow up, as an effort to encourage potty training. There is also a lot of other tactics that are intended to encourage potty training, that basically create a negative association with diapers. Sometimes, children are encouraged to potty train earlier than they are even ready, in order to save money by not having to buy diapers. Things like that can cause a bit of trauma in a child to develop inadequacy, or on the other end, a deep need to appear super adult.
2. Also, diapers could be discouraged in your mind socially because of religious expectations. Growing up in an environment where alternative lifestyles are frowned on, such as bondage, or homosexuality, or fetishes of many kinds. It is easy for a religious group to look at these types of things and deem them all as bad, even though they don't harm anybody, and can be contained within a honorable relationship.
3. Then there is the aspect of social fear, fear of being discovered, or fear of never finding somebody who accepts you for who you are. So far, I did go through a lot of trouble with my parents for coming out as an Adult baby, but since that has cooled down, however, the rest of my family was pretty accepting, especially one of my sisters and her husband. As well, all of my friends have been very accepting of it. Most of the current 'young' generation has grown up to be fairly accepting of others, and it isn't really that big of a deal to find out. Finding a life partner is difficult, but would it really be worth living with somebody for your whole life who you weren't capable of sharing your deepest secrets with? You might find that you are happy with somebody who accepts you, but doesn't really want to participate, or you might find that you want to be with somebody who is a part of your little side. Either way, it is totally possible to find somebody when you look for people who are open minded.
4. The fear that people might interpret your liking of diapers as a form of pedophilia. This is just straight up ignorance, and isn't true. Anybody who can't get past that notion and realize it is false, is somebody that you probably wouldn't want to associate with much anyway.
So:
I would assume that one of these 4 reasons, or some, or all of them, are why you are feeling anxious about wearing diapers, and even finding sexual gratification out of them. The reason why I suspect that you hate them for social reasons, is because it just doesn't make sense why you would both personally love diapers, and personally hate them. Also, it isn't sensible to think that you socially love diapers, so that wouldn't explain it.
So do you want to live a "normal" life? Or do you want to live a Genuine life. The thing about being normal, is that it isn't a real thing. There is no such thing as normal, normal is fluid, it changes whenever culture changes.
The best thing you can do for yourself, is to start finding a balance, make sure you aren't throwing away stuff, that just makes your life more frustrating. Tell yourself, "I get to wear diapers on such and such days at such and such times," and be consistent. By giving yourself a schedule when you can let the little out, you can have less anxiety.
Also, if it is available to you for cheap, consider going to a liberal minded therapist if you start getting depressed over this 'battle.' A therapist would easily point out that, yeah this is an unusual lifestyle, but no, there isn't really a reason to be ashamed of it or afraid.
Life is so much better when you can just let yourself be yourself.