Because biology and nature. Think of it like a separate axis. People turn into adults whether they want to or not and soft plushy diapers or being controlled or belittled can be a turn on for someone who is otherwise asexual. Nature finds a way etc.
ABDLs struggle with guilt and self loathing and accepting themselves because of the unwanted association with children in their own mind. It takes years of realizing it's not going away and that it's never been about children and it's just a harmless quirk they enjoy in private as a consenting adult.
If normies could peek into the mind of a ABDL and see all the self hate and rejection and self disgust they go through when we discover this part of ourselves, they would be 100% empathetic and understand just how strongly ABDLs don't want it to have anything to do with children either.
It's because the association with children is so repulsive that ABDLs struggle so hard with this stuff themselves and don't want this. "Omg I'm a freak why do baby things excite me like this, I can't indulge in this it's wrong", etc