Maybe the babyish association has something to do with proportions and control as well. If you see a baby in a wet diaper, it looks huge on them and is possibly hanging to their knees, or maybe mid calf for a toddler. I think that's a part of it, for me at least. I think that a baby or a toddler wearing a diaper certainly can sense they are wearing one, as mentioned above, but its more than that... its like this thing that they have no control over, a heavy thing that sort of hangs on them. If our larger bodies have a regular diaper on, maybe there's something missing when we have that perceived control- a totally manageable nappy just isn't as exciting.
Compare that to when you've got a heavy and thick diaper on, and you feel its weight between your legs, the cushioning effect of the swollen padding, and that slight loss of control over your surroundings because of an accessory that you, of course, don't want to control. It feels like its the right size in proportion to your body, and you feel smaller or more babyish in effect.
I know that this explanation has a really heavy AB feel to it, I'm not sure how differently a pure DL would answer this question.