I think they size on the small side. The mediums are especially small mediums. The larges are closer to "typical" large size. So that means there's a big gap between their medium and large.
So if the M4 fits but the lower tapes are barely making it to the front, you can go for large. But If the lowers are reaching with any reasonable room to spare, the larges will be very big on you.
IIRC, the lower tapes on me BARELY fully tape to the front panel on the mediums, and they're uncomfortably tight. The larges on the other hand, have the top tapes within about 6" of each other. (wore one last night, still wearing this morning!) So that's arguably at the very high end of medium, and very close to the low end of large. There's very little overlap.