If you really always wear stretchy shorts without a belt, anoher optino that would work really well for you is a Thunderwear holster.
This is a denim cloth lined holster that has a "belt" secured via velcro aorund your waist and a 2-compartment pouch that can carry a firearm left handed or right handed, that you position above the crotch. It conceals even rather large firearms well. With larger firearm, it actually merely makes you look, uh, well endowed.
It's amazingly comfortable. When I wore mine, I would actually forget it was there - until I needed to go to the restroom. At that point, you merely open up, or open and drop the pants, and then "flip" the holster upward away from your plumbing. You can sit in a restroom stall without your neighbor seeing a holster and gun on the floor, as the holster remains hund on your waist.
The draw would NORMALLY be slower than from a belt holster, BUT if your shorts are actually elastic or stretchable, you could simply pull them forward with offhand while drawing with strong hand. Should be pretty rapid.
I am looking for a different solution myself only because I'd lack a faster draw, and my consistent use of a heavy belt makes that impossible with the Thunderwear. The fastest presentation I've found with the Thunderwear when worn with a strong pants belt is to simply yank open the fly on the pants and pull the gun from there. That's still slow. But, your stretchy shorts nullify that problem.
Jim G