Maybe because my vision isn't limited to 'street and casual use'?
The camera I want is something that I can take anywhere, use anywhere, and have with me without needing to lug around a heavy bag. And it should have the maximum performance consistent with those qualities. I don't just want a street camera; I want a camera I can take into a museum without disturbing the other guests, or carry into a packed venue like a sports arena or an airshow without trouble, or tuck into a carry-on bag, and have it perform well under all of those circumstances. I took the Pen-F with me on my last trip to Florida, and while I could have wished for better low-light performance at KSC, it made a great kit with the Lumix 15/1.7, 20/1.7 and Oly 12-40 Pro. (And the old OM Zuiko 100/2.8 for some tele reach in a tiny package.) It was a superb travel camera.*
As for high frame rates and fast readout... how about Precapture and multi-shot computational modes? Those are photo-centric features, and very good ones - I'd even say they're good for the 'casual' use you name - and they're definitely improved by a stacked sensor. And I'd imagine even street photographers run across a bit of action from time to time where high frame rates would be nice. The world is always in motion around us...