I haven’t seen anybody mention failing systems. These consoles are all old enough that some TLC may be required soon to replace failing capacitors, etc. If you’re good with that and the (light) headaches involved in connecting old consoles to new TVs, there are solid recommendations in this discussion.
If you’re okay with emulation, the Anbernic RG351MP (or the newer 353) will play games from Atari 2600 through PlayStation 1, including arcade machines and old DOS games. The Anbernic hardware is very nice and exceptionally portable.
Steam Deck and Nintendo New 3DS are also phenomenal emulators for old games, but require some setup.
Rockbox is a bad benchmark due to its insane versatility. Just because modern players “should” support something doesn’t mean the manufacturer put the work in to do so, or even do it well.
That said, when I buy a portable music player the decision is always based on whether or not there’s a Rockbox port for it.