My guess is that the heat capacity of the gas is much more significant than the conductivity
I suspect conductivity plays the major role because it is much higher for helium than any other gaseous element than hydrogen.
He has a heat capacity at constant volume of 12.5 J/mol-K (same for all the noble gases, which are near-ideal). Thermal conductivity is 0.1513 W/m-K.
For N
2, a diatomic gas, heat capacity at constant volume is higher than He: 19.9 J/mol-K, but its molar mass is considerably higher (28 vs 4). OTOH, the number of moles would be the same regardless of gas for the same temperature, pressure and volume: PV = nRT. But conductivity is 25.83 mW/m-K, considerably less than He.
I know that some divers inflate their suits with argon because it has an even lower thermal conductivity than air. The tanks are hardwired and/or carefully marked so they can't accidentally breathe it; it's even more narcotic than nitrogen. Double-pane windows are often argon-filled.
SF
6 would be a poor choice: thermal conductivity is 11.627 mW/m-K, less than half that of nitrogen. Also, it is almost as narcotic as nitrous oxide. (I'm not sure those inhaling it for comic effect know this). And finally, while chemically almost inert, electric arcs can generate disulfur decafluoride, which is so toxic it was considered as a war gas.