Also, the CM didn't just use aluminium in its construction. There was a stainless steel plate in it too. Also, the other components of the spacecraft provided shielding. The SM - 38 tonnes - shielded the CM from one end, the LM - 15 tonnes - from the other on the outbound trip. Only the sides were "open" to space.
How does that factor into your calculations?