On the contribution of quiet Sun magnetism to solar irradiance variations: Constraints on quiet Sun variability and grand minimum scenarios