"Is it true that chlorine can be evaporated from tap water by just leaving it setting out for 24 hours?"
- Yes!
"So I'm good then!"
- Absolutely not.
In addition to chlorine, tap water also contains chloramines which take much longer to evaporate than chlorine - at least 2 weeks. The effect these lovelies are having once absorbed into your soil (outside of slowly poisoning your plant) is to bind with and lock out many of the nutrients you're working so hard (or not so hard, apparently) to provide the plant with. In short, if you don't kill the plant outright, watering in this fashion will make any supplemental feeding you plan to do a nightmare and leads most commonly to nutrient burn due to overcompensation. Notwithstanding, of course, any sodium overdosing you're inadvertently giving your plants with this water.
Your most cost-effective best bet in the long run is to just shell out the 75 cents/gallon for the carbon-filtered Reverse Osmosis at your local grocer.