The first probe to hit the Moon (Luna 2, on September 13th, 1959) took an even straighter trajectory and hit the moon just 33.5 hours (less than a day and a half) after launch.
This is fine when you only intend to hit the moon. It's possible to get there even faster; the fastest launch ever conducted, the New Horizons probe to Pluto, crossed the moon's orbit only 10 hours after launch.
But if you want to soft land or decelerate into orbit around the moon, you have to use a less energetic (and more time-consuming) orbit or you'll have a lot of excess velocity to get rid of with your lunar landing or orbit insertion burn.
Apollo's lunar trajectory, besides being designed to provide a free return, was a compromise between the propellant required both for TLI and LOI and the consumables required to sustain the crew in the meantime. You can get there faster (and save consumables) by spending more propellant. Or you can save propellant by taking longer (and using more consumables) but only to a point; if you take too long, you'll never reach the moon at all, but fall back to earth before reaching it.
It's a common misconception that TLI injected Apollo into an earth escape trajectory. It actually went into a highly elliptical earth orbit with an eccentricity of about 0.97. (Escape would be 1.0 or more.) It had its apogee beyond (but not too far beyond) the moon's orbit, meaning it couldn't have taken too much longer than it did or it never would have gotten there at all.