Even at the time, there were people who were mad that as much was spent on Apollo as was. People thought the money could better be spent on Earth (neglecting to understand, of course, that they didn't just fire the money out into space, and that every penny of it was spent on Earth and provided jobs and so forth) and wanted the program scrapped in favour of practically anything else, the "something else" varying depending on the particular hobbyhorse of the person in question. Getting the political will to keep funding the program once Nixon wanted it scrapped simply wasn't there...
With the greatest of respect to Gillianren, I've never liked this argument.
Sure, the money was all spent here on Earth, and sure it gave jobs to hundreds of thousands of people. But the infrastructure they built
was (to a large extent) "fired out into space".
The key point about people saying "the money should have been spent on Earth" when they criticise Apollo is not the implication that Apollo's money wasn't spent on Earth, but that the infrastructure didn't stay on Earth (and didn't provide any real direct benefit to the nation).
If Kennedy's challenge had been to build a network of hospitals or schools, instead of Apollo, then hundreds of thousands of people could still have been employed and billions of dollars spent, and on top of that the nation would have had an enormous suite of useful stuff that would have provided a much bigger return on investment than Apollo managed.
For me, another way of looking at it is Public Art. The government of the Australian Capital Territory, where I live, is often coming in for criticism for its funding of public art. Over the years, a couple of million dollars have been spent commissioning artists to create sculptures which are dotted around the city. Sure, it's kept a few artists employed, but what they've constructed doesn't serve any practical purpose - it's just intended to make the city more interesting. But as some have pointed out, in theory the money could have been allocated to funding other activities that might have employed/benefited more people directly. (Personally I like most of it.)