In general leisure time has been increasing rather
than decreasing at least for the last few thousand years and yet the modern
human still spends about 40 hours per week at work.

  This event has happened in the past few hundred years, at most, and only
with the privileged. Have you ever worked a farm? Most people in the world
still have to work their tails off(no pun intended) even today.

It has always been my assumption that the advent of modern civilization as
we know it was due in large part to the additional free time we gained as a
result of agriculture, animal husbandry and the subsequent organized
distribution of labor. As a result of this free time we found ourselves with
the time to develop intellectual skills like reading, writing, mathematics,
architecture, etc. Am I wrong?

 All those things you list, are the ways we tried to "Correct" the negative
aspects of eating an improper diet, and becoming entrapped in keeping it up.
Most people, did not work in an office, in the 1800's.

If I am wrong then I must wonder why humans took a million+ years to get
around to learning how to write. If we hunted and gathered only for one hour
per day then on what pursuits were we wasting all our free time? What were
we thinking about for the remaining hours of the day during which we were
awake but not looking for food? Did we spend a million years or more just
contemplating our navels? I doubt it.

  Singing and dancing sound fun to me, perhaps a drum circle. Can't wait for
that gathering, just think, single paleo woman!