1650s, "public department in charge of letter-carrying," from post, n.3 + office. Meaning "building where postal business is carried on" is from 1650s. In slang or euphemistic sense of "a sexual game" it refers to an actual parlor game first attested early 1850s in which pretend "letters" were paid for by kisses.