1560s, from button, n. + hole, n.. The verb, also buttonhole, meaning "to detain (someone) in conversation against his will" (1862) was earlier button-hold (1834), from button-holder (1806, in this sense). The image is of holding someone by the coat-button so as to detain him.