Duffy is based upon the real-life character Albee Baker. The story actually happened in the mid-'50s and was recounted by Albee to the writer, 'Pierre de la Salle', who came to know Baker in 1956 whilst living in New York.
When Duffy introduces Stefane as a "Mr. Eugenides" and describes him as "a Smyrna merchant", he is surreptitiously quoting a poem by T.S. Eliot.
Filmed under the title "Avec-Avec", a term used in the game of chemin de fer; some journalists mistakenly thought it was the French for "with-it". The title was changed at the last minute just before the film opened, a move roundly condemned by the film's leading actors. Susannah York publicly said the new title was "boring and square" and was reluctant to do any promotional work for the film.