Martin Scorsese wanted to direct after Mean Streets (1973), but the producers wanted Sydney Pollack. Scorsese is on record that he would very much have liked to direct the film and was disappointed that he was passed over. However, he got to direct Alice n'est plus ici (1974) instead after being sought out by Ellen Burstyn. "Alice" ended up making more than 20 times its budget and won Burstyn an academy award, while this film became a box office bomb.
Sydney Pollack spoke no Japanese, and his cinematographer, Kôzô Okazaki, spoke no English. "We communicated in a very strange but effective way." They each had a gray-scale card going from white to black in ten stages, and they used that to communicate light density in various areas of a shot. "He did a lovely job."
This is film critic-turned-screenwriter Paul Schrader's feature-film writing debut. He also had written Taxi Driver (1976) for Martin Scorsese and Obsession (1976) for director Brian De Palma. On this story, Schrader had worked with his brother Leonard Schrader who had been an American teacher in Japan. Together they came up with the premise involving a former American soldier who travels back to Japan to help a friend is in trouble with the yakuza. The Schrader brothers sold this idea to Warner Brothers for $300,000 (plus 30% of the net profit for his original screenplay), an unheard-of sum for a script at the time, particularly for two brand-new screenwriters. Schrader fleshed out their story with Robert Towne brought in for rewrites. Also Towne was also about to have several hits, including Chinatown (1974) released the same year as Yakuza (1974).
"He was capable of a lot," says Sydney Pollack about Robert Mitchum, "but you had to push him." He thinks the actor, who often referred to himself as "an actress," didn't consider himself to be all the good without being ridden hard. "He was a real mule. He would give you what you wanted, but you had to beat him."
The star who was originally set to play the central character was Lee Marvin with his The-Dirty-Dozen director Robert Aldrich also attached. Warner Brothers wanted Charles Bronson for the lead, and screenwriter Paul Schrader preferred Robert Mitchum. Warner Brothers refused to make the script changes Marvin wanted so he dropped out then Aldrich too, replaced by Mitchum with director Sydney Pollack. William Holden was also considered for leading man before the part finally went to Mitchum. Pollack briefly considered Robert Redford for the lead role.