Virdon initially signed with the Yankees, but he never played for the big league club. As a minor league prospect, he was traded to the Cardinals in a multi-player deal before the 1954 season for veteran outfielder Enos Slaughter, now a member of the Baseball Hall of Fame. After one more year of seasoning in the minors, Virdon joined the Cardinals in 1955 and was named National League Rookie of the Year. He was traded to Pittsburgh in May 1956. A left-handed batter (who threw righthanded), Virdon's career batting average was .267 with 91 home runs during his 1,583-game NL career. He was the starting center fielder on the 1960 world champion Pirates, batting .241 in seven World Series games. His ground ball in Game 7 took a bad hop and struck Yankees shortstop Tony Kubek in the throat, enabling Virdon to reach base and the Pirates to mount a furious rally in a game (and series) eventually won by Bill Mazeroski's dramatic home run.

As a manager, he led the Pirates to the 1972 NL East title, but the Buccos dropped the NLCS to the Cincinnati Reds when Pittsburgh pitcher Bob Moose unleashed a wild pitch in the final inning of Game 5, allowing the winning run to score. His 1980 Astros won the NL West championship, but fell to the Philadelphia Phillies in a five-game NLCS. His career managerial record, over all or parts of 13 seasons, was 995-921 (.519). He also served three different terms as a Pirates coach.