Dillard, who grew up on a farm near Salem, Missouri, began learning guitar and fiddle at age five, and banjo at age 15. He began playing in the family band, with his father Homer Sr. on fiddle, his mother Lorene on guitar, and his older brother Earl on keyboards.

His banjo heroes were Earl Scruggs, Ralph Stanley, and Don Reno. After corresponding with Scruggs, Dillard persuaded his parents to drive him to Scruggs' home in Madison, Tennessee, where Scruggs installed "Scruggs Tuners" on Dillard's banjo.

By age 19, Dillard was performing regularly on a weekly radio show hosted by Howe Teague on KSMO, a Salem radio station.