Born in Memphis, Tennessee, Grimes first performed in public at the age of 12, with Rufus Thomas. By his late teens he recorded regularly on sessions for Satellite Records, which later became Stax, where he was tutored by Al Jackson, Jr.. He also began working with bandleader and record producer Willie Mitchell at Hi Records, taking over as Mitchell's main drummer after Jackson's death. As a key member of the house band at Mitchell's Royal Recording Studios in Memphis, Grimes was instrumental in creating some the most memorable songs and soul grooves of the 1970s.
After the demise of Hi Records in the late 1970s, Grimes continued to perform with Mabon "Teenie" Hodges and other members of the Hi house band, credited as Hi Rhythm or, on the 1994 album Perfect Gentlemen, the Hodges Brothers.