Despite achieving limited mainstream success, Yo La Tengo has been called "the quintessential critics' band" and maintains a strong cult following. The band is renowned for its encyclopedic repertoire of cover songs both in live performance and on record.

Ira Kaplan and Georgia Hubley, a husband/wife duo, formed the band in 1984. They chose the name "Yo La Tengo" (Spanish for "I have it") (which in Spanglish can also translate to "she is mine") in an effort to avoid any connotations in English. The name came from a baseball anecdote. During the 1962 season, New York Mets center fielder Richie Ashburn and Venezuelan shortstop Elio Chacón found themselves colliding in the outfield. When Ashburn went for a catch, he would scream, "I got it! I got it!" only to run into Chacón, who spoke only Spanish. Ashburn learned to yell, "¡Yo la tengo! ¡Yo la tengo!" instead. In a later game, Ashburn happily saw Chacón backing off. He relaxed, positioned himself to catch the ball, and was instead run over by left fielder Frank Thomas, who understood no Spanish and had missed a team meeting that proposed using the words "¡Yo la tengo!" as a way to avoid outfield collisions. After getting up, Thomas asked Ashburn, "What the heck is a Yellow Tango?".

They placed an advertisement to recruit other musicians who shared their love for bands such as The Soft Boys, Mission of Burma, and Arthur Lee's Love. The group's debut recording was a 7" single entitled "The River of Water" backed with a cover of Arthur Lee's "A House Is Not a Motel" released in late 1985 with Dave Schramm on lead guitar and Dave Rick on bass. After recording "Private Doberman" for inclusion on a Coyote Records compilation entitled Luxury Condos Coming to Your Neighborhood, Rick left the band and was replaced by Mike Lewis, the founding bass player of Boston garage-punk bands DMZ and Lyres, who was also a member of Brooklyn garage rock band The A-Bones throughout his tenure.

In 1986, Yo La Tengo released their first LP, Ride the Tiger on Coyote Records. Produced by former Mission of Burma bassist Clint Conley who also took over bass duties on three songs, the album "marked Yo La Tengo as a band with real potential" according to reviewer Mark Deming. Kaplan was credited as "naive guitar" on the sleeve, and in the liner notes for the 1993 reissue of the album on City Slang Records, went so far as to say "Dave's guitar playing is inarguably the best thing about the record."

Schramm and Lewis left the band after the album's release, with Kaplan subsequently taking on the role of lead guitar and Stephan Wichnewski joining to play bass. The group's next album New Wave Hot Dogs (1987) sold poorly, but in the words of Mark Deming, "was a quantum leap over the sound of their debut."

The release of President Yo La Tengo in 1989 did much to establish the band's reputation among rock critics including Robert Christgau who praised the "mysterioso guitar hook" in the first song. Produced by Gene Holder of The dB's, the album was the band's last release on Coyote. Despite the positive reception of the album, sales were still poor and Wichnewski left the band not long after. Hubley and Kaplan carried on as a duo and began playing two-electric-guitar shows. Kaplan, though typically a pragmatist, started carrying a bug trapped in amber in his pocket for luck.

Yo La Tengo reunited with Dave Schramm in 1990 to record Fakebook, an album of mostly acoustic tunes, including covers of Cat Stevens, Gene Clark, The Kinks, Daniel Johnston, among others, with five original songs by the band themselves. Again produced by Gene Holder, the album's folk sound was a change of pace for the band. Years later, Kaplan recalled that the album was "just me and Georgia looking for an excuse to record with Dave Schramm and Al Greller" who played guitar and double bass on the album, respectively.

In 1991, with Dave Schramm in tow, Yo La Tengo collaborated with Daniel Johnston on the song "Speeding Motorcycle" which was released as a single. The band also released a 7" single on Bar/None Records with the song "Walking Away from You" backed with a cover of Beat Happening's "Cast a Shadow." Gene Holder produced the single and played the bass. The That Is Yo La Tengo EP released later that year included some tracks that would end up on the group's next LP.

After the release of That Is Yo La Tengo, James McNew (who also records under the solo moniker Dump) began playing bass with the band, forming the trio that continues to make up the band today. According to McNew,

The band recorded May I Sing with Me in Boston with Holder producing and Lou Giordano engineering. The album was released on Alias Records in 1992. Two of the album's eleven songs ("Swing for Life" and "Five-Cornered Drone") were carried over from the That Is Yo La Tengo EP and feature Holder on bass. The Upside-Down EP was released on CD in support of the album, rounding out the band's releases on Alias.