A goal of HIV vaccine development is to elicit antibodies with neutralizing breadth. Broadly neutralizing antibodies (bNAbs) to HIV often have unusual sequences with long heavy-chain complementarity-determining region loops, high somatic mutation rates and polyreactivity. A subset of HIV-infected individuals develops such antibodies, but it is unclear whether this reflects systematic differences in their antibody repertoires or is a consequence of rare stochastic events involving individual clones. We sequenced antibody heavy-chain repertoires in a large cohort of HIV-infected individuals with bNAb responses or no neutralization breadth and uninfected controls, identifying consistent features of bNAb repertoires, encompassing thousands of B cell clones per individual, with correlated T cell phenotypes. These repertoire features were not observed during chronic cytomegalovirus infection in an independent cohort. Our data indicate that the development of numerous B cell lineages with antibody features associated with autoreactivity may be a key aspect in the development of HIV neutralizing antibody breadth.