Under normal operation the seals run with clearance and an oil film makes the seal. They are not loaded against the shaft at all. Nearly any material would work and brass is softer than the shaft so it won't score it if contact is made. Brass is also reasonably corrosion resistant, easy to machine, and inexpensive.