justice of the peace

All lower case. The plural form is justices of the peace. Oddly, the plural abbreviation is JPs.