For me it's generally a textural thing. If I'm fermenting either whole peppers or a puree for a hot sauce only the stems get removed. At the other end of the spectrum if I'm adding chopped peppers to scrambled eggs I'll remove the seeds and pith.
Also, as pointed out above, some peppers, such as scotch bonnets, have flavour elements that get lost with too much heat.