I'll add that the nicest breading I've gotten on fish is with Vivian Howard's rice-crusted catfish recipe from her book Deep Run Roots. She submerges the fish in buttermilk for a hour to overnight in the fridge. Her breading is 1/2 cup long grain rice, ground in a spice mill to the texture of coarse cornmeal, plus 1/2 cup cornmeal, seasoned with salt, cayenne and paprika. After draining and dredging the fish and knocking off the excess, she puts the filets on a rack in the fridge for at least an hour.
She contends that the ground rice behaves differently from rice flour. Maybe does something similar to the potato flakes you are using. I dunno but the result is shatteringly crisp and not heavy at all.