Are fish really disturbed by noise outside the water?

When I was a kid, my dad and other fishermen would always say, "Be really quiet or you'll scare the fish." But surely if the fish are underwater, they can't hear you outside on the bank? I've been underwater and couldn't hear anything topside. Was it just a way to keep the kids quiet?