Why Fuss over a Little Leaven (Yeast)?

Let the coming chag not be polluted by old yeast, but that you observe the Festival of Unleavened Bread like the Most High wants us to remember the exodus of the Bnei Yisroel.