You will have less power, may be tough to climb super steep hills (you will get up them, just with less power). You will save money in gas and pollute less.
Four cylinders are fine, Detroit marketing makes us think we need 6 and need to spend the extra cash on it that we do not have.
