Is a little gravel ok in a garden?

[pkmoo3]

We recently moved into a house that has gravel in the front and back of the house. We would like to plant a garden in the back yard and put flowers in the front but have no idea about what to do with the gravel. The gravel is medium sized and we removed most of it from where we want the garden bed but there are still a few that raking won't get up. Should we leave them for drainage? We also have received record setting rain fall in AZ this year and weeds are going crazy in our yard..honestly if we mowed it, it would look like grass! What can we do that won't cost us an arm and a leg, we would like to do it ourselves. We have tried good ol' fashioned pulling and in the gravel this is quite difficult plus now that we have a forest of weeds it's impossible. Thanks!