I first read the Giver in elementary school, then again as a junior in high school. I enjoyed it in elementary school but my teacher didn't necessarily have the necessary discussions with our group. Back in that time, it was simply enough to hand students a book and give them a test at the end. There were no discussions.
I think the lack of discussions does a disservice to students.
Obviously I wouldn't have the same discussion with fifth graders and I would with high school students, but that doesn't mean elementary students can't handle this book in a very structured, scaffolded setting.
I've made my teacher notes available here.
Hope your students enjoy this book as much as mine do! I've used it both as a whole class read aloud and in small groups.