I had some help from a very enthusiastic three year old with the last few lumps of rubble (thanks Thomas!), but still had a fair bit left. No matter, simply bung it in the spaces around the concrete support pillars - I can use it to help support the hearth when I pour it.
I'll fill the last foot or so with more rubble once the mortar's set on the top stones. I'm pretty pleased with that for a first effort - the uneven levels on the walls will be taken care of when I pour the (level!) hearth. If anyone else asks me how I'm going to get the pizzas to fit in that little brick hole, I'm going to hit them.