Bryce is a 32 bit application, and so it can only use up to 4 GB of memory depending on what OS you are using.
You can make it Large Address Aware (LAA), but I don;t know if that helps much, I will leave it to somone else to explain how to do that, and if it will help in your case.
Actually unaltered the maximum amount of memory Bryce can use is 2GB’s. If you make it Large Address Aware then it can use approximately 3.5GB’s
The LAA tool does help quite a bit if one keeps things reasonable. Also it’s good to understand that Bryce does everything in memory including what it saves in the undo buffer. So if you have been tweaking your scene alot and know you’re about to do something that’s going to be hard on memory like importing a detailed mesh with a high rez texture applied, via the Bryce/Studio Bridge then it’s a good idea before you do that to save the scene you’re working on, completely exit Bryce (to clear out the undo buffer). Then reopen Bryce, reload your scene and proceed with whatever memory intensive thing you were planning on doing.