There are two things you can do (probably more than two but these are the most relevant at this point).
Firstly you may need to increase the resolution of the height map.
Secondly, you will most certainly need to increase the frequency of the material you apply to it.
Increasing the resolution of the heightmap
Select your terrain object and then click the [E]dit button in it’s little floating sub menu.
This takes you to the terrain editor when you will see the heightmap that is used to determine which bits of the terrain are high and low.
There are several floating pallets in the terrain editor, but you only need to look at the tool bar one on the far left.
At the bottom of the vertical column of tools you will see grid of squares icon which you need to click.
‘512 - Ultra Fine’ is set as the default, alter this to ‘1024 - Massive Resolution’ and then click the tick in the bottom right to apply the changes.
You may need to increase it further if these changes don’t make enough difference, but now you know how it’s done, if you need to increase if further, you can simply repeat the process taking it up another step later.
Increasing the material’s frequency
With your terrain still selected, click the [M]aterials Lab button in the terrain object’s little floating menu.
This takes you to the Mat Lab where you can alter the properties of any material applied to which ever object is selected.
On the right hand side of this window, you will see a block containing three little thumbnail pics of texture elements and a series of of little round buttons.
You need to click the one at top left in this block. When you put your mouse over the button it lights up slightly and the legend just above it will read ‘Transformation tools’
Click that button and another sub-window will open allowing access to these tools.
The uppermost of the set of three tool is the one you need to alter.
To the right of the tool icon there are a set of numbers for X, Y and Z coordinates. you need to make these numbers higher to increase the frequency of the material. If you click and drag directly on the center of the tool icon you can alter the numbers or you can type in numbers manually into the number fields by clicking on these individually.
Increasing the frequency will have the effect of making the terrain material smaller and less likely to pixelation though it will only have the desired effect on Bryce procedural materials. If your terrain is mapped with a picture, you will need to edit that in a photo editor and re-import it in.
Hope this helps.