It's an inverse problem.
You can't extract information from the results directly like you can with X-ray or electron, which directly from images or spectra that can be directly measured and interpreted. It is literally just like looking at it with your eyes, just enhanced a bit by your instrument.
For optical CD, the optical scattering patterns by itself is meaningless, since you physically cannot extract sub wavelength imaging information directly. Instead there will be hints, but just hints, in the scattering patterns. You have to theoretically calculate the scattering patterns with N free parameters, and see if they fit your model.
My brain has trouble with trusting inverse problems as there's always the doubt of "but what if the model is wrong/incomplete".
Imaging and spectral data is different. You are directly observing the property of interest. It is model free. The CD is directly observed.
It is like the difference between directly observing a crime happen and piecing a crime together with a forensic team. There's some cases that are airtight, some cases confuse for decades, and sometimes bias puts the wrong guy in jail. But direct observation always works.