Caleb A. Scharf writing for Scientific American:
One of those hurdles is radiation. For reasons unclear to me, this tends to get pushed aside compared to other questions to do with Mars's atmosphere (akin to sitting 30km above Earth with no oxygen), temperatures, natural resources (water), nasty surface chemistry (perchlorates), and lower surface gravitational acceleration (1/3rd that on Earth).
...
The bottom line is that the extremely thin atmosphere on Mars, and the absence of a strong global magnetic field, result in a complex and potent particle radiation environment. There are lower energy solar wind particles (like protons and helium nuclei) and much higher energy cosmic ray particles crashing into Mars all the time. The cosmic rays, for example, also generate substantial secondary radiation - crunching into martian regolith to a depth of several meters before hitting an atomic nucleus in the soil and producing gamma-rays and neutron radation.
...
However, if we consider just the dose on Mars, the rate of exposure averaged over one Earth year is just over 20 times that of the maximum allowed for a Department of Energy radiation worker in the US (based off of annual exposure).
And that's for a one-off trip. Now imagine you're a settler, perhaps in your 20s and you're planning on living on Mars for at least (you'd hope) another 50 Earth years. Total lifetime exposure on Mars? Could be pushing 18 sieverts.
Now that's kind of into uncharted territory. If you got 8 sieverts all at once, for example, you will die. But getting those 8 sieverts spread out over a couple of decades could be perfectly survivable, or not. The RAD measurements on Mars also coincided with a low level of solar particle activity, and vary quite a bit as the atmospheric pressure varies (which it does on an annual basis on Mars).
Of course you need not spend all your time above surface on Mars. But you'd need to put a few meters of regolith above you, or live in some deep caves and lava tubes to dodge the worst of the radiation. And then there are risks not to do with cancer that we're only just beginning to learn about. Specifically, there is evidence that neurological function is particularly sensitive to radiation exposure, and there is the question of our essential microbiome and how it copes with long-term, persistent radiation damage. Finally, as Hassler et al. discuss, the "flavor" (for want of a better word) of the radiation environment on Mars is simply unlike that on Earth, not just measured by extremes but by its make up, comprising different components than on Earth's surface.
To put all of this another way: in the worst case scenario (which may or may not be a realistic extrapolation) there's a chance you'd end up dead or stupid on Mars. Or both.