Is it the primary sex organs that determines sex? Or is it hormone levels? Or is it chromosomes?

Each of these are non binary and have outliers and exceptions. All three together play a role in determining what we think of as being sex.

It’s not binary, unless your looking at it with a grade school level understanding of biology.