One aspect of your question that seems to have been ignored by most of the other answers (I believe that Carl Mummert touches on the idea) and comments here is that it is not just due to the fact that $ZFC$ does not decide $CH$ that leads some mathematicians, logicians, and philosophers to believe that $CH$ does not have a determinate truth value. It is also the fact that $CH$ has proved incredibly resilient in it's ability to elude decidability when in the presence of large cardinal axioms. So in response to your question "why do people disagree with my argument" the answer is "because your premise that they do so solely on the justification that $CH$ is independent of $ZFC$ (the second "fact" you mention should be noted) is a false assumption." They do believe that the 'responsibility lies with $ZFC$', but the massive amount of work that has happened since the '60s in extending $ZFC$ with larger axioms of infinity have still left $CH$ undecidable.
After Gödel proved that $L$ is a model of $(ZFC+CH)$ and before Cohen had invented forcing, Gödel was still skeptical that $CH$ would turn out to be decided by $ZFC$. In his 1947 essay What is Cantor's Continuum Problem? he states this belief explicitly in Section 4 Some observations about the question: In what sense and in which direction may a solution of the continuum problem be expected?:
So from either point of view, if in addition one takes into account what was said in Section 2, it may be conjectured that the continuum problem cannot be solved on the basis of the axioms set up so far, but, on the other hand, may be solved with the help of some new axiom which would state or imply something about the definability of sets.
The latter half of this conjecture has already been verified; namely, the concept of definability mentioned in footnote 17 (which itself is definable in axiomatic set theory) makes it possible to derive, in axiomatic set theory, the generalize continuum hypothesis from the axiom that every set is definable in this sense.
In a footnote he then goes on to say:
On the other hand, from an axiom in some sense opposite to this one, the negation of Cantors conjecture could perhaps be derived. I am thinking of an axiom which (similar to Hilbert's completeness axiom in geometry) would state some maximum property of the system of all sets, whereas axiom A [his axiom $(V=L)$] states a minimum property.
So, even in 1947, Gödel understood that his $(V=L)$ is a minimal statement about the nature of $V$ and that there could in principle be a maximal statement about the nature of $V$ where there are more sets than just the definable ones (meaning $L$) which he believed would result in more subsets of $\mathbb{R}$ and therefore that $CH$ would be false. Under the necessary consistency assumptions, if $(ZFC + Minimal)$ implies $CH$ and $(ZFC + Maximal)$ implies $\lnot CH$, then $CH$ is undecidable in $ZFC$. This is what Gödel believed would be the case.
Gödel went on to suggest that one approach to solving the problem would be to assume stronger axioms of infinity, which is referred to as his program for large cardinals. Peter Koellner writes in his article The Continuum Hypothesis for the Stanford Encyclopedia of Philosophy:
Gödel's program for large cardinal axioms proved to be remarkably successful. Over the course of the next 30 years it was shown that large cardinal axioms settle many of the questions that were shown to be independent during the era of independence. However, CH was left untouched. The situation turned out to be rather ironic since in the end it was shown (in a sense that can be made precise) that although the standard large cardinal axioms effectively settle all question of complexity strictly below that of CH, they cannot (by results of Levy and Solovay and others) settle CH itself. Thus, in choosing CH as a test case for his program, Gödel put his finger precisely on the point where it fails. It is for this reason that CH continues to play a central role in the search for new axioms. [Emphasis mine]
Not only is Koellner explicit about large cardinal axioms being unable to give us a definite resolution to $CH$, he implicitly states that there is still a search for new axioms to extend $ZFC$. In other words, $ZFC$ is not where the buck stops in terms of asking whether or not $CH$ has a definite truth value.
There are many different schools of thought about whether or not $CH$ has a definite truth value, but like with any realism vs. anti-realism question they can be placed into two camps: realism (it does) or anti-realism (it does not). The terminology that Koellner uses in the above article, as well as two other SEP articles Large Cardinals and Determinacy and Independence and Large Cardinals, dealing with the same sort of topics, are non-pluralism and pluralism. From the latter:
The main question that arises in light of the independence results is whether one can justify new axioms that settle the statements left undecided by the standard axioms. There are two views. On the first view, the answer is taken to be negative and one embraces a radical form of pluralism in which one has a plethora of equally legitimate extensions of the standard axioms. On the second view, the answer is taken (at least in part) to be affirmative, and the results simply indicate that ZFC is too weak to capture the mathematical truths.
Pluralism is the view that there are many different universes of sets, there are many different formalizations and axioms that describe these different universes, and all of them are equally equivalent in terms of mathematical truth. Pluralism is the anti-realist view, it states that in some universes $CH$ is true, in other's it is false, and that is all that can be said of the matter. Each system, under this view, is equally privileged to make the claim about $CH$. What I believe is probably the quintessential explication of pluralism is Joel Hamkin's paper The Set Theoretic Multiverse. Consider the abstract:
The multiverse view in set theory, introduced and argued for in
this article, is the view that there are many distinct concepts of set, each instantiated in a corresponding set-theoretic universe. The universe view, in contrast, asserts that there is an absolute background set concept, with a corresponding absolute set-theoretic universe in which every set-theoretic question has a definite answer. The multiverse position, I argue, explains our experience with the enormous range of set-theoretic possibilities, a phenomenon that challenges the universe view. In particular, I argue that the continuum hypothesis is settled on the multiverse view by our extensive knowledge about how it behaves in the multiverse, and as a result it can no longer be settled in the manner formerly hoped for.
The non-pluralist view, or in Hamkin's terminology, the universe view, is that there is one conception of set, and therefore questions such as $CH$ do have a definite answer. Gödel himself was of this view, owing to the fact that he was a mathematical platonist. Gödel believed that mathematical objects are real, they are abstract objects that exist, and therefore any question about their nature has a determinate truth value. He believed, therefore, that $CH$ being independent of $ZFC$ just means that we need new axioms. Hugh Woodin's contemporary work (1, 2 on the large cardinal program is part of an attempt to create $L$ like models for every large cardinal axiom. His (relatively) recent proposal is that there is an ultimate enlargement of L once one reaches an $L$ like supercompact cardinal, leading to the axiom that $V=Ultimate$ $L$ which would give a non-pluralistic decidability of $CH$. From his article Strong Axioms of Infinity and the Search for $V$:
Godel’s Axiom of Constructibility, ¨$V = L$, provides a conception of the Universe of Sets which is perfectly concise modulo only large cardinal axioms which are strong axioms of infinity. However the axiom $V = L$ limits the large cardinal axioms which can hold and so the axiom is false. The Inner Model Program which seeks generalizations which are compatible with large cardinal axioms has been extremely successful, but incremental, and therefore by its very nature unable to yield an ultimate enlargement of $L$. The situation has now changed dramatically and there is, for the first time, a genuine prospect for the construction of an ultimate enlargement of $L$.
Whether or not one accepts pluralism or non-pluralism, they are still prompted to confront the fact that $ZFC$ is not where the buck stops. It's true that almost all of contemporary mathematics can be decided by $ZFC$, but it has never been a secret that there are many results which are not decided (See Finite functions and the necessary use
of large cardinals by Harvey Friedman and of course all work related to Gödel's incompleteness theorems).
In summation, the main thrust of my answer is that your presupposition that it is because $CH$ is independent of $ZFC$ that people who understand the issues believe that $CH$ has no definite truth value is false. $ZFC$ was proposed as a foundational system because the logicians working at the time (starting from the foundational crisis up to whenever we want to decide that set theory became it's own distinguished branch of mathematics, probably in the '60s after Cohen) believed that the $ZFC$ axioms did capture everything we wanted to capture about the intuitive notion of set. At least they believed this until we had concrete independence results that showed the opposite. So the idea that $ZFC$ perfectly captures our idea of set turned out to not be true and the search for new axioms is an implicit motivation in a lot of current set theory (although, of course, there are set theorists who do not care about foundations and are doing the work for it's own sake).
All of this has come about due to a vast amount of philosophical, logical, and mathematical thinking about the results of set theory and the pluralist or multiverse view (the view you claim to have issue with in your question) did not arise simply because $CH$ is independent of $ZFC$.