Confirmatory factor analysis is some of the most widely used statistical techniques in the social sciences. Frequently, variables (i.e., items) stemming from questionnaires are analyzed. Two competing approaches for estimating confirmatory factor analysis can be distinguished. First, ordinal variables could be treated as in the case of continuous variables using Pearson correlations, and maximum likelihood estimation method would be applied. Second, an ordinal factor analysis based on polychoric correlations can be fitted. In the majority of the psychometric literature, there is a preference for the ordinal factor analysis based on polychoric correlations because the continuous treatment of variables results in biased factor loadings and biased factor correlations. This article argues that it is not legitimate to speak about bias when comparing the two competing factor analytic approaches because it depends on how true model parameters are defined. This decision can be made individually by a researcher. It is shown in simulation studies and analytical derivations that treating variables ordinally using polychoric correlations instead of continuous using Pearson correlations can also lead to biased estimates of factor loadings and factor correlations. Consequently, it should only be stated that different model parameters are defined in a continuous and an ordinal treatment, and one approach should not generally be preferred over the other.