In Taiwan, English is regarded a key subject in high school curriculum. Hence, before entering university, high school seniors are required to take an English subtest (ES) of the General Scholastic Ability Test (GSAT). Their scores on the GSAT-ES, besides other subtests, influence considerably their future academic pursuit and career path. To them, an important question is: What does the GSAT-ES really measure – several aspects of their English proficiency or just their overall English proficiency? This study was aimed to address, among others, such a question and focus particularly on the two types of cloze items in the GSAT-ES. For this study, a simpler version of Purpura’s (2004) model of grammatical knowledge was adopted to classify the cloze items. Classification was done by a group of five experienced university-level English teachers. Then confirmatory factor analysis (CFA) was applied to investigate the degree of fit between the raters’ classifications and the test takers’ responses. The CFA findings indicated that the raters’ classifications failed to fit the test takers’ responses and that the one-component model best portrayed the response data. The finding of a one-component model agreed with the unitary competence hypothesis of Oller (1976, 1979), who advocated using cloze items for measuring overall language proficiency. This study concluded with some relevant implications for English teaching and testing.