Objectives: This study takes the elderly outpatients of a medical hospital as subjects, analyzes the effects of various dimensions on their sleep quality, and examines how such effects differ by sex and modification. Methods: By employing a cross-sectional design, this study collected data by obtaining clinical measurements related to metabolic syndrome in patients attending outpatient geriatrics, neurology, and psychiatry departments at a medical center in Southern Taiwan. Data were collected using a questionnaire containing a demographic scale, a lifestyle scale, the Pittsburgh Sleep Quality Index, the Epworth Sleepiness Scale, the Geriatric Depression Scale, a sleep-disordered breathing scale, and an activities of daily living functional assessment scale. Results: Women had more problems with low sleep quality. Women with irregular exercise, BMI ≥ 27 kg/m^2, an abnormal depression evaluation, metabolic syndrome, and moderate-to-high risk of sleep-disordered breathing had low sleep quality, as were men with an abnormal depression assessment and moderate-to-high risk of sleep-disordered breathing. In the analysis of all dimensions of sleep quality, women had significantly poorer scores than did men in subjective sleep quality, sleep latency, habitual sleep efficiency, sleeping medication use, and daytime dysfunction. In a logistic regression analysis on sex modification in various aspects of sleep quality, men and women with abnormal geriatric depression results had higher risks of poor subjective sleep quality, sleeping medication use, and daytime dysfunction than did those with normal depression results. Men and women with moderate-to-high risks of sleep-disordered breathing and daytime dysfunction, respectively, had higher risks of sleep disturbance than did their counterparts with low risks. Conclusions: The most common sleep problems in men were sleep disturbance and longer sleep latency, whereas those in women were longer sleep latency and sleeping medication use. Both men and women frequently experienced low subjective sleep quality. Abnormal depression assessment was the major factor influencing the subjective sleep quality of men and women, as indicated by their use of sleeping medication and daytime dysfunction. Sleep-disordered breathing among men and long sleep latency and short sleep time among women were the major factors affecting sleep distress and daytime functioning.