Americans Speak on Health Reform: Report on Health Care Community Discussions

Appendices

B. Methodology

The Health Policy Transition Team's review of Health Care Community Discussion reports consisted of three parts: an analysis of group reports submitted by hosts to Change.gov; an analysis of individual Participant Surveys submitted by hosts to Change.gov; and an analysis of the host sign-ups and participants. The Transition Team received approximately 4,100 Health Care Community Discussion group reports through the reporting Web site on Change.gov, either from uploaded documents or comments in a text box. These submissions were screened by Health Policy Transition Team members and volunteers to determine if they were a group report from a Health Care Community Discussion. The review team determined that approximately 825 documents were not group reports²⁰. As such, the Health Policy Transition Team and the trained volunteers read through and analyzed 3,276 Health Care Community Discussion group reports submitted to Change.gov.

With guidance from qualitative research experts, trained volunteers systematically labeled or "coded" sections of text in each of the group submissions using Atlas.ti, a computer software program designed to analyze written documents²¹. These codes provided an organized and comprehensive list of the topics participants discussed and the nature of those comments, which helped to identify major themes or distinct and recurring ideas expressed across all of the reports. The Health Policy Transition Team and qualitative research experts developed 95 manual codes to apply to various words and ideas in the group submissions. These codes, the critical ingredient in qualitative analysis, were generated by reviewing the topics in the Participant Guide and by reviewing a large sample of the group summaries to identify responses to those topics as well as other comments, ideas, and solutions. These codes were organized into six categories:

1. Biggest problem (including costs, access, quality, and problems with the overall system);

2. Other major problems or major concerns (such as unhealthy behaviors, shortages of key providers, and lack of information);

3. Impact of problems on various groups (such as state government, small businesses, providers, patients, families, the uninsured, and hospitals);

4. Hopes and visions for a reformed health system (such as less complex/simpler, comprehensive coverage, emphasis on wellness/prevention, and patient-centered);

5. Roles and responsibilities of various groups moving ahead (such as consumers, patients, employers, doctors, churches, businesses, hospitals, insurers, government, and schools); and

6. Specific suggestions or recommendations (such as Health IT, wellness education, public health improvement, and building (or not building) on the experiences of other states or countries).

After entering these codes and all of the Health Care Community Discussion group reports into Atlas.ti, the Health Policy Transition Team and trained volunteers read through thousands of Health Care Community Discussion group submissions on computers and applied codes to relevant sentences or paragraphs by highlighting the relevant text and selecting the applicable code. For example, a paragraph that discussed the shortages of hospitals and doctors in rural areas would be coded with "Access To: Hospitals, Doctors, Rural Concerns, Shortages." In addition, group reports were coded to identify whether the majority of a meeting's participants were everyday Americans, providers, or members of an advocacy group.

In addition to manually coding each document, the reviewing team also used the "autocode" feature of Atlas.ti, which searches for words, variations of words, or phrases and then applies the relevant autocode. The Health Policy Transition Team and a team of volunteer qualitative researchers helped develop "autocodes" to systematically capture themes. The autocodes covered a single-payer system, veterans, women's health, mental health, and malpractice. For example, several group submissions discussed veteran's care. An Atlas.ti autocode searched for the word "veteran" and then placed the appropriate code on the sentence or paragraph where the words appeared.

After the thousands of group reports were read, analyzed, and coded, the reviewing team ran searches by codes and code combinations in the Atlas.ti database to view the written text from the group reports associated with a particular comment or idea in order to identify the major themes. The software also has the ability to conduct simple counts, cross-tabulations, and export data to Excel or other software like SAS to conduct basic descriptive statistics (e.g., correlations) to better understand the major themes discussed by group participants and the range of views expressed. For example, the coding system gives a count of the number of times Health Care Community Discussion group reports highlighted that the biggest problem of our current health care system is cost, access, quality, or the nature of the overall system. The coding system also allowed the team to assess whether there were systemic differences in perspective or opinion based on group characteristics or where the Health Care Community Discussions took place.

The code information was then exported from Atlas.ti and analyzed by the volunteer team, including volunteer qualitative research experts. The volunteers compared the coding results by region, population type, per capita income, and unemployment and looked for trends and differences between the percentages of responses of each code for each of the above categories²². For example, the group compared the percent of reports from the Northeast that mentioned "Suggestion_Education" to the percent of reports from the South, West and Midwest that said "Suggestion_Education." They also compared the percentages within a code by region, population type, income, and unemployment. For example, within the Southern Region, the researchers looked at what code had the highest percentage of documents coded with that response and whether that was the same code for each region, population type, income bracket, and unemployment bracket. The researchers also looked for correlations between codes to identify trends and interactions. For instance, the researchers analyzed the values and solution categories to determine if there was a correlation between the "Values_Prevention" code and the "Suggestion_Education" code. The researchers asked: Was a report more likely to mention education as a solution if they mentioned prevention as a key value for the health care system?

The Health Policy Transition Team also received Participant Survey responses uploaded by hosts through the reporting Web site on Change.gov. After eliminating outlier responses, 30,603 responses were used in the analysis²³. Following the same procedure as the code analysis, the researchers analyzed the participant responses by region, population type, income, and unemployment. The results of the Participant Survey analysis were then compared to the results of the code analysis. The team looked for similarities and differences between the two analyses because the code analysis was conducted on the reports from open-ended, group discussions and allowed for multiple codes in a single category, and the Participant Survey responses were limited to one response per participant per question.

The Participant Survey and Health Care Community Discussion reports are distinct but complementary sources of information about the views of the public who chose to participate in this forum. The individual survey permits each participant who responded to express his or her opinions. The group reports capture the results of a dialogue among individuals and permit the expression of more complex points of views and differences of opinion on issues. For example, the Participant Survey addressed the issue of the "biggest problem" and permitted respondents to pick one item. In contrast, the "biggest problem" discussion in the groups generated responses on multiple problems and included responses not in the Survey response categories (e.g., underlying system structure or values) and responses on the interactions among those problems (e.g., because there is no real system or a system that prioritizes sickness instead of wellness or prevention, health care is costly and the system impersonal and hard to navigate). The other Participant Survey questions focused on other important topics related to the process of moving forward on health care reform, including how people would like to participate and what kinds of information would help them participate. Health Care Community Discussion reports also provided complementary information on these subjects.

The Health Policy Transition Team and qualitative research experts also analyzed the diversity of the people who signed up to be hosts and the participants who submitted Participant Surveys. Using the same categories as the code and Participant Survey response analysis, the researchers looked at the regional distribution, population type distribution, per capita income distribution, and unemployment rate distribution of the hosts and participants.

The quotes used in the report were edited to correct spelling, grammatical mistakes and for format; brackets were used to add language for clarity.