Request a DemoCSO Research, Inc.’s comments on the reauthorization of the Higher Education Act related to the collection of consumer information
Matthew Berndt, Vice President
CSO Research, Inc.
I am writing to comment on the importance of valid, reliable and comparable career outcomes data on college graduates and to encourage the federal government to focus policy-making efforts on establishing and enforcing clear and transparent data collection and reporting standards when reauthorizing the Higher Education Act.
I am a higher education professional with more than 20 years of experience working in college career services, workforce development, and software-as-a-service, all focused on assisting the emerging college-educated workforce to bring their educational and career goals into alignment and launch and manage successful careers. I am also the parent of a high school senior who has recently completed the process of selecting a university and major field of study and will soon be embarking on her college career.
As a parent, as a career services educator, and as a technology solutions provider serving the higher education market, I am keenly aware of the importance to the consumer of valid, reliable, comparable and transparent career outcomes data when evaluating and selecting college and major fields of study, while pursuing a college education and discovering the career options said education will offer, and after graduating – when data on long term career outcomes is essential to effective career/life planning and management.
Current data collection methods at the campus-level lack consistency, clarity and statistical reliability and validity and result in data that misrepresent actual graduate career outcomes.
Apart from standards for career outcomes data collection for law and MBA graduates, set respectively by the National Association for Law Placement (NALP) and the MBA Career Services and Employer Alliance (MBA CSEA), well-established data collection standards have not existed for the remainder of the higher education community.
In January 2014, the National Association of Colleges and Employers (NACE) introduced its first effort toward a uniform set of data collection standards when it released its First Destination Survey Standards and Protocols. NACE’s standards are, by their own admission, “aspirational” in nature; a first attempt to bring standardization to first destination career outcomes data collection for postsecondary education. This is an historic and important effort to be sure, and one which should be recognized as such, however the NACE standards do not currently adhere to best practice methodology for institutional survey research, nor expect such adherence from their reporting institutions.
This lack of data collection standards makes any accurate comparison between institutions difficult at best and misleading at worst. This issue is often magnified by the common practice among institutions of including data from secondary or tertiary sources such as employers, social media, family and friends. There is no clear standard for vetting such information and the decision to include or not include such information is largely subjective. Given the vested interest institutions have in reporting high rates of employment and high salaries, this data collection practice creates a clear conflict of interest.
There are also inconsistencies in terms of what may be reported via such third party data sources, as one is more likely to post about a new job on LinkedIn or Facebook than to post about continuing unemployment. This data collection practice produces results which are positively biased and therefore unlikely to present an accurate representation of post-graduation career outcomes.
More often than not, data on income, employability, and career outcomes for new college graduates are simply not available and when they are, these data are neither comparable nor representative.
Absent any clear standards, many schools sacrifice data integrity in an effort to achieve high data collection rates which are often misrepresented as response rates reflective of actual placement rates.
Existing federal and state-level data collection initiatives also fall short of the goal of collecting reliable and representative career outcomes data.
In his March 25, 2015 Chronicle of Higher Education article “States Are Eager to Collect Graduates’ Job Data. Here’s Where That Effort Stands”, Lance Lambert presents the shortcomings of current federal and state level initiatives career outcomes data collection.
In many state-level efforts, graduates who work out of state, work out of the country, work for the federal government, or work part-time are not included in reporting. Most reports do not account for graduates who pursue additional higher education or those who pursue service or other gap year programs, all of which are common activities for many new college graduates.
Further, most state and federal level initiatives are predicated on the notion that there is (or perhaps should be) a linear relationship between what a student studies in college and the career(s) that same student pursues upon graduation.
While one would clearly expect that individuals majoring in accounting and engineering will most likely pursue careers in these professional fields, the same expectation cannot be had of individuals pursuing degrees in the liberal arts and sciences.
Likewise, one should not expect that the motivations and expectations of individuals pursuing college degrees are directly comparable.
Lastly, and regardless of a student’s chosen major, their first destination may or may not be indicative of their long term career path, employability, earning potential or professional motivations. Longitudinal career outcomes data – data five, ten and fifteen years post-graduation – are more instructive and beneficial to the career decision-making process that begins as students are considering their options for postsecondary education.
Students need, and do in fact have a fundamental right to relevant data to evaluate their postsecondary educational and career options prior to enrolling in a program of higher education, but the notion that all students’ decision-making criteria are identical is fundamentally flawed.
Assessing the value of a college education primarily on the earning potential of the resulting career options places undue value on the highest paying career paths, devaluing the role of teachers, social workers and many public servants.
Assessing the value of a college education primarily on the quality of the educational experience had in pursuit of the degree places undue value on the simple pursuit of education for education’s sake.
A balance needs to be struck that recognizes the complexity of the process, the simplicity of the goal, and the need for data everyone can trust and compare.
The need for universal career outcomes data collection standards and protocols.
My company, CSO Research Inc., recognized the need for valid, reliable, trustworthy and transparent career outcomes data and developed The Outcomes Survey® (TOS) to collect and report career outcomes data in a manner that meets the institutional needs of U.S. colleges and universities, meets current reporting standards, and addresses the emerging federal reporting requirements. As such, we have created a national standard for first destination career outcomes data-gathering and beyond.
Data on post-graduation career outcomes is only valuable to the extent that it is comparable between institutions. Institutions using The Outcomes Survey utilize the same survey questions on core career outcomes, making comparison and aggregation more meaningful. In addition to the core career outcomes questions, institutions using TOS can gather comparable data on student engagement in their search for a post-graduation occupation, the role of experiential education in their securing a first destination, their satisfaction with their first destinations, and their motivations in pursuing their first post-graduation occupations.
This standardization allows for national aggregation and the production of meaningful data on post-graduation outcomes across the country. Currently, more than 120 institutions in 38 states are using TOS to collect data for spring 2015 and beyond.
Commentary on specific recommendations in the “Consumer Information: Concepts and Proposals” white paper.
Following are comments and observations directly related to the specific recommendations in the “Consumer Information: Concepts and Proposals” white paper.
“Increase data quality and transparency for federal program management and for informed consumer decision-making.” (p. 9)
“1. Allow the new Outcomes Measures Integrated Postsecondary Education Data System survey that is the result of the 2008 reauthorization to go into place before moving forward on new improvements. Based on the limitations of the new Outcomes Measures survey, evaluate the next iteration of data quality improvement to move towards making sure all students are counted.” (p. 9)
The existing IPEDS Postsecondary Outcomes Measures related to career outcomes (employment rate, unemployment rate and average salary; all by major field of study) do, in fact, have significant limitations that should be addressed prior to adoption of any improvements or enhancements.
I recommend reviewing existing education industry standards to better understand how colleges and universities are seeking to gather standardized career outcomes data. These standards include those developed, endorsed, and often used for accreditation by associations including the National Association of Colleges and Employers (NACE), the MBA Career Services and Employers Alliance, the National Association for Law Placement (NALP), and NASPAA.
“4. Create a national graduate survey to voluntarily collect information from students regarding their careers post-graduation.” (p. 11)
I recommend that the federal government establish reporting standards and guidelines and require schools to adhere to these standards rather than creating a national voluntary survey that will ultimately be supplementary to surveys already conducted by many universities. The mandate should come from the government. The responsibility for fulfillment should belong to the individual schools, which will be in the best position to determine the most effective way to meet the requirement in a manner that also meets campus-level data needs related to accreditation and continuous improvement.
“5. Rely on third-party data that is not technically federal data in order to evaluate institutional success.” (p. 10)
I strongly recommend relying on third-party collected data rather than federal data or campus-collected data. Federal data is very limited in scope and therefore also in quality, and data collected independently by the schools themselves will always be subject to scrutiny around reporting bias, as schools have a vested interest in reporting positive data.
“Consider requiring that non-federal data on student success includes all students, be provided in the aggregate, as well as by agreed upon subgroups, with some federal oversight of privacy, collection and statistical practices.” (p. 10)
I agree that non-federal data on student success should be provided in aggregate so as to protect the identity of individual respondents. These data should be reported by school, degree(s), major field(s) of study, gender, ethnicity, veteran status, disabled status, geographic location, and grade point average.
“Institutions could be allowed to submit information from existing databases maintained by third-party organizations or states to provide a fuller picture of all students’ success and progress towards degrees or certificates.” (p. 10)
As a third-party provider of survey research services to colleges and universities, I am very much in favor of allowing institutions to submit data collected by third-parties on their behalf, provided the data gathering is conducted according to best practices for human subjects institutional survey research. Sound survey research practices yield valid, reliable, trustworthy and generalizable data.
Non-verified data on graduate career outcomes culled from various public sources (e.g., social media sites) is not by definition valid or reliable and therefore cannot be trusted or generalized in the same manner. It is valuable, anecdotal information beneficial for broadening one’s general awareness, but it is not data upon which policy decisions should be made.
“Institutions could permit third-party organizations or states to maintain and report student success measures on behalf of institutions to the federal government.” (p. 10)
It is our preference that institutions own the responsibility for reporting their data to the federal government. To assist them in reporting, our company prepares customized one-click reports of their data based upon prescribed reporting requirements. We have done this successfully to allow our clients to easily compile and submit annual NACE and MBA CSEA report.
