Gray Decision Intelligence (Gray DI) is pleased to present this overview of PES Markets, a module of our Program Evaluation System (PES) software application.

Gray DI PES helps higher-education institutions make better-informed decisions that improve growth, efficiency, and student success. The Gray DI academic Program Evaluation System (PES) is the only platform that provides data by program on external markets and institutional performance.

PES Markets includes data on student demand, jobs, skills, and competition for over 1,500 academic programs. It predicts potential enrollment for your current and new programs. It corrects labor supply data; for example, 200,000+ online students are usually shown at headquarters’ locations; Gray DI reports them where they live.

Data Dashboards: Your subscription to PES Markets includes access to Gray DI’s Data Dashboards, which offer robust datasets on Job Postings, Alumni Insights, Program Enrollment, Google Keyword Searches, International Student Demand, Athletics Benchmarking, and NonDegree Demand

PES Markets has at least three independent data sources on each of these topics so that program decisions are not vulnerable to the idiosyncrasies of a single source. For example, our student demand data includes IPEDS completions, Google search volumes, and enrollment from the National Student Clearinghouse. PES Markets also provides detailed information on competition, including program size, growth, modality (online vs. on-campus), US News Rankings, and selectivity. PES Markets is now used by over 100 higher education institutions in the US, ranging from some of the largest online universities to local community colleges. Every year, we also research emerging programs that are too new to be included in traditional data sources. PES Markets will address the following:

▪ Provide regional and national academic data to position your institutional brand in those markets

▪ Analyze historical data and information to assess the University’s current academic program offerings, including market share and future demand potential

▪ Evaluate the strength of current program offerings

▪ Identify opportunities for program growth and enhancement with the current portfolio of programs

▪ Student demand for different types of academic programs and the development of strategies to gain a greater market share by program

▪ Assess workforce demand and alumni outcomes by program and key components in the curriculum to ensure alignment with workforce needs

The goal of a program portfolio strategy fueled by PES is to enable investments in programs that drive institutional growth, financial vitality, and realization of the mission. PES Markets brings data, rigor, speed, and scale to program analysis. Schools no longer need to evaluate one new program at a time—and hope to hit a winner. You can evaluate all potential new programs and pick the best for development. Schools can also evaluate all current programs by campus and decide where to invest and where to cut. The time required to make programmatic decisions can be reduced from months to days (or even minutes), which is increasingly important as the pace of change in education continues to accelerate and competition becomes more intense.

Fundamentally, PES Markets enables you to:

A successful new program can attract hundreds of students, stimulate millions in tuition and alumni giving, and generate considerable prestige. Using PES, institutions can consider all programs and select the best for your school.

Poor program decisions incur actual losses and opportunity costs. The most visible penalties are for new programs that fail. Fortunately, these failures tend to be small, since few students enroll, but they are expensive, somewhat embarrassing, and may increase regulatory risk.

Traditional, one-at-a-time program evaluations allow colleges to identify good programs to launch; however, they are often not the best programs. This leads to the most expensive error in program strategy: missing the biggest opportunities for growth. PES, combined with the judgment of faculty and institutional leaders, helps schools choose the best programs.

PES contains metrics on student demand including Google keyword searches, job postings, market completions, and enrollment. This information helps your team evaluate student demand for current and new programs.

Existing programs fall into three groups: Start, Stop, or Grow. Deciding which programs fall into each category ensures that the right programs receive the resources they need.

Program research and development become more expensive as the work on a program proceeds. PES lets you quickly select the programs that merit more expensive research and development.

Gray’s PES approach to academic program evaluation and prioritization combines comprehensive and customized data, advanced analytics, and a robust BI platform to improve and accelerate the evaluation of current and potential programs.

PES Markets combines over a dozen data sources on the market drivers of a successful academic program, including enrollment, Google keyword searches, demographics, competition, job openings, job postings, wages, and placement rates. We have mapped all the data down to the census tract level so you can pull data for the exact geographies you serve or aspire to enter. We have developed sophisticated crosswalks that link related variables, such as jobs, skills, and wages, to academic programs. We have invested in a BI tool (Astrato) that enables you to rigorously screen hundreds of programs in several markets at one time.

Often, institutions research a few new programs per year and rely on manual processes to pull data from industry and regulatory databases. This approach has several weaknesses. It tends to be slow, error-prone, and difficult to scale up. Resource constraints may lead to simplistic analyses, such as lists of competitors that miss IPEDS data on program size. BLS data may be pulled for the most common job for a program, ignoring other fields that compete for these jobs, and other jobs that may be appropriate for graduates. Since not all program options can be explored manually, programs that emerge from this approach may be good, but far from the best programs the school could have chosen, leaving millions of dollars in potential revenue on the table. In contrast, PES enables you to evaluate and compare all potential IPEDS programs for each individual local market using the best available data on student demand, competitive intensity, and job opportunities.

PES provides its data using secure, private, cloud-based servers and a BI application called Astrato. This combination ensures that you have fast, reliable access to PES from any device, anywhere there is internet access. In particular, Astrato allows you to point and click to select data (no SQL needed) and pull a custom report in a few seconds from the millions of records in our dataset.

To customize PES Markets for your institution, we pull data for your markets and work with you to tailor a program scoring rubric for your school. Your markets are custom-defined, typically a radius around your main campus and other campus locations or geographic regions you serve. Using the data on these markets, we score current programs and work with you to refine the scoring rubric until it correctly evaluates your current programs. The validated scoring rubric is then applied to over 1,400 IPEDS programs in Markets. A typical work plan is outlined below:

This meeting will help Gray begin to set up your instance of PES. We will review the tasks and schedule for your system installation. You will let us know the users you would like trained. We ask for the data needed to define custom markets. We will also give your team an overview of

We work with you to clarify the geographic scope for program analysis (e.g., within 30 miles of a city center). In most cases, our clients provide a list of campus locations and a file of starts that Gray analyzes to find a radius that covers substantially all your students.

For the selected geographic areas, Gray creates a unique instance of PES with the relevant student demand, competition, and employment data for all programs in IPEDS. This includes metrics created from National Student Clearinghouse enrollment, IPEDS completions, BLS employment and job openings, job postings, and wages.

Once Markets is customized and configured, we provide you with secure login details and one half-day of training for your users. In the training session, we start with an explanation of the data we use and its limitations. We review the scoring system and help your users learn to pull program reports by program and location.

As part of your subscription, we refresh your instance of Markets as new data becomes available. Our PES Markets Data Dashboards are updated monthly. We update monthly for keywords, job postings data, international page views, and non-degree demand, and annually for data derived from IPEDS and BLS. Enrollment data is updated three times per year by term.

Gray’s Customer Support Hub is available 24/7 on our website, and our Customer Success team hosts monthly office hours to answer questions and discuss new features or use cases. We provide a limited amount of coaching to help your users get the most value from our systems. For example, on request, we will walk them through how to pull custom data and reports. We also answer questions about data interpretation. We have also reviewed and commented on draft analyses using data from these systems. This service is intended to support your use of PES but is not a substitute for a consulting or analytical support engagement. Therefore, Gray reserves the right, at Gray’s sole discretion, to limit the amount of coaching provided.

PES allows for custom market analytics. Because we map market data down to the census tract level (tracts are twice as precise as zip codes), PES can be customized to your exact market definitions. Using census tracts also ensures PES data aligns precisely with US Census data on population, ethnicity, income, and other factors. Gray will work with you to clarify the geographic scope for program analysis (e.g., within 30-35 miles of your campuses for on-ground programs, or a larger radius for online programs). This task may involve discussions about whether to include or exclude particular markets or sites. We can create multiple market definitions to align with different campuses, programs, degree levels, and/or modalities.

In the example below, 90% of students come from within 25 miles of a campus. In this case, a 25-mile radius would be appropriate for defining markets since it would include the vast majority of students without accidentally including potential students, competitors, or jobs that are not really part of the college’s market.

For the selected geographic markets, Gray creates a dataset for all programs in IPEDS. We pull all the data for the radius you choose, around each of your campuses. Using Astrato, you can evaluate campuses one at a time, in groups, or in total.

Our program evaluation criteria fall into four categories: student demand, employment, competitive intensity, and degree fit. Within each category, we use several different metrics to cross-validate the information and provide a comprehensive rating. We can also construct additional or different metrics to better align with your view of the market. For example, different institutions target very different mixes of award levels for their programs. The chart below illustrates the categories and metrics we normally use.

PES produces a Program Ranking Report. Once you set your award level and select a market, the Program Ranking Report scores and ranks your current programs, all other IPEDS programs, and any other groups of programs you choose. This report allows you to identify current programs with strong growth potential and the best new programs to offer in this market. It will also identify current programs that may not be a good fit for the market.

The example below shows the top-scoring bachelor's programs in California. The report ranks programs by overall market score (top-to-bottom). The bar colors represent the scores the program received on each individual component. This allows you to see, at a glance, how each demand category contributes to the overall market score for the program.

A unique feature of PES is the availability of data on student demand. Many program assessments focus on labor market statistics to ensure program graduates can find jobs. While vitally important, PES also tracks student demand to ensure there is enough interest among prospective students to fill seats.

The ranking is a starting point for program decisions. PES also provides a Program Scorecard with data and your scores on over 50 metrics for each program in a single view. Below is an illustrative Program Scorecard for bachelor’s-level Computer Science programs in the California market.

The scorecard is broken into quadrants that correspond to the market demand categories: student demand, competitive intensity, and employment opportunities. Within each quadrant, the raw value for each criterion is shown in the Value column, and the score associated with that value is shown in the Score column. The Percentile column indicates how each category's value and score compare to all other programs in the market for that criterion. A percentile color key is included at the bottom of the scorecard.

For example, the scorecard below shows there were 3,935 new student enrollments at the bachelor’s level for this program; this enrollment volume is in the 99th percentile for all programs in the market and, therefore, is shaded in green and receives a score of 5.9 for that metric.

Aggregated scores for each demand category are shown next to the category label and an Overall Score for the program is shown at the top of the scorecard with a color-coded box to indicate the program’s percentile rank compared to all other programs in the market.

The Student Demand metrics included in PES (referenced on the sample scorecard on the previous page and shown in more detail below) quantify student interest in academic programs by location, degree level, and modality. PES includes several data sources to triangulate on total volume and trends in student demand. The most authoritative and complete source is IPEDS (Integrated Postsecondary Education Data System), which includes data on completions by program for all Title IV institutions. It tracks completion data by program and degree level, from certificates to post-docs, so you can tell roughly how big a program is, and whether it has been growing.

Unfortunately, IPEDS data is intrinsically old: people choose their major years before they graduate, so program completions reflect decisions made four years earlier, or more. To address this issue, Gray DI developed a data set using National Student Clearinghouse enrollment data, which we update with new enrollment volumes three times a year. We also track Google search volumes for over 15,000 program keywords for over 900 IPEDS CIP codes, which we update with new search volumes each month. International student demand captures the number of pages viewed by program, country of origin, and destination location for students around the world. This data includes information on the student’s location, degree level sought, program of interest, and whether they want to take it online or on-campus.

Unique to PES, we have also added estimates of online completions by in-market students, based on data from IPEDS, NC-SARA, and our proprietary brand database, to understand how many in-market students are completing the program online, including those that complete at out-of-market institutions. This can be an important indication of how many in-market students are “exporting” tuition revenues to out-of-market institutions and can help colleges and universities prioritize distance education efforts
​
The system includes all current data for these metrics, as well as year-over-year changes, so you can see if demand is trending up or down in each category.

The Employment Opportunities data in the scorecard quantifies labor market data for all academic programs in a geographic market. Gray compiles data from several sources, including our proprietary job postings database, an enhanced CIP-SOC crosswalk based on ~70 million profiles in our Alumni Insights dashboard, Bureau of Labor Statistics employment and wage data, and American Community Survey (ACS). We also provide metrics on Job Postings and Job Openings per Graduate.

A sample of the Employment section of a program scorecard is shown below.

PES includes job posting data in the Employment quadrant of the program scorecard. This data includes:

▪ Count of annual job postings

▪ Job postings per graduate

▪ Job openings per graduate

▪ BLS Wages (early- and mid-career wages, wages by educational attainment)

Unlike BLS, this data is current (to the most recent quarter) and tracks actual postings, rather than survey data. The example below shows past-year job postings in California, by program. Data is customized for each institution to pull job postings by geographic market, award level, and program.

The scorecard includes several sources of data on competitors. PES tracks completions in all markets and average and median program size. Additionally, we have developed metrics to assess market saturation and competitive intensity using data from IPEDS, Google, and the Census. For example, PES tracks completions per capita for over 300 markets, to compare and gauge local market saturation.

PES also tracks national online completions for the program, including the number of institutions nationally that offer the program online and the number of institutions with in-market graduates.

For each program and market, the system produces a Report on Competition. The report provides the name, degree level, and number of completions for every competitor for the last five years (including online). Institutional and demographic data is also available, such as sector, tuition rates, selectivity metrics, graduation rates, student body composition by ethnicity, gender, and age, and school rankings.

This report lets users quickly identify relevant competitors, whether they offer the program online, the size of each competitor’s program, which institutions and programs are growing or declining, who has entered or exited the market, and how each institution compares on institutional and demographic metrics.

The example below shows a Report on Competition for bachelor’s-level Computer Science programs in the California market.

Once you have identified a program of interest, you can quickly get a summary report that provides an overview of market demand along key dimensions.

Within PES, you can select the program and click a button to get an AI-generated text report on the program in your market. The text summary highlights key market metrics and trends. You may request an unlimited number of reports. Each report takes a minute or two to produce. A text report summarizes the data in a program scorecard (see next page).

If you want to examine individual metrics more closely, our dashboards provide detailed information on student demand and labor market data. For example, in our Program Enrollment Dashboard, you can pull data on new and total enrollment by academic year and term, market, and award level.

One of the most valuable features of PES is the ability to pull custom data and reports from the comprehensive database underlying the system. Users can use filters to select data by program code, program title, campus or geographic market, or a larger program grouping such as 2-digit or 4-digit CIP code. The example below shows data for Computer & Info Sciences (CIP 11) in California.

For the selected programs, users can then select the types of data to include in the table. The categories include the four categories used for program scoring, and the user can select any combination of data elements within each category. The example shows selected data elements: Google search volume and YoY change; job postings total for one year, job postings per graduate, 25th percentile and median wages, in-market campuses with graduates, median program sizes, and online competition.

As with all data tables in PES, users can also download data directly to an Excel file for further review and analysis.

Additional Questions? Please contact support at support@graydi.us or register for a Daily Office Hour Session!