Benchmarking: Real World Results from a Real World Attempt
By J. Gary Augustson, Executive Director of Computer and Information Systems, The Pennsylvania State University
EDUCOM '94
Wednesday, November 2, 1994
By J. Gary Augustson, Executive Director of Computer and Information Systems, The Pennsylvania State University
EDUCOM '94
Wednesday, November 2, 1994
In the Spring of 1994, the information technology organization at Penn State embarked upon an aggressive program of benchmarking to determine how Penn State compared to other major research universities and to discover what practices other institutions used to achieve "best-in-class" status in using information technology to support institutional goals. This endeavor quickly became an active collaboration among the participating institutions.
Penn State initiated this benchmarking effort as a result of our more-than-decade long commitment to strategic planning and our more recent Total Quality Management activities. For several years to come, benchmarking will be the focus of our strategic planning efforts.
In selecting "best-in-class institutions" to benchmark against, we looked for institutions that were vigorously and successfully pursuing information technology solutions in a broad range of areas, rather than those "best-in-class" only in a specific area. Priority was also given to universities that most closely matched Penn State in mission and size--a large, complex, public land-grant university.
The five institutions selected to benchmark against were
Many of you I am sure are here to see the numbers we came up with. After all, isn't benchmarking about numbers? Well, here are some of our numbers!
The first thing we discovered is that the real value of benchmarking does not lie in the numbers. Lucky for us--because most all of the numbers we gathered were
Why? Because we are all different institutions, each is a product of its culture. Some practices that serve a particular environment well, may not be applicable in others. Simply put, there is no single "right way" to deal with the immense changes occurring in information technology. The primary driving force at all of the institutions is the need to adapt rapidly and effectively to technological change. The challenge is to manage change, rather than to be overwhelmed or outpaced by it. Each institution is finding ways to meet this challenge that suit its individual character.
These differences in institutional cultures made the comparison of numerical data difficult. For example, some institutions include capital expenses in the information technology budget; some do not. We found that, with the exception of some very simplistic measures, we were sometimes not just comparing apples and oranges, but apples and orangutans--in fact, Nittany Lions and Wolverines (and we all know how that comparison works out--even in Ann Arbor).
In fact, the numbers proved so troubling to the group that it was agreed that we would not publish them. We promised to dispose of them in a fitting manner. So Doug, George, Charlie, Mark, and Elazar and others--as promised--we are publicly burning the numbers. But, Doug and Sam, we didn't burn all the numbers.
The numbers aside, this benchmarking effort produced many positive results. The six institutions involved have begun building relationships that will yield increasing benefits over time. We know more about each other's successes and tribulations and we have made the commitment to work together over the coming years to look at issues and problems on a collaborative basis. I expect that this will be the major benefit of the time and effort invested by all of us.
We also uncovered some useful predictors of "best-in-class" operations. I will briefly touch on the most significant of these in this talk.
So what did we discover? We were able to loosely categorize our findings into four areas of activity that we believe to be critical to achieve the type of information technology excellence that we perceive to characterize "best-of-class" institutions. Thus, in general, we can say that "best-in-class institutions":
Let's look at these four areas in some detail, starting with some policy issues.
All of the benchmark institutions emphasized that, as the importance of information technology increases, more direct involvement of the senior information technology officer with senior university management levels is essential. For example, as information technology tools become more broadly disseminated and become key factors in every discipline, they demand a larger proportion of the university's budget. Formal relationships must be established to enable frequent exchanges between the Chief Financial Officer and the senior information technology officer and between their staffs. However, this is a particularly sensitive area and only a few of the six benchmark institutions have moved to do this.
Although even fewer benchmark institutions include the senior information technology officer as a member of the "president's cabinet," all perceived this to be an extremely critical move if they are to fully realize the potential of information technology. Wisconsin has moved to place the Senior Information Technology Executive as a member of appropriate University-wide, strategic/advisory/ planning committees. One observation made during our deliberations was ["If those who sit at the table understand the importance of information technology, the seat at the table is less necessary."] But in how many institutions is that actually true today?
Senior-level management support is also required for successful process re-engineering.
At UCLA, the Senior Vice President for Finance has taken a leadership position in eliminating paper and converting business practices to electronic processes. Particularly noteworthy is UCLA's adoption of post-audit procedures for electronic forms. Rather than forcing all actions through a lengthy pre-approval process, actions are executed, with copies provided to individuals who would have been in the approval chain. Problems are handled by exception, rather than holding each form to await a lengthy review process. UCLA expects to save $500,000 per year and recognize significant reduction in business office staff as a result of this practice.
Process re-engineering is also occurring in the classroom. Classroom instruction is, of necessity, evolving from the standard lecture format to one of collaborative learning, where the instructor is more of a mentor or coach than a dispenser of knowledge. This transition will rely heavily upon investments in information technology. The benchmark institutions have recognized the need to both evaluate and encourage the changes in instructional delivery that technology may enable. Several benchmark institutions have made significant progress in this area.
The increasing reliance on information technology demands an increased emphasis on development of information technology staff and human resource policies that recognize the value of such development. The University of Texas has a novel and effective approach to information technology staff development that is only possible through the active cooperation and support of Human Resources. At Texas, Human Resources is viewed as an asset and partner.
While I said before that the value of comparing raw numbers is extremely limited, we did come away from our exercise with some useful observations related to budget.
Some interesting observations include:
Central funding support for student microcomputer labs was shown to be critical to ensuring equitable access for students in all disciplines-- particularly freshmen and sophomores.
Access to special computing resources, such as those that support numerically intensive computing, often is best supported centrally. Penn State has begun to break down discipline barriers by centrally supporting a distributed computing environment where researchers throw their local workstation cycles into a pool--with the knowledge that they will receive more cycles in return from "the commons" than they throw in.
Finally, our most discouraging observation in the budget area was that all benchmark institutions are struggling with how to fund the regular replacement of information technology equipment as it ages.
We found that:
Some of the benchmark institutions are examining what the role of their information technology organizations really should be now and in the future. Two have restructured their organizations in somewhat dramatic fashion, based on the results of this process of introspection. Both are moving towards flatter organizations and are strongly emphasizing responsiveness to customer needs and customer satisfaction as critical measures of success. It is too early to gauge success or failure of these efforts. They are works in progress.
Quality improvement is a continuous process. Many benchmark institutions are adopting a formal quality improvement process to improve service to customers. Some have made a stronger commitment to formal quality improvement efforts than others, but, in general, this strategy is becoming a routine element of the information technology agenda.
Those of us from non-urban areas understand the value of infrastructure more than others. ["If you live at the end of a dirt road, you have to provide more infrastructure."] The prime example we found of the value of early investment in information technology infrastructure was in the area of telecommunications. Placing a high priority on early and sufficient funding for telecommunications infrastructure leads to faster completion and availability. While the approach to obtaining funding is highly dependent on institutional culture, five of the six benchmark institutions have essentially completed installation of basic cabling infrastructure both between and within buildings and are now able to turn attention to other critical needs, such as facilitating greater use of the technology.
Priority investment strategies are also being used to extend full network connectivity to student residence halls, sometimes taking advantage of non-traditional funding opportunities.
Turning to the area of customer service. . . .
Most benchmark institutions are strongly encouraging a customer focus, meaning to "serve the customer not control the customer." It is recognized that users have many choices to support their information technology needs. They are not necessarily bound to solutions developed centrally. Therefore, the relationship between the central information technology organization and its users must be viewed as collaborative and cooperative, not dictatorial. These organizations exist to serve users, not the other way around.
There is clearly an increased emphasis on what we refer to as the "Help Desk" function. Help desks are critically important--the more you help at the Help Desk, the more you save overall in secondary assistance.
Some institutions are consolidating their help desk functions, providing the user with a single point of contact, and lessening user confusion about which office to call in a given situation.
Some of the benchmark schools have large, centralized user support organizations. By institutionalizing and elevating the role of the user service organization, they have been able to emphasize its role as the primary contact between the information technology organization and the customer. In addition to consulting, the organizations handle training, software site licensing, sales and services, departmental services, public site management, and planning with the Schools and Colleges.
Many institutions are seeking better ways to support student users within the residence hall environment.
While universities have traditionally allocated minimal support for training, many of the benchmark institutions are developing creative means of coping with the increased demands for information technology training.
For example, Texas has developed an Internet access training model where technical tools and their use are taught by the information technology organization, and Library personnel, in an extension of their traditional role of facilitating user access to information, teach how to use the tools to access different subject areas.
Another approach is to enlist the aid of students in instructing other students. Most courses "for students, by students" are free with content being driven by peer needs, thus resulting in a more collaborative learning environment.
Many are seeking a reliable means to provide more secure authentication applicable to the highly heterogeneous environments that typify large research universities.
The schools most actively engaged in examining network security problems are beginning to explore options for encrypting at least some data. The concern driving examination of encryption options is not only with the information being read (confidentiality) but also with it being changed (integrity).
At present there is no firm consensus on how to accomplish this objective technically, nor even on the degree of technical difficulty.
That concludes my 15-minute summary of an incredibly intense process.
So what do we do next? Is this the end of the process?
First, we decided that there were areas where we ought to attempt to collaborate to help address some of the common problems that we face. While there are numerous areas where we might collaborate, we decided to first focus on how we can better support the incredible growth in our user community, both in its size and its demands. As a first step, the University of Wisconsin has agreed to take leadership in putting together a Union Catalog of User Support Materials and Tools. Our intent is to reduce redundant effort by sharing user support materials that each of our institutions uses to support our user community.
In examining our benchmark process, we determined that the major reason our comparative numbers were useless was that we had asked the wrong type of questions. We had asked about "input measures" rather than "quality indicators." In essence, we focused on "how many" rather than "how effective." We worried about the extent to which students had access to computing resources rather than what impact these resources were having on their educational experience. Thus, we agreed, as a group, to try to develop a set of quality indicators over the course of the rest of this academic year that we can use to survey faculty, students, and staff across our six institutions to determine the kind of impact information technology is having on our institutional mission.
This, to say the least, is a non-trivial task. We are not at all sure what a quality indicator is. We won't know how successful we are until we develop them and survey against them and look at ourselves next year to see whether we have learned anything. The University of Michigan has agreed to be the coordinating point for our six institutions in gathering and consolidating our suggestions for quality indicators.
So--at this point, we need to ask--was our benchmark effort worth it? From Penn State's perspective, the answer is yes. Thus far we have learned several things. First, we have found that identifying useful quantitative measures of quality is extremely difficult or in the words of Charlie Warlick, ["the only numbers that seem to be interesting are those that are difficult to find."] Secondly, we have seen a great deal of consistency in what institutions are doing, but not in how they are doing it. There is immense value in recognizing the cultures of each institution and the ways in which each is accommodating change. Third, both the benchmarker and the benchmarkees have benefited by sharing findings.
For those considering whether "to benchmark or not to benchmark," the message from our experience is that the real value transcends the numerical data collected. The most important elements lie in synthesizing the experiences of institutions of similar size, and finding common processes that may benefit all. As we continue working to develop useful "quality indicators," we do hope that at some time we will be able to share additional information about how information technology is helping to fulfill our respective missions.
Now, my thoughtful colleagues had agreed that I was the stuckee that had to give this presentation since I'm the one responsible for involving them in this activity. It was also agreed that my partners in this effort would participate in the discussion during the question-and-answer period which we have attempted to leave at the end of this. Most of them are sitting up in the front of the auditorium. What I would like to do now is throw the session open for discussion and invite you to direct questions to any of us.
The Pennsylvania State University ©1998
Design and content are provided by the Office of University Relations.
Technical support is provided by Information Technology Services (ITS).This page was last updated on Thursday, April 30, 1998.
Have a question? See our list of contacts.