You are currently browsing the tag archive for the ‘research’ tag.
So this afternoon, an unusual request sent me down a rabbit hole.
A colleague over in ITS asked me what seemed on the surface a relatively simple question: how many students that enroll come from rural environments within Pennsylvania?
During the course of the conversation, I expanded that to include those who visit our website but might not end up enrolling who were from Pennsylvania, since a lot of Penn State’s services also assist those outside of the officially-affiliated Penn State community. Not to mention that was an easier number to come by, relatively speaking.
To that end, I went to our web analytics and expanded to look over the course of a year. Just on the high end of traffic, it was pretty easy to see the major population centers in the commonwealth. Likewise, by going to the low end of traffic, you could see the less densely populated areas, though this was less accurate. But here we run into the problem that folks from rural environments may not be online, and thus not even making it onto this count.
So what about our enrollment numbers? There’s no checkmark (rural/urban) for us to bounce against, thus the Penn State Fact Book doesn’t list it, nor have we ever run these numbers before. After talking to one of my data-minded colleagues, our best solution was to tap into our sociology folks at the university for a list of “rural” zip codes: once we had that, from there we could run these numbers with some degree of accuracy. Problem is, even a ballpark was tough to come by until we had some definitions like this.
Simple question, yes? Not so very.
Prompted by an excellent post by Shelby Thayer, as well as a meeting today with IntelliResponse, our knowledge base provider, I’d like to dig into what you can find out from and how you can fine-tune a knowledge base.
I personally find it fun and entertaining to track our top ten over time and see how things shift. Yes, I’m a strange bird. I know this and embrace it. But watching “how do I apply” crawl up over inquiries for our academic calendar shows the shift in the year again: that shift in what students and parents are thinking about. Likewise, inquiries for campus tours.
And I was thrilled to see “is Penn State a multicampus system” actually gain enough traction to make it to one of our top tens (the one that people actually type in, rather than browsing the provided questions). Thrilled. Essentially, that means that the message that Penn State is not one place has gotten out there.
But I found, when I looked farther, that “2+2” goes unanswered if someone enters it. Likewise, “campus offer.” And these oversights (my own fault, mostly) are pretty egregious.
Now, those not familiar with the Penn State system might say “huh?” But both those questions should likely point to the “is Penn State a multicampus system” answer, or one like it. (2+2 is what we call our system of two years at a smaller campus, followed by two years at a larger one to finish an undergraduate degree; campus offer is an offer of admission to one of our campuses.)
As much help as we’ve gotten from IntelliReponse (and we’ve gotten a ton: they’ve been great), you really need someone who knows the institution and office in question pretty much inside and out to evaluate your answers on a regular basis. I’ve found miscategorized questions, questions that don’t get answered, and quite a few that are really just garbage–often someone with a juvenile sense of humor trying to see what they can get the system to answer to pretty ridiculous questions.
The miscategorized and unanswered worry me the most. That said, we’re almost at a 90% response rate, and haven’t been up and running too terribly long, so I’m not losing sleep over it. Yet.
I’ve seen as many misspellings as I think might exist for “Schreyer” (referring to our Honors College), including “shriars,” “shryer,” and “scyers” (virtually never capitalized, I might add), as well as referral to the aforementioned as “honros colelge.” I also know the Penn State system is confusing, but I really wish we could get enough info out there that people wouldn’t be so bewildered as to ask “what college is the university campus on?”
I’m also sincerely hoping that “convicted of a felony” does not become a common question to turn up.
Examples aside, paying attention to these and getting them categorized appropriately not only helps your numbers, it helps your community, and that has real results when it comes to customer service. I’ve found that it’s not just prospective students and parents using our knowledge base: it’s also Penn State staff and faculty, some of them internal to admissions.
And people are using it all hours of the day, from all around the world. So rather than staying up until it’s business hours on our side of the world, a student from Thailand or China can get the answer they need right then. Likewise those that are in our time zone and don’t want to pick up the phone or write an email. Not to mention those right at any one of our Penn State campuses.
So in the end, to serve your customer, you have to find out what they want or need and try to deliver it to them in the way that’s easiest for them to use, not the easiest for you to manage, although the two often intersect. Keep in touch with your community and serve them well, and you’ll find the benefit is enormous.
So as of yesterday, I’ve been approved to move forward with some usability testing on our online application for admission. As an excellent bonus, I’ll be allowed to waive the application fee of the students who help us by letting them observe them.
Our Directors Planning Team, which collectively makes most of the important decisions in the office (which I’m a member of), was somewhat characteristically interested in the details. Who are we testing? Just students who visit? Out-of-state students? Students in the different regions of Pennsylvania? Those who visit our Community Recruitment Centers? I took it a step farther to make it clear that breadth isn’t necessarily our aim considering our resources for the project–as of yet, I have no further funding approved–by suggesting transfers and international students. Further than that, we should look at adult students. But if I intend to test 5 – 10 students, there’s no way we’ll hit all these populations. How do we balance my funding, resources, support, and time? Not to mention equipment and travel, possibly?
The easy answer is: we don’t. Keeping it simple and keeping it close to home (so to speak) keeps it feasible. And we’ll get plenty of useful feedback just with that, potentially more than we can change in the time we have. If I get the funding and time I need, though, the equation changes completely.
This morning (during the Spend a Summer Day admissions presentation, I’ll admit, with the caveat that I’ve seen it five times now) I started mulling over my own questions about the project, which in some ways I find more important. What should we ask on a pre- or post-survey? Do we really strip this down to make it fast and easy and not have either? Do I want to publish/speak on this topic and thus slow us down with the IRB process? (No, probably not. What we do now falls under process-improvement and hasn’t required one thus far.)
Should I contact my Usability Engineering instructor from last semester and see if he knows a student who might be interested in helping with this very important research? Do we want to try to include someone with accessibility issues to make the point that we need to work on that end of things? (One rule of usability testing is not to go in with an agenda, so I probably shouldn’t pursue this.)
The questions go on and on.
In the end, we’ll probably end up on the end of a stripped-down version, because doing that way gets it done. Getting ambitious may mean it doesn’t happen, and I’d hate for that to occur. We need to know what real students do rather than continuing to make suppositions.