The Perplexing Role of Metrics in the Arts

Metrics are everything these days — they’re used to find ways to measure outcomes, finding ways to prove that your effects are having an impact, finding [insert buzzword and buzzverb], or [insert another buzzword and buzzverb], etc, etc. If in no other setting, most people are at least vaguely aware that metrics are a huge part of the major conflicts that have been taking place in the nation’s education system — specifically, the way in which many so-called “reforms” in education tie programmatic or teaching success to standardized testing scores. But issues around metrics exist in every sector, including the arts. The stakes in the arts may not seem as dire or far-reaching as in education, for instance, but it’s all part of a societal shift toward relying on fairly abstract numbers to dictate how decisions are made around funding, opportunity, and what groups of people are targeted in programming.

… it’s all part of a societal shift toward relying on fairly abstract numbers to dictate how decisions are made around funding, opportunity, and what groups of people are targeted in programming.

And the biggest issue around metrics, is that, when it comes down to it, it’s rarely entirely clear or consistent exactly how the numbers of collected, whether or not they’re valid, and, most especially, whether or not the numbers being used are really all that useful.

So, let’s get straight to one concrete and contemporary example I want to explore when it comes to metrics and the arts.

The Brooklyn Arts Council (BAC) serves what is probably the largest populations of artists in the US and possibly the world. Founded in 1966, the organization endeavors to grow the arts community in the borough by funding and presenting work for Brooklynites, providing services and grants to Brooklyn artists, and also engaging in arts education programming (among other things). And, in my experience, they work pretty hard to be inclusive in their activities, to reach artists and audiences in every part of the borough, and they embrace an expansive definition of what art is, including a robust folk arts program. I haven’t done any kind of formal analysis of the organization, so I’m sure others with a deeper relationship with them could give a more balanced picture, but the point of all of this is to say, they seem to do pretty good work and take the call to serve an incredibly diverse population very seriously. The point here is that my goal is not to slander or attack the BAC. In fact, I’m using them as example precisely because they seem to have a fairly progressive and robust approach.

What’s the issue, then? The Brooklyn Arts Council has a couple of granting programs where they make small bits of money available to artists and arts organizations to help fund the creation and/or presentation of art in the borough. Chief among them is the Community Arts Grants program, which disperses $350,000+ in small grants (ranging from $1,000-5,000) to Brooklyn-based artists and small arts organizations (those with operating budgets under $100K), and just closed its application process for 2013 last week. This is a re-grant program, which essentially means they get a big chunk of money (in this case, the primary named funding source for the program is the Greater New York Arts Development Fundof the New York City Department of Cultural Affairs), and then they divide a bunch of that money up into little bitty chunks that they give out as grants. Re-grants, particularly when it comes to government money, are pretty much standard operating procedure these days, but it would require a whole other big, long essay to go into how and why that came to be, so I’ll just leave it at that for now.

There’s an application process that you have to go through to apply for the BAC Community Arts Grants, with eligibility requirements, and mandatory attendance at one of the seminars that take place all over the borough where they explain the whole process. Again, pretty standard these days, though few organizations require the informational seminar — that’s another example of the BAC trying to make sure people have the support and info they need to get through the application process, which many funding organizations are not good at, at all.

What’s strange about the whole thing is a couple of questions that come up on the otherwise fairly concise and straight-forward application:

Age Group Served (by number):

Children (0-18)
Adult (19-64)
Seniors (65+)

Ethnic Groups Served (by percentage):

Asian
Black/African-American
Hispanic/Latino
White, Non-Hispanic
Mixed Ethinicty or Other

Now, it’s not unusual at all that an agency wants to get a sense of what groups in the population are being served by the programming that it’s working on and/or funding. You could quibble with the specific breakdown, but, ultimately, these days, breakdowns like this are de rigueur.

What is surprising is that they are asking for this information from an artist or an organization who is proposing a potential, future project that has yet to be realized, and, given the reality of most arts funding and creation, is likely to change in at least a couple substantive ways between the time it was proposed in this application and the time it is presented to the public. Anyone who has ever filled out a grant application of any kind, in any field, knows that things said in an application can often bear little resemblance to the final product — a week of torrential rain cancels the entire run of a four-day outdoor dance performance or the idea for the script that a playwright had only just begun at the time of the grant application morphs from a three person one-act into a 24 person musical in three acts.

And perhaps more importantly, what would an artist or arts organization base these predictions on? Almost none of the artists I know actively survey their audiences for demographic information, and many of them performance outside traditional arts venues. A couple of performance venues that I go to regularly do actively survey their audiences, but it’s optional, there’s no incentive to fill out the survey, and most of the questions are marketing-related, not about demographics.

In an effort to better understand why the BAC includes these two questions on their application, I sent over some questions by email to the director of the Community Arts program. Sara DeRose, their communications director, was kind enough to take the time to respond.

The most important point that came across in her answers to my questions is that the reason that the BAC asks those questions is because their funders require them to report on demographic information related to age and ethnicity (see the full list of funders for the Community Arts Program at the bottom of the first page of this PDF and a further list of BAC supporters can be seen here). In other words, BAC’s funders are measuring its success, in part, based on demographic data. Again, not so surprising, nor is it controversial — the hope, I think we can safely assume, is that this is helping to drive a push toward spreading programming and money out more equitably within a given population.

What is unusual is that the first and primary method by which that demographic data is being collected is through predictions made by artists, who, by and large, I’m going to assume, are not trained to interpret census data or likely to take the time out to survey their audiences when they’re already struggling to get people to show up in the first place, not to mention, rehearsing or working in the studio, maintaining a paying job, producing the show, etc..

Here is an excerpt from Sara’s email in response to some of my questions around the application’s request for demographic predictions:

Q: Do you provide any sources for artists to extrapolate that data from in the materials or seminars that you offer to applicants?

A: At our grant application seminars we encourage applicants to base the size and demographic breakdown of their projected audience on their typical audience for past work, on how they plan to market the project, and where it will take place. We receive a lot of one-on-one questions from applicants on this particular question, and we remind them that what we’re looking for is their best educated guess since the exact audience is impossible to predict. We also encourage applicants to base their numbers on information about typical audience size/demographics from their partner venue(s). In cases of public art installation where the audience is people walking by on the street, we refer them to census numbers for the neighborhood in which their project will take place.

Q: How do you make use of the demographic data artists report in the grant application process and after—i.e. do you report the predicted numbers in your own literature and/or to funders?

A: Whenever possible, we use the actual reported numbers on the demographic breakdowns in an awardee’s final report instead of their projected numbers. In the final reports our grantees fill out we ask for actual numbers of who showed up to an event, so would expect higher accuracy there than in an application. The numbers are primarily for our own analysis of our programs and reporting to funders.

Q: What, if any, mechanism is in place to ensure that their predictions match with their actual audiences once the project is realized?

A: BAC makes recommendations to artists and organizations for how to best estimate their numbers in the first place (see answer above), then requires demographic breakouts and actual audience size on final reports from our grantees.

We are required by our funders to have BAC representatives attend many of the events we fund. We have BAC staff, past panelists, and volunteer auditors attend funded events and report back on their overall impression of the event as well as size and perceived demographics of the audience.

Q: Do you have a sense as to whether or not people inflate or change the percentages they list based on assumptions they have/make about what the priorities of BAC are?

A: We rely more on final report demographics than application demographics whenever possible. At seminars and in one-on-one conversations, we encourage applicants to be honest and straightforward in describing the project that they actually want to do, and not to describe audience outreach they’re not really committed to, or move the project to a neighborhood they think they’ll be more competitive in if it doesn’t make sense for the project, etc.

We try to clarify that while BAC’s mission is to spread grant funding across a variety of cultural communities and neighborhoods in the borough, any one individual project does not have to accomplish all of these goals.

I believe BAC is trying to go through due diligence in following up on these numbers and even doing the work of genuinely trying to figure out how to represent demographics and ensure inclusion in their programming.

The issue is that all of this shows just how weak the use and collection of on-the-ground metrics really is, even in the best of circumstances. Reading DeRose’s answers it’s clear that BAC does rely on these predictions in some part of the grant-making decision process and at least some of their external and internal reporting. And they also rely on “perceived demographics” in some instances, which I’m sure we can all recognize are shaky, at best.

Metrics are not all they’re cracked up to be, even when put to good use, because for the majority of people, they’re consuming a simple set of numbers in an annual report or a newspaper article. People want to believe in the veracity of those numbers and make assumptions about them being a kind of scientific indicator of truth, when the reality is, as we can see here, they are often best guesses, or worse, fabrications.

The goal here is not to stand in the way of an organization trying earnestly to increase the inclusion in its programming, what I want to do is point out the fact that even an earnest and responsible arts agency runs into trouble when it comes to accurately sourcing the metrics by which they, and, importantly, their funder, measure their success, which raises some concerns about what less scrupulous organizations are doing when it comes to numbers like this, and how that rolls up to the funders who base the spending of millions of dollars in arts funding on these numbers. And what does the top-down dictation of metrics mean, when the people at the top, i.e. the funders above BAC, hand down requirements like this, but take on none of the burden of actually assessing the numbers and their usefulness.

… so much of grant-writing already feels like writing the story that other people want to hear instead of the story you want to tell …

And when it comes to this specific instance, I feel like the Brooklyn Arts Council should know the census data for different neighborhoods in the borough and have access to those numbers. Artists and arts organizations shouldn’t be charged with trying to do the work of interpreting census data on their own, it adds needless complication and puts an undue burden on artists in the application process. An agency charged with knowing and understanding its community, should, at the very least, be able to provide demographic data about the neighborhoods it operates within.

Also, if they’re asking artists and arts organizations to estimate future demographics based on past demographics, then they should simply have the organizations report past demographics or have the artist report the average demographics for the venue as such, not as estimates for a future, possible project. Phrasing the question in this way feels a little like passing the buck, as if they’re trying to put all the responsibility for the numbers they report back up to their funders on the artist, which again, is this top-down pressure that puts the greatest burden on the people who receive the smallest amounts of money but are required to do most of the work.

I can remember feeling, as an artist, when I filled out one of these applications in 2007, like a stooge, like no matter what number I put in there, I had no real basis in reality for any it. And when so much of grant-writing already feels like writing the story that other people want to hear instead of the story you want to tell, having to try to present a piece of that story as a kind of statistic just felt really yucky.

And lastly, if demographics like these are a priority in all of BAC’s reporting, then it behooves them, and the funders who require these numbers of them, to work with venues and artists across the borough to discuss how to accurately and actively survey an audience — what methodologies are appropriate and robust (social scientists do this kind of thing as their job, and New York is jammed with them in many of the universities around the city, surely a couple of them would help out), how to handle audience surveys in a public environment where there is no fixed audience, and also push for organizations and venues to make those numbers publicly available so that artists have access to them.

All that said, I do suspect that in some small way, these questions might actually do their own work via the power of suggestion. They serve as a reminder to any artist filling out the application that they should be more inclusive, that it’s not enough to just accept a status quo audience, that they need to think about why it is that their work reaches or doesn’t reach a certain segment of the population and think critically about that.

And I want to be clear that these numbers are not the only criteria on the application — I’m picking out only two questions on a longer, comprehensive application.

But the reason that I’m bringing this up is because I remember how weird and confusing it was to be asked, as an artist who was just settling into Brooklyn and grant-writing, to provide numbers and percentage estimates for a hoped-for future audience. And my sense is that there are a lot of artists, independent/unaffiliated artists, who have never considered applying for grants, who would be intimidated or put off by the grant application process, and who are often from the very groups that are the target of community arts funding programs. It seems like continuing to look for ways of to fund arts within communities that aren’t project and application-based might also be a good strategy for ensuring that programming and money reaches across the borough.