Modeling (Sustainability)
April 12, 2011, 9:34 pm
Filed under: Brain dump | Tags: , , ,

So you want to build a model that rates companies?! (Please nod your affirmation, or this whole post will be pointless.) Well, there are three steps you’ll need to follow. We’ll go through them first in principle, and then I’ll hopefully follow-up to add in some examples.

Step 1. Identify your issues.

Step 2. Choose your indicators.

Step 3. Define (mathematical) relationships between them to generate a score.

Step 1: Identify your issues

The starting point for the model is to lay out what the model will evaluate. Let’s say: sustainability (as opposed to ethics or social justice or financial solvency), and further limit it to the sustainability of companies in mineral extraction industries (e.g. fossil fuels, precious and base metals). There are two directions from which to approach the identification of issues: (1) top-down, looking for indications of problems and then see which ones link to extractive companies; and (2) bottom-up, analyzing the operations of extractive companies and trace all the externalities. The optimal approach is likely a rationalized combination of the two.

The Top-Down Approach: Looking for Signs of Concern

Sustainability can be a difficult concept to grasp, so I prefer to use the metaphor of health. There is more to health than physical health of individuals – there is mental health, financial health, public health, and the health of ecosystems and economies. Now, we can use this health metaphor to help us evaluate sustainability, by asking ourselves the question, if we have a problem with our health, how would we know?

1. Diagnosis on the basis of direct measurement. In the case of medicine, this would include visible signs of disease on the skin or lab results showing harmful bacteria in the blood. In the case of sustainability, we can look to figures on mortality and morbidity, inequality (e.g. the GINI coefficient), the stocks of environmental resources such as clean water or forests, average regional temperatures, and other direct indicators that something is fundamentally wrong.

2. Diagnosis on the basis of mitigating efforts. The presence of antibodies is often a clue that an infection is present. In the case of sustainability, we can look to the activity of our existing systems to mitigate problems as signs that all is not well. These include fines and regulatory actions, NGO campaigns, litigation, legislation and regulation proposed and/or passed, and debt.

3. Diagnosis on the basis of adaptation. If a patient has tooth pain and disproportionately uses the other side of her mouth to chew, that adaptation can cause problems itself, but the root cause is the pain that caused the initial change in behavior. Similarly with society, if we experience problems, we adapt by turning to religion, therapy, donations to charities, sacrifices in spending habits (including on essentials like health and education), alcohol, tobacco, gambling, television and other escapist entertainment, discretionary consumption, research and development (if we’re a company that sees an opportunity in solving a problem), employee and customer turnover, migration, medical or alternative treatments, etc.

So if we see these types of behaviors, correlated with a company’s operations [understanding that at least some of the figures mentioned above are more systemic / hard to trace to a particular industry or company], then we should evaluate further to see if there’s a problem. If a company is experiencing high turnover in employees, maybe there’s a problem. If there is relatively high mortality in proximity to a company’s mines or refineries, maybe there’s a problem. If there is a large number of regulatory actions or shareholder resolutions, maybe there’s a problem. Once you have identified a number of these potential problems, you can classify them and identify them as issues on which the model will evaluate the companies in question.

It can be difficult to trace from the figures and behaviors above to a particular cause, so there’s definitely an art to this more than a science. And when defining problems, here are two common mistakes to be careful to avoid: First, don’t define problems as a lack of the solution du jour (e.g. “the problem is a lack of renewable energy capacity” or “the problem is a lack of organic agriculture”).  Secondly, try to get to the root issue in drawing the boundaries of your problem definition (e.g. “too large a population” is not a problem so much as a contributing factor to problems, because (a) the real problems are the resource shortages and waste generated — and experienced — by the population, which (b) share other contributing factors such as the intensity of resource consumption by the population — i.e. if we consumed less, there could be more of us).

[We should keep in mind that “problems” do not necessarily imply blame. Water shortages can be a problem for companies, even if they have equitably and responsibly shared the resource with other members of a community. Climate change can be a problem for insurance companies, even if they did not have a large direct contribution to the stock of greenhouse gases in the atmosphere. I should also add, and it might as well be here, that these issues could also be opportunities, whether an opportunity for a company to differentiate itself by reducing the effect of a problem on its own operations (e.g. by pioneering a manufacturing process that uses far less water) or by helping others to mitigate or adapt to problems that they face (e.g. insurance companies providing drought insurance, even if they didn’t themselves cause the drought, or solar cell companies providing cleaner energy).]

[I’ll be honest: I’ve been meaning for some time to go through this exercise, but it is time consuming. So please feel free to visit GapMinder or NationMaster and get started yourself.]

The Bottom-Up Approach: Breaking Down the Value Chain

Life-cycle assessment (LCA) is an established practice (with various options on methodology) that breaks down the steps of the value chain from producers to consumers to product end-of-life and, in so doing, identifies the positive and negative effects on the surrounding system, e.g. providing employment and tax dollars, polluting rivers and air, and requiring treatment of hazard waste. LCA and similar analyses often extend beyond the legal borders of a corporate entity to look at issues facing its suppliers, distributors, and customers.

Where you define your borders in an LCA analysis determines the cost, time, and accuracy of the results. Tracing a Patagonia’s operations from retailers to distributors to design offices to specific factories to specific cotton farms and rubber producers provides a much clearer picture of the company’s operations, but at the expense of cost and time. As a result, there are a number of models which make simplifying assumptions, e.g. the environmental impacts of cotton generally, rather than cotton from Nike’s factories’ suppliers, to provide a more cost-effective and flexible analysis, with specificity reserved for operational areas with known high impacts or risks. Carnegie Mellon’s Economic Input Output LCA model is an example of such an approach.

For our purposes, we can think at a higher level about what inputs (and anything else that’s consumed, if unintentionally and unnecessarily) and outputs (desired and “side effects”) mineral extraction involves. Inputs include labor, mineral reserves, energy, water, land and biodiversity, various chemicals (e.g. solvents), financial capital and insurance, security and other services, etc. Outputs include end products (e.g. coal, petroleum products, diamonds), wages and taxes, political contributions / bribes, effluents and waste (direct and through product use), risks to health and safety of employees and communities (e.g. through mine collapses), profits to shareholders, etc. It’s also relevant to note differences in the types of operations / products (e.g. offshore oil vs mountain-top removal coal) and the locations of operations and customers (e.g. Nigeria vs West Virginia).

Clearly, there are a lot of potential issues here. It is usually wise to limit focus on the basis of the scale of impact, the probability / level of certainty of impact, the time horizon, the ease and cost of reversing any damage, and the capacity of your model to properly assess the company’s performance. [Read this post at OpenIDEO to see what other people think are the most pressing issues.]

[There are, of course, other ways to identify issues at a more macro level. For instance, some investors exclude all companies operating in Burma or the Sudan, to protest government abuses in those countries. Investors may also choose to exclude entire industries, such as tobacco and weapons, but also perhaps oil and other fossil fuels. These exclusions are generally more relevant to values-focused as opposed to sustainability-focused investment approaches, so they’re less likely to be found in a “best in class” model such as the one we’re in the process of describing.]

Step 2. Choose your indicators.

In an ideal world, at least as regards depletion of renewable resources, you would evaluate sustainability by comparing rate of withdrawal to rate of renewal. So, for water use at a factory, you’d want to know whether the factory and its neighbors is withdrawing water faster than the aquifer or lake or reservoir is being recharged (or will be recharged in the future, if climate change is factored in). And use of non-renewable resources is, by definition, unsustainable. This is one reason why we use a “best in class” approach: we’re not saying that the best oil company is magically “sustainable,” just that it is better positioned than its sector peers.

Of course, our real-world situation is non-ideal in a variety of senses: we also don’t have full access to information. We probably don’t know the rate of renewal of a company’s water sources; we probably don’t even know the exact location of the company’s water sources, or the amount it and its neighbors withdraw, though hopefully the company itself is at least tracking these figures, if water is indeed an issue in its operations.

So, what do we do to evaluate environmental and social issues in this non-ideal world?

1. Where we can, we track actual performance / impacts with precise numbers. Greenhouse gas emissions, water use, energy use, hazardous waste generated, fatalities. [Often, we’d want to normalize these, say, against revenues or employee counts, to more directly evaluate efficiency of operations rather than size, though absolute size of course can be relevant as well. The rate of change of these figures is also relevant.]

2. Where no precise figures are available or where we hope to get a sense of future impacts, we look to efforts the company has made. These include some policies, structures, commitments, processes. Here, we consider certification schemes, commitments to standards set by respected groups (perhaps enforced by audits), etc. [It is possible that a particular effort will fail to have the desired or expected result; it is also possible that the desired result could be produced by other efforts. The model will not be perfect but can be improved over time.]

3. We also look to anecdotal evidence as a gauge of the effectiveness of corporate efforts at managing the issues. These may include newspaper articles or NGO campaigns, product recalls, or litigation. These indicators are incomplete, so cannot give us a sense of, say, what percentage of employees are happy, but it can give us a glimpse as to where the company’s efforts may be falling short.

Ideally, the standards and processes in point 2 would be validated over time against actual results. For instance, do real estate portfolios with a higher percentage of LEED-certified buildings use less energy and water than other portfolios with fewer LEED-certified buildings? Do fuel-efficient cars result in fewer greenhouse gas emissions or simply more miles driven? Do ISO or SAI certifications lead to fewer spills or safety incidents? Unfortunately, many of the indicators are not so validated, due to the complexity of the systems involved and the incompleteness of the data [maybe also limited by availability of resources and interest to pursue the question].

Step 3. Define (mathematical) relationships to generate a score.

In general, with modeling, there is usually no single “right” answer. You could argue multiple classifications of issues and indicators – and, believe me, people do. But nowhere is there less of a “right answer” than in defining the mathematical relationships of the model. What scale is used? What weights among the issues? Are positive impacts weighed equally to negative impacts? Are scores added or averaged or multiplied … or shall we get into logarithms and vectors? Are ratings relative to the set of companies being analyzed, or should the scale be fixed somehow? What do you do when data are unavailable? How do you evaluate conglomerates, with very different lines of business? What do you do when a highly rated company suffers a large incident, such as BP’s Macondo well blow out? If it becomes apparent that a Nike supplier has a problem with human trafficking, is it more responsible of Nike to stay with the supplier and help them get better, or to abandon them for another supplier?

There is a reason why Moody’s and S&P and FICO defend their models as opinions (less subject to liability if their results don’t predict future behavior).

More important than the individual formulas is to stay consistent and be transparent. Wherever possible, you should try to validate your model, both comparing the results to what you would have expected and, later, to actual company performance. It is worth testing, as well, to see how sensitive the model is to changes in the various indicators, or to changes in who is entering (i.e. interpreting) the data, as well as to the presence or absence of any estimates you use as indicators where reported data are not available.

How you use the output of such best-in-class models, we’ll get to in a subsequent post. In the meantime, it’s worth reading Malcolm Gladwell’s warnings on what happens when a model combines heterogeneity and comprehensiveness (“The Order of Things” in The New Yorker, Feb. 14, 2011). And, to the extent that companies work to improve their scores in these models or gain inclusion in ESG indexes, we must worry about the danger of “seeking the wrong goal” as Donella Meadows puts it (Thinking in Systems: A Primer, p.138).


1 Comment so far
Leave a comment

It’s worth mentioning that I’m by no means the first or only person using a health metaphor for sustainability. One example from a textbook on conflict resolution: “[M]edical professionals who had an understanding of the physical and mental costs of war … were struck by the medical analogy: if war was like disease, then knowledge of symptoms and aetiology should precede diagnosis and therapy or cure.” (p. 35 of Contemporary Conflict Resolution by Ramsbotham, Woodhouse, & Miall)

Comment by essareye

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: