Text Size:   A+ A- A   •   Text Only
Find     
Site Image

Step 2: Selecting Evaluation Criteria
/ODOT/TD/OSTI/PublishingImages/SPG%20bubble%20steps.png

Summary

/ODOT/TD/OSTI/PublishingImages/rsz_step_2_circle.png This step involves translating the goals identified in Step 1 into a set of guiding principles that will serve as the objectives to guide the process. Metropolitan areas, or local jurisdictions within the area, may have recently developed guiding principles from comprehensive plans, TSPs or community visions that already support scenario planning goals. If so, it may not be necessary to develop new guiding principles, but rather assemble existing guiding principles. This step also discusses the process for establishing the evaluation criteria that will be used to analyze the alternative scenarios and communicate their results in Step 5. The guiding principles established in this step will be used to help develop and prioritize the evaluation criteria.

Preview

At the crux of scenario planning is the ability to objectively compare a range of possible futures. In Step 1, the organizational structures of the effort were agreed upon. This step discusses how to measure success by creating guiding principles and translating them into a set of evaluation criteria to measure scenario outputs. While the reduction of GHG emissions will be a central evaluation criterion, measuring and communicating other important local priorities and criteria will be essential to garnering widespread community interest. A common set of evaluation criteria are recommended in this step, but should be adjusted or augmented to fit the community. An assessment of the evaluation criteria will be calculated later using Metropolitan GreenSTEP and a sketch planning tool.

Develop Guiding Principles

One of the keys to scenario planning and implementation is clearly defining the outcomes that will signify the success of the project. In order to do this, development of a series of guiding principles for the scenario planning project is recommended. Guiding principles are broadly stated objectives or precepts to guide the scenario planning process. Developing a set of guiding principles will help to identify and prioritize evaluation criteria that will be used later to assess and compare scenarios. The guiding principles will serve as a touchstone throughout the project in all circumstances, irrespective of changes in its goals, strategies, type of work, or management. These can be new guiding principles developed specifically for the scenario planning process or assembled from objectives identified in existing plans.

First, comb through existing plans for guiding principles shared by the participating jurisdiction(s). Work with the advisory committee developed in Step 1 to expand that list by brainstorming other potential guiding principles. Then work with the leadership team, also developed in Step 1, to narrow the list. It is best to select and organize the guiding principles into categories, such as: economy, transportation, environment, equity and opportunity, and community and housing. Below are four examples of guiding principles, a full sample list is located in the Technical Appendix.
  • The transportation network should ensure efficient and cost effective movement of goods and people.
  • The transportation network will ensure safe and timely access to housing, jobs, shopping, services, and recreation.
  • The metropolitan region promotes a full-range of housing types to fit every income, household and preference.
  • Ensure there are parks, trails and open spaces that are easily accessible to residents throughout the community.
On page 42, the guidelines will use these examples to demonstrate how guiding principles and evaluation criteria can be linked.

Establish Evaluation Criteria

A set of evaluation criteria is a benchmark or yardstick against which the performance of the scenarios can be measured. Establishing evaluation criteria that resonate with members of the community is essential to a successful process. While GHG reduction is an important evaluation criteria in a scenario planning project, it is just one of several that will make this effort meaningful. Develop criteria tailored to the needs and values of the participating local jurisdiction(s) using the key issues identified in Step 1 and the guiding principles as a guide.

The most effective evaluation criteria can be measured objectively and are meaningful to people. Objective measures allow for clear assessment of scenario performance and highlight tradeoffs between policy choices. Evaluation criteria should address a range of community objectives such as public health, a stronger local economy, average household fuel expenditures, access to transit, access to bicycle and pedestrian paths and trails, and preservation of natural areas, as well as GHG reduction. Many of the strategies that help meet the evaluation criteria will have more than one benefit. For example, an evaluation criterion calling for calculating the reduced amount of driving one might experience in a scenario can resonate with many members of the community primarily in terms of money saved on fuel and secondarily on benefits to the environment through GHG reduction.

The evaluation criteria in the scenario planning process are not designed to assess specific impacts and precise outcomes such as congestion at a specific intersection or roadway level of service. Specific impacts and outcomes typically are assessed when developing an implementation plan or updating existing plans as part of an implementation phase. The evaluation criteria will be used to gauge big-picture items such as the amount of daily driving or the potential consumption of land. Objective evaluation of scenarios will provide a general sense of the potential impacts of actions and programs, such as those described in the Oregon GHG Reduction Toolkit, as they pertain to the goals set forth in the guiding principles. Through comparison of several evaluation criteria, tradeoffs between actions and programs can also be examined and better understood.
Recommended Evaluation Criteria, Corresponding Unit of Measure and Source
/ODOT/TD/OSTI/PublishingImages/Evaluation%20criteria%20table%20from%20Step%202-sm.jpg
Potential Evaluation Criteria
Scenario planning tools create the potential to develop and apply dozens of different evaluation criteria. The ability to construct computer simulations of metropolitan areas has advanced tremendously over the past two decades. However, even prior to recent technological advances in modeling, the Oregon planning system required planners to develop an estimate of future conditions (e.g., land consumed by urbanization). The federal transportation planning system also requires that future modeling demonstrate compliance with the Clean Air Act and other legislation. As a result RTPs have trackable performance measures. Existing plans in Oregon jurisdictions include many evaluation criteria, so the notion of using predictive evaluation to analyze plans is not new. It will be helpful to carry forward some of the evaluation criteria the metropolitan area is currently using into the scenario planning process. Tracking some consistent measures will aid in communicating the scenario analysis with stakeholders already familiar with some of these well-established criteria.

This step also recommends new evaluation criteria that can be employed in the scenario planning process. The evaluation criteria will be used to track a range of potential outcomes expressed by the scenarios and evaluated through the sketch planning tool and Metropolitan GreenSTEP. The scenarios will express the future in terms of land uses, including the types, locations and intensities of uses, and the transportation network with its accompanying infrastructure from roads and trails to trains and buses. Using Metropolitan GreenSTEP and a sketch planning tool, the scenario comparison can include measures such as: residential and transportation-related GHG emissions, land consumption, land use patterns (e.g., housing and job distribution, density), mobility (e.g., delay and travel times), travel behavior (e.g., vehicle miles traveled, walking, bicycling, transit ridership, carpooling, drive alone trips), mobility corridor performance and regional travel patterns.
Recommended Evaluation Criteria
The process of comparing multiple future scenarios should employ a set of criteria that addresses the full range of needs and desires expressed through the guiding principles. This set of criteria should be used consistently for apples-to-apples comparisons of scenarios. The best practice is to select the number of evaluation criteria based on the guiding principles and what is important to the community. The same evaluation criteria may apply to multiple guiding principles. Use of too many indicators can overload the team as they may start to focus too much on mathematical evaluations, rather than the bigger picture planning concepts being discussed. When sharing results with the community, a smaller subset of the selected evaluation criteria may be more appropriate. Step 6 discusses in greater detail guidelines for effectively communicating the evaluation criteria to the public.

The final set of evaluation criteria should be drawn from measures established in existing plans, and calculated by Metropolitan GreenSTEP and the chosen sketch planning tool. The accompanying table presents a suggested set of criteria likely to support the selected guiding principles. However, as always, this list will need to be adjusted to address the community’s specific guiding principles. For example, the Envision Central Texas project added measures related to the health of the region’s aquifer because water quality was a top priority for the public and stakeholders.

Link Guiding Principles and Evaluation Criteria

Evaluation criteria can help community members and stakeholders understand the benefits or trade-offs of a scenario as they relate to guiding principles. In some instances, evaluation criteria can be used to assess the performance of a scenario in relation to more than one guiding principle. In this section, the four examples of guiding principles from page 40 are linked to related evaluation criteria examples to highlight their relationship.
Guiding Principle Example #1
The transportation network should ensure efficient and cost effective movement of goods and people.

Associated Evaluation Criteria:
  • Vehicle miles traveled per capita
  • Average trip time
  • Vehicle hours under congestion or vehicle hours of delay region wide
  • Amount of new development located within proximity of transit
  • Average speed by travel mode
  • Cost of transportation improvements and maintenance
Guiding Principle Example #2
The transportation network will ensure safe and timely access to housing, jobs, shopping, services, and recreation.

Associated Evaluation Criteria:
  • Number of people with 20-minute access to employment centers
  • Average trip time
  • Vehicle hours under congestion or vehicle hours of delay
  • Amount of new development located within proximity of transit
  • Total time traveled per capita, by travel mode
Guiding Principle Example #3
The metropolitan region promotes a full-range of housing types to fit every income, household and preference.

Associated Evaluation Criteria:
  • Housing mix by type
  • Housing tenure
  • Housing affordability by income level
  • Housing costs
Guiding Principle Example #4
Ensure that there are parks, trails and open spaces that are easily accessible to residents throughout the community.

Associated Evaluation Criteria:
  • Households within a ¼ and ½ mile walk distance of parks, trails and open space

Emerging Indicators

/ODOT/TD/OSTI/PublishingImages/7%20cell%20graphic-Step%202.pngThe practice of linking community goals to measurable evaluation criteria is expanding as modern technology introduces new data and tools for analysis. The following case studies describe examples of indicators other regional efforts selected to measure.
Salt Lake City Region Scenario Planning Process
One of the reasons that it is now easier to link broad community goals with land use and transportation planning is that there has been a great deal of research over the last decade correlating urban form to a number of measures. These measures range from travel behavior, to health and obesity, to employment resilience. The Salt Lake City region recently received a $5 million dollar HUD Sustainable Communities grant, of which a substantial portion is being used by the University of Utah to build 18 modules to calculate a host of new evaluation criteria. These modules are open source and will be usable with Envision Tomorrow and other sketch planning tools (described further in Step 3):
  • Growth location prediction
  • Transportation effects using seven factors (more accurate than current five factors)
  • Return on investment
  • Housing transportation costs
  • Air quality and CO2 impacts
  • Fiscal impact
  • Public health
  • Employment growth
  • Employment resilience
  • Development capital attraction
  • Redevelopment timing
  • Water consumption
  • Transportation safety
  • Workforce housing production
  • LEED-ND
  • Arts and public amenities impacts
  • Local jobs-housing balance
Metro’s Scenario Planning Process
One example can be seen in Metro’s Climate Smart Communities scenario planning process. Metro’s honeycomb graphic illustrates the six desired regional outcomes which serve as guiding principles for its Climate Smart Communities project and its ongoing making a great place work. Metro’s scenario planning process identified evaluation criteria that could assess scenario performance on all of the region’s six desired outcomes.
Cumberland Region Tomorrow Scenario Planning Process
Another example of planning for multiple objectives can be seen in the Middle Tennessee region. A multi-faceted coalition brought together by the non-profit Cumberland Region Tomorrow identified six issues of regional importance:
  • Transportation/transit
  • Land use/quality growth/sustainable development
  • Infrastructure
  • Open space conservation
  • Air and water quantity and quality
  • Economic competiveness
The region has worked together, planning and implementing through workshops, plans, handbooks and lobbying. They consider these six key issues and how the region handles them vital to Middle Tennessee’s continued economic success, community health and resource sustainability.
 

Refine Evaluation Criteria

It is likely that a large number of evaluation criteria will be employed for the technical scenario analysis. However, when communicating the results of the scenario comparisons with residents of the community, the list should be pared down. Key criteria, most closely linked to the highest priority guiding principles, should be identified. At the same time, it is important that the full range of issues within the guiding principles be addressed. Measures communicated to the community should be those that affect their lives, and be communicated in terms to which they can relate. For example, in estimating transportation outcomes, it would be useful to report on annual fuel cost per household, or time spent behind the wheel in addition to VMT per capita. A short list of well-chosen evaluation measures will help the public relate to the scenarios.

In addition, since many evaluation measures are calculated from a small number of model outputs, these measures are likely to be closely related and increase or decrease together. For example, land consumption and density are closely related and likely to communicate similar messages when charted or graphed. While both may be useful for a few stakeholders, it is better to pick a single measure to represent the related ones, and use the measure the public can relate to the most.

Conclusion

Completion of this step results in the selection of a series of guiding principles that provide the focus for the scenario planning effort. Evaluation criteria, linked to these principles, will be used to measure the relative performance of the alternative scenarios in Steps 4 and 5. A subset of the evaluation criteria, which best resonate with the community, will be selected to be used in communications describing scenario performance. These criteria will communicate to the community members, stakeholders and decision makers the benefits, impacts and tradeoffs of different policy choices and investments within each alternative scenario, based on the metropolitan area’s goals and values.

With guiding principles identified and evaluation criteria established, the next step is to prepare for actual scenario planning by choosing and calibrating planning tools, then collecting and preparing the needed data.