Streamlining SEM Program Evaluation with Energy Sensei

SEM impact evaluations are a crucial part of ensuring programs operate with the highest level of excellence. However, an examination of recently published SEM impact evaluations revealed consistent recommendations for better data management, measurement, verification, and project documentation. Providing these improvements means streamlining SEM program evaluation with less time and resources needed to establish important program metrics like gross savings and effective useful life (EUL).  

Cascade’s energy management software, Energy Sensei, helps program administrators, evaluators, and implementers collaborate effectively through a central, online platform. Consequently, the outcome is lower evaluation costs for any size SEM program. Additionally, eradicating missing information lowers uncertainty, greatly benefiting all parties. 

Centralized, Secure, Accessible 

For program managers, using Energy Sensei also helps reduces costs and time. Moreover, evaluators need a lot of information to complete their job. Energy Sensei centralizes the needed information, enabling evaluators to get that data with less lift. Additionally, as a SOC II Type 2 certified software, all program data is secure. Say goodbye to versioning issues, aggregation errors, and misfiled reports. Instead, Energy Sensei provides evaluators access to only specified data, meaning pre-loaded analysis views are easy to work with and require less effort to rebuild themselves. 

 Recommendations from recent evaluations for ComEd (2022), Energy Trust of Oregon (2019), California Public Utility Council (2021), and Bonneville Power Administration (2020) point to several critical needs for effective evaluation: 

• Raw Data Management: Incoming raw data needs to be received at the finest interval available and properly aggregated to the modeling interval. For EUL studies, availability of data post-reporting period is desirable. 
• Opportunity Registers: To confirm savings and establish EUL, evaluators need access to clear measure completion documentation. For savings confirmation, Sensei makes cross-referencing completed measures with measure savings easy and provides an implementer-focused template to record key measure details. Persistence strategies are documented here, to help ensure savings persist and providing evaluators critical interview information for EUL studies. 
• Non-routine events (NREs): recent evaluation recommendations come right from the International Measurement and Verification Protocol’s (IPMVP’s) NRE Guide. Sensei aids implementers and participants in early NRE identification, meaning there’s less potential for post-reporting adjustments and discrepancies between reported and evaluated savings. 
• Measurement and Verifications: As the primary method for most M&V, top-down models must be readily re–creatable. This means all assumptions and data streams should be transparently explained and documented. Outliers, exclusions, and other adjustments must be clearly documented and the math needs to be properly executed. 
• Reporting: Pulling together a full year, or multiple years of information for reporting is hard. Sensei streamlines the process by centralizing everything in a secure environment. From data streams to persistence strategies, we have you covered. 

Working Together 

Let’s talk through some examples of how Energy Sensei helps evaluators and program managers work together in streamlining SEM program evaluation processes.  

Raw Data Management 

Accepted best practices recommend providing the finest interval data available. Yet, aggregation errors are a common source of discrepancy in evaluation. As programs scale up, manual management of data streams becomes more challenging to manage, with more potential for aggregation or meter to meter error. 

 Energy Sensei accepts fine interval data and automatically aggregates it to the level needed for measurement and verification. Automated data feeds through Green Button and secure file transfer protocols (sFTP) ensure data is attributed to the right meter for each participant, yielding clean datasets and less manual data handling. Additionally, automating data also facilitates extending data collection in preparation for persistence studies. 

 Whether the data is energy, occupancy, or production, it can be visualized, tracked, and reviewed in the Sensei platform. The platform also provides temperature data, taking that data stream off everyone’s plate. 

Reliable Updating 

Opportunity Registers and Measure Tracking 

If you’ve ever wondered, “Is this the right version?” when looking at an opportunity register, Sensei is for you. A program, cohort, or site’s opportunity register can be viewed with the latest updates. Energy teams and coaches update the register during regular check-ins, leading to more complete information.  

 Here are a few key evaluation-specific benefits of Sensei’s register: 

1. Standard template to track critical information like implementation date, progress notes, description, estimated savings, measure type, and persistence strategies.  
2. Bottom-up savings information for measurement and verification can be housed here (images, calcs, supporting documents) and bottom-up project savings are automatically aggregated for claims. 
3. Cross-validate completed measures with the energy model’s cumulative sum of savings (CUSUM) chart. Completed measures are automatically flagged on the CUSUM to help cross-reference measure implementation dates with CUSUM inflections, making verification of measure impact a simple task. 

Opportunity register information is easily filterable and can be reviewed within the platform or downloaded for more detailed analysis. 

 Routine and Non-Routine Events 

Speaking of inflections, Sensei empowers users to flag events for further investigation. Program implementers and SEM participants can create ‘Events’. Each Event includes a date range and creates a flag on timeseries charts. Consequently, this promotes early discussion between SEM participants and coaching teams on changes in energy intensity. 

This capability is key in streamlining SEM program evaluation. Several recent evaluations (SCE, ComEd) encountered non-routine events that could have been flagged and discussed during the program. Sensei helps find these with ‘Data Inspector’ which identifies outlier residuals and data points. As a result, post-project non-routine event adjustments were among the most impactful differences between evaluation and program reported savings results. Sensei provides tools to better identify NREs and to document the whole process. 

Elevating Models and Reports  

 Modeling and Adjustments 

Energy Sensei also generates energy models using temperature and other energy driver data streams. Hypothesis models are created, tested, and honed within the software. Standard statistics for model fitness are automatically generated and provided along with critical information like date ranges, historical data, and acceptable variable ranges. Energy, temperature, and dependent variables are visualized in time series and x-y plots. Additionally, the viewer can customize the interval viewed to help better identify atypical data. 

Sensei tools include outlier inspection and consistent documentation and handling of missing, excluded, adjusted, or non-programmatic savings values. Evaluators can view all of this information within Sensei. If additional analysis is needed information can be provided in a consistent, transparent manner via a bulk database export (program-wide analysis) or project-specific workbooks (deep dive on individual projects). 

Reporting 

Savings claims are tracked within Energy Sensei. Using Sensei as the claim record reduces errors and provides a traceable pathway to verify both top-down and bottom-up savings. Furthermore, each claim includes additional detail of dates and the method used to arrive at the savings value.  

Energy Sensei is a robust platform any program can use to standardize their reporting and delivery. Program implementers all use the same platform and tools to deliver and report results. Savings, measures, and other key metrics can be viewed at a participant, cohort, or program-level, giving you better access to your program potential and progress. As a result, updates made by coaches and participants take effect immediately, so you’re always seeing the most up-to-date information.  

Collective Wins 

SEM impact evaluators serve an important role as we all seek to bring excellence to clients and programs. Reliable updating, raw data management and elevated modeling are all recommended elements of improvement. Striving for these together is how we achieve collective wins. Let Energy Sensei do the heavy lifting in elevating both SEM implementation and streamlining SEM program evaluation. Freeing your team from logistics allows you to fully focus on your goal: saving energy.   

Reach out here to learn more about Energy Sensei!