Jeopardy answer: Annual program appraisals
Contestant response: What did fire departments worldwide begin pencil whipping in 2016?

I am sure several progressive departments were already diligently performing formal and documented program appraisals before the 9th edition of the Commission on Fire Accreditation International’s Fire and Emergency Services Self-Assessment Manual was released. Still, I am equally confident that the freshly added core competencies mandating the process catalyzed program reviews at a far greater number of headquarters. In the latter group’s defense, most accreditation-minded departments were already tracking short-term and strategic goal progress through monthly or quarterly meetings; creating a formal assessment document was the only uncharted water. Many accreditation managers were left to ponder what an appraisal should contain and how it should be used.

Now that we are midway through the mandate’s second cycle, I am confident that there are currently three camps regarding annual program assessments. The first camp is only interested in checking the CFAI box. The second camp (which I believe is the majority) takes the document seriously but needs help moving past simply reporting on fundamental measures such as short-term performance, long-term goal progress, and significant events. Their documents are mainly historical records to be used in annual reviews or for presentation bullet points. The last camp has embraced the grind and found that a well-designed program assessment can dramatically improve communication, define defensible budget priorities, and ensure the department’s strategic plans remain aligned with the community’s greatest needs.

A thoroughly performed APA seeks feedback from a wide array of stakeholders. Department members from every level are invited to participate by providing insights and shaping future initiatives. Partner agencies and community allies are also asked to be at the table. This inclusive approach improves idea dissemination and provides a sense of ownership of the program and department. An analysis of current performance levels is combined with service needs projections and future challenges to determine potential gaps. Gaps can be addressed through plans and action items, and budgets are then built off of these plans and action items.

A great example of this approach can be found at Gwinnett County Fire and Emergency Services. Chiefs Russell Knick, Fred Cephas, and Chad Sheppard have built and currently oversee an annual program appraisal process that covers almost every aspect of their department, breaks down information silos, and generates buy-in from both internal and external stakeholders. Committees prepare APA rough drafts that are reviewed and finalized through open, web-cast meetings involving all interested department personnel and key external partners.

“If you pull more hands onto the team, that’s how things get done,” GCFES Chief of Staff Chad Sheppard said. When asked about their process, he stated, “We get lots of feedback and then are able to put it into action. Actions speak louder than words.”

I will cover the specifics of their approach in greater depth later this summer, but for a department looking to ‘up their APA game,” the GCFES’s APA documents are a great launch point. They range in length from a few pages to over 20. The common elements one would expect to find in any APA are all still there: significant events, short-term performance data, strategic goal progress, and future plans. In this way, they still capture all of the essential historical information they may need to reference later. The added value in their approach comes from the additional categories they consider.

Committees begin their work well before the day of the formal APA meeting. They typically include a broad cross-section of department members and contemplate their section’s training, equipment, facilities, and staffing needs. They give consideration to each sub-area’s budget, purchases, and projects. Performance measures are considered for training, and recommendations are updated for equipment and facilities. Staffing changes are acknowledged, and future staffing needs are discussed.

Guidelines, standards, and regulations (internal and external) are covered. Were changes made? Are regulatory or other legal changes coming? Do we need to make any changes? Do we have any external agreements, and if so, how close are they to expiring?

The insights gained from these discussions are then added to the gaps found in the short-term performance and strategic plan assessments. This comprehensive list drives the debate regarding the section’s plans moving forward and, therefore, its budgetary needs. The committee weighs its options regarding their impact on the community, department, section, and external partners before settling on new action items. Each action item should be directly tied to an expected outcome and cost.

“This process helps people understand what it takes to get things done. They also better understand that your plans aren’t just affecting your section, and there are impacts to all other divisions,” Sheppard explained.

At the Indianapolis Fire Department, we have made significant changes to our APA approach in 2023 based on GCFES’s documents. Our document has been reworked to include many of the same focus areas. In each one, we look at what happened, what is about to happen (short-term), and what will happen down the road (long-term). We note changes and new developments and discuss budget-related needs for the forthcoming year. The most significant change to our process is that we review this document with the program leads every month. The monthly meetings (limited to an hour per program) are a significant draw on the planning division’s time. Still, the impact of regularly having these forward-thinking conversations is already tracking to substantially impact the forthcoming 2024 budget preparation.

While we are a ways away from mimicking GCFES’s success, I can attest that we see a significant return on our time investment. Communication has improved up and down the chain of command, and problems are solved before frustration reaches a boiling point. As an organization, we are learning to think with an eye toward the future and better articulate our needs. These open conversations have also provided perspective on how intertwined our efforts are. I was anxious to take such a deep dive into the APAs, but the payoffs are already proving substantial.

GCFES APA Doc Outline
There are several slightly different variations of the GCFES APA document, but the overall content is consistent. The top of the page contains the program name, date, and attendees. From there, the categories breakdown is similar to the following:

• Program Summary
  • The core competency (if applicable) with its description
  • Significant events
  • Significant changes
• Strategic goals
  • Goal description and aligning county priority
  • Performance measure (type and benchmark) and assessment
  • Plan for improvement
• Other goals and objectives
  • Strategic recommendations and updates
  • Program goals
    • Goal description
    • Performance measures and assessments
• Equipment
  • Budget
  • Purchases
  • Issues and recommendations
• Facilities
  • Budget
  • Projects
  • Issues and recommendations
• Staffing
  • Changes
    • Additions
    • Subtractions
    • Movement
• Performance improvement measures
  • Recommended goals
    • Title
    • Description
    • Recommendations
  • CPSE charts (if applicable)
• Needs and performance gaps
• Guidelines
  • General orders
  • Public
• External agreements
  • Partner organizations
  • Effective date
  • Expiration date
  • Purpose served
• Outcome measures and recommendations
• Follow-up and action items
  • Description
  • Action item