PAAB : Resources : PAAB Quarterly CEI Report

PAAB Quarterly CEI Report - 2024

A Quarterly Review of the eFiles Customer Experience Index (CEI) - Q1

This review of the CEI data accounts for the data captured between January 1 – March 31, 2024.

231

Completed Surveys

January 1 to March 31, 2024

Averages of the CEI question survey results by question:

1. Staff connected with this eFile (e.g., file coordinators, reviewers, senior reviewers, etc.) were helpful and responsive

4.6/5

Indicating an average response between “Agree” and “Strongly Agree”

2. Comments and suggestions in response letters, calls and messenger were clear and actionable.

4.4/5

Indicating an average response between “Agree” and “Strongly Agree”

3. I felt the review was:

1. Highly inconsistent
2. Somewhat inconsistent
3. Somewhat consistent
4. Highly Consistent
5. I don't know

3.7/4*

Indicating an average response between “Somewhat consistent” and “Highly Consistent”

*This average rating is calculated from Responses 1-4 as including ratings of 5 (or “I don’t know”) would have skewed the average upward. There was 4 ratings of “I don’t know” in this data set.

4. Please provide any other feedback specific to this file: [optional open text field]

See feedback themes below.

5. Please rate your overall experience with this particular review

1 – highly negative experience

10 – highly positive experience

8.8/10

Indicating a positive average overall experience.

Key Themes and Actions Taken from Open-text Feedback:

Perceived inconsistencies with past rulings (n= 6) There have been six comments pertaining to inconsistencies or perceived inconsistencies with previous rulings.
Action Taken: Reviewers have been reminded to review backfiles when appropriate to ensure consistency where appropriate. In most instances, when the reviewer was made aware of the inconsistency, the copy was ultimately accepted.
Reviews could have been completed in less rounds (n=4) In four instances, the submitters felt the review could have been completed in fewer rounds. Reasons for the perceived delays included images being FPO in the layout (n=1), the submitter not having access to the appropriate eFiles associated with linked APS (n=1), submission-related issues (n=1) and a late comment from the reviewer (n=1).

Action Taken: Reviewers have been reminded to ensure review letters are complete upon sending to avoid late comments. We encourage submitters to ensure to review the Guidance on Submission Process and Format Requirements and ensure that the submission is complete to avoid delays. If the APS includes links to other PAAB-accepted APS, it is important to ensure those eFile numbers are included within the copy to allow for assessment of the linkages. PAAB has created a manufacturer portal where the sponsor has access to all their ongoing and past files. If a manufacturer is unable to locate and provide a past eFile number to the agency, our file coordinators may be able to help find this information for the new agency of record if provided written permission from the manufacturer. Please note that this option should be utilized only in exceptional circumstances. As a long-term solution, PAAB is diligently working on AI that will help automate these processes in the future.

For FPO images, we understand that often submitters will not purchase the stock image until PAAB approval. To avoid delays or further questions, please confirm in your cover letters that these are intended to be the final images for purchase and will not be changed post-approval. We have also reminded reviewers that generally images with an FPO watermark are the intended final images and that any issue with the FPO image should be raised on review of the layout to avoid delays.
Comment lacked clear substantiation from the PAAB code (n= 1):

Action Taken: Reviewers have been reminded to ensure all relevant code sections are included to support review comments.

Key Takeaways:

Survey Completion Rate is 18.5%, with this data capturing 235 responses out of 1254 surveys sent. Data should be interpreted with this in mind, as this is a relatively low sample size.
Ratings for all questions, on average, have been positive. This data set is reflective of the complete quarter. Results have remained generally positive and consistent with the data from all quarters of 2023.

Have your voice heard! Help us in continually improve by completing your CEI surveys. You can find them in the “My CEI Surveys” Tab in the top navigation bar in eFiles. This helps us identify trends and implement quality improvement initiatives both internally and externally.

We continue to encourage you to be as specific as possible when providing feedback in order to best understand your experience with PAAB and create a meaningful action-plan to improve or disseminate best practices. Thank you for your continued participation in the CEI surveys!

How to make your feedback count!

*Management provided variations on comments received to ensure no identifiers

Please be specific!

“All staff involved were very responsive. However, the reviewer assigned to the file was not very helpful during the review.”

TIP: Add detail! If you were happy with most of the staff, but didn’t find the reviewer particularly helpful, tell us what happened. What made you feel that way? What part of the review was unhelpful?

“N/A” or similar (accompanied by a rating with an asterisk denoting a lower score)

TIP: Put yourself in our shoes. If you received a low score on your performance rating, but were not told why, would you know how to improve to better your score in the future? CEI scores are part of our team’s performance evaluations and are used to guide company-wide process improvement and best-practice sharing. The more detail we have, the better we can guide these initiatives!

Most of the staff involved in my eFile were very responsive. The file coordinators noticed A PM update was missing and worked with me to quickly get the proper file uploaded and into the queue. However, in the second round of review, I requested a call with my reviewer to discuss Comment 2 of their letter, and the only availability they provided was 4 days after my initial request. It’s my understanding that reviewers are to at least provide options for a return-call within 24 hours, so I was disappointed by this as it impacted my overall timeline.

Why this works:

It’s detailed! We now understand what the issue is that caused the responder to feel that the reviewer was not helpful and can dig into why this ticket call took so long to book and return.
Both positive and constructive feedback was provided. We appreciate that not all cases will have both positive and constructive feedback, but this was particularly helpful to understand the ranking and identify areas for improvement. Where there is positive feedback, it helps us amplify these best practices since we know what is helpful to you. Constructive feedback helps us understand what the challenges were and improve on an individual level and a systems level.

Confidence in confidentiality

As a reminder, client tags trigger internal audits for validation by PAAB’s Director of Pre-clearance Services, Yin Man. Any tags pertaining to Yin are validated by the Commissioner and removed from the report provided to Yin. No Reviewer, Senior Reviewer, or Director is EVER aware of tags generated by clients. You can be confident in the confidentiality of the tagging system. For additional reassurance, the tagging system, tag assessments, and documented actions taken will periodically be reviewed by an external auditor.

What does PAAB use the tags for?

Staff and system performance metrics
To identify trends and training opportunities
To stay inform on what is going well (best practices) and areas for improvement

If you’d like to learn more about the client tagging system, check out the Client Tagging System Advisory. You’ll also find links to useful videos on tagging a review and tagging phone calls.