A Quarterly Review of the eFiles Customer Experience Index (CEI) - Q4

The CEI Survey launched on February 13, 2023. This review of the CEI data accounts for the data captured between October 1 – December 31, 2023.

215 Completed Surveys October 1 to December 31, 2023

Averages of the CEI question survey results by question:

1. Staff connected with this eFile (e.g., file coordinators, reviewers, senior reviewers, etc.) were helpful and responsive	4.4/5   Indicating an average response between “Agree” and “Strongly Agree”
2. Comments and suggestions in response letters, calls and messenger were clear and actionable.	4.3/5 Indicating an average response between “Agree” and “Strongly Agree”
3. I felt the review was: 1. Highly inconsistent 2. Somewhat inconsistent 3. Somewhat consistent 4. Highly Consistent 5. I don't know	3.7/4*   Indicating an average response between “Somewhat consistent” and “Highly Consistent”
*This average rating is calculated from Responses 1-4 as including ratings of 5 (or “I don’t know”) would have skewed the average upward. There was 4 ratings of “I don’t know” in this data set.
4. Please provide any other feedback specific to this file: [optional open text field]	See feedback themes below.
5. Please rate your overall experience with this particular review  1 – highly negative experience 10 – highly positive experience	8.6/10   Indicating a positive average overall experience.

 

Key Themes and Actions Taken from Open-text Feedback:

1. Reviewer overly consistent and perceived to have defaulted to previous rulings (n= 2) There have been three comments pertaining to reviewers being overly consistent with previous rulings and they were perceived to not consider new claims or arguments.

Action Taken: Reviewers have been reminded that agencies and manufacturers may revisit and adapt creative and messaging, and that new rationale and presentations should be considered with fresh eyes. We will continue to map this issue to see if trends develop.

2. Reviews could have been completed in less rounds (n=5) In five instances, the submitters felt the review could have been completed in fewer rounds.

Action Taken: Reviewers have been reminded that while we do not review layout for copy correctness, they should be acknowledged, and any easily identifiable issues should be raised as early as possible. Reviewers were encouraged to make mention of issues even if they expect the copy to change, if they do not perceive that change to impact the flow of the piece. Similarly, reviewers were asked to advise clients if they did not perform the French review and to perform the French review if the remaining change would not impact the French (i.e. revision of a number or update repositioning of a footnote).

3. No comments (n= 5): Five comments read “no comment” or “straight renewal”.

Action Taken: We would like to ask that if clients are filling out the CEI and there were no issues with the review, could clients please provide suggestions that would have improved the interaction or what limited the experience to a “3” so that we can try to take action.

 

Key Takeaways:

• Survey Completion Rate is 8%, with this data capturing 215 responses out of 1067 surveys sent. Data should be interpreted with this in mind, as this is a relatively low sample size.
• Ratings for all questions, on average, have been positive. This data set is reflective of the complete quarter. Results have remained generally positive and consistent with the data from Q1 through Q3, with all scores being similar across the quarters.

Have your voice heard! Help us in continually improve by completing your CEI surveys. You can find them in the “My CEI Surveys” Tab in the top navigation bar in eFiles. This helps us identify trends and implement quality improvement initiatives both internally and externally.

We continue to encourage you to be as specific as possible when providing feedback in order to best understand your experience with PAAB and create a meaningful action-plan to improve or disseminate best practices. Thank you for your continued participation in the CEI surveys!

A Quarterly Review of the eFiles Customer Experience Index (CEI) - Q3

The CEI Survey launched on February 13, 2023. This review of the CEI data accounts for the data captured between July 1 to September 30, 2023.

256 Completed Surveys July 1 to September 30, 2023

Averages of the CEI question survey results by question:

1. Staff connected with this eFile (e.g., file coordinators, reviewers, senior reviewers, etc.) were helpful and responsive	4.5/5   Indicating an average response between “Agree” and “Strongly Agree”
2. Comments and suggestions in response letters, calls and messenger were clear and actionable.	4.4/5 Indicating an average response between “Agree” and “Strongly Agree”
3. I felt the review was: 1. Highly inconsistent 2. Somewhat inconsistent 3. Somewhat consistent 4. Highly Consistent 5. I don't know	3.8/4*   Indicating an average response between “Somewhat consistent” and “Highly Consistent”
*This average rating is calculated from Responses 1-4 as including ratings of 5 (or “I don’t know”) would have skewed the average upward. There was 4 ratings of “I don’t know” in this data set.
4. Please provide any other feedback specific to this file: [optional open text field]	See feedback themes below.
5. Please rate your overall experience with this particular review  1 – highly negative experience 10 – highly positive experience	8.9/10   Indicating a positive average overall experience.

Quick Complete CEI Data:

Upon the launch of the CEI Survey dashboard and their associated emails, a “quick complete” function was added in the emails only. This was added inadvertently and made live prior to its intended launch. It has since been deactivated. However, based on the data collected in this time frame, 36 CEIs were submitted via “quick complete” which indicate that these submitters were “happy with the overall experience with this eFile”.

Key Themes and Actions Taken from Open-text Feedback:

1. Clarity of comments (n= 3): Open text responses suggested that in a few instances, reviewer comments could have been clearer. In all instances, the issues were clarified and resolved with the help of a phone call.

Action Taken: Reviewers have been reminded to ensure comments are written in clear, concise language and to provide examples and suggestions when possible.

2. Reviewer overly consistent and perceived to have defaulted to previous rulings (n= 3) There have been three comments pertaining to reviewers being overly consistent with previous rulings and they were perceived to not consider new claims or arguments.

Action Taken: Reviewers have been reminded that agencies and manufacturers may revisit and adapt creative and messaging, and that new rationale and presentations should be considered with fresh eyes.

3. Inconsistencies identified within the same brand (n=5) In five instances, there were comments made by the reviewer that were inconsistent with previous rulings for the same brand. In three cases, when the reviewer was notified, the reviewer ultimately accepted the copy in question. In the other cases, the context of the current APS was different and the reviewer continued to question the copy.

Action Taken: When it is brought to a reviewer’s attention that their current comment may be inconsistent with previous rulings for the brand, unless there is a significant context difference that impacts the ruling, the comment will be brought to the Director of Preclearance Services to ensure the comment is necessary. Reviewers have also been reminded to try to elaborate when context is different to help facilitate understanding of why a ruling may be perceived to be different.         

 

Key Takeaways:

• Survey Completion Rate is 6%, with this data capturing 256 responses out of 1452 surveys sent. Data should be interpreted with this in mind, as this is a relatively low sample size.

• Ratings for all questions, on average, have been positive. This data set is reflective of the complete quarter. Results have remained generally positive and consistent with the data from Q1 and Q2, with all scores being the same or slightly higher than the average scores reported in Q2. We will continue tracking for trends through Q4.

Have your voice heard! Help us in continually improve by completing your CEI surveys. You can find them in the “My CEI Surveys” Tab in the top navigation bar in eFiles. This helps us identify trends and implement quality improvement initiatives both internally and externally.

While we have received a relatively low amount of open-text feedback, we have noticed an improvement in the quality of the comments, rendering the feedback more actionable. We continue to encourage you to be as specific as possible when providing feedback in order to best understand your experience with PAAB and create a meaningful action-plan to improve or disseminate best practices.

A Quarterly Review of the eFiles Customer Experience Index (CEI) - Q2

The CEI Survey launched on February 13, 2023. This review of the CEI data accounts for the data captured between April 1 to June 30, 2023.

291 Completed Surveys April 1 to June 30, 2023

Averages of the CEI question survey results by question:

1. Staff connected with this eFile (e.g., file coordinators, reviewers, senior reviewers, etc.) were helpful and responsive	4.4/5   Indicating an average response between “Agree” and “Strongly Agree”
2. Comments and suggestions in response letters, calls and messenger were clear and actionable.	4.4/5 Indicating an average response between “Agree” and “Strongly Agree”
3. I felt the review was: 1. Highly inconsistent 2. Somewhat inconsistent 3. Somewhat consistent 4. Highly Consistent 5. I don't know	3.7/4*   Indicating an average response between “Somewhat consistent” and “Highly Consistent”
*This average rating is calculated from Responses 1-4 as including ratings of 5 (or “I don’t know”) would have skewed the average upward. There was 4 ratings of “I don’t know” in this data set.
4. Please provide any other feedback specific to this file: [optional open text field]	See feedback themes below.
5. Please rate your overall experience with this particular review  1 – highly negative experience 10 – highly positive experience	8.7/10   Indicating a positive average overall experience.

Quick Complete CEI Data:
Upon the launch of the CEI Survey dashboard and its associated emails, a “quick complete” function was added in the emails only. This was added inadvertently and made live prior to its intended launch. It has since been deactivated. However, based on the data collected in this time frame, 19 CEIs were submitted via “quick complete” which indicate that these submitters were “happy with the overall experience with this eFile”.

Key Themes and Actions Taken from Open-text Feedback:
1. Perceived Creative Subjectivity (n= 2): Reviewing creative concepts can be challenging, and that was noted as a perceived issue in the open-text comments regarding larger creative campaigns and the use of icons.

Action Taken: The Creative Imagery Committee was formed and is comprised of industry experts on creative imagery use in Pharmaceutical Advertising. The objectives of the committee are to explore the global landscape and the needs of the Canadian Market, and propose standards which can be used by agencies when developing/Canadianizing creatives.

2. Late comments (n= 2) In a few instances, there were comments brought up by the reviewer after the first response letter that were necessary for the acceptance of the piece.

Action Taken: We understand late comments can cause delays. Reviewers have been reminded that any late comments should be first reviewed by the Director of Preclearance Services to ensure the comment is valid.

3. Inconsistencies identified within the same brand (n=2) In a few instances, there were comments made by the reviewer that were inconsistent with previous rulings for the same brand. In one case, when the reviewer was notified, the reviewer ultimately accepted the copy in question. In the other case, the context of the current APS was different and the reviewer continued to question the copy.

Action Taken: When it is brought to a Reviewer’s attention that their current comment may be inconsistent with previous rulings for the brand, unless there is a significant context difference that impacts the ruling, the comment will be brought to the Director of Preclearance Services to ensure the comment is necessary. Reviewers have also been reminded to try to elaborate when context is different to help facilitate understanding of why a ruling may be perceived to be different.

Key Takeaways:

• Survey Completion Rate is 17.6%, with this data capturing 291 responses out of 1647 surveys sent. Data should be interpreted with this in mind, as this is a relatively low sample size.
• Ratings for all questions, on average, have been positive. This data set is reflective of the complete quarter. Results have remained generally positive and consistent with the data from Q1, although all scores are slightly lower than the average scores reported in Q1. We will continue tracking for trends through Q3 and provide actionable items at that time.

See "How to make your feedback count" and "Confidence in confidentiality" below. 

A Quarterly Review of the eFiles Customer Experience Index (CEI) - Q1

The CEI Survey launched on February 13, 2023. This review of the CEI data accounts for the early data captured in the first quarter.

141 Completed Surveys February 13 to March 31, 2023

Averages of the CEI question survey results by question:

1. Staff connected with this eFile (e.g., file coordinators, reviewers, senior reviewers, etc.) were helpful and responsive	4.5/5   Indicating an average response between “Agree” and “Strongly Agree”
2. Comments and suggestions in response letters, calls and messenger were clear and actionable.	4.4/5 Indicating an average response between “Agree” and “Strongly Agree”
3. I felt the review was: 1. Highly inconsistent 2. Somewhat inconsistent 3. Somewhat consistent 4. Highly Consistent 5. I don't know	3.7/4*   Indicating an average response between “Somewhat consistent” and “Highly Consistent”
*This average rating is calculated from Responses 1-4 as including ratings of 5 (or “I don’t know”) would have skewed the average upward. There was 1 rating of “I don’t know” in this data set.
4. Please provide any other feedback specific to this file: [optional open text field]	No responses provided
5. Please rate your overall experience with this particular review  1 – highly negative experience 10 – highly positive experience	8.9/10   Indicating a positive average overall experience.

Key Takeaways:

• Survey Completion Rate is 16%, with this data capturing 141 responses out of 886 surveys sent. Data should be interpreted with this in mind, as this does not capture the full quarter and is a relatively low sample size.
• Ratings for all questions, on average, have been positive. As this data is reflective of an incomplete quarter, we will continue tracking for trends through Q2 and provide actionable items at that time.

Have your voice heard! Help us in continually improve by completing your CEI surveys. You can find them in the “My CEI Surveys” Tab in the top navigation bar in eFiles. This helps us identify trends and implement quality improvement initiatives both internally and externally.

We received a low amount of actionable, open-text feedback. In order to best understand your experience with PAAB and create a meaningful action-plan to improve or disseminate best practices, detailed, specific feedback is very important.

How to make your feedback count! *Management provided variations on comments received to ensure no identifiers Please be specific!  “All staff involved were very responsive. However, the reviewer assigned to the file was not very helpful during the review.” TIP: Add detail! If you were happy with most of the staff, but didn’t find the reviewer particularly helpful, tell us what happened. What made you feel that way? What part of the review was unhelpful?   “N/A” or similar (accompanied by a rating with an asterisk denoting a lower score) TIP: Put yourself in our shoes. If you received a low score on your performance rating, but were not told why, would you know how to improve to better your score in the future? CEI scores are part of our team’s performance evaluations and are used to guide company-wide process improvement and best-practice sharing. The more detail we have, the better we can guide these initiatives! Most of the staff involved in my eFile were very responsive. The file coordinators noticed A PM update was missing and worked with me to quickly get the proper file uploaded and into the queue. However, in the second round of review, I requested a call with my reviewer to discuss Comment 2 of their letter, and the only availability they provided was 4 days after my initial request. It’s my understanding that reviewers are to at least provide options for a return-call within 24 hours, so I was disappointed by this as it impacted my overall timeline. Why this works: It’s detailed! We now understand what the issue is that caused the responder to feel that the reviewer was not helpful and can dig into why this ticket call took so long to book and return. Both positive and constructive feedback was provided. We appreciate that not all cases will have both positive and constructive feedback, but this was particularly helpful to understand the ranking and identify areas for improvement. Where there is positive feedback, it helps us amplify these best practices since we know what is helpful to you. Constructive feedback helps us understand what the challenges were and improve on an individual level and a systems level.

 

Confidence in confidentiality As a reminder, client tags trigger internal audits for validation by PAAB’s Director of Pre-clearance Services, Yin Man. Any tags pertaining to Yin are validated by the Commissioner and removed from the report provided to Yin. No Reviewer, Senior Reviewer, or Director is EVER aware of tags generated by clients. You can be confident in the confidentiality of the tagging system. For additional reassurance, the tagging system, tag assessments, and documented actions taken will periodically be reviewed by an external auditor. What does PAAB use the tags for? Staff and system performance metrics To identify trends and training opportunities To stay inform on what is going well (best practices) and areas for improvement If you’d like to learn more about the client tagging system, check out the Client Tagging System Advisory. You’ll also find links to useful videos on  tagging a review and tagging phone calls.