Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
For Mozambique LLIN Implementation
April 27th, 2023, April 28th, 2023
Maputo
Register 5 households residing in village A in Tete. You must capture the address details, how many people are living in the household, and fill in details (age, gender, mobile number) for at least 1 member living in the household. The first member you register in the household must be the head of the household. Then you proceed to deliver bed nets:
a. Which is less than the number of bed nets suggested by the app for 2 households.
b. Which is more than the number of bed nets suggested by the app for 1 household.
c. Which is same as the number of bed nets suggested by the app for 2 households.
You realise that you made a mistake while registering one member and want to correct your mistake. Search for a household head you have registered which has individual members added. The mistake was in the mobile number- Change the mobile number you entered previously with a new mobile number.
As a distributor you are required to do the distribution in village B. Change the location from village A to village B and register 2 households. This time add only the household head and no individual members in the household. Also, skip adding the landmark in the household details page and skip adding mobile number, age, gender while registering the household head. Deliver the exact number of bed nets as suggested by the app.
Search for a household you have delivered. You entered the household head for person A. But when you visited the household again, you needed to change the household head to another person. How will you do it?
Search for a household you have delivered. Now, delete a member from a household that you have delivered. How will you do it?
For one the deliveries recorded, you realise you have entered an incorrect number of bed nets. Instead of 2, you recorded 3. How will you correct the mistake?
Sync all the records pending to be synced in the application.
You are a registador and you forgot your password. What will you do?
You are not able to see your assigned village in the list of villages. What action would you take?
You are a registador and you realise, one of the members you registered has already been registered by your colleague and is a duplicate entry. What will you do to remove the duplicate entry from your records?
You are the supervisor for the team working in village A for Angonia district, and your team members are in the field distributing nets. As per protocol, you are expected to go and observe them work and record your observations in the registration and distribution checklist. Fill out the checklist and submit your observations for the following checklists. Mark ‘No’ as a response for atleast 2 questions in each checklist and provide the reasons.
a. Registration & distribution monitoring - Make this checklist for village A in Angonia district.
b. District Wwrehouse monitoring - Make this checklist for district Angonia.
c. Local monitors training monitoring - Make this checklist for administrative post A in district Angonia.
d. Registration & distribution training monitoring - Make the checklist for village B in Angonia district.
You are the supervisor for the team working in village A for Angonia district, and you have made 4 checklist entries so far. You are supposed to make 5 entries for the checklist "Registration & Distribution monitoring" daily. How will you check how many entries you have made so far for today and for which boundaries?
Sync all the records pending to be synced in the application.
You are a warehouse manager for a warehouse in administrative post A and received a stock of 2 bales today in your community warehouse from a district warehouse in Angonia. Enter the receipt of this in the app.
You are a local monitor for administrative post A and received a stock of 45 bed nets back from the distribution team. Enter the receipt of this for the community warehouse you are managing in the app as a return.
You are a district warehouse manager and you distributed the following
a. 200 nets to the delivery team A.
b. 5 bales to the community warehouse A.
You are inspecting the warehouse at the end of the day as a community warehouse manager and you counted the number of nets in stock at the end of the day. How will you make an entry for this for the following cases?
a. There are 50 more nets in the warehouse that you counted than the number of nets suggested by the system.
b. There are 10 less nets in the warehouse that you counted than the number of nets suggested by the system.
Sync all the records pending to be synced in the application.
(On 5 point scale: Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree)
I think I would like to use this tool frequently.
I found the tool unnecessarily complex.
I thought the tool was easy to use.
I think that I would need the support of a technical person to be able to use this app.
I thought there was too much inconsistency in this tool.
I would imagine that most people would learn to use this tool very quickly.
I found the tool very difficult to use.
I felt very confident using the tool.
I needed to learn a lot of things before I could get going with this tool.
I could efficiently complete my tasks using the system
On a scale of 1 to 10, how well did the UAT task represent real-world scenarios and user interactions with the system?
Was the UAT task comprehensive, covering all key functionalities and features of the system?
Did the UAT include a variety of scenarios, data inputs, and user interactions to thoroughly test the system?
Was the UAT task representative of different user roles, permissions, and use cases that are relevant to the system?
Did the UAT task adequately address any specific requirements or criteria that were defined for the UAT phase?
All UAT test cases completed: Verified that all UAT test cases, as defined in the UAT plan, have been executed and completed.
Business requirements validated: Validated that all business requirements, as defined in the requirements documentation - All features, functions, workflows, calculations, translations have been thoroughly tested and verified.
Compatibility tested: Verified that the application has been tested on the specified device which is used for the distribution (Samsung A09) and operating system (Android) and any compatibility issues have been addressed.
All feedback has been identified and documented. Agreed on the priority.
Documentation updated: Ensured that all relevant documentation, including user manuals, has been updated to reflect any changes made during UAT.
UAT summary report prepared: Prepared a UAT summary report that includes the UAT results, findings, and any outstanding issues or risks identified during UAT.
No P1 bugs found during the execution of the test scenarios.
90% of the total test cases could be executed successfully and the observed behaviour was found to match the expected results.
The purpose of the User Acceptance Test (UAT) is to verify that the system meets the business needs as defined in the functional requirements (product requirement document or PRD) and instills confidence in its usage. Additionally, the UAT helps to identify any defects with associated risks, suggest potential enhancements (via change requests), communicate any known defects or enhancements to the project team, and ensure that all issues are resolved before go-live in an appropriate manner.
The UAT process included testing the following user flows:
Registration & Delivery
Inventory Management
Supervision
The participants validated the end-to-end flow and features of the application to:
Verify the end-to-end business flow for each of the identified user flows.
Ensure that all required fields were present.
Confirm that the system accepts only valid values.
Test that the user could save after filling all mandatory fields in the form.
Check the correctness of labels and their translations.
Separate demo and/or training sessions will be conducted for the following functionalities, as they were not included in the scope of UAT:
Helpdesk for complaints and user management
a. User management module for adding/updating/activating/deactivating users and role mapping.
b. Handling complaints via inbox functionality.
Central, provincial, and district dashboards, and reports
These targeted sessions will ensure that the relevant user groups are trained on the proper use and functionality of these features.
Ines Juleca, Catarina, Bernardo Espinosa, Aderito Joaquim Suaze, Marta Chande, Joao Dombo, Custodio Mondlane, Guidion Mathe, Luis Ismael, Gerito Augusto, Silva Pedro, Gercio Machva, Aurelio Carlos Marindze, Mariana da silva, Figuereido Mussambala, Carla Bambo, Sergio Tsabete.
The selection of participants for the UAT was decided by NMCP, and specifically targeted members of the "Nucelo Duro" group. A total of 17 participants took part in the UAT and were supported by a three-member team from eGov and CHAI.
Moderators: Abhishek Suresh, Mrunal Surve, Frank Cumaio
The UAT planning was comprehensively documented in the following documents.
Before commencing testing, the participants were provided with a brief overview of the health mission and the benefits of DIGIT's platform approach. Each user flow was demonstrated to the participants by the moderator, and they were then tasked with executing each testing scenario under the three user flows.
A pre-approved list of test cases/scripts was executed to verify the behavior and performance of various functionalities. All observations from each participant during testing were captured and compiled. These observations will be subsequently classified as either defects or enhancements based on internal discussions within eGov. While some of the enhancements may be implemented using configuration changes, others (change requests) may require going through a formal change management process.
#
Observation type
Description
Addressing mechanism
1
Defects
Any observation pertaining to a feature or functionality not working as expected or agreed at the time of scope review, will be classified as an issue.
The observations classified as defects will be taken up by the eGov programme team for further validation, prioritisation and fixing. Minor issues originating due to incorrect configurations or erroneous labels, or translations will be fixed and made available for re-testing during the next UAT cycle.
2
Change requests (CR)
Any recommendation to enable or disable a functionality from the initial requirements or a new functionality to be added.
These will be handled via the change control process as per the SoP defined, and will be evaluated based on their impact and effort.
eGov and NMCP together will prioritise and classify defects. Defects found in the UAT can be assigned one of three (3) levels of severity:
P1, Work halted – Testing defects that due to the complexity of the function or the scheduled dates are putting the implementation date at risk. No workaround exists.
P2, Work slowed – Testing defects occurring in a less complex function of the application with sufficient time to resolve before the implementation date – but must be implemented as scheduled. A workaround has been identified and is listed in the defect.
P3 and lower, Work uUnaffected – Testing defects occurring in a function that are simple to fix or could be excluded if not resolved by the scheduled implementation date.
CRs will be clearly captured and reported for analysis to identify effort and impact to the eGov team. Each CR submitted will be validated and categorised for acceptance or rejection and then assigned with a priority. The development team will work on it and will be made available for testing. Following is a snapshot of the standard CR lifecycle.
eGov, in consultation with NMCP, will decide acceptance and categorisation of change requests. Change requests found in UAT can be assigned one of three (3) levels of category:
Must have – Change requests that are needed for the success of a campaign. No workaround exists.
Should have – Change requests that are required for better tracking and monitoring and will increase the ease of use of the system for users. A workaround has been identified and is listed in the CR.
Good to have – Change requests that are simply for better visualisation and reporting. It could be excluded if not resolved by the scheduled implementation date.
eGov to endeavour to cover "Must Have" changes before distribution. Lower priority changes will be taken through the eGov Gating process for planning subsequent releases.
The participants provided highly positive feedback, which was gathered through feedback forms accessed via QR codes presented during the session. To enable this, QR code scanners were installed on MISAU mobile devices using the scale fusion software. The feedback forms included:
System Usability Scale (SUS) feedback form: This form aimed to measure the overall usability of the system as perceived by the participants.
Overall UAT process feedback form: This form was designed to assess the effectiveness and efficiency of the UAT session, with the goal of incorporating feedback into future sessions.
The average SUS score for all participants was 81.25, which exceeded the standard acceptable range:
0-59: Very Poor
60-69: Poor
70-79: OK
80-89: Good
90-100: Excellent
“Show features like dashboards”
“I think I would need to see the dashboard for decision making”
“For me everything is fine“
“Include transport management to better support planning“
“Place visualisation panels and interoperability with other platforms“
“Extend to the dashboard to be more complete“
“Reports should appear in graphics for better monitoring“
“We should have more time and not happen between campaigns“
“Submit the application to practice before the start of the UAT“
“The language issue makes interaction with the team a little difficult“
“Must be included another forms suggested to be more a creaky“
Moving forward, the following needs to be checked for UAT or any other event:
Availability of a dedicated translator.
All documents and presentations to be made available in Portuguese.
A terms of reference (TOR) to be shared before the event
The eGov team will categorise the noted observations as either defects or change requests, and subsequently develop a plan to address them within an agreed-upon timeline. Following the resolution of identified bugs and change requests from UAT-1, which have been mutually agreed to by both eGov and NMCP, a second round of UAT is scheduled for June 2023.
The UAT session will be conducted physically in Maputo at the MOH office. The detailed plan for the UAT session is as follows:
The CR analysis may take more than the anticipated timelines. If the CR analysis exceeds more than day 2, then the analysis results, acceptance and prioritisation will be discussed in a subsequent meeting.
SUPERVISION
1
Supervisor
You are the supervisor for the team working in village A for Angonia district, and your team members are in the field distributing nets. As per protocol, you are expected to go and observe them work and record your observations in the registration and distribution checklist. Fill out the checklist and submit your observations for the following checklists. Mark ‘No’ as response for atleast 2 questions in each checklists and provide the reasons.
Registration and distribution monitoring - Make this checklist for village A in Angonia district.
1. Login credentials of a supervisor 2. Click on "My Checklists" 3. Click on the registration and distribution checklist 4. Fill in all the responses in yes or no 5. In the case of no, fill in the reason 6. Click on submit
District warehouse monitoring - Make this checklist for district Angonia.
1. Login credentials of a supervisor 2. Click on "My Checklists" 3. Click on the warehouse checklist 4. Fill in all the responses in yes or no 5. In the case of no, fill in the reason 6. Click on submit
Local monitors training monitoring - Make this checklist for administrative post A in district Angonia
1. Login credentials of a supervisor 2. Click on "My Checklists" 3. Click on the warehouse checklist 4. Fill in all the responses in yes or no 5. In the case of no, fill in the reason 6. Click on submit
Registration and distribution training monitoring - Make the checklist for village B in Angonia district
1. Login credentials of a supervisor 2. Click on "My Checklists" 3. Click on the registration and distribution checklist 4. Fill in all the responses in yes or no 5. In the case of no, fill in the reason 6. Click on submit
1
Warehouse manager
You are a warehouse manager for a warehouse in administrative post A, and received a stock of 2 bales today into your community warehouse from a district warehouse in Angonia. Enter the receipt of this in the app.
1. Login as a warehouse manager 2. Click on manage stock 3. Click on record stock receipt 4. Enter managed warehouse details 5. Click on next 7. Enter stock receipt details - Includes warehouse details and the mode from where the bales came 9. Click on submit button 9. Record created successfully
Warehouse manager should be able to create the stock received record.
2
Warehouse manager
You are a local monitor for administrative post A, and received a stock of 45 bed nets back from the distribution team. Enter the receipt of this for the community warehouse you are managing in the app as a return.
1. Login as a warehouse manager 2. Click on manage stock 3. Click on stock returned 4. Enter the details of the warehouse managed by you 5. Enter the details of stocks that have been returned - Includes the mode of transport by which the stock have been returned and the warehouse details from where they have been returned to. 6. Click on submit 7. Record successfully created
Warehouse manager should be able to record the stocks returned after entering details in the stocks returned form.
3
Warehouse manager
You are inspecting the warehouse at the end of the day as a community warehouse manager, and you counted the number of nets in stock at the end of the day. How will you make an entry for this for the following cases? a. There are 50 more nets in the warehouse that you counted than the number of nets suggested by the system. b. There are 10 less nets in the warehouse that you counted than the number of nets suggested by the system.
1. Login as a warehouse manager 2. Click on manage stock 3. Click on stock reconciliation in home screen 4. Check the total count of bed nets suggested for the warehouse(total inflow - total outflow = stock on hand) 5. Enter the physical count 50 more than the suggested total count. 6. Enter the reason in the comments behind entering more number of stocks in warehouse 7. Click on the submit button
Warehouse manager should be able to enter the stocks as per the manual count over the count of the stocks suggested.
1. Login as a warehouse manager 2. Click on manage stock 3. Click on stock reconciliation in home screen 4. Check the total count of bed nets suggested for the warehouse (total inflow - total outflow = stock on hand) 5. Enter the physical count 10 more than the suggested total count 6. Enter the reason in the comments behind entering less number of stocks in warehouse 7. Click on the submit button
Warehouse manager should be able to enter the stocks as per the manual count over the count of the stocks suggested.
#
Activity
Approximate Duration
Time
Day 1- 27 April 2023
1
Introductions
15 minutes
8.00 am-8.15 am
2
Introduction to the UAT Ssssion plan
Walkthrough of the mobile app and test cases
Walkthrough of the defect/CR reporting templates
Distribution of the templates for capturing feedback
Distribution of the test cases and mobile devices
60 Minutes
8.15 am-9.15 am
Tea break-1
9.15 am-9.45 am
3
Execution of the test cases
Registration & distribution
Capture feedback in the Google form
120 Minutes
9.45 am-11.45 am
4
Review and collection of test results
30 minutes
11.45am-12.15 pm
Lunch break
75 minutes
12.15 pm-1.30 pm
5
Orientation for the next set of test cases
15 minutes
1.30 pm-1.45 pm
6
Execution of Ttest cases
Supervision
Warehouse Manager
Capture defects/ CR in the Google form
Close the day
90 minutes
1.45 pm-3.15 pm
Tea break-2
30 minutes
3.15 pm-3.45 pm
7
Collation and digitisation of defects and CR by eGov & CHAI team
3.45 pm- 4.45 pm
Day 2 - 28th April 2023
1
Defect review: Clarifying Q&A on defects raised the day before
60 Minutes
8.15 am-9.15 am
Tea break-1
30 minutes
9.15 am-9.45 am
2
Execution of pending test cases if any
Capture defects/ CR in the Google form
60 minutes
9.45 am-10.45 am
3
Focus group discussion
SUS
UAT process feedback
Collection and review of the test results
60 minutes
11.00 am-12:00 pm
4
Close Day 2 testing session
30 minutes
12:00 pm-12.30 pm
Lunch break
75 minutes
12.30 pm-1.30 pm
Plan for UAT 1 & 2
Identify UAT Team
Team will include participants from eGov, CHAI and NMCP, as per the profiles suggested in section 4.1.
06-Mar
UAT Plan
SOP for UAT test plan and share it with CHAI/NMCP for review.
06-Mar
UAT Plan Team Review
Session with CHAI and NMCP to review the plan and provide feedback and sign-off.
15-Mar
UAT 1
UA Test Case Preparation
A document that details each specific test case that will be performed during the UAT process and user manual.
17-Mar
Test Data Acquisition
Receipt of accounts and test environment data from QA required to execute test scripts.
27-Mar
UAT Environment Setup
Validation of connectivity and expected results in the test environment for each end user participating in testing.
03-Apr
Availability of Mobile devices with App installed
Availability of mobile devices with app installed for testing.
03-Apr
UAT Testing
Completion of all test scripts execution by test team.
10-Apr
Defect Analysis and Approval / Prioritisation
Analysis of defects/ observations identified for approval and prioritisation.
14-Apr
UAT 2
UA Test Case Preparation
A document that details each specific test case that will be performed during the UAT process and user manual.
27-Mar
Test Data Acquisition
Receipt of accounts and test environment data from QA required to execute test scripts.
03-Apr
UAT Environment Setup
Validation of connectivity and expected results in the test environment for each end user participating in testing.
17-Apr
Availability of Mobile devices with App installed
Availability of mobile devices with app installed for testing.
17-Apr
UAT Testing
Completion of all test scripts execution by the test team.
24-Apr
Defect Analysis and Approval / Prioritisation
Analysis of defects/ observations identified for approval and prioritisation.
27-Apr
UAT 3
UA Test Case Preparation
A document that details each specific test case that will be performed during the UAT process, and user manual.
01-May
Test Data Acquisition
Receipt of accounts and test environment data from QA required to execute test scripts.
08-May
UAT Environment Setup
Validation of connectivity and expected results in the test environment for each end user participating in testing.
22-May
Availability of Mobile devices with App installed
Availability of mobile devices with app installed for testing.
22-May
UAT Testing
Completion of all test scripts execution by the test team.
29-May
Defect Analysis and Approval / Prioritisation
Analysis of defects/ observations identified for approval and prioritisation.
02-Jun
UAT Sign-Off
Formal sign-off indicating the system satisfies the needs of the business as specified in the functional requirements and provides confidence in its use.
12-Jun
Registration & Distribution
Registrar (Registration & Distribution)
Register one household
1. Login into the application through given credentials. 2. Click on view beneficiary 3. Search for the beneficiary 4. Click on register button 5. Enter registration details asked in the screen 6. Click on save button. Repeat the same steps to register four more households
Registrar should be able to register a new household
Pass
1. Ponto de referencia - make no mandatory; 2. Detalhes de membros do agregado familiar - replace (+; -) by Numbers, creat validation for 1 to 20 members per household;
1. Reference point - do not make mandatory; 2. Details of household members - replace (+; -) with Numbers, create validation for 1 to 20 members per household;
Registrar (Registration & Distribution)
Add members to each households registered.
1. Login into the application 2. Search the household 3. Click on open button right to the household name 3. Click on the add member button 4. Enter the details asked in the screen 5. Click on the submit button
Registrar should be able to add members to each household that has been created
Pass
1. Name is mandatory field. It is suggested to add all the details in the form. But if the other details are not available then the member can be added only by name. Fill the rest of the details for atleast one member in each household. For example, if there are 3 members in a household, as part of UAT, please fill in all the details in atleast one member, you can skip filling the other details other than names for the rest. 2. It is not mandatory to add details or add members to each household. The number of members in a household is enough for the bed nets to be distributed. As part of UAT, we ask you to add members to each household.
1. Nome do individuo - consider name and surname, because now allows to fil only one name but we need to see the surname; 2 Data de nascimento - translate the calendar to portuguese
Registrar (Registration & Distribution)
In two households, deliver bed nets less than the application mentions to deliver
1. Click on beneficiaries 2. Search for beneficiary 3. Click on open (right to the household name) on the household where the bed nets are to be delivered 4. Click on "Deliver intervention" button. 5. Enter the number of bed nets to be distributed less than the number mentioned 5. Select delivery comments 6. Select the reason "Insufficuient resource" 7. Click on submit.
Registrar should be able to enter the number of bed nets delivered less than what is suggested
Pass
Repeat the steps for two households.
None
Registrar (Registration & Distribution)
In one household, deliver bed nets more than the application mentions to deliver
1. Click on beneficiaries 2. Search for beneficiary 3. Click on open (right to the household name) on the household where the bed nets are to be delivered 4. Click on the "Deliver intervention" button. 5. Enter the number of bed nets to be distributed more as the number mentioned 5. Select delivery comments 6. Select the reason <> 7. Click on submit.
Registrar should be able to enter the number of bed nets delivered more than what suggested
Pass
1. Numero de redes distribuidas - change (+;-) by numbers; 2- comentario de envio - change by Comentarios and include validation rule to display only when numero de redes distribuidas is different from the Numero de redes para entrega
Registrar (Registration & Distribution)
In two households, deliver bed nets same as the application mentions to deliver
1. Click on beneficiaries 2. Search for a beneficiary 3. Click on open (right to the household name) on the household where the bed nets are to be delivered 4. Click on the "Deliver intervention" button. 5. Enter the number of bed nets to be distributed same as the number mentioned 5. Select delivery comments 6. Select the reason "Successfull Delivery" 7. Click on submit.
Registrar should be able to enter the number of bed nets delivered same as suggested
Pass
Same as above
Registrar (Registration & Distribution)
As a distributor you are required to do the distribution in village B. Change the location from village A to village B and register 2 households. This time add only the household head and no individual members in the household. Also, skip adding the landmark in the household details page and skip adding mobile number, age, gender while registering the household head. Deliver the exact number of bed nets as suggested by the app.
1. Login as registrar into the application 2. Select the location from the dropdowns mentioned in the location picker screen 3. Click on the selected location on the top right of the home screen. 4. Change the village name in the location picker screen 5. Click on submit 6. Click on view beneficiary 7. Search for the beneficiary 8. Click on register and next 9. Add the number of members 10. Click submit without adding any details in the individual screen 11. Click on open next to the beneficiary name 13. Click on deliver intervention 14. Enter the number of bed nets distributed as suggested 15. Select delivery comments as successful delivery 16. Click on the submit button
Registrar should be able to change the village. Registrar should be able to register the household without adding any indivdual members and household details. Registrar should be able to deliver and records the number of bed nets suggested
Pass
Repeat the steps from 7 to 15 twice.
None
Registrar (Registration & Distribution)
You realise that you made a mistake while registering one member and want to correct it. Search for a household head you have registered which has individual members added. The mistake was in the mobile number- Change the mobile number you entered previously with a new mobile number.
1. Launch the application 2. Enter credentials 3. Click on the login button 4. Search for the registered household using the household head's name 5. Click on the edit button next to the member whose mobile number has to be changed 6. Enter details for the updation 7. Click on the update button
Registrar should be able to change the mobile number of any member of the household, and the change mobile number should reflect
Pass
None
Registrar (Registration & Distribution)
Search for a household you have delivered. You entered the household head for person A. But when you visited the household again, you needed to change the household head to another person.
1. Launch the application 2. Enter credentials 3. Click on the login button 4. Search for the registered household using the household head's name and open the card. 5. Tap on edit for the household head details 6. Add members to the household. 7. For the newly-added member, tap on edit and assign the newly-created individual as the head of household. 8. Perform a search based on the newly-added individual's name
Registrar should be able to change the household head of the household
Pass
None
Registrar (Registration & Distribution)
Search for a household you have delivered. Now, delete a member from a household that you have delivered.
1. Launch the application 2. Enter credentials 3. Tap on Login button 4. Search for the registered beneficiary 5. Tap on edit household button 6. Tap on the delete button 7. On the confirmation message, tap on 'Delete' 8. On the "Reason For Deletion" screen, select an option and tap on 'Delete'
Registrar should be able to delete the household
Pass
Registrar (Registration & Distribution)
For one the deliveries recorded, you realise you entered an incorrect number of bed nets. Instead of 2, you recorded 3.
1. Launch the application 2. Enter credentials 3. Tap on Login button 4. Search for the registered beneficiary 5. Tap on update delivery details 6. Change the number in the number of bed nets distributed. 7. Click on Submit
Registrar should be able to update the delivery detail
Pass
Personally I think that the already synchronised forms should not have the option to edit later; should only work for non-synchronised forms.
