Measuring the usability of the Pydroid 3® app using SUS method

1Departamento de Investigación, Instituto Superior Tecnológico Japón, Quito, Ecuador.

2Universidad Laica Eloy Alfaro de Manabí, Extensión Chone, Ecuador.

3Carrera de Desarrollo de Software, Instituto Superior Tecnológico Japón, Santo Domingo, Ecuador.

*Autor para correspondencia

Como citar el artículo:

Marcillo Vera, F., Andrade Acosta, N., Vaca Escoba, P., & Viteri Alcívar, Y. (2023). Measuring the usability of the Pydroid 3 ® app using SUS method. Informática y Sistemas: Revista de Tecnologías de la Informática y las Comunicaciones, 7(1), 1–6. https://doi.org/10.33936/isrtic.v7i1.5791

Enviado: 02/12/2022;

Aceptado: 25/12/2022;

Publicado: 02/01/2023

Medición de la usabilidad de la aplicación Pydroid 3® utilizando el método SUS

Autores

1*Fabricio Marcillo Vera

2Nilo Andrade Acosta

3Patricio Vaca Escobar

2Yanina Viteri Alcívar

Resumen

Hoy en día, las empresas de tecnología desarrollan aplicaciones móviles con el fin de satisfacer diversas necesidades de los usuarios. Estas aplicaciones móviles son diseñadas en base a ciertos parámetros de estudio como es la usabilidad. La usabilidad de una aplicación móvil se refiere a la magnitud con la cual el usuario satisface sus necesidades en un contexto específico. La usabilidad de una aplicación móvil está regulada por la eficiencia, eficacia y el grado de satisfacción del usuario como lo establece la norma ISO 9241-11. En este estudio, se pretende medir la usabilidad de la aplicación móvil Pydroid 3 ® mediante el método SUS, donde la variable de estudio fue el grado de satisfacción del usuario, en este caso la población de estudio fue homogénea, fueron estudiantes de nivel universitario. Según los resultados obtenidos, se obtuvo un grado de satisfacción entre el 40 % y 50 %, estos denotan que la aplicación presenta complejidad en el uso y requiere de soporte técnico para mejorar la usabilidad. En conclusión, la aplicación móvil presentó una usabilidad media considerando el nivel de satisfacción obtenido.

Palabras clave: Usabilidad; Sistema; Satisfacción; Usuario.

Abstract

Nowadays, technology companies develop mobile applications in order to satisfy various user needs. These mobile applications are designed based on certain study parameters such as usability. The usability of a mobile application refers to the extent to which the user satisfies his needs in a specific context. The usability of a mobile application is regulated by the efficiency, effectiveness and degree of user satisfaction as established by the ISO 9241-11 standard. In this study, we intend to measure the usability of the mobile application Pydroid 3 ® using the SUS method, where the study variable was the degree of user satisfaction, in this case the study population was homogeneous, they were university level students. According to the results obtained, a degree of satisfaction between 40 % and 50 % was obtained, these denote that the application presents complexity in use and requires technical support to improve usability. In conclusion, the mobile application presented an average usability considering the level of satisfaction obtained.

Keywords: Usability; System; Satisfaction; User.

1. Introduction

Mobile devices offer multiple services through mobile applications, in short, a mobile application can execute a specific task through an operating system. In the market, there are various operating systems, which stand out, Android, iOS, OS, Windows Phone, among others. Today, mobile applications must comply with multiple aspects so that it is liked by the user and generates greater economic value in the market. Multiple companies develop mobile applications in order to satisfy educational, data storage, data analysis, and entertainment needs, among others (Hoehle & Venkatesh, 2015).

Mobile applications are developed based on quality standards, according to the ISO 9241-11:1998 standard modified in 2018, this standard is oriented towards quality in usability and ergonomics for technology products and services, both in software and hardware. These regulations are based on standards such as ISO/EEC 15288:2008 and ISO/EEC 12207:2008, which allow the software to provide the concept of quality, establishing whether the requirements are correct, complete, precise, consistent, and verifiable (Fathiyyah et al., 2022; Sigalingging et al., 2022).

The ISO 9241 standard focuses specifically on human-centered design and a determining factor is usability. Usability refers to the extent to which a product can be used by specific users in a specific context (Fathiyyah et al., 2022). Also, it should be noted that usability is influenced by constant technological progress and offering a product that satisfies the user need (Fathiyyah et al., 2022). Based on the aforementioned ISO standard, to measure usability, the parameters to take into consideration are effectiveness, which is related to the precision and completeness with which users use the application. Also, efficiency is defined as the relationship between effectiveness and the resources used in the development of the mobile application. Finally, satisfaction is the degree to which the user feels satisfied when using the application to achieve the defined objective (Fathiyyah et al., 2022).

There are different methods to evaluate the usability of a mobile application, considering that there is no single test to do it, companies like Google, Apple, Yahoo!, among others, use usability techniques based on specific needs. According to Kaya et al. (2019) it establishes that the utility scalability of a system (SUS) method is an alternative to determine usability through the level of user satisfaction, where it will be possible to understand the problems that users face when using a mobile application.

Hoehle & Venkatesh (2015) used the SUS method to measure the usability of the top ten mobile applications on iOS and Android platforms for smartphones and tablets. The results of their study show that mobile apps on the iOS platform are easier to use than Android-based apps.

Google Android and Apple iOS have their own user interface guidelines that developers must follow to launch their mobile apps on the Apple and Google stores (Rafifing et al., 2022; Ratnawati et al., 2020). In addition to these guidelines, in the literature, there are several mobile application usability guidelines developed by researchers based on this type of user interface guidelines (Darmawan et al., 2021; Dian Martha et al., 2021; Wahyuningrum et al., 2020).

Kortum & Sorber (2015) developed nineteen first-order constructs such as instant startup, effort minimization, concise language, and 6-second constructs such as application layout, UI graphics for mobile applications, such as those based on Apple general usage guidelines. They validate their conceptualization by applying surveys to American consumers who use social networking applications.

This study aims to evaluate the usability of the Pydroid 3 ® mobile application, this mobile application is available in the Android operating system, developed by the company IIEC ®. Pydroid 3 ® is a mobile application that offers users to learn programming with Python (IIEC, 2023). The usability of the mobile application was evaluated through the SUS method applied to university level students in Ecuador, mentioning that the application was used from May 2019 to January 2023 in order to improve the teaching-learning processes of software development. in college-level students.

2. Method

2.1. Study population

For this study, the mobile application Pydroid 3 ® was evaluated on the Android operating system. The study population were students of the Software Development career (n = 96) of the Higher Technological Institute Japan, in Ecuador. In this study the students taken into consideration were in technical training prior to the use of the mobile application.

2.2. Determination of the user satisfaction level

For this study, the mobile application Pydroid 3 ® was evaluated on the Android operating system. The study population were For the study, the SUS method established by Brooke in 1996 was used, which consists of a set of questions, which evaluate positive and negative aspects of the mobile application. The even-numbered questions represent the evaluation of negative aspects, and the odd-numbered questions represent the evaluation of positive aspects. The dependent variable was the degree of user satisfaction, which is represented on a scale of 0 to 5 points. As detailed in Table 1.

For the interpretation of the results, the scheme proposed by Ahmad et al. (2023) was used, which establishes a scale of 0 to 4 points, i.e., for the score of the even numbered questions, the user score will be subtracted from the maximum score, i.e., five points As detailed in Table 2. For odd numbered questions, one point will be subtracted from the user score. Additionally, Ahmad et al. (2023) and Kaya et al. (2019) state that the sum of the scores obtained, multiplied by a constant (2.5), these values were represented as a percentage.

2.3. Data analysis

Within the statistical analysis, a hierarchical model was used, supported by scatter plots, the study variable was the percentage of user acceptance with respect to the question asked, then a maximum likelihood scheme of variables was elaborated by means of a dendogram.

3. Results

According to the results obtained, ninety-six participants were surveyed, of which 80.2 % are male and 19.8 % are female, these data refer to the number of students enrolled from 2019 to the present date as shown in Figure 1.

3.1. Level of user satisfaction

Based on the results obtained in the application of the questionnaire, it was determined that the percentage of user satisfaction is in a range between 40 % and 50 % as shown in Figure 2. Within question 1, users stated that they would use the system again with a tolerable frequency. In question 2, users stated that they were dissatisfied as they considered it to be a complex system. In question 3, users reported that the ease of use of the system is tolerated. In question 4, users stated that usability regarding technical support is dissatisfied as technical support is required. In question 5, users stated that various integrated functions are observed so usability is tolerable.

Likewise, as shown in Figure 2, in question 6, users reported several inconsistencies, but stated that these were tolerable. In question 7, the users reported that they are tolerable, however, they do not consider that it is easy to learn for all the public. In question 8, they reported that the system is cumbersome to use and were dissatisfied. In question 9, it was reported that users are tolerable in reference to the use of the system from the perspective of the security offered by the service. Finally, in question 10, users stated that learning previous concepts is required for the use of the Pydroid 3 ® system, however, it is tolerable.

3.2. Analysis of results

According to the results obtained, users reported complexity in the use of the mobile application, this may be due to multiple factors, lack of prior knowledge is one of them, although the group was in contact with the mobile application for about 4 months and its use was challenging during didactic classes. Pydroid 3 ®, being a mobile application for software development, requires technical support, especially if the user is a student. In addition, users reported different integrated functions, this refers to the fact that Pydroid 3 ® offers multiple tools for software development (Al-Omar, 2018; Hidayat et al., 2022).

Users indicate that there are inconsistencies in the system, this may be due to new application or mobile updates or compatibility with mobile devices. Pydroid 3 ® being an educational mobile application requires previous knowledge of software development to improve its usability, this is corroborated by the results obtained. Users reported that the system can be cumbersome to use, this may be due to the lack of experience in software development. In relation to data security, Pydroid 3 ® is dependable as stated by the users (Ismail et al., 2021).

In reference to the plausibility of the degree of user satisfaction as seen in Figure 3, it was determined that the odd number questions are grouped in general clade as well as the even number questions. Considering that it is not the same trend between question eight and question nine, however the context of both questions refers to the security of the data and the difficulty of using the application, in itself, there is a proportional correlation since the users corroborate that they feel safe using the application despite the difficulty of its use (Hadiwiyanti et al., 2022; Rafifing et al., 2022).

According to data provided by Google, the Pydroid 3 mobile application has a degree of user acceptance of 90 % considering 33500 user opinions, i.e., it has a high usability, considering the degree of user satisfaction. In comparison to the results obtained, within the study population, a range of user satisfaction between 40 % and 50 % was obtained, i.e., it presents a medium usability. This may be due to the fact that the study population is homogeneous, i.e., they are students in the stage of acquiring knowledge, which may hinder the usability of the mobile application (Martono et al., 2022; Pradini et al., 2019).

4. Conclusions

It is concluded that the degree of user satisfaction is between 40% and 50%, which shows that the usability of the mobile application was average considering the aspects evaluated by the SUS method. Based on the results obtained, the usability of Pydroid 3 ®, the application requires updates and technical support. As this influences the level of user satisfaction.

The SUS method allows to evaluate the usability of a mobile application, through the degree of user satisfaction, although it is a parameter to be considered in the usability, it should also take into consideration the efficiency and effectiveness of a mobile application as a parameter of study of the usability of a mobile application.

It is important to determine the usability of educational mobile applications since new teaching-learning techniques can be employed considering high usability. In future studies, we intend to evaluate the usability of educational mobile applications in different operating systems in order to establish usage trends in gamification processes in students.

Acknowledgments

As part of this study, special thanks are due to the Higher Technological Institute of Japan for the resources allocated to the research project.

Contribution of the authors

Fabricio Marcillo Vera: supervision, writing - drafting and editing of the article. Nilo Andrade Acosta: conceptualization and methodology. Patricio Vaca Escobar: visualization and research. Yanina Viteri Alcívar: software and formal analysis.

Conflicts of interest

The authors declare no conflict of interest.

Bibliographic references

Ahmad, A. E., Kusrini, K., & Sudarmawan, S. (2022). Usability evaluation of office stationery procurement service and management system using System Usability Scale. 2022 6th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE), 498–502. https://doi.org/10.1109/ICITISEE57756.2022.10057706

Al-Omar, K. (2018). Evaluating the usability and learnability of the “Blackboard” LMS using SUS and data mining. 2018 Second International Conference on Computing Methodologies and Communication (ICCMC), 386–390. https://doi.org/10.1109/ICCMC.2018.8488038

Darmawan, A. K., Hamzah, M. A., Bakir, B., Walid, M., Anwari, A., & Santosa, I. (2021). Exploring Usability Dimension of Smart Regency Service with Indonesian Adaptation of the System Usability Scale (SUS) and User Experience Questionnaire (UEQ). 2021 International Conference on Computer Science, Information Technology, and Electrical Engineering, ICOMITEE 2021, 74–79. https://doi.org/10.1109/ICOMITEE53461.2021.9650086

Dian Martha, A. S., Budi Santoso, H., Junus, K., & Suhartanto, H. (2021). Usability Evaluation of the MeMo Tutor: A Scaffolding-Based Pedagogical Agent to Facilitate Learning. Proceedings - 2021 International Conference on Software Engineering and Computer Systems and 4th International Conference on Computational Science and Information Management, ICSECS-ICOCSIM 2021, 360–364. https://doi.org/10.1109/ICSECS52883.2021.00072

Fathiyyah, D., Sulthon Diani, M. D., Ayuning Saputri, Z., & Sunardi. (2022). Usability Evaluation on Life Insurance Application Using System Usability Scale and ISO 9241-11. Proceedings of 2022 8th International HCI and UX Conference in Indonesia, CHIuXiD 2022, 94–99. https://doi.org/10.1109/CHIuXiD57244.2022.10009774

Hadiwiyanti, R., Suryanto, T. L. M., & Safitri, E. M. (2022). Evaluation of Campus Event Management Information System Using System Usability Scale Method. Proceeding - IEEE 8th Information Technology International Seminar, ITIS 2022, 350–353. https://doi.org/10.1109/ITIS57155.2022.10009978

Hidayat, A. S., Santosa, P. I., & Hidayah, I. (2022). Usability Testing of MOOC Prototype Using SUS (System Usability Scale) Method. Proceedings - IEIT 2022: 2022 International Conference on Electrical and Information Technology, 290–294. https://doi.org/10.1109/IEIT56384.2022.9967901

Hoehle, H., & Venkatesh, V. (2015). Mobile application usability: Conceptualization and instrument development. MIS Quarterly: Management Information Systems, 39(2), 435–472. https://doi.org/10.25300/MISQ/2015/39.2.08

IIEC. (2023). Pydroid 3 - IDE for Python 3 (No. 3). IIEC. https://play.google.com/store/apps/details?id=ru.iiec.pydroid3&hl=es_EC&gl=US

Ismail, I. E., Nalawati, R. E., & Putra, A. (2021). System Usability Scale and Net Promoter Score on Donation Application of Toddlers Equipment. Proceedings - 2021 4th International Conference on Computer and Informatics Engineering: IT-Based Digital Industrial Innovation for the Welfare of Society, IC2IE 2021, 170–174. https://doi.org/10.1109/IC2IE53219.2021.9649186

Kaya, A., Ozturk, R., & Altin Gumussoy, C. (2019). Usability measurement of mobile applications with System Usability Scale (SUS). En F. Calisir, E. Cevikcan, & H. Camgoz Akdag (Eds.), Industrial Engineering in the Big Data Era (pp. 389–400). Springer International Publishing.

Kortum, P., & Sorber, M. (2015). Measuring the Usability of Mobile Applications for Phones and Tablets. International Journal of Human-Computer Interaction, 31(8), 518–529. https://doi.org/10.1080/10447318.2015.1064658

Martono, K. T., Prasetijo, A. B., & Distira, A. K. (2022). Analysis of Usability Game Educational Learning of Wayang Characters Using Usability Scale System. International Conference on Electrical, Computer, and Energy Technologies, ICECET 2022. https://doi.org/10.1109/ICECET55527.2022.9873447

Pradini, R. S., Kriswibowo, R., & Ramdani, F. (2019). Usability Evaluation on the SIPR Website Uses the System Usability Scale and Net Promoter Score. Proceedings of 2019 4th International Conference on Sustainable Information Engineering and Technology, SIET 2019, 280–284. https://doi.org/10.1109/SIET48054.2019.8986098

Rafifing, N., Mphale, O., & Asare, S. D. (2022). Exploring User perceptions of an E-Government System in Botswana Using System Usability Scale Model. Proceedings of the IEEE International Conference on Software Engineering and Service Sciences, ICSESS, 2022-Octob, 195–198. https://doi.org/10.1109/ICSESS54813.2022.9930194

Ratnawati, S., Widianingsih, L., Anggraini, N., Marzuki Shofi, I., Hakiem, N., & Eka M Agustin, F. (2020). Evaluation of Digital Library’s Usability Using the System Usability Scale Method of (A Case Study). 2020 8th International Conference on Cyber and IT Service Management, CITSM 2020. https://doi.org/10.1109/CITSM50537.2020.9268801

Sigalingging, F. A. L., Alibasa, M. J., & Nuha, H. H. (2022). Usability Analysis of My TelU Application Using System Usability Scale. International Conference on Electrical Engineering, Computer Science and Informatics (EECSI), 2022-Octob, 244–249. https://doi.org/10.23919/EECSI56542.2022.9946493

Wahyuningrum, T., Kartiko, C., & Wardhana, A. C. (2020, October 20). Exploring e-Commerce Usability by Heuristic Evaluation as a Compelement of System Usability Scale. 2020 International Conference on Advancement in Data Science, E-Learning and Information Systems, ICADEIS 2020. https://doi.org/10.1109/ICADEIS49811.2020.9277343