Automatic execution of tests in enterprise production environments for software
DOI:
https://doi.org/10.33936/isrtic.v3i1.1585Keywords:
ejecución automática de pruebas; generación automática de casos de prueba; integración continua.Abstract
ABSTRACT This paper presents a set of good practices for introducing automatic test execution in software devel- opment organizations. The objective is to integrate the tests with the work environment to reach higher levels of coverage and to assist the developers and testers in the design and execution of the test cases. The proposal contemplates environments of continuous integration of applications, with models for the automatic generation and execution of test cases. An analysis is made of the existing proposals in this area, their fundamental contributions and limitations; as a starting point for the presentation of the model for the automatic execution of software tests.The Mtest.search model contains procedures and methods for generating and executing test cases in- serted in a seamless integration environment within the application development process itself. This proposal can be adapted to the specific conditions of each company according to its own development platform.The experiences of application of the model in an environment of university development are exposed. KEYWORDS: automatic execution of test, automatic generation of test cases, continuous integration.Downloads
Download data is not yet available.
References
[1] GJ Myers, C Sandler y T Badgett. The art of software testing. 3rd. New Jersey, 2011. isbn: 978- 1-118-03196-4.
[2] Ting Chen y col. “State of the art: Dynamic symbolic execution for automated test generation”. En: Future Generation Computer Systems 29.7 (sep. de 2013), págs. 1758-1773. issn: 0167739X. doi: 10.1016/j.future.2012.02.006. url: https://linkinghub.elsevier.com/retrieve/ pii/S0167739X12000398.
[3] Frank Elberzhager y col. “Reducing test effort: A systematic mapping study on existing ap- proaches”. En: Information and Software Technology 54.10 (oct. de 2012), págs. 1092-1106. issn: 09505849. doi: 10.1016/j.infsof.2012.04.007. url: https://linkinghub.elsevier.com/ retrieve/pii/S0950584912000894.
[4] A.M. Memon, M.E. Pollack y M.L. Soffa. “Hierarchical GUI test case generation using automa- ted planning”. En: IEEE Transactions on Software Engineering 27.2 (2001), págs. 144-155. issn: 00985589. doi: 10.1109/32.908959. url: http://ieeexplore.ieee.org/document/908959/.
[5] L. Morales. “Componente de generación automática de valores de pruebas”. Tesis doct. Instituto Superior Politécnico José Antonio Echeverría, 2012.
[6] A. Moreira. “Entorno de integración continua basado en software libre para el desarrollo de apli- caciones Java”. Tesis doct. Instituto Superior Politécnico “José Antonio Echeverría”, 2016.
[7] Beatriz Lamancha Pérez y Macario Polo. “Generación automática de casos de prueba para Líneas de Producto de Software Automatic test case generation for software product lines”. En: Revista Española de Innovación, Calidad e Ingeniería del Software 5.2 (2009). issn: 1885-4486. url: http: //www.redalyc.org/pdf/922/92217153004.pdf.
[8] Tang Rongfa. “Adaptive Software Test Management System Based on Software Agents”. En: 2012, págs. 1-9. doi: 10.1007/978-3-642-25437-6_1. url: http://link.springer.com/10.1007/ 978-3-642-25437-6%7B%5C_%7D1.
[9] Soma Sekhara Babu Lam y col. “Automated Generation of Independent Paths and Test Suite Optimization Using Artificial Bee Colony”. En: Procedia Engineering 30 (2012), págs. 191-200. issn: 18777058. doi: 10.1016/j.proeng.2012.01.851. url: https://linkinghub.elsevier. com/retrieve/pii/S1877705812008612.
[10] Ying XING y col. “Intelligent test case generation based on branch and bound”. En: The Journal of China Universities of Posts and Telecommunications 21.2 (abr. de 2014), págs. 91-103. issn: 10058885. doi: 10.1016/S1005-8885(14)60291-0. url: https://linkinghub.elsevier.com/ retrieve/pii/S1005888514602910.
[11] Zhiqiang Zhang y col. “Generating combinatorial test suite using combinatorial optimization”. En: Journal of Systems and Software 98 (dic. de 2014), págs. 191-207. issn: 01641212. doi: 10 . 1016 / j . jss . 2014 . 09 . 001. url: https : / / linkinghub . elsevier . com / retrieve / pii / S0164121214001939.
[12] S Bergmann. Sitio oficial de PHPUnit. 2015. url: http://phpunit.de/ (visitado 15-10-2015).
[13] JUnit. JUnit 5. url: https://junit.org/junit5/ (visitado 09-10-2015).
[14] NUnit. NUnit.org. url: http://nunit.org/ (visitado 09-10-2015).
[15] Bestoun S. Ahmed y Kamal Z. Zamli. “Comparison of metahuristic test generation strategies based on interaction elements coverage criterion”. En: 2011 IEEE Symposium on Industrial Electronics and Applications. IEEE, sep. de 2011, págs. 550-554. isbn: 978-1-4577-1417-7. doi: 10 . 1109 / ISIEA.2011.6108773. url: http://ieeexplore.ieee.org/document/6108773/.
[16] Saswat Anand y col. “An orchestrated survey of methodologies for automated software test case generation”. En: Journal of Systems and Software 86.8 (ago. de 2013), págs. 1978-2001. issn: 01641212. doi: 10 . 1016 / j . jss . 2013 . 02 . 061. url: https : / / linkinghub . elsevier . com / retrieve/pii/S0164121213000563.
[17] Fabrice Bouquet y col. “A test generation solution to automate software testing”. En: Proceedings of the 3rd international workshop on Automation of software test - AST ’08. New York, New York, USA: ACM Press, 2008, pág. 45. isbn: 9781605580302. doi: 10.1145/1370042.1370052. url: http://portal.acm.org/citation.cfm?doid=1370042.1370052.
[18] José Carlos Bregieiro Ribeiro. “Search-based test case generation for object-oriented java softwa- re using strongly-typed genetic programming”. En: Proceedings of the 2008 GECCO conference companion on Genetic and evolutionary computation - GECCO ’08. New York, New York, USA: ACM Press, 2008, pág. 1819. isbn: 9781605581316. doi: 10.1145/1388969.1388979. url: http: //portal.acm.org/citation.cfm?doid=1388969.1388979.
[19] Gustavo Carvalho y col. “NAT2TESTSCR: Test case generation from natural language require- ments based on SCR specifications”. En: Science of Computer Programming 95 (dic. de 2014), págs. 275-297. issn: 01676423. doi: 10.1016/j.scico.2014.06.007. url: https://linkinghub. elsevier.com/retrieve/pii/S0167642314002858.
[20] Eugenia Díaz y col. “A tabu search algorithm for structural software testing”. En: Computers & Operations Research 35.10 (oct. de 2008), págs. 3052-3072. issn: 03050548. doi: 10.1016/j.cor. 2007.01.009. url: https://linkinghub.elsevier.com/retrieve/pii/S0305054807000214.
[21] Roger Ferguson y Bogdan Korel. “The chaining approach for software test data generation”. En: ACM Transactions on Software Engineering and Methodology 5.1 (ene. de 1996), págs. 63-86. issn: 1049331X. doi: 10.1145/226155.226158. url: http://portal.acm.org/citation.cfm?doid= 226155.226158.
[22] Mark Harman. “Automated Test Data Generation using Search Based Software Engineering”. En: Second International Workshop on Automation of Software Test (AST ’07). IEEE, mayo de 2007, págs. 2-2. isbn: 978-1-5090-8886-7. doi: 10.1109/AST.2007.4. url: https://ieeexplore.ieee. org/document/4296713/.
[23] Mark Harman, S. Afshin Mansouri y Yuanyuan Zhang. “Search-based software engineering: Trends, techniques and applications”. En: ACM Computing Surveys 45.1 (nov. de 2012), págs. 1-61. issn: 03600300. doi: 10.1145/2379776.2379787. url: http://dl.acm.org/citation.cfm?doid= 2379776.2379787.
[24] I. Hermadi, C. Lokan y R. Sarker. “Dynamic stopping criteria for search-based test data generation for path testing”. En: Information and Software Technology 56.4 (abr. de 2014), págs. 395-407. issn: 09505849. doi: 10.1016/j.infsof.2014.01.001. url: https://linkinghub.elsevier.com/ retrieve/pii/S0950584914000123.
[25] Muhammad Zohaib Iqbal, Andrea Arcuri y Lionel Briand. “Empirical investigation of search algo- rithms for environment model-based testing of real-time embedded software”. En: Proceedings of the 2012 International Symposium on Software Testing and Analysis - ISSTA 2012. New York, New York, USA: ACM Press, 2012, pág. 199. isbn: 9781450314541. doi: 10.1145/2338965.2336777. url: http://dl.acm.org/citation.cfm?doid=2338965.2336777.
[26] Laura Lanzarini, Juan Pablo y La Battaglia. “Dynamic Generation of Test Cases with Metaheu- ristics”. En: Optimization 10.2 (2010), págs. 91-96.
[27] C.C. Michael, G. McGraw y M.A. Schatz. “Generating software test data by evolution”. En: IEEE Transactions on Software Engineering 27.12 (2001), págs. 1085-1110. issn: 00985589. doi: 10 . 1109/32.988709. url: http://ieeexplore.ieee.org/document/988709/.
[28] Ankur Pachauri y Gursaran Srivastava. “Automated test data generation for branch testing using genetic algorithm: An improved approach using branch ordering, memory and elitism”. En: Jour- nal of Systems and Software 86.5 (mayo de 2013), págs. 1191-1208. issn: 01641212. doi: 10 . 1016 / j . jss . 2012 . 11 . 045. url: https : / / linkinghub . elsevier . com / retrieve / pii / S0164121212003263.
[29] Roy P. Pargas, Mary Jean Harrold y Robert R. Peck. “Test-data generation using genetic algo- rithms”. En: Software Testing, Verification and Reliability 9.4 (dic. de 1999), págs. 263-282. issn: 0960-0833. doi: 10.1002/(SICI)1099- 1689(199912)9:4<263::AID- STVR190> 3.0.CO;2- Y. url: http://doi.wiley.com/10.1002/%7B%5C%%7D28SICI%7B%5C%%7D291099- 1689%7B%5C% %7D28199912%7B%5C%%7D299%7B%5C%%7D3A4%7B%5C%%7D3C263%7B%5C%%7D3A%7B%5C%%7D3AAID- STVR190%7B%5C%%7D3E3.0.CO%7B%5C%%7D3B2-Y.
[30] R. S. Pressman. Software Engineering: A Practitioner’s Approach. 7th. New York, USA: McGraw- Hill, 2010. isbn: 978–0–07–337597–7.
[31] Praveen Ranjan Srivatsava, B. Mallikarjun y Xin-She Yang. “Optimal test sequence generation using firefly algorithm”. En: Swarm and Evolutionary Computation 8 (feb. de 2013), págs. 44-53. issn: 22106502. doi: 10.1016/j.swevo.2012.08.003. url: https://linkinghub.elsevier. com/retrieve/pii/S2210650212000612.
[32] Abdelilah Sakti, Yann-Gaël Guéhéneuc y Gilles Pesant. “Boosting Search Based Testing by Using Constraint Based Testing”. En: 2012, págs. 213-227. doi: 10.1007/978-3-642-33119-0_16. url: http://link.springer.com/10.1007/978-3-642-33119-0%7B%5C_%7D16.
[33] Sapna Varshney y Monica Mehrotra. “Search based software test data generation for structural tes- ting”. En: ACM SIGSOFT Software Engineering Notes 38.4 (jul. de 2013), pág. 1. issn: 01635948. doi: 10.1145/2492248.2492277. url: http://dl.acm.org/citation.cfm?doid=2492248. 2492277.
[34] Joachim Wegener, Andre Baresel y Harmen Sthamer. “Evolutionary test environment for automatic structural testing”. En: Information and Software Technology 43.14 (dic. de 2001), págs. 841-854. issn: 09505849. doi: 10.1016/S0950-5849(01)00190-2. url: http://linkinghub.elsevier. com/retrieve/pii/S0950584901001902.
[35] Vogella. Eclipse, Android and Java training and support. url: https : / / www . vogella . com/ (visitado 09-10-2015).
[36] Paul M. Duvall, Steve Matyas y Andrew Glover. Continuous Integration: Improving Software Qua- lity and reducing risks. Ed. por Inc. Pearson Education. Second Pri. Boston, 2007. isbn: 13: 978- 0-321-33638-5.
[37] Sonarqube. Continuous Inspection | SonarQube. url: https://www.sonarqube.org/ (visitado 12-12-2018).
[2] Ting Chen y col. “State of the art: Dynamic symbolic execution for automated test generation”. En: Future Generation Computer Systems 29.7 (sep. de 2013), págs. 1758-1773. issn: 0167739X. doi: 10.1016/j.future.2012.02.006. url: https://linkinghub.elsevier.com/retrieve/ pii/S0167739X12000398.
[3] Frank Elberzhager y col. “Reducing test effort: A systematic mapping study on existing ap- proaches”. En: Information and Software Technology 54.10 (oct. de 2012), págs. 1092-1106. issn: 09505849. doi: 10.1016/j.infsof.2012.04.007. url: https://linkinghub.elsevier.com/ retrieve/pii/S0950584912000894.
[4] A.M. Memon, M.E. Pollack y M.L. Soffa. “Hierarchical GUI test case generation using automa- ted planning”. En: IEEE Transactions on Software Engineering 27.2 (2001), págs. 144-155. issn: 00985589. doi: 10.1109/32.908959. url: http://ieeexplore.ieee.org/document/908959/.
[5] L. Morales. “Componente de generación automática de valores de pruebas”. Tesis doct. Instituto Superior Politécnico José Antonio Echeverría, 2012.
[6] A. Moreira. “Entorno de integración continua basado en software libre para el desarrollo de apli- caciones Java”. Tesis doct. Instituto Superior Politécnico “José Antonio Echeverría”, 2016.
[7] Beatriz Lamancha Pérez y Macario Polo. “Generación automática de casos de prueba para Líneas de Producto de Software Automatic test case generation for software product lines”. En: Revista Española de Innovación, Calidad e Ingeniería del Software 5.2 (2009). issn: 1885-4486. url: http: //www.redalyc.org/pdf/922/92217153004.pdf.
[8] Tang Rongfa. “Adaptive Software Test Management System Based on Software Agents”. En: 2012, págs. 1-9. doi: 10.1007/978-3-642-25437-6_1. url: http://link.springer.com/10.1007/ 978-3-642-25437-6%7B%5C_%7D1.
[9] Soma Sekhara Babu Lam y col. “Automated Generation of Independent Paths and Test Suite Optimization Using Artificial Bee Colony”. En: Procedia Engineering 30 (2012), págs. 191-200. issn: 18777058. doi: 10.1016/j.proeng.2012.01.851. url: https://linkinghub.elsevier. com/retrieve/pii/S1877705812008612.
[10] Ying XING y col. “Intelligent test case generation based on branch and bound”. En: The Journal of China Universities of Posts and Telecommunications 21.2 (abr. de 2014), págs. 91-103. issn: 10058885. doi: 10.1016/S1005-8885(14)60291-0. url: https://linkinghub.elsevier.com/ retrieve/pii/S1005888514602910.
[11] Zhiqiang Zhang y col. “Generating combinatorial test suite using combinatorial optimization”. En: Journal of Systems and Software 98 (dic. de 2014), págs. 191-207. issn: 01641212. doi: 10 . 1016 / j . jss . 2014 . 09 . 001. url: https : / / linkinghub . elsevier . com / retrieve / pii / S0164121214001939.
[12] S Bergmann. Sitio oficial de PHPUnit. 2015. url: http://phpunit.de/ (visitado 15-10-2015).
[13] JUnit. JUnit 5. url: https://junit.org/junit5/ (visitado 09-10-2015).
[14] NUnit. NUnit.org. url: http://nunit.org/ (visitado 09-10-2015).
[15] Bestoun S. Ahmed y Kamal Z. Zamli. “Comparison of metahuristic test generation strategies based on interaction elements coverage criterion”. En: 2011 IEEE Symposium on Industrial Electronics and Applications. IEEE, sep. de 2011, págs. 550-554. isbn: 978-1-4577-1417-7. doi: 10 . 1109 / ISIEA.2011.6108773. url: http://ieeexplore.ieee.org/document/6108773/.
[16] Saswat Anand y col. “An orchestrated survey of methodologies for automated software test case generation”. En: Journal of Systems and Software 86.8 (ago. de 2013), págs. 1978-2001. issn: 01641212. doi: 10 . 1016 / j . jss . 2013 . 02 . 061. url: https : / / linkinghub . elsevier . com / retrieve/pii/S0164121213000563.
[17] Fabrice Bouquet y col. “A test generation solution to automate software testing”. En: Proceedings of the 3rd international workshop on Automation of software test - AST ’08. New York, New York, USA: ACM Press, 2008, pág. 45. isbn: 9781605580302. doi: 10.1145/1370042.1370052. url: http://portal.acm.org/citation.cfm?doid=1370042.1370052.
[18] José Carlos Bregieiro Ribeiro. “Search-based test case generation for object-oriented java softwa- re using strongly-typed genetic programming”. En: Proceedings of the 2008 GECCO conference companion on Genetic and evolutionary computation - GECCO ’08. New York, New York, USA: ACM Press, 2008, pág. 1819. isbn: 9781605581316. doi: 10.1145/1388969.1388979. url: http: //portal.acm.org/citation.cfm?doid=1388969.1388979.
[19] Gustavo Carvalho y col. “NAT2TESTSCR: Test case generation from natural language require- ments based on SCR specifications”. En: Science of Computer Programming 95 (dic. de 2014), págs. 275-297. issn: 01676423. doi: 10.1016/j.scico.2014.06.007. url: https://linkinghub. elsevier.com/retrieve/pii/S0167642314002858.
[20] Eugenia Díaz y col. “A tabu search algorithm for structural software testing”. En: Computers & Operations Research 35.10 (oct. de 2008), págs. 3052-3072. issn: 03050548. doi: 10.1016/j.cor. 2007.01.009. url: https://linkinghub.elsevier.com/retrieve/pii/S0305054807000214.
[21] Roger Ferguson y Bogdan Korel. “The chaining approach for software test data generation”. En: ACM Transactions on Software Engineering and Methodology 5.1 (ene. de 1996), págs. 63-86. issn: 1049331X. doi: 10.1145/226155.226158. url: http://portal.acm.org/citation.cfm?doid= 226155.226158.
[22] Mark Harman. “Automated Test Data Generation using Search Based Software Engineering”. En: Second International Workshop on Automation of Software Test (AST ’07). IEEE, mayo de 2007, págs. 2-2. isbn: 978-1-5090-8886-7. doi: 10.1109/AST.2007.4. url: https://ieeexplore.ieee. org/document/4296713/.
[23] Mark Harman, S. Afshin Mansouri y Yuanyuan Zhang. “Search-based software engineering: Trends, techniques and applications”. En: ACM Computing Surveys 45.1 (nov. de 2012), págs. 1-61. issn: 03600300. doi: 10.1145/2379776.2379787. url: http://dl.acm.org/citation.cfm?doid= 2379776.2379787.
[24] I. Hermadi, C. Lokan y R. Sarker. “Dynamic stopping criteria for search-based test data generation for path testing”. En: Information and Software Technology 56.4 (abr. de 2014), págs. 395-407. issn: 09505849. doi: 10.1016/j.infsof.2014.01.001. url: https://linkinghub.elsevier.com/ retrieve/pii/S0950584914000123.
[25] Muhammad Zohaib Iqbal, Andrea Arcuri y Lionel Briand. “Empirical investigation of search algo- rithms for environment model-based testing of real-time embedded software”. En: Proceedings of the 2012 International Symposium on Software Testing and Analysis - ISSTA 2012. New York, New York, USA: ACM Press, 2012, pág. 199. isbn: 9781450314541. doi: 10.1145/2338965.2336777. url: http://dl.acm.org/citation.cfm?doid=2338965.2336777.
[26] Laura Lanzarini, Juan Pablo y La Battaglia. “Dynamic Generation of Test Cases with Metaheu- ristics”. En: Optimization 10.2 (2010), págs. 91-96.
[27] C.C. Michael, G. McGraw y M.A. Schatz. “Generating software test data by evolution”. En: IEEE Transactions on Software Engineering 27.12 (2001), págs. 1085-1110. issn: 00985589. doi: 10 . 1109/32.988709. url: http://ieeexplore.ieee.org/document/988709/.
[28] Ankur Pachauri y Gursaran Srivastava. “Automated test data generation for branch testing using genetic algorithm: An improved approach using branch ordering, memory and elitism”. En: Jour- nal of Systems and Software 86.5 (mayo de 2013), págs. 1191-1208. issn: 01641212. doi: 10 . 1016 / j . jss . 2012 . 11 . 045. url: https : / / linkinghub . elsevier . com / retrieve / pii / S0164121212003263.
[29] Roy P. Pargas, Mary Jean Harrold y Robert R. Peck. “Test-data generation using genetic algo- rithms”. En: Software Testing, Verification and Reliability 9.4 (dic. de 1999), págs. 263-282. issn: 0960-0833. doi: 10.1002/(SICI)1099- 1689(199912)9:4<263::AID- STVR190> 3.0.CO;2- Y. url: http://doi.wiley.com/10.1002/%7B%5C%%7D28SICI%7B%5C%%7D291099- 1689%7B%5C% %7D28199912%7B%5C%%7D299%7B%5C%%7D3A4%7B%5C%%7D3C263%7B%5C%%7D3A%7B%5C%%7D3AAID- STVR190%7B%5C%%7D3E3.0.CO%7B%5C%%7D3B2-Y.
[30] R. S. Pressman. Software Engineering: A Practitioner’s Approach. 7th. New York, USA: McGraw- Hill, 2010. isbn: 978–0–07–337597–7.
[31] Praveen Ranjan Srivatsava, B. Mallikarjun y Xin-She Yang. “Optimal test sequence generation using firefly algorithm”. En: Swarm and Evolutionary Computation 8 (feb. de 2013), págs. 44-53. issn: 22106502. doi: 10.1016/j.swevo.2012.08.003. url: https://linkinghub.elsevier. com/retrieve/pii/S2210650212000612.
[32] Abdelilah Sakti, Yann-Gaël Guéhéneuc y Gilles Pesant. “Boosting Search Based Testing by Using Constraint Based Testing”. En: 2012, págs. 213-227. doi: 10.1007/978-3-642-33119-0_16. url: http://link.springer.com/10.1007/978-3-642-33119-0%7B%5C_%7D16.
[33] Sapna Varshney y Monica Mehrotra. “Search based software test data generation for structural tes- ting”. En: ACM SIGSOFT Software Engineering Notes 38.4 (jul. de 2013), pág. 1. issn: 01635948. doi: 10.1145/2492248.2492277. url: http://dl.acm.org/citation.cfm?doid=2492248. 2492277.
[34] Joachim Wegener, Andre Baresel y Harmen Sthamer. “Evolutionary test environment for automatic structural testing”. En: Information and Software Technology 43.14 (dic. de 2001), págs. 841-854. issn: 09505849. doi: 10.1016/S0950-5849(01)00190-2. url: http://linkinghub.elsevier. com/retrieve/pii/S0950584901001902.
[35] Vogella. Eclipse, Android and Java training and support. url: https : / / www . vogella . com/ (visitado 09-10-2015).
[36] Paul M. Duvall, Steve Matyas y Andrew Glover. Continuous Integration: Improving Software Qua- lity and reducing risks. Ed. por Inc. Pearson Education. Second Pri. Boston, 2007. isbn: 13: 978- 0-321-33638-5.
[37] Sonarqube. Continuous Inspection | SonarQube. url: https://www.sonarqube.org/ (visitado 12-12-2018).
Published
2019-01-30
How to Cite
[1]
Larrosa Uribazo, D., Verona Marcos, S., Fernández Oliva, P. and Delgado Dapena, M.D. 2019. Automatic execution of tests in enterprise production environments for software. Informática y Sistemas. 3, 1 (Jan. 2019), 32–44. DOI:https://doi.org/10.33936/isrtic.v3i1.1585.
Issue
Section
Regular Papers
License
Articles submitted to this journal for publication will be released for open access under a Creative Commons Attribution Non-Commercial No Derivative Works licence (http://creativecommons.org/licenses/by-nc-nd/4.0).
The authors retain copyright, and are therefore free to share, copy, distribute, perform and publicly communicate the work under the following conditions: Acknowledge credit for the work specified by the author and indicate if changes were made (you may do so in any reasonable way, but not in a way that suggests that the author endorses your use of his or her work. Do not use the work for commercial purposes. In case of remixing, transformation or development, the modified material may not be distributed.
