Seminar Software Engineering

Developping software components using machine learning requires a workflow of tasks like data cleaning, features engineering, model selection, etc. that needs to be regularly performed, monitored and tracked. The current popularity of machine learning methods have lead several companies to develop new workflow orchestration and model management tools such as DVC, mlflow, kubeflow, etc. therefore surveys are currently missing. The goal of this seminar is gain an overview of the different tools and methods used to manage data science workflows.

Bachelor - literature overview of workflow orchestration and model management systems

Master - literature overview + MLOps

Literature :

1. Shearer, Colin (2000): The CRISP-DM Model: The New Blueprint for Data Mining. In Journal of Data Warehousing 5 (4), pp. 14–22.
2. Continuous Delivery for Machine Learning (2020). Available online at martinfowler.com/articles/cd4ml.html, updated on 5/3/2020, checked on 6/3/2020.
3. Data Science Workflow Tools - Chris Kenwright - Medium (2020), updated on 6/3/2020, checked on 6/3/2020.
4. Data Version Control · DVC (2020). Available online at dvc.org, updated on 6/3/2020, checked on 6/3/2020.
5. Christopher Bergh; Gil Benghiat; Eran Strod: The DataOps Cookbook. Methodologies and Tools that Reduce Analytics Cycle Time While Improving Quality, checked on 5/3/2019.
6. Jpe316 (2020): MLOps: ML-Modellverwaltung - Azure Machine Learning | Microsoft Docs. Available online at docs.microsoft.com/de-de/azure/machine-learning/concept-model-management-and-deployment, updated on 6/3/2020, checked on 6/3/2020.

Incorporating machine learning components in a software system inherently brings new risks and challenges for engineers developing and operating such systems. In this seminar, the goal is to identify and classify the different challenges brought by machine learning software engineering.

Bachelor - only literature review of current challenges

Master - literature review of current challenges + potential solutions

Literature :

1. Shearer, Colin (2000): The CRISP-DM Model: The New Blueprint for Data Mining. In Journal of Data Warehousing 5 (4), pp. 14–22.
2. Manifesto for Agile Software Development (2001). Available online at agilemanifesto.org, updated on 3/31/2019, checked on 7/19/2019.
3. Rahman, Md Saidur; Rivera, Emilio; Khomh, Foutse; Guéhéneuc, Yann-Gaël; Lehnert, Bernd (2019): Machine Learning Software Engineering in Practice: An Industrial Case Study. Available online at arxiv.org/pdf/1906.07154v1
4. Saltz, J.; Heckman, R.; Shamshurin, I. (Eds.) (2017): Exploring how different project management methodologies impact data science students. Proceedings of the 25th European Conference on Information Systems, ECIS 2017. Available online at https://www.scopus.com/inward/record.uri?eid=2-s2.0-85058023852&partnerID=40&md5=95248678a298270f33b1e15a21cfe095.
5. Wan, Zhiyuan; Xia, Xin; Lo, David; Murphy, Gail C. (2019): How does Machine Learning Change Software Development Practices? In IIEEE Trans. Software Eng., p. 1. DOI: 10.1109/TSE.2019.2937083.
6. Belani, Hrvoje; Vukovic, Marin; Car, Zeljka (2019 - 2019): Requirements Engineering Challenges in Building AI-Based Complex Systems. In : 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW). 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW). Jeju Island, Korea (South), 9/23/2019 - 9/27/2019: IEEE, pp. 252–255.
7. Zhang, Jie M.; Harman, Mark; Ma, Lei; Liu, Yang (2020): Machine Learning Testing: Survey, Landscapes and Horizons. In IIEEE Trans. Software Eng., p. 1. DOI: 10.1109/TSE.2019.2962027.

The process of building software components using machine learning methods can be seen as a workflow of more or less tedious tasks (a directed acyclic graph, or DAG) which can be (partially) automated. In the last years several automated tools have emerged (as known as AutoML). In this seminar, the goal is to gain an overview of the features proposed by AutoML tools and discuss their use by data scientist.

Bachelor - literature recherche: features of AutoML

Master - literature recherche + usage of AutoML by data scientists

Literature :

1. Shearer, Colin (2000): The CRISP-DM Model: The New Blueprint for Data Mining. In Journal of Data Warehousing 5 (4), pp. 14–22.
2. Weidele, Daniel Karl I.; Weisz, Justin D.; Oduor, Erick; Muller, Michael; Andres, Josh; Gray, Alexander; Wang, Dakuo (2020): AutoAIViz: Opening the Blackbox of Automated Artificial Intelligence with Conditional Parallel Coordinates, pp. 308–312. DOI: 10.1145/3377325.3377538.
3. Doris Jung Lin Lee; Stephen Macke; Doris Xin; Angela Lee; Silu Huang; Aditya G. Parameswaran (2019): A Human-in-the-loop Perspective on AutoML: Milestones and the Road Ahead. In IEEE Data Eng. Bull. 42, pp. 59–70.
4. He, Xin; Zhao, Kaiyong; Chu, Xiaowen (2019): AutoML: A Survey of the State-of-the-Art. Available online at arxiv.org/pdf/1908.00709v4.
5. Tuggener, Lukas; Amirian, Mohammadreza; Rombach, Katharina; Lorwald, Stefan; Varlet, Anastasia; Westermann, Christian; Stadelmann, Thilo (2019): Automated Machine Learning in Practice: State of the Art and Recent Results, pp. 31–36. DOI: 10.1109/SDS.2019.00-11.

A self-adaptive system (SAS) is capable of adjusting its own behavior in response to any change in its operating environment. It is able to automatically adapt at runtime by adjusting its own parameters or components of the system in order to cope with the dynamic surrounding environment. However, adaptation actions should be identified as per the particular change in the environment, i.e. “What kind of change is required?”. The aim of this literature study is to address the scope of dynamic integration of the new parameter or new component at runtime in SAS. The following set of questions is essential to fulfilling this literature study.

•  Which adaptation techniques are better?
•  How much reliable and safe it is?
•  Is it feasible to implement combination of techniques in SAS?
•  What are the impacts and cost factors?
•  How to increase the performance of the SAS with adaptation techniques?

Literature :

1. Salehie, M., & Tahvildari, L. (2009). Self-adaptive software: Landscape and research challenges. ACM transactions on autonomous and adaptive systems (TAAS), 4(2), 14.
2. Krupitzer, C., Breitbach, M., Roth, F. M., VanSyckel, S., Schiele, G., & Becker, C. (2018). A survey on engineering approaches for self-adaptive systems (extended version).
3. Handte, M., Schiele, G., Matjuntke, V., Becker, C., & Marrón, P. J. (2012). 3PC: System support for adaptive peer-to-peer pervasive computing. ACM Transactions on Autonomous and Adaptive Systems (TAAS), 7(1), 1-19.
4. McKinley, P. K., Sadjadi, S. M., Kasten, E. P., & Cheng, B. H. (2004). Composing adaptive software. Computer, 37(7), 56-64.
5. Epifani, I., Ghezzi, C., Mirandola, R., & Tamburrelli, G. (2009, May). Model evolution by run-time parameter adaptation. In 2009 IEEE 31st International Conference on Software Engineering (pp. 111-121). IEEE.
6. Weyns, D., Schmerl, B., Grassi, V., Malek, S., Mirandola, R., Prehofer, C., & Göschka, K. M. (2013). On patterns for decentralized control in self-adaptive systems. In Software Engineering for Self-Adaptive Systems II (pp. 76-107). Springer, Berlin, Heidelberg.

The goals of the literature research are to review the fundamentals of diversity and multi-version systems (MVS) to analyze potential use cases related to their application in autonomous systems. The literature research must be included the main issues of safety, security, and survivability in the event of accidental or deliberate faults within the autonomous systems. The following set of questions is essential to fulfilling this literature study.

• How diversity can decrease the common-cause fault (CCF)?
• Which techniques are required to assess diversity?
• What needs to be done to improve safety and security in MVS?
• Limitation of diversity application in different industrial cases
• How to choose variants in MVS?

Literature :

1. Littlewood, B. (1996). The impact of diversity upon common mode failures. Reliability Engineering & System Safety, 51(1), 101-113.
2. Vilkomir, S., & Kharchenko, V. (2012, June). A diversity model for multi-version safety-critical i&C systems. In Proceedings of the 11th International Probabilistic Safety Assessment and Management Conference and the Annual European Safety and Reliability Conference, PSAM-ESREL 2012, Helsinki, Finland.
3. Bakhmach, E. S., Siora, A. A., Tokarev, V. I., Kharchenko, V. S., Sklyar, V. V., & Andrashov, A. A. Safety Critical FPGA-based NPP I&C Systems: Assessment, Development and Implementation. In 17th Pacific Basin Nuclear Conference Cancún, Q.R., México, October 24-30, 2010
4. Avizienis, A. (2000). Design diversity and the immune system paradigm: Cornerstones for information system survivability. In Information Survivability Workshop.
5. Kharchenko, V. (2016, October). Diversity for safety and security of embedded and cyber physical systems: Fundamentals review and industrial cases. In 2016 15th Biennial Baltic Electronics Conference (BEC) (pp. 17-26). IEEE.
6. Deswarte, Y., Kanoun, K., & Laprie, J. C. (1998, July). Diversity against accidental and deliberate faults. In Proceedings Computer Security, Dependability, and Assurance: From Needs to Solutions (Cat. No. 98EX358) (pp. 171-181). IEEE.

Advancement of technology and their impact on the real-world can be assessed by how safe it is. It is a primary role of a safety engineer to assess the risk via a process relevant data and learning from past lessons. Due to advanced and autonomous technology, a study for continuous and dynamic risk assessment is required. Aim of this literature study is to address the dynamics of risk and AI-based assessment techniques. The following set of questions is essential to fulfilling this literature study.

•   What are the dynamics of risk?
•   Which quantitative/qualitative risk assessment techniques are available?
•   How AI-based risk analysis is feasible?
•   What is the impact of human factors on AI-based risk assessment?
•   Which AI-techniques can be useful for risk assessment?

Literature :

1. Hegde, J., & Rokseth, B. (2020). Applications of machine learning methods for engineering risk assessment–A review. Safety science, 122, 104492.
2. Paltrinieri, N., Comfort, L., & Reniers, G. (2019). Learning about risk: Machine learning for risk assessment. Safety science, 118, 475-486.
3. Apostolakis, G. E. (2004). How useful is quantitative risk assessment? Risk Analysis: An International Journal, 24(3), 515-520.
4. Aven, T., & Krohn, B. S. (2014). A new perspective on how to understand, assess and manage risk and the unforeseen. Reliability Engineering & System Safety, 121, 1-10.
5. Villa, V., Paltrinieri, N., Khan, F., & Cozzani, V. (2016). Towards dynamic risk analysis: A review of the risk assessment approach and its limitations in the chemical process industry. Safety science, 89, 77-93.
6. Adedigba, S. A., Khan, F., & Yang, M. (2017). Dynamic failure analysis of process systems using neural networks. Process Safety and Environmental Protection, 111, 529-543.
7. Rausand, M. (2013). Risk assessment: theory, methods, and applications (Vol. 115). John Wiley & Sons.

Twitter user Paul Franks tweeted at Musk on Friday night with a request for Tesla: “Can you guys program the car once in park to move back the seat and raise the steering wheel? Steering wheel is wearing.” Musk responded: “Good point. We will add that to all cars in one of the upcoming software releases.” And they did! Your job in this seminar topic is to explain how this was possible with the help of the continuous practices, i.e., continuous integration, delivery, and deployment.

Literature :

1. Vöst, S., & Wagner, S. (2016). Towards continuous integration and continuous delivery in the automotive industry. arXiv preprint arXiv:1612.04139. arxiv.org/abs/1612.04139
2. Estivill-Castro, V., Hexel, R., & Stover, J. (2015, October). Modeling, validation, and continuous integration of software behaviours for embedded systems. In 2015 IEEE European Modelling Symposium (EMS) (pp. 89-95). IEEE. ieeexplore.ieee.org/abstract/document/7579811
3. Shahin, M., Babar, M. A., & Zhu, L. (2017). Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE Access, 5, 3909-3943. ieeexplore.ieee.org/abstract/document/7884954

Integrating Agile Software Development (ASD) with Software Product Line Engineering (PLE) has resulted in proposing Agile Product Line Engineering (APLE). The goal of combining both approaches is to overcome the weaknesses of each other while maximizing their benefits. However, combining them represents a big challenge in software engineering. Your job in this topic is to investigate the up-to-date methodologies and approaches for APLE and to assess their feasibility.

Literature :

1. Haidar, H., Kolp, M., & Wautelet, Y. (2017). Agile Product Line Engineering: The AgiFPL Method. In ICSOFT (pp. 275-285).  https://www.semanticscholar.org/paper/Agile-Product-Line-Engineering%3A-The-AgiFPL-Method-Haidar-Kolp/682f730b7cb9a0e2f99a58576b3236f8b69fb3dd
2. Díaz, J., Pérez, J., Alarcón, P. P., & Garbajosa, J. (2011). Agile product line engineering—a systematic literature review. Software: Practice and experience, 41(8), 921-941.  https://onlinelibrary.wiley.com/doi/full/10.1002/spe.1087

Achieving a safe autonomous vehicle is not something that can be solved with a single technological silver bullet. Rather, it as a coupled set of problems that must be solved in a coordinated, cross-domain manner. Your job in this seminar is to investigate the challenges that face safety arising from different interrelated fields.

Literature :

1. Koopman, P., & Wagner, M. (2017). Autonomous vehicle safety: An interdisciplinary challenge. IEEE Intelligent Transportation Systems Magazine, 9(1), 90-96. (https://ieeexplore.ieee.org/document/7823109)

How does the agile Method Scrum look like in a remote Setting? This question should be answered based on the literature suggestions and additional literature identified in a small literature search. The seminar should summarize existing experiences and highlight challenges caused by the remote setting, and solutions (adaptation to the original Scrum) to cope with those challenges. In the focus should be literature dealing with a full remote setting, meaning that all team members are working remotely (e.g., not only a sub-team working remotely from another country.). Results should provide an overview to agile teams how to adapt their Scrum approaches to deal with remote collaboration, which could be a nice guiding document in the current situation. Experience with Agile methods (e.g., in lectures process management) would be ideal

Literature :

1. Scrum abandonment in distributed teams: A revelatory case.
2. Challenges on adopting scrum for distributed teams in home office environments
3. Using Scrum in Distributed Agile Development: A Multiple Case Study

How does effective and efficient communication of an agile team looks like in a remote setting? This question should be answered based on the literature suggestions and additional literature identified in a small literature search. The seminar should summarize existing experiences and highlight challenges caused by the remote setting, and recommendations for adequate communication. Experience with Agile methods (e.g., in lecture process management) would be ideal.

Literature :

1. Waste identification as the means for improving communication in globally distributed agile software development
2. Effective Communication in Distributed Agile Software Development Teams
3. Communication and awareness patterns of distributed agile teams

Wie werden agile Methoden bzw. Agilität in der Verwaltung eingesetzt? Analyse vorhandener Erfahrungsberichte und Zusammenfassung von Herausforderungen, Erfolgsfaktoren und zukünftigen Forschungsthemen. Die Quellen sind eher in deutsch, das Seminar kann aber auch in Englisch verfasst werden. Vorkenntnisse in agilen Methoden (z.B. aus der Vorlesung Process Management) von Vorteil.

Literature :

1. Agilisierung einer kommunalen Verwaltung – das Beispiel Ängelholm (Schweden)
2. Chancen und Risiken von agilen Methoden in der Verwaltung
3. Agile Projekte in öffentlichen Verwaltungen – Eine Bestandsaufnahme

The analysis of Deep neural networks (DNN) is crucial to ensure the quality of the trained model. In order to find different and test different input classes, multiple methods exists. The student should investigate, how the methods described in the reference list work and how they are interrelated. Beyond the given papers, further work should also be examined.

Literature :

1. Investigating Layer Activation Patterns in Neural Networks for Classification Error Detection. Victor DEMUYSERE. 2018.  dial.uclouvain.be/memoire/ucl/en/object/thesis%3A17232/datastream/PDF_01/view
2. Runtime Monitoring Neuron Activation Patterns. Chih-Hong Cheng, Georg Nührenberg, Hirotoshi Yasuoka. 2018.  arxiv.org/abs/1809.06573
3. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, Rory Sayres. 2018.  arxiv.org/abs/1711.11279
4. Testing Deep Neural Networks. 2019. Youcheng Sun, Xiaowei Huang, Daniel Kroening, James Sharp, Matthew Hill, Rob Ashmore. arxiv.org/abs/1803.04792
5. Samarasinghe S. (2016) Order in the Black Box: Consistency and Robustness of Hidden Neuron Activation of Feed Forward Neural Networks and Its Use in Efficient Optimization of Network Structure. In: Shanmuganathan S., Samarasinghe S. (eds) Artificial Neural Network Modelling. Studies in Computational Intelligence, vol 628. Springer, Cham
6. Not Just a Black Box: Learning Important Features Through Propagating Activation Differences. Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, Anshul Kundaje. 2017.  arxiv.org/abs/1605.01713
7. M. Kahng, P. Y. Andrews, A. Kalro and D. H. Chau, "ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models," in IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 1, pp. 88-97, Jan. 2018. doi: 10.1109/TVCG.2017.2744718 URL:  ieeexplore.ieee.org/stamp/stamp.jsp
8. Youcheng Sun, Min Wu, Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska, and Daniel Kroening. 2018. Concolic testing for deep neural networks. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018). ACM, New York, NY, USA, 109-119. DOI:  doi.org/10.1145/3238147.3238172

The release of autonomous vehicles requires a sufficient degree of testing. The performed testing has to fulfill certain characteristic to reach the wanted level of confidence. Current research is aiming to determine of required testing. As argumentation framework safety cases of autonomous vehicles should be used. The safety case will determine which V&V activities during development time as well as runtime mechanism are required to assure a sufficient level of confidence, even in the presence of unknown events and uncertainties. Among these activities this work should identify and compare which implications the upcoming safety standards for autonomous vehicle impose on testing. (Work split for students: one for safety for AV in general and one with testing requirement focus) Out of scope: explaining how simulations can be setup.

Literature :

1. Safety First for autonomous driving, www.daimler.com/innovation/case/autonomous/safety-first-for-automated-driving.html
2. Winner H., Lemmer K., Form T., Mazzega J. (2019) PEGASUS—First Steps for the Safe Introduction of Automated Driving. In: Meyer G., Beiker S. (eds) Road Vehicle Automation 5. Lecture Notes in Mobility. Springer, Cham, link.springer.com/chapter/10.1007/978-3-319-94896-6_16
3. Koopman P., Ferrell U., Fratrik F., Wagner M. (2019) A Safety Standard Approach for Fully Autonomous Vehicles. In: Romanovsky A., Troubitsyna E., Gashi I., Schoitsch E., Bitsch F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2019. Lecture Notes in Computer Science, vol 11699. Springer, Cham
4. UL4600: edge-case-research.com/ul4600/
5. Uber safety case: www.uber.com/us/en/atg/safety/

Autonomous vehicle will require simulations to be tested. The student should identify which challenges exist and how solution approaches tackle them. The challenges include test specification, execution as well as the definition of how to integrated testing knowledge from component level testing to the vehicle level. The work should also explain how machine learning can be used to interpret simulations. Out of scope: explaining which level of testing is necessary for safety cases.

Literature :

1. C. E. Tuncali, G. Fainekos, H. Ito and J. Kapinski, "Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components," 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, 2018, pp. 1555-1562. ieeexplore.ieee.org/abstract/document/8500421
2. Studying the Safety Impact of Autonomous Vehicles Using Simulation-Based Surrogate Safety Measures www.hindawi.com/journals/jat/2018/6135183/
3. S. Zhang, H. Peng, D. Zhao and H. E. Tseng, "Accelerated Evaluation of Autonomous Vehicles in the Lane Change Scenario Based on Subset Simulation Technique," 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, 2018, pp. 3935-394 ieeexplore.ieee.org/abstract/document/8569800
4. P. M. Chu, M. Wen, J. Park, H. Kaisi and K. Cho, "Three-Dimensional Simulation for Training Autonomous Vehicles in Smart City Environments," 2019 International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), Atlanta, GA, USA, 2019, pp. 848-853. ieeexplore.ieee.org/abstract/document/8875271