DOI10.5281/zenodo.1288390Zenodo1288390MaRDI QIDQ6699222FDOQ6699222
Dataset published at Zenodo repository.
Tomás Miranda, Miguel Goulão, Moharram Challenger, Vasco Amaral, Baris Tekin Tezel, Geylani Kardas, Omer Faruk Alaca
Publication date: 13 June 2018
Copyright license: Creative Commons Attribution 4.0 International
Improving the Usability of a MAS DSML Toms Miranda, Moharram Challenger, Baris Tezel, Omer Faruk Alaca, Vasco Amaral, Miguel Goulo, and Geylani Kardas Paper accepted for publication in the 6th International Workshop on Engineering Multi-Agent Systems (EMAS 2018) Abstract Context:A significant effort has been devoted to the design and implementation of various domain-specific modeling languages (DSMLs) for the software agents domain. Problem:Language usability is often tackled in an ad-hoc way, with the collection of anecdotal evidence supporting the process. However, usability plays an important role in the productivity, learnability and, ultimately, in the adoption of a MAS DSML by agent developers. Method:In this paper, we apply the principles of The Physics of Notations (PoN) to improve the visual notation of a MAS DSML, called SEA_ML and evaluate the result in terms of usability. Results:The evolved version of the language, SEA_ML++, was perceived as significantly improved in terms of icons comprehensibility, adequacy and usability, as a direct result of employing the principles of PoN. However, users were not significantly more efficient and effective with SEA_ML++, suggesting these two properties were not chiefly constrained by the identified shortcomings of the SEA_ML concrete syntax. Study Participants Summary.This evaluation is comprised of a set of studies, each with its own subset of participants. No participants were involved in more than one study. Symbol selection study (25 participants) Concrete syntax evaluation experiment (24 + 12 participants) Participants in the symbol selection study.The symbol selection experiment had 25 participants, 20 males, 5 females, all trained in Computer Science. Of these, 5 had completed their MSc, 16 their BSc and 4 were undergraduate Computer Science students at UNL. 11 had learned about MAS in the context of a course (although not using any of the particular languages in this experiment). The remaining 14 were not familiar with MAS. 9 participants had learned about the semantic web in the context of a course, 9 had only informal knowledge about it and the remaining 7 were not familiar with the semantic web. More details concerning the development of the SEA_ML++ version of SEA_ML can be found in: SEA_MLPoN.pdf . Participants in the concrete syntax evaluation experiment.This evaluation experiment was run as a pair of replicated studies. The first replica was run in Universidade Nova de Lisboa, in Portugal. The second replica was run in the International Computer Institute, Ege University, Izmir, in Turkey. UNL evaluation.At UNL we had 24 participants, including 19 males and 5 females, all trained in Computer Science. Of these, 8 have completed their MSc, 15 their BSc and 1 is an undergraduate Computer Science student. 12 of them had learned about MAS in the context of a course (although not using any of the particular languages in this experiment), and 1 had only informal knowledge about MAS. The remaining 11 were not familiar with MAS. 14 participants had learned about semantic web in the context of a course, 5 had informal knowledge about it, while 4 were not familiar with the semantic web. EGE evaluation.At Ege University, we had 12 participants, all Computer Science graduate students who had learned about MAS in the context of a course. Case studies Music Trader.In this case study, participants are requested to develop a system that allows agents to trade their music albums without using any currency. Agents want to trade their music albums for other albums, with this trade being made on an N to N basis (Agent A wants to trade the album A1 for the album B1 from Agent B and vice versa). Agents are not able to trade more than one album for only one album.) Expert Finding.In this case study, participants are requested to develop a system that allows agents to find information about other agents that they are searching for in order to communicate with them. Agents have some information about the other agent they are looking for (they are family related or were friends at the past), which is crucial in order to find the correct SemanticWeb Service to search the right person. The communication between agents can be made through Social Networks, E-Mail, VoIP or Phone Call. This case study is an adaptation of the case study Expert Findingfor an evaluation of SEA_ML. Exercises Each case study presents the participants with 2 different exercises (4 exercises in total). Each exercise has a description that defines all variables of the system to be modeled. For each exercise, an incomplete version of this system is presented. Participants should read the description that is provided to them and compare to the model they have in hands. When the participant thinks the model is according to the description, the exercise is complete, passing to an inquiry about the system they have modeled and afterward to the next exercise. Each exercise should take around 10 minutes to be completed. The total experiment should take around 40 minutes. A short description of each exercise is as follows: 1. Music Trader: Exercise 1 In this exercise, the participants will be modeling the M.A.S viewpoint. An environment and one customer are missing from the original model. A model ispresented to the participant for this exercise (either in SEA_ML or in SEA_ML++). 2. Music Trader: Exercise 2 In this exercise, the participants will be modeling the Agent-SWS viewpoint. A SS_RegisterPlan is missing from the original model. A model ispresented to the participant for this exercise (either in SEA_ML or in SEA_ML++). 3. Expert Finding: Exercise 1 In this exercise, the participants will be modeling the Agent Internal viewpoint. A goal, a belief and a behavior (and its respective connections) are missing from the original model. A model ispresented to the participant for this exercise (either in SEA_ML or in SEA_ML++). 4. Expert Finding: Exercise 2In this exercise, the participants will be modeling the Ontology viewpoint. A fact and a semantic web organization are missing from the original model. A model ispresented to the participant for this exercise (either in SEA_ML or in SEA_ML++). How does the usage of SEA_ML vs SEA_ML++ affect the effectiveness and efficiency of developers? Summary.We briefly present the data analysis for comparing the effectiveness and efficiency of developers with the case studies, using SEA_ML and SEA_ML++. The data collection was performed in two replicate experiments: one conducted in Universidade Nova de Lisboa, the other in EGE University. The tasks and experimental design were identical and planned so that we are able to aggregate the results. In the end, we outline how the data collected in each of these replications compares with each other. Descriptive statistics.We start by presenting the descriptive statistics for the correctness and duration of the task. The correction is measured as the percentage of elements correctly identified by participants in our case studies. The duration measures how long it took them to perform thetask (see CorrectnessEfficiencyDescriptives.pdf). Normality tests. We conducted a Kolmogorov-Smirnov and a Shapiro-Wilk tests to check for normality. Neither the correctness nor the duration have a normal distribution, p 0.001 (see CorrectnessEfficiencyNormality.pdf). Hypothesis testing.Our null hypotheses are that: Using SEA_ML and SEA_ML++ has no impact in the correctness of the answers provided by our participants. Using SEA_ML and SEA_ML++ has no impact in the duration of the tasks performed by our participants. We summarize the main descriptive statistics here. We note a slight increase from .8021 to .8368 in the achieved correctness and a slight decrease in the mean time to complete the tasks, from 14m48s to 13m20s (see CorrectnessEfficiencySummary.pdf). We can visually compare the distributions of correctness and duration, depending on the particular version of the language. Concerning correctness, as we can observe, the mode is 1.00 for both languages, although the distribution seems to be more skewed towards the highest rank. The values below 1.00 are considered extreme values, for SEA_ML++, meaning they are relatively scarce. It was relatively infrequent not to obtain maximum correctness with SEA_ML++ (see CorrectnessBoxplot.pdf). Concerning duration, the distributions are even more similar (see EfficiencyBoxplot.pdf). In order to assess whether there is any statistically significant difference between correctness using SEA_ML vs SEA_ML++ and between duration, again using SEA_ML vs SEA_ML++, we conducted a Welch T test, which is robust to deviations from normality, unequal sample sizes and does not assume equal variances of the compared variables (see WelchTTestCorrectnessEfficiency.pdf). The correctness does not differ significantly, according to Welchst-test,t(141.968) = .417,p=.519 from the SEA_ML (M = .80, SD=.32)to the SEA_ML++ (M=.84,SD= .32) concrete syntax. These results suggest that there was no difference between the two concrete syntaxes, in terms ofcomplexity. The duration does not differsignificantly,according to Welchst-test, t(122.030) = 1.180,p=.280 from the SEA_ML (M = 14 : 48,SD = 09 : 32) to SEA_ML++ (M = 13 : 20,SD = 06 : 12) concrete syntax.These results suggest that there was no difference between the two concretesyntaxes, in terms of duration. Conclusions The slight differences are not statistically significant. We found no evidence supporting the hypothesis that using SEA_ML++ instead of SEA_ML leads to more correct models. Likewise, we also did not find evidence supporting the hypothesis that using SEA_ML++ instead of SEA_MLwill make practitioners reduce the duration of this task. Acknowledgments The authors would like to thank the followings: the Scientific and Technological Research Council of Turkey (TUBITAK) under grant 115E591 Portuguese grants NOVA LINCS Research Laboratory (Grant: FCT/MCTESPEst UID/ CEC/04516/2013) and DSML4MA Project (Grant: FCT/MCTESTUBITAK/0008/2014)
This page was built for dataset: Improving the Usability of a MAS DSML
