Analysis of The User Experience of a Mobile Application System: A Case Study of Xyz

This research intention is to comparing user experience value of mobile application using User experience(UX) result from new and old system, then doing gap analysis to determine how much improvement of user experience in the new system compared to old system. By using User Experience Questionnaire (UEQ), this research is to evaluate which aspect of mobile application is looked for by its user. Survey is conducted to 60 people to answer what experience they have on old application, new application and user expectation for application and compare them. The result shows that there ’ s much improvement from old application to new application, and for expectation user towards new and old application, the gap between new app to expectation become closer if compare with old app to expectation.


Introduction
Retail company usually have partnership with retail store to distribute their goods. The company will provide sales manpower to their partner store while the store will highlight the company's product on that store. The sales will also promote the company's product to the store customer. In order to measure the performance of sales manpower in the partner store, company will require their sales to track their absences, sales, and problems. However, as the number of store partner and sales manpower grow, it become harder to track sales performance. So, to address this problem, company uses information system with mobile app as the user interface to track their sales manpower performance.
User experience plays a big role in every website and every mobile application. It is important because it encompasses all aspect of the user's interaction with the company, the service and the products. The requirement to having a good user experience is firstly it needs to meet the customer's needs; it has to do its jobs without trouble.
These days it not only about the product development and the features of the product that is important because the product must be enjoyable and support the fundamental of the user needs. Therefore, user experience is one of the key parts in product development.

Research Problem
There are big number of small to medium enterprise that relies on custom made mobile appbased information system. The information system is critical for reporting and tracking sales created by the employee. However, the system interface itself has not undergone quality assurance test, resulting in many flaws and general system unreliability. So, in order to increase system reliability and improving the user experience, the system will be replaced with the new one. The reason for system replacement are poor current mobile application user interface, lack of advanced reporting features, lacking proper dashboard features, and unresponsive backend system. However, principal needs a quantitative method to measure the user experience of old system and comparing them to the replaced system. Principal also need feedback from stakeholder about the weakness of the old system so the new system can address those weaknesses.

Research objectives
There are multiple objectives that this research wants to achieve. First is to obtain the quantitative user experience (UX) score of old tracking system and new tracking system using user experience analysis framework. Then comparing user experience value from old and new system to measure how much improvement on the new system after the feedback is implemented. After that, determine the impact of user experience score changes on user satisfaction using new system. Finally, determine how much user expectation for new applications and compare it to user experience score on new application.

Significance of study
This research can be used as a guide to improve the quality of mobile application system using standard set by the framework, thus increasing productivity of the user of this system.  (Lukita, 2016) The purpose of this study is to define the attributes of the Nappa Milano online store that needs most improvement based on the result of the user experience evaluation. Based on the User Experience Questionnaire (UEQ) framework, there are several attributes on a website such as attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty. The nature of the research is same, but the subject is rather different, this research objective is to define the attributes which needs to be improved based on the result of the user experience evaluation. This related work would be used for user experience analysis framework (Ros, 2016) The exponential growth of mobile usage in developing countries, the tendencies between the usage of Mobile App and Responsive Website, the characteristic as well as advantages and disadvantages between the two, urge this study to be conducted. This study deems to compare the effectiveness, the efficiency and satisfaction (usability) between different mobile solutions: The so-called Apps and the Responsive Website. The purpose of this study almost same, because the difference is only this study compare between mobile app and website. (Pratama, 2016) Internet economy is growing in Indonesia, and so is the smartphone usage. The popularity of online marketplace is getting bigger by the day yet the market is not utilized as best as it could. A lot of new smartphone user may not be familiar with the concept of mobile application, yet they browse for products to purchase using their mobile web browser. Therefore, a mobile website User Experience (UX) evaluation on online marketplace in Indonesia is needed by using UEQ framework. The nature of the research is same but have different respondents, this study can use everyone as respondent but author only have fixed respondent. This related work can be use as how to compare data using UEQ framework.

User Experience Questionnaire
The concept of user experience combines well-known aspects like efficiency and effectiveness which often referred as pragmatic quality aspects, and additional criteria like creativity, joyof-use which often referred as hedonic quality aspect. (Hassenzahl, 2001). In the Internet context, the user experience is of the main factors contributing to the distinction between two different Internet surfing behaviors which is goal-directed and exploratory. This research is conducted using the UEQ framework. According to (Hassenzahl, 2001), the most important element of a user experience is the attractiveness, and it is also important to consider both the pragmatic quality and the hedonic quality aspect if we want to measure how satisfied the user is with the current system. The Attractiveness of the system concerns the impression of the users to the current system. The pragmatic quality aspect concerns task-oriented quality such as the easiness to learn, the efficiency, and the dependability of a system, while Hedonic quality aspect concerns the non-task quality such as the system Stimulation and the aesthetic impression of the system.

Gap Analysis
Gap analysis determine the space between where we are and where we want to be and serves as a means to bridge that space." A gap analysis is conducted to determine the difference between the actual state and a desired future state of system." (Gomm, 2009). With this methodology the gap between user expectation and the new application will be revealed whether the current application quality has fulfilled the user expectation or not. This methodology using survey with the same question to collect the data from user. VOL.4, NO. 2, 2017

Methodology overview
The research follows the methodology described in figure 1. The research cover first phase of defining research problem, including the significance of this research and the hypothesis of this research. Next, the review of theoretical foundation of this research and the review of preceding works related to this research. The research methodology and the overview of the research design is defined. The next phase of the research which is data collection, data analysis, and result. Finally, the conclusion and improvement of this research. The following diagram.

Research Framework Determination
Research framework used in this research is UEQ framework. UEQ framework is chosen because of its comprehensive nature and simplicity to use. UEQ is a questionnaire that is meant to measure UX that include aspects of pragmatic and hedonic quality according to Laugwitz et al. (2008). Originally, the UEQ is in German language but has been made available in English. The questionnaire contains 26 items that is in the Likert scale fashion. Each one has a scale from 1 to 7 to measure the value of user satisfaction or dissatisfaction to each question component in the system tested.

Questionnaire Creation
The questionnaire is broken down into 2 sections including the background question of the respondent to get their anonymized demographic information and the user experience questionnaire where the respondent scores their experiences in browsing both mobile websites during testing process

Data Gathering from Old System
The questionnaire is created on Google Forms and distributed to each user by the project manager by user's email. After clicking the questionnaire link on the email, user will be redirected to Google Form page where they can start to fill the questionnaire

Old System Data Analysis
There are some steps involved in the data analysis process, they are: 1. Data transformation 2. Mean and standard deviation calculation 3. Confidence interval determination 4. Scale consistency checking 5. Data benchmarking

Feedback Gathering
The result from data analysis step is interpreted by author and the interpretation result is used as the basis for generating feedback for the news system. The feedback covers weak point from the old system in the user experience area and includes the open suggestion from project user and stakeholder. The open suggestion result come from interview process in the previous step. However, due to qualitative nature of the feedback and open suggestion, we can't precisely measure how much impact of this feedback to the improvement of the new system. The impact of feedback can only determine by measuring the data analysis result change from old system to new system

New System Data Gathering
This process is similar to data gathering process in previous step. However, this time, the object of data gathering is the user who already installed and use the new system for their daily task. The questionnaire and the distribution target is exactly the same with previous process to ensure the parity of research data

New System Data Analysis
This process is also similar to data analysis process in previous step. The difference of this process and previous process is the data source for the analysis. The analysis data come from questionnaire result of the new system user. To ensure parity with old system result, this step also use the exact same methodologies with the previous step

Gap Analysis
The questionnaire is broken down into 2 sections including the background question of the respondent to get their anonymized demographic information and the user experience questionnaire where the respondent scores their experiences in browsing both mobile websites during testing process

Result Comparison
The process is the final process of this research. The purpose of this process is how much improvement of new system compared to old system. This process relies on data from two of previous step, old system data analysis result and new system data analysis result. The result of those two processes are the quantization of application user experience, so those data can be compared directly by subtracting value from new system result to old system result. The formula for comparison is on equation 1.
(1) VOL.4, NO. 2, 2017 Where ∆x representing the improvement for each UEQ component. After obtaining the ∆x value, we can calculate how much improvement of the new system using simple percentage formula is on equation 2. (2) Where i is the improvement value in percent, rmax is the maximum value in the UEQ scoring range and rmin is the minimum value in the UEQ scoring range.

Result analysis
The data from data gathering process is analyzed using UEQ framework tools and will be compared to each other.

Sample size determination
The sample size is obtained from Slovin's sampling formula. The desired error rate is 5 persons, Sample size for this research is 60 persons, and total number of populations is 70 persons. The calculation for sample size is shown on equation 3. ( That can be rounded to 60 persons.

Validity measurement
Pilot study is conducted to 10% of sample size before the actual questionnaire being distributed to respondents. The number of respondents for this pilot study is 6 persons, chosen randomly from 70 persons in population size. The questions used in pilot study is the same with the question used on actual questionnaire to ensure the parity. The pilot study validity is measured using Cronbach Alpha coefficient. The result of validity for each category is shown on table 1.

Data Transformation
After all the result collected, the next step is transforming raw data into suitable data for UEQ processing. The data transformation is done on UEQ questionnaire result section, where the value from raw data is shifted to left by 4 points to achieve the UEQ result range.

Feedback Gathering and Result Comparison
Interview result with project stakeholder is summarized and the summary result is compiled into open suggestion for improving the new application. The feedback result is the following: 1. Can you please train the users of this application? 2. Please make the dashboard workflow similar to our standard workflow 3. Fix the fatal bug on the old application 4. Make more reporting data available on dashboard

Mean and Standard Deviation Value Comparison
Raw data of the UEQ is first transformed in order to calculate further. Raw data value is from 1 to 7, and the transformed data should be from -3 to 3. The mean value of each item is calculated, and the result is ranged between -3 to 3. Mean values from -0.8 to 0.8 are considered neutral evaluation. Values below -0.8 are negative evaluation, and values above 0.8 are positive evaluation.  Per category value on figure 3 shows the comparison between new system and old system mean value. The horizontal line represents the category, and vertical line represent mean value. Positive value means that user satisfied while negative value in old system means that user dissatisfied. Image above shows the old application's main user interface. This interface shows all the features of the application, but without any discernable "eye candies" that make the application looks good. New application also lacks advanced feature on the application itself, as the user needs to access dashboard via sitemap menu that opens the browser to access advanced feature. VOL.4, NO. 2, 2017 Revamped application user interface is shown on image above. Compared to old application's user interface, this app looks more colorful and feels "alive" with images on the menu. This application also incorporates more advanced feature into the application itself, so user only doesn't need to access dashboard to use more advanced feature.

Confidence Interval
The confidence level used in this data is 95%. Each component of UEQ scale have their own confidence interval that old app has can be seen on table3. The confidence interval of the user's answer suggests that majority of user using this application tends to think that this application is unsatisfactory to use in their daily works. The confidence interval of new app that can be seen on table 4. the user's answer suggests that majority of user using this application tends to think that this application is somewhat satisfying, even though there are still some aspects that can be improved, such as creativity seen in number 3, usefulness seen in number 5, and innovation seen in number 26.

Gap Analysis
For gap analysis, respondents are asked to fill another questionnaire used to measure their expectation for the old system. Their perception value for the new system is obtained from new system data gathering process in the previous step. Both questionnaire questions and respondent is the same to ensure valid result.  Table 5 shows the expectation value analyzed from UEQ on each category. User expects the perspicuity and dependability of the application, as it shows the highest value from user expectation from new app.    The table 7 shows the user perception value toward new application. As seen on table above, the attractiveness and perspicuity category shown the biggest increase compared to old application.  Table 8 shows the comparison of gap value between old app perception value compared expectation value and new app perception value compared to user expectation value. As seen, new application managed to close the gap of user expectation, although t=it doesn't meet the expectation. This result is predictable, as the timeline for application development is rushed by the owner of application. The red line represents user's expectation towards the app and the bars represents the perception value toward the application. From figure 5, almost all aspects of the app don't meet user's expectation, but the new app perception value is approaching the user's expectation.

Focus Group Discussion
Based on the UEQ result, they like the attractiveness aspect on the new application. Attractiveness aspect have the highest improvement when comparing those 2 applications. Attractiveness aspect improvement can be seen on the new application main interface that more stylish and more visual compared to the old application.

Conclusion and future work
Improvement for the application can be called success, because user like the new application compared to the old application. This research focuses on evaluating Company XYZ mobile application UX using UEQ. It begs the question about how well the performance of mobile application itself, and whether the mobile application able to fulfill their intention and expectation the user. The UEQ result show on how the UX of both old and new application.
VOL.4, NO. 2, 2017 61 What the respondents answer is reflected within the UEQ score. Users do not like the performance of the old one because too many problems within. The new application performance much better than the old one because not only it shrinks down the expectation user between new and old application, it also fulfils many requirements as the function work well.