Main authors: Luuk Fleskens, Coen Ritsema, Zhanguo Bai, Violette Geissen, Jorge Mendes de Jesus, Vera da Silva, Aleid Teeuwen, Xiaomei Yang
iSQAPERiS editor: Jane Brandt
Source document: Fleskens, L et al. (2020) Tested and validated final version of SQAPP. iSQAPER Project Deliverable 4.2, 143 pp

 

The SQAPP has been through a number of rounds of evaluation by different groups of users stakeholders during its development. These include: a field evaluation of SQAPP performance; a formal evaluation of the beta version by some 90 European stakeholders (researchers, farmers, students, advisory services and policy makers) in locations in Slovenia, Poland, Portugal, Greece, Spain, France, Estonia, Romania and Netherlands; an evaluation by some 220 participants in the 11 study site Demonstration Events. The feedback from each evaluation was assesed and used in the further development of the app (see »User feedback and SQAPP development). Here we provide details about how issues raised during the user evaluations were addressed in the SQAPP development.


Contents table
1. SQAPP review focus
2. Student evaluation of the beta version with farmers in the Valencia region, Spain 
3. Formal evaluation of SQAPP beta-version in case study sites 
4. SQAPP feedback at demonstration events 

1. SQAPP review focus

Review of SQAPP performance has focused on three dimensions: accuracy and relevance of the information provided, as well as its functionality as a tool in the hands of end-users. These dimensions were defined as follows (van den Berg et al., 2018):

Accuracy

Accuracy as a concept can be defined as the difference between the estimated value and the true value. In order to determine accuracy, quantitative measurements of data are necessary. This is expressed by knowing or estimating the actual (true) value (Walther and Moore, 2005). As such, the accuracy in this case can refer to the ability of the app to report information to the best estimation of the true value. Reference values were taken from laboratory analysis of soil samples, or from field measurement or visual soil assessment.

Relevance

The relevance of the information provided by the app can be analysed using the concept of actionable knowledge as stated by Cash et al. (2003): “Science and technology must play a role in sustainable development whilst effectively managing the boundaries between knowledge and action in ways that simultaneously enhance salience, credibility and legitimacy of the information produced”. Actionable Knowledge as a concept defines the boundaries of stakeholder participation in the decision making process and fostering solutions together (Geertsema et al., 2018).

The concept of actionable knowledge was channeled into the investigation of relevance. Relevance, in the context of this evaluation process, is used to describe the meaningfulness of the information provided by the app and subsequent action through adjustments in management practices by the end-users. This was tested via discussion of the recommendations with app users during interviews. We also assessed the recommendations provided based on various levels of suitability to the specific farmers’ context.

Functionality

Functionality can be defined as the effectiveness of the app as a medium, the accessibility of the language used, and the usability of the tool. Within the concept of usability, Iwarsson and Ståhl (2003) suggest four components that need to be satisfied: a personal component related to human functioning, an environmental component related to barriers within the environment that may inhibit action, an activity component related to the activities that need to be performed, and, finally, an analysis of the three aforementioned parts ensuring individual and group preferences are met within the targeted environment. That means, that the functionality of the app is not only limited by the design of the interface, but it also encompasses the socio-environmental context in which the user is placed, as well as the individual attitude towards the technology.

Accuracy assessment was an essential part of the SQAPP evaluation process as it was raised as an important issue by stakeholders. It was in several of the rounds of evaluation integrally included in the assessment with app users, but as it relied on testing against other data, this part is reported in »SQAPP testing. The evaluation with researchers in the project plenary meeting in Ljubljana and conference participants in the Soil Horizons workshop was labelled as peer review and is reported in »SQAPP peer review.

2. Student evaluation of the beta version with farmers in the Valencia region, Spain

This evaluation sought to understand the accuracy and relevance of the app through assessing how soil properties, threats, and suitable management practices are uniquely interpreted and reported by the application, by field measurements and observations, and by farmers and landowners themselves (Figure 1).

D4.2 2 fig01
Figure 1

 

For the full report see »Field evaluation of SQAPP performance (beta version) in the greater Albaida region. The main recommendations stemming from this report are shown below with an indication on how this was included in SQAPP (Table 1).

 

Table 1: Core recommendations for the improvement of SQAPP: the soil quality app

Area Recommendation no. Details How it was dealt with in SQAPP development  
Accuracy A1 Insert the the option to let users search and specify location using direct address and coordinate entry  This functionality has been added in SQAPP.
  A2

Reconsider source and evaluate accuracy of datasets for:

  • ‘Soil pH’
  • Nutrient Availability: ‘Exchangeable Potassium’ ‘Phosphorus using the Olsen method’ ‘Total Nitrogen’
  • ‘Electrical Conductivity’
  • ‘Wind erosion vulnerability (classified)’  
No better datasets were currently available. Users can update global data with their own data. The level of reliability of datasets has been indicated in SQAPP.
  A3

Work to fill gaps in datasets for:

  • 'Rainfall data’
  • ‘Altitude’
  • ‘Soil Wind Erosion in Agricultural Soil’
  • ‘Wind erosion vulnerability (classified)
Rainfall and altitude data were missing from several coastal areas due to coarse-scale native datasets. These were extrapolated to give a better coverage. Soil quality data is available in these locations but no cumulative probability density functions have been produced, which leads to lower functionality (no recommendations for improvement of soil quality parameters).
Relevance R1 Allow for entry of optional field characteristics including crop type and AMPs Crop types and use of pesticides can now be specified by the user, as well as interest in specific AMP recommendation domains.
  R2 Allow user to specify ‘user type’ during profile creation (Farmer vs Researcher) and curate recommendations accordingly Modular user specifications have not yet been implemented. The app is intuitively organised so that different types of users can easily find the information they are interested in.
  R3 App should not give 10 AMPs arbitrarily, instead listing all that exceed a score threshold A standard list of 10 AMPs ordered in descending order of relevance is maintained. As the AMP database has grown extensively, the risk of including less suitable AMPs has diminished.
  R4 Require that ‘land cover’ be manually entered rather than auto filling ‘arable’ or ‘other’ Further specification of land cover is now enabled, so that users can verify whether the correct land cover is selected and change manually if required.
Functionality F1 Include a detailed/guided tutorial that can be reviewed by the user at any time Information buttons have been added in SQAPP to briefly explain the elements of the app. A FAQs section has also been provided. On iSQAPERIS app users can find detailed explanation and a video tutorial about the SQAPP.
  F2 Soil properties terminology has to be clarified and links to more information could be provided Through information buttons the soil properties are explained.
  F3 ‘Landscape position’ should be auto filled (locked to latitude and longitude) and not allowed to be changed manually Landscape position is now auto-filled, but editable by the user in case the global data are found to be incorrect.
  F4 Re-specifying coordinates for a saved location should automatically update field characteristics to match (altitude, precipitation, landscape position, and slope) This has been implemented.


3. Formal evaluation of SQAPP beta-version in case study sites

As part of the process of developing SQAPP, the beta version was formally evaluated by some 90 European stakeholders (researchers, farmers, students, advisory services and policy makers) in locations in Slovenia, Poland, Portugal, Greece, Spain, France, Estonia, Romania and Netherlands. The evaluation used a standardized interview protocol and questionnaire to collect feedback from the stakeholders on the quality and accuracy of information provided by SQAPP and the benefits and disadvantages of its different features. The results of this evaluation are reported in »SQAPP (beta version) - questionnaire»SQAPP (beta version) - feedback results and »SQAPP (beta version) - gender disaggregated feedback results. Key issues identified and how they were considered in SQAPP development are listed below in Table 2.

Table 2. Key issues emerging from the formal SQAPP beta-version and how they were addressed in SQAPP development.

Issue How it was dealt with in SQAPP development 
Not all soil parameters relevant Most soil parameters were deemed relevant by users. When not deemed relevant, e.g. electrical conductivity, this was due to a particular aspect of soil quality (e.g. soil salinity) not being an issue locally. Still, all soil parameters are shown when data is available (and confirming that the indicator is indeed in the desired range).  
Meaning of the probability density functions The concept has been explained in information buttons and a video tutorial. 
Missing soil parameters: (a) magnesium and sulphur (5 responses); (b) aggregate stability and soil structure (5 responses); (c) soil compaction (4 responses); and (d) the methods used, e.g. for the analysis of potassium, phosphorus, pH in calcium or chloride suspension (2 responses). For magnesium and sulphur no spatial data was available. For aggregate stability and soil structure visual soil assessment methods need to be applied, for which also no spatial data is available. Soil compaction risk can be predicted, but requires management information and information on weather conditions which is too complex for SQAPP – instead susceptibility to soil compaction is included. The methods used have been specified.
There is a lack of clarity in the units that should be better presented and adapted. Description of the units should be added. Some units should be presented according to the location under consideration (e.g. soil nutrient in Slovenia is expressed with mg/100gr soil). The units are explained and further specified in the information button for the soil properties. Some common conversion factors are provided in FAQs section.
The source of data should be mentioned. This lack of information sows doubt about the reliability of the parameters. For this purpose, it was suggested to include a reference range (high / medium / low) or an index of estimated accuracy level to indicate the reliability of the results. The source of data has been mentioned, and an indication of the accuracy of each parameter tested is given (if available).
SQAPP should be available in the local language. SQAPP has been made available in 14 languages
Soil threat threshold values are not realistic (too high) in the local context and should be in accordance with national legislation. The threshold levels were documented in »Impact of promising agricultural management practices. They are used as a global reference – indeed local classifications and legislation can deviate from these, but the current version of SQAPP cannot consider nationally and regionally different threshold values.
Difficulty in understanding the outcomes of the threshold values (in a practical way) suggesting appropriate information on the thresholds values, colours (red for high risk, yellow for moderate risk, and green for low risk for threshold). The low, medium and high risk areas are consistently colour-coded and in the overview page of soil threats each local score is labelled with the respective colour code as well. The colour coding system is explained in FAQs and video tutorial.
Include some other soil threats such as “susceptibility to pests”, contamination with pesticides and other relevant organic components recommendations should be restricted to the most important and innovative AMPs that can be implemented in any one location, instead of listing many practices that are out of context. Susceptibility to pests was not deemed a soil threat per se and spatial data on soil pests is not available. For pesticide contamination risk a new module was developed and included in SQAPP. The majority of users found the number of listed AMPs (top-10) appropriate. Some users would like the recommendations to focus on fewer more focused AMPs. The top-10 is ordered in sequence from most to less relevant. The user can indicate a specific category of AMPs of interest and these will be highlighted in bold. Also the number of AMPs considered and the scoring system have been expanded, which has led to lower likelihood for inappropriate AMPs being recommended.
The level of innovation of AMPs is insufficient. The AMPs have a quite broad range. The examples highlight innovative approaches within a specific AMP. A huge effort has been made to prepare 381 specific examples of AMPs, providing more inspiration to app users.
Enable the cropping system to be entered manually in order to refine the recommendations. A further specification layer of the broad land cover categories has been implemented.
SQAPP recommends converting arable land to forest/grassland in areas where the farmers already do not have enough arable land. The option to convert land use has been removed from the list of AMPs. Although this can be a valid option and is sometimes recommended by researchers and advisors, it was deemed offensive to farmers and land users. Hence, SQAPP now aims to provide recommendation for a given land use system, and not to change land use.
Include more information such as: yield (which crop should be planted in this area), costs, how much manure should be used per ha and what kind is recommended, the possibility to compare results from different areas. As SQAPP is about soil quality, plant-specific aspects such as crop choice, yield expectations and fertilization levels were beyond the scope of the app development. Costs are also highly variable in practice, but a complexity factor giving an aggregate indication of the magnitude of costs, effort and know-how has been added to the AMP examples.


4. SQAPP feedback at demonstration events

In the final phase of iSQAPER, demonstrations events were organised in all the study sites (with the exception of Zhifanggou Watershed) to demonstrate and discuss the local soil quality assessment and recommended management practices provided by SQAPP (among other things). See »»Demonstration events - feedback results.

A total of 483 people participated in 11 events including representatives from all the target groups of stakeholders (farmers, advisors, suppliers, researchers, students, policy makers and administrators). During the events feedback was collected from some 220 of the participants in response to the following questions.

What aspect of the SQAPP app interests you most?

The three most frequent response types from the Europeans were:

  • Data provided on soil properties “The availability of soil data for specific area.” (male agronomist, Crete)
  • Management recommendations “Tips on how to improve soils.” (male farmer, Slovenia)
  • Soil quality evaluation provided “Fast results about soil quality." (male student, Estonia)

The three most frequent response types from the Chinese were:

  • Data provided on soil properties “The database is very powerful.” (male farmer, Qiyang/Gongzhuling)
  • Potential to add own data “All users can update the data.” (female researcher Qiyang/Gongzhuling)
  • (equally): Soil quality evaluation and Accessibility of data, ease of use “I can know the quality of my farmland by the APP.” (female farmer, Qiyang/Gongzhuling) “Data download is very convenient.” (male agro-technician Qiyang/Gongzhuling)

Are there any improvements or changes you think should be made to SQAPP to make it a tool that you would use regularly?

The most frequently requested improvement from users in both Europe and China was to have more of the text translated from English into their own language. The two other most frequently requested improvements from the Europeans were:

  • User input data “Inputs from users should be checked by experts since there is always a risk for not valid data or data entry mistakes.” (female agronomist, Crete)
  • More specific recommendations for local methods/practices “Recommendations that are more suitable in Estonian conditions.” (female student, Estonia)

The two other most frequently requested improvements from the Chinese were:

  • Android version Because, at the time of testing, SQAPP was only available on Apple’s App Store in China, the second most frequent request was for an Android version of the app.
  • Soil data temporal/spatial resolution “I think it needs improvement in accuracy.” (male researcher, Qiyang/Gongzhuling)

Regarding the suggested improvements, we have worked on the translations of app commands, AMP recommendations and additional information in 14 languages. We have prepared guidelines for different types of users, in which farm advisors could play a role in supporting farmers and land users to operate SQAPP effectively, and we have expanded the portfolio of AMPs and specific examples to provide to app users. The Google Play store is not accessible in China – an apk installer file has been shared. Accuracy issues have been highlighted by adding a qualification for the reliability of each data layer for which we could perform tests in iSQAPER sites.

 


References cited in this article

  • Cash, D.W., Clark, W.C., Alcock, F., Dickson, N.M., Eckley, N., Guston, D.H., Jäger, J. and Mitchell, R.B., (2003). Knowledge systems for sustainable development. Proceedings of the national academy of sciences, 100(14), pp.8086-8091.
  • Geertsema, W., Rossing, W.A., Landis, D.A., Bianchi, F.J., Rijn, P.C., Schaminée, J.H., Tscharntke, T. and Werf, W., (2016). Actionable knowledge for ecological intensification of agriculture. Frontiers in Ecology and the Environment, 14(4), pp.209-216.
  • IWARSSON, S. and STÅHL, A. (2003). Accessibility, usability and universal design—positioning and definition of concepts describing person-environment relationships. Disability and Rehabilitation, 25(2), pp.57-66.
  • van den Berg, P., Steward, S. R., Vidal Morant, M., Ongus, E., van der Zaan, T. 2018. Evaluation of the performance of the Soil Quality App (SQAPP) for the greater Albaida region. Report submitted to Wageningen University and Research as part of the authors' MSc programme
  • Walther, B. and Moore, J. (2005). The concepts of bias, precision and accuracy, and their use in testing the performance of species richness estimators, with a literature review of estimator performance. Ecography, 28(6), pp.815-829.