--- title: "Regreening App: Exploring restoration data in R" subtitle: "Download data from DRS, and run analyses on it in R" format: typst: toc: true section-numbering: 1.1.a code-fold: false execute: warning: false message: false --- # About this notebook In this practical, you will work with **Regreening App** monitoring data downloaded from the **Data Reporting System (DRS).** If you are new to R, or want extra practice, we encourage checking out [Notebook 1 - Basics of R and analysis with climate datasets](https://k4ggwa.thegrit.earth/content/state_of_land/tutorials/_downloads/Notebook1_R_Basics_Climate.qmd), or some of the other notebooks available on the [Practicals Gallery](https://k4ggwa.thegrit.earth/content/state_of_land/tutorials/gallery.html) of the K4GGWA Platform's Evidence Hub. You will learn how to: 1. download and inspect your Regreening App project data 2. clean up your data 3. explore species, management and restoration attributes 4. turn the recorded wkt geometry into spatial polygons 5. make interactive maps of your restoration plots 6. render this notebook /outputs into a shareable report ::: callout-note This notebook focuses on accessing, downloading and exploring Regreening App field data from the DRS and ingesting it into R, producing a final, shareable report. If you are interested in combining Regreening App data with rasters and other spatial data for more advanced monitoring analyses, you can check out the [Combining Regreening App data with GGW raster data notebook](https://k4ggwa.thegrit.earth/content/state_of_land/tutorials/_downloads/N2_RA_STAC.qmd). ::: #### Before getting started If you have just opened this file inside RStudio, you can click on the *Visual* button in the top left corner of the notebook window (next to *Source*). This will allow you to work in a notebook-style environment with clearly visible text, code, and output sections. ------------------------------------------------------------------------ ## 1. Download your project data Open the [Regreening App Data Reporting System (DRS)](https://radrs.icraf.org/permission/dashboard). If you are not yet familiar with the DRS Platform, [see this tutorial video explaining how to navigate and access your project.](https://www.youtube.com/watch?v=q6_24YmqGrU) As a Project Manager (PM): 1. Open the DRS and project manager dashboard 2. Select the project you want to analyse 3. Download the project data as a csv file 4. Save the csv somewhere easy to find (a project folder) The exported file contains all the information recorded in the app about the restoration plot, including trees species, management practices, uses, location and polygon geometries. The kinds of data available depend on the app module that was used (Tree Planting; FMNR; Nursery; Rangeland). ::: callout-warning Remember, best practice is to **set up a project folder** where you store your `.qmd` notebook, data, and outputs. That way, you can easily find all of our project documents and data in one place, and we can set our R session working directory to this folder (**Session \> Set Working Directory \> Choose Directory**). See the example below. ::: ``` text my_restoration_project/ │ ├── data/ └── outputs/ ``` **Note:** If you have not yet created a project on the DRS, but you want to explore an example dataset, you can [download an example dataset here](https://drive.google.com/drive/folders/1zagSkg53XSGFttooknQrQJt5ahSTj7hW?usp=sharing). The example dataset is called `ra_plots_example.csv`. Save this file in your data folder that you created above. #### Note on the example dataset The example dataset is from a real FMNR restoration project in northern Ethiopia that has been anonymised. The names of the data collectors and farmers have been removed, and the georeferenced plots have had their geometry modified for privacy. Therefore, the polygons appear as hexagons (H3 grid), not the original geometry - your dataset will however show the real geometry of your plots/polygons. The code in this notebook generally uses variable names that refer to data captured using the FMNR module. Change these according to your own data. ------------------------------------------------------------------------ ## 2. Bring the data into R ### 2.1 Install and load the packages ```{r} #install.packages(c("tidyverse", "sf", "leaflet", "janitor") ``` ### 2.2 Load the packages ```{r} #| eval: FALSE library(tidyverse) library(sf) library(leaflet) library(janitor) ``` ### 2.3 Import your csv Replace `"ra_plots_example.csv"` with the name of your downloaded project csv file, if using your own file. Replace `"project_name"` with the name of your project. ```{r} project_name <- read.csv("data/ra_plots_example.csv") View(project_name) ``` ```{r} project_name <- project_name |> clean_names() # clean_names() is a function from the janitor package. We can use the it to easily clean/standardize our column names ``` ```{r} # Let's check our data to see what we're working with project_name |> summarise( records = n(), unique_plots = n_distinct(plot_name), unique_species = n_distinct(scientific_name, na.rm = TRUE), missing_geometry = sum(is.na(plot_geometry)) ) ``` In certain app modules, a single restoration plot can contain multiple records. For example, you can record several trees under management in the FMNR module, or many trees planted or water points captured under the Tree Planting or Rangeland modules, for instance. The number of rows in your csv file may therefore be much larger than the number of unique plots. The `ra_plots_example.csv` file contains restoration data from FMNR activities, so we will be expecting columns/variables with information on FMNR. ```{r} project_name |> count(plot_name, sort = TRUE) |> slice_head(n = 10) ``` ------------------------------------------------------------------------ ## 3. Explore the restoration data Lets check which species have been recorded. We can start with a simple number count, and then look at the species names. ```{r} species_counts <- project_name |> filter(!is.na(scientific_name), scientific_name != "") |> # Here we are telling R to return us a list of all the tree species recorded, by filtering out missing entries count(scientific_name, sort = TRUE) species_counts |> slice_head(n = 15) ``` Let's now visualise the most frequently recorded species: ```{r} species_counts |> slice_head(n = 15) |> # slice_head() tells R that we want to see the top 15 sp. ggplot(aes(x = reorder(scientific_name, n), y = n, fill = scientific_name)) + geom_col() + # geom_col() means we want a bar chart coord_flip() + # coord_flip means we want the bars to be horizontal labs( title = "Most frequently recorded tree species", x = NULL, y = "Number of records" ) + theme( plot.title = element_text(hjust = 0.5), # hjust tells R we want to centre our title legend.position = "none" # Here we tell R, we don't want to see the legend ) ``` ### 3.1 Data quality - checks and cleaning Monitoring datasets, especially those that have been generated by citizen scientists, often contain spelling, duplication or other errors. We can observe this in our species data. For example, the same species can appear as: ``` text Acacia etbaica acaciaetbaica Acaica etbaica acciaetbaica ``` These will be treated as different categories by R. We should therefore clean these up to make sure R doesn't count them twice, and to make sure we capture the full list of species. ### Cleaning up names We can standardise capitalisation and remove accidental spaces: ```{r} project_name <- project_name |> mutate( species_clean = scientific_name |> # Here, we use mutate to create a new column, called "species_clean", in which we pipe our cleaned species names using the two functions: str_squish and str_lower str_squish() |> str_to_lower() ) ``` For known variants, we can then make a small manual correction dictionary like so: ```{r} # This example 'dictionary' is for the species recorded in the example dataset. You will need to modify this according to your own data. species_corrections <- c( "acaciaetbaica" = "Acacia etbaica", "acaicaetbaica" = "Acacia etbaica", "acciaetbaica" = "Acacia etbaica", "eucleca racemosa" = "Euclea racemosa", "euclca racemasa" = "Euclea racemosa", "Euclearecemosasalus" = "Euclea racemosa" ) ``` ```{r} # Now we tell R to put our corrected, cleaned and standardised species names into our species_clean column, using the recode function project_name <- project_name |> mutate( species_clean = recode( species_clean, !!!species_corrections, .default = str_to_sentence(species_clean) ) ) ``` Now compare the cleaned counts: ```{r} species_clean_counts <- project_name |> filter(!is.na(species_clean), species_clean != "") |> count(species_clean, sort = TRUE) species_clean_counts |> slice_head(n = 15) ``` And plot them: ```{r} # Having now cleaned our species records for duplicates, spelling errors, and missing entries, we can re-create the plot from above species_clean_counts |> slice_head(n = 15) |> ggplot(aes(x = reorder(species_clean, n),y = n, fill = species_clean)) + geom_col() + coord_flip() + labs( title = "Most frequently recorded tree species", x = NULL, y = "Number of records" ) + theme( plot.title = element_text(hjust = 0.5), legend.position = "none") ``` ::: callout-warning This is a very simple example of taxonomic name-cleaning. Simple string cleaning can fix capitalisation, spacing and some known typos, but it can't reliably pick up whether multiple entries with similar spellings, especially in very large datasets, are actually the same species. For a formal analysis, species names should be checked against an authoritative taxonomic reference and use something called *fuzzy-string matching* function or similar. For this, check out the `taxify` and `WorldFlora` R packages. For the purposes of this practical, we can focus on learning the steps in the workflow and document any errors you find or corrections you make, so that it may be revisited later. ::: ------------------------------------------------------------------------ ### 3.2 Which management practices are being used? The `management_practices` field can contain several practices in one cell. For example: ``` text PRUNING FENCING WEEDING ``` While it can be valuable information to know which kinds of management practices are being done together, it can also be useful to separate these into individual practices: ```{r} management <- project_name |> filter(!is.na(management_practices)) |> separate_rows(management_practices, sep = " ") |> count(management_practices, sort = TRUE) management ``` Visualise them: ```{r} management |> slice_head(n = 10) |> ggplot(aes(x = reorder(management_practices, n),y = n, fill = management_practices)) + geom_col() + coord_flip() + labs( title = "Recorded management practices", x = NULL, y = "Number of records" ) + theme( legend.position = "none", plot.title = element_text(hjust = 0.5) ) ``` ::: callout-tip You can try changing `slice_head(n = 10)` to `slice_head(n = 15)` or remove it entirely - what happens to the figure? ::: ------------------------------------------------------------------------ ### 3.3 What are trees being used for? We can try this with `tree_usage` too: ```{r} usage <- project_name |> filter(!is.na(tree_usage)) |> separate_rows(tree_usage, sep = " ") |> count(tree_usage, sort = TRUE) usage |> slice_head(n = 10) ``` Make a chart: ```{r} usage |> slice_head(n = 10) |> ggplot(aes(x = reorder(tree_usage, n),y = n, fill = tree_usage)) + geom_col() + coord_flip() + labs( title = "Commonly recorded tree uses", x = NULL, y = "Number of records" ) + theme( legend.position = "none" ) ``` ------------------------------------------------------------------------ ### 3.4 Explore restoration plot characteristics The dataset also contains information about the plots themselves, like whether there are crops or fencing of some kind. We can explore these too. ### Crops and fencing ```{r} plots <- project_name |> distinct( plot_name, plot_has_crops, plot_has_fence ) plots ``` A simple summary: ```{r} plots |> summarise( plots = n(), crops_present = sum(plot_has_crops == TRUE, na.rm = TRUE), fenced = sum(plot_has_fence == TRUE, na.rm = TRUE) ) ``` You can also compare the two characteristics: ```{r} plots |> count(plot_has_crops, plot_has_fence) |> ggplot(aes(x = plot_has_crops, y = n, fill = plot_has_fence)) + geom_col() + labs( title = "Cropping and fencing across restoration plots", x = "Crops present", y = "Number of plots", fill = "Plot has fence" ) + theme( plot.title = element_text(hjust = 0.5) ) ``` ### Explore plot size If plot size is recorded consistently in your project, inspect its distribution: ```{r} project_name |> distinct(plot_name, plot_actual_size) |> filter(!is.na(plot_actual_size)) |> ggplot(aes(x = plot_actual_size)) + geom_histogram(bins = 25) + labs( title = "Distribution of restoration plot size", x = "Actual plot size (ha)", y = "Number of plots" ) ``` ## 4. Turn WKT into spatial polygons (with `sf`) The `plot_geometry` column contains polygons stored as **WKT (Well-Known Text)**. For example: ``` text POLYGON((41.0162303 11.7928467, 41.0161877 11.7928214, 41.0161457 11.7928436, 41.0160885 11.7928373, 41.0162303 11.7928467)) ``` The coordinates are stored as: ``` text longitude latitude ``` We can convert these strings into spatial polygons using `sf`. ### 4.1 Create one row per plot A plot can have several tree records, so we first create a plot-level dataset. ```{r} plots <- project_name |> filter(!is.na(plot_geometry)) |> group_by(plot_name, plot_geometry) |> summarise( Country = first(na.omit(country)), Region = first(na.omit(county_region_provence)), Crops = first(na.omit(plot_has_crops)), Fence = first(na.omit(plot_has_fence)), Plot_Size = first(na.omit(plot_actual_size)), Photo = first(na.omit(photo)), TSpecies = first(na.omit(species_clean)), .groups = "drop" ) head(plots) ``` ### 4.2 Convert wkt to `sf` ```{r} plots_sf <- st_as_sf( # Here, we tell R that our dataset is spatial, and we need to map it - so take our wkt coordinates and turn them into spatial features plots, wkt = "plot_geometry", crs = 4326 ) plots_sf ``` Check the coordinate reference system and geometry validity: ```{r} st_crs(plots_sf) sum(!st_is_valid(plots_sf)) ``` If the second result is greater than zero, some geometries may need repair before further spatial analysis. ------------------------------------------------------------------------ ## 5. Map your restoration plots ```{r} plots_sf |> leaflet() |> addProviderTiles("Esri.WorldImagery") |> setView( lng = 39.2, lat = 13.6, zoom = 10) |> addPolygons( weight = 2, fillOpacity = 0.4 ) ``` You should now see the polygons collected through the Regreening App. ------------------------------------------------------------------------ ### 5.2 Add information to the popups A map becomes much more useful when clicking a polygon gives you information about the plot. Let's create pop-ups for our individual plots, so we can quickly check key information. ```{r} # First, we create a new object called 'popup_text' in which we specify the text fields we want to see in our pop-up. We can then call this object when building our leaflet map. popup_text <- paste0( "Plot:", plots_sf$Plot_Name, "
Region:", plots_sf$Region, "
Crops:", plots_sf$Crops, "
Fence:", plots_sf$Fence, "
Tree species:", plots_sf$TSpecies, "
Plot size:", round(plots_sf$Plot_Size, 2) ) ``` ```{r} # Create the leaflet map with pop-up data plots_sf |> leaflet() |> addProviderTiles("Esri.WorldImagery") |> setView( lng = 39.2, lat = 13.6, zoom = 10) |> addPolygons( weight = 2, fillOpacity = 0.5, popup = popup_text ) ``` ------------------------------------------------------------------------ ### 5.3 Colour the plots by a characteristic or attribute We can use different colours to show a particular variable from the monitoring data. For example, to distinguish plots with and without crops: ```{r} plots_sf |> leaflet() |> addProviderTiles("Esri.WorldImagery") |> setView( lng = 39.2, lat = 13.6, zoom = 10) |> addPolygons( weight = 2, fillOpacity = 0.6, color = ~ifelse(Crops, "green", "grey"), popup = popup_text ) |> addLegend( position = "bottomright", colors = c("green", "grey"), labels = c("Crops present", "No crops"), title = "Plot characteristics" ) ``` ## 6. Add photographs to your map Some Regreening App records include a `Photo` URL. First check how many records contain a link, to see if it will be possible to map them. ```{r} sum(!is.na(plots_sf$Photo) & str_detect(plots_sf$Photo, "^https?://") # We tell R to detect any rows in our dataset that start with this string ) ``` If your project contains working photo links, we can include the image directly in the popup. ```{r} # Here, we create a new pop-up for our markers on our map. So when you click on the marker, a photo will show, with some information on the plot of interest. photo_popup <- ifelse( !is.na(plots_sf$Photo) & str_detect(plots_sf$Photo, "^https?://"), paste0( "
Region: ", plots_sf$Region, "
Species: ", plots_sf$TSpecies, "
" # Specify here your photo dimensions ), popup_text ) ``` ```{r} plots_sf |> leaflet() |> addProviderTiles("Esri.WorldImagery") |> setView( lng = 39.2, lat = 13.6, zoom = 10) |> addPolygons( weight = 2, fillOpacity = 0.5, popup = photo_popup ) ``` ------------------------------------------------------------------------ ## Turn your notebook into a report So far, you have been working in a notebook where the R code is visible, which is useful while learning because you can see exactly how each analysis was done and can revisit and tinker with sections of your work. Once you have finished your analysis, however, you may want to create a cleaner report that you can share with a colleague, or other stakeholder. With Quarto (the `.qmd` extension we use for our notebook file is a Quarto file), you can use the same analysis to create an HTML or PDF report where: - you can hide your R code - keep your plots/charts/other outputs visible - you can add written interpretation between code cells and outputs - the report can be rendered as a standalone HTML or PDF file ::: callout-note Learn more about Quarto and the different types of file formats and other publishing options [here.](https://quarto.org/docs/get-started/authoring/vscode.html) ::: ### Create a report version of your notebook A very simple approach is to make a copy of your completed `.qmd` file and call it something like: ``` text restoration_data_report.qmd ``` At the top of the new file, change the YAML header to control how the report is produced. The YAML specifies what format you want your report to be rendered in, the title, author, and other publication information, as well as what to do with code cells and outputs. Don't forget to change your title, subtitle, author name and date to match your own, personal analysis and work. ### PDF report You can create a PDF report by changing the format: ``` yaml --- title: "REPORT TITLE HERE" subtitle: "REPORT SUBTITLE HERE" author: "AUTHOR NAME HERE" date: today format: typst: toc: true section-numbering: 1.1.a code-fold: false execute: warning: false message: false --- ``` In the example above, we specify in our YAML, our format as *typst.* *Typst* is an open-source typesetting distribution that renders Quarto documents into clean, visually appealing PDF formats. You can customise the styling of your report in the YAML (the chunk of code we include at the top of our notebook, as you see above) - [see here for more information on how to do that.](https://quarto.org/docs/output-formats/typst.html) For a more interactive report, that retains your interactive maps and data, you might opt for a HTML report instead. ### HTML report ``` yaml --- title: "REPORT TITLE HERE" subtitle: "REPORT SUBTITLE HERE" author: "AUTHOR NAME HERE" date: today format: html: toc: true code-fold: true code-tools: false execute: echo: false warning: false message: false --- ``` This tells Quarto not to show the R code in the rendered report however behind the scenes your code still runs, it is just hidden from the reader - i.e., your plots, tables and other outputs will still appear. ### Include markdown text between your code as report sections You can write normal markdown text between your R code cells, which will render as normal report text sections. For example: ``` markdown ## Species composition The analysis identified 24 recorded species across the restoration plots. The most frequently recorded species was *Acacia etbaica*. ### Interpretation The most frequently recorded species was XYZ. The dominance of these species may reflect XYZ. An area for further investigation going forward is XYZ. ``` When the report is rendered, the reader sees the heading, plot and written interpretation, but not the R code that was used to generate any data visualisations. Visit the link to the Quarto website included above, to explore more options on formatting markdown text in Quarto. ### Render your report There are several ways to render a Quarto document **Option 1: Use the Render button** If you are working in RStudio, you will see a button 'Render' with a small blue arrow next to it, at the top of the page. > **Render** Quarto will run the R code and create the output specified in your YAML (i.e., depending what render type you have specified, HTML, PDF). **Option 2: Use the terminal** You can also render directly from the terminal: ``` bash quarto render restoration_data_report.qmd ``` The rendered file will be created in your project folder. ### Publish your report **For PDF:\ **If you specified PDF as your render format, a PDF report will be rendered and saved to your project folder! **For HTML:**\ A rendered HTML file can either be shared directly, or Quarto can publish it to the web. For example, from the terminal: ``` bash quarto publish ``` Quarto will guide you through selecting a publishing service and setting up the destination. Once published, you will have a web link that you can share with colleagues or partners. To share directly, you can simply use the Render button and the file will be saved to your project folder. ------------------------------------------------------------------------ ### Well done! If you have arrived at the end of this notebook, you have successfully: - [ ] Accessed and downloaded your Regreening App restoration data from the DRS - [ ] Opened your `.qmd` file in R-Studio and set-up the project folder - [ ] Ingested your Regreening App data into R - [ ] Used R to perform analyses on your restoration data and generated different plots and maps - [ ] Saved your work in a shareable report format We encourage you (data privacy considerations permitting!) to share your reports and outputs with your GGW networks, including the Spatial Data for GGW Restoration WhatsApp community, including any challenges, extra analyses or other interesting things you found when running through this notebook.