R Exchange 2024
Lillian Lu
Data Analyst
Ministry for the Environment, Manatū Mō Te Taiao
Lillian is a senior data analyst at the Ministry for the Environment. With experience in various government organisations, she has worked with health, environment, and tourism datasets. Coming from a marketing background, Lillian is passionate about data communications. She is also a keen developer of Shiny and how data scientists or analysts can leverage it to achieve an end-to-end analytics workflow within R or Python.
"Maintaining order amidst chaos with open-source workflow"
The National Monitoring System (NMS) project is an annual audit of local government performance under the Resource Management Act. A total of 78 councils are required to collect their NMS data in a spreadsheet template, and submit it to the Ministry for the Environment (MfE) through a mailbox. The data team at MfE then reviews, processes, and consolidates the data for various reporting purposes. There is extensive communication between MfE and councils throughout this process. The form of an excel spreadsheet also creates significant challenges when it comes to data cleaning and processing.
Lillian Lu is a senior data analyst from MfE and the technical lead for the NMS project. In this talk, Lillian will walk you through how the team leverages different functionalities in R to manage the complex NMS workflow, including Shiny app and R package development. While the current solutions are not perfect, as the root of the problem lies in how the data is collected, the team hopes to demonstrate the potential of utilising open-source technology to maintain order amidst the chaos, especially for low-budget projects.