VAGO report user research + optimisation

The Victorian Auditor-General's Office (VAGO) provides independent assurance to the Victorian Parliament and the public regarding the performance of the Victorian public sector.

VAGO operates under the principle that when the government spends significant public money on a major project, Parliament and the community should be provided with clear updates on the project’s progress and outcomes.

Each financial year, VAGO assesses whether Victoria's public sector has transparently reported the performance of major capital projects against cost, time, scope and benefits. (A major project is defined as a capital investment worth $100 million or more in Total Estimated Investment (TEI).)

Findings and information relating to their assessment is provided in VAGO’S Major Projects Performance Report.

There are two products relating to this report:

1. The PDF report

A digital PDF version of the report tabled in parliament.

2. The digital dashboard

An interactive Microsoft Power BI report housed on the VAGO website designed to be used as a companion to the PDF report.

VAGO engaged us to audit these two products—the PDF and the dashboard—as they wanted to understand how they could improve the report to better serve their audience needs

Firstly, we conducted qualitative user research to better understand the experience, pain points, and expectations of VAGO’s key audience segments when trying to access and understand the Major Projects Performance Report.

We then completed a best-practice audit of the PDF report and the accompanying companion dashboard, highlighting opportunities to optimise usability, readability, navigation, accessibility and overall design and presentation.

Finally, we prepared a series of indicative wireframes to illustrate our suggested improvements into the 2024 companion dashboard.

User Interviews

To fully understand how VAGO’s audience uses the report and companion dashboard, their experience and pain points, we began by conducting a round of user interviews.

Participants came from three main groups:

• Members of Parliament
• Analysts
• Electoral staff

MP and electoral staff participants came from a range of political parties.

We began our interview sessions by asking participants a series of questions to gain a deeper understanding of their role, how the Major Project Performance Report intersects with their work and the purposes for which they use it. From there, we observed the participants as they worked their way through the content of both the PDF report and the dashboard. We took note of how they engaged with the information presented, and in the case of the companion dashboard, how they interacted with the available functionality.

Through this process, we identified which aspects of the reporting were most valuable to the people it was ultimately intended for and which areas were hindering its effectiveness.

Perhaps even more importantly, it shed light on how people used the information in this report and the challenges in getting it to people in a manner compatible with their roles.

These insights helped form the foundations of the recommendations that would drive future reporting to be more effective and more responsive to the practical needs of those using it.

Best practice auditing

As important and valuable as the insights from the user-focused studies were, they could only cover some aspects of the reports' content, format and delivery.

To uncover further insights, we reviewed the reporting products using our expertise in user experience, accessibility, and design thinking.

Our review found numerous opportunities to improve the reporting in tangible, meaningful, and, most importantly, actionable ways.

From layout and design considerations to the technical implementation of the reports, we ensured our focus resulted in specific enhancement activities rather than broad or vague recommendations, empowering VAGO to make the changes required to maximise the report's value and utility.

Through our user interviews and auditing, we uncovered various opportunities to strengthen the report products.

The most important finding from our work was an understanding that we can't assume the audience's knowledge. It is unrealistic to expect that members of the audience, even members of parliament, are so immersed in the language and ways of working of large project spending that they will understand the presented data without appropriate framing and explanations. Instead, the report needs to support the building of knowledge, helping audience members understand what the report presents. It should give them the tools to come to their own conclusions regarding what it means.

This understanding drove much of our approach to developing the new dashboard wireframes, which were the ultimate outcome of this project.

Our work emphasised simplicity of navigation, prioritisation of the content most important to the dashboard's users, accessibility, support and data clarity. We recognised that users came from a range of backgrounds and did everything available within the limitations of the PowerBI framework to ensure people could access information quickly, intuitively and without a background in data analysis.

With wireframes and a set of clear, actionable, evidence-based recommendations, VAGO can optimise its 2024-25 Major Projects Performance Report for maximum usability, accessibility and comprehension.

A clearer, easier to use and understand report and companion dashboard can strengthen the efforts of VAGO when assessing whether Victoria's public sector has transparently reported the performance of major capital projects. Such improvements are a win for transparency and the democratic process as a whole.