Hailiang Chen, Miao Yu: When it comes to handling enquiries for government services, a Retrieval-Augmented Generation (RAG) framework is capable of resolving more complex issues, while still providing accurate answers. The time required, computing power, and costs are also far lower than Incremental Pre-training and Fine-tuning methods.
Can general LLMs be directly applied to government services?
In recent years, Large Language Models (LLMs) like ChatGPT have rapidly gained worldwide popularity and shown great potential. Through massive training using vast amounts of data, these models can produce coherent and semantically sound scripts, equipping the scripts with excellent question-and-answer capabilities. When it comes to government services, members of the public generally use government websites or mobile applications to enquire about policies, regulations and procedures. People may also seek support from physical help centres.
While traditional government services are primarily delivered over a physical counter or by phone, service efficiency and response speed can often face limitations. This is especially true when complicated questions come up, which can lengthen queuing times and result in lower public satisfaction. As technology develops, especially from the emergence of LLMs, government services are gradually moving toward more intelligent automation. By combining LLMs with Conversational AI, government services have become more efficient and can significantly enhance the user experience.
Although LLMs perform well in handling common enquiries, they still face many challenges in the more specialised fields of government policies and regulations. LLM training relies primarily on data from public content that is online. The lack of in-depth, professional knowledge could lead to inaccurate answers or contradictory responses. Under certain scenarios, the models may even have “hallucinations” that provide false information – with total confidence. These issues may be difficult for the average person to detect if they do not have professional knowledge in a specialised subject.
Moreover, government service policies are constantly changing. If LLMs fail to include the latest policy updates, their answers could be at odds with existing policies. In this regard, it is not enough to just ensure the accuracy of answers of government service AI systems. A boost to the systems’ interpretability is essential to help the public understand the basis for each answer. This will also help improve the systems’ transparency and credibility.
Application characteristics of RAG in government services
The RAG framework was created to fully use LLMs in government service scenarios. The RAG technology optimises answers through two steps. First, by searching for documents and text extractions relevant to the user’s question. Second, based on these results, the reply is then generated using an LLM. The introduction of RAG has resolved some existing issues in government services and has shown important advantages (Note 1).
RAG has drastically increased the accuracy of answers by integrating an external database with references to the latest and authoritative policies and regulations. RAG also supports more complex queries, as it is able to handle a multi-level and multi-dimensional problem, instead of relying solely on keyword matches. Through the combined search results and generated responses, RAG has not only improved the accuracy of its replies but also strengthened the users’ trust in AI systems. This allows people to see the justification of each answer and reduces concerns caused by LLM hallucinations.
Furthermore, RAG technology greatly cuts costs and power consumption. Compared to Incremental Pre-training and Fine-tuning approaches, RAG does not need to retrain large models. Instead, it improves the quality of its answers by supplementing them with search results from external data sources. Given this, the RAG framework requires far less time, computing power and running costs than the Incremental Pre-training and Fine-tuning methods.
Foundational model, Fine-tuning model and RAG framework: Which one is best?
To further evaluate the RAG framework’s performance in a legal service setting, the authors made a comparative analysis by grading its semantic rules and factual consistencies. This was done to identify any discrepancies in response quality and accuracy between the base model, the fine-tuned model and the RAG framework.
Research results showed that the Fine-tuned model performed best in dealing with semantic queries, due to its linguistic style and wording being closest to the model answer. However, there were glaring problems, as hallucinations were more severe than in the Base model, generating answers that often included information that was inconsistent with facts. This affected its reliability in practical applications.
In comparison, the Base model was able to give relatively concise answers and with less information. It still had limited ability for handling complicated questions and it performed below par on high-precision tasks.
As for the RAG framework, it was effective in reducing hallucinations. Responses were less contradictory to answers generated by the Fine-tuned model. RAG also provided more accurate explanations with high factual consistency. While ensuring for accuracy and consistency, the RAG framework can reduce hallucinations, making it especially suitable for complicated tasks that demand external knowledge support, such as consultations and analyses on government services.
In terms of cost, using a comprehensive analysis of performance and resource consumption, the Fine-tuned model was found to be five times that of the RAG framework, whereas the cost of the RAG framework was similar to the base model. On top of ensuring generational quality, the RAG framework also significantly cut costs, making it an ideal choice in striking a balance between efficiency and cost. Crucially, data safety is of critical importance to governments and enterprises. The RAG framework can be applied through a fully private deployment, so it does not need to upload sensitive, internal data to third-party platforms. This effectively reduces the risks of data leakage (Note 2).
The future of government services: From automation to personalisation
Presently, government service chatbot supported by RAG technology can effectively address shortfalls in conventional service delivery such as delayed information, inefficient searches, and human intervention. As technology advances, the RAG system is expected to broaden its application in wider domains and provide more personalised government services like smart content recommendations and ‘digital humans’. Not only will this help realise low-latency voice interaction, but it can also customise a government service experience to meet the user’s needs, allowing for a more personalised and targeted service.
Note 1：Special Report: “A New Chapter in Smart Governance: AI Empowering Innovation in Government Services”
https://fwik3jehaxr.feishu.cn/file/FFCjbsGLzoiHuOxKcTPcNqn1nMb
Note 2：Special Report: “A New Chapter in Smart Governance: AI Empowering Innovation in Government Services”
https://fwik3jehaxr.feishu.cn/file/FFCjbsGLzoiHuOxKcTPcNqn1nMb
Professor Hailiang Chen
Assistant Dean (Taught Postgraduate)
Director of AI Research Institute
Professor in Innovation and Information Management
Miao Yu
Research Postgraduate Student, HKU Business School
This article was also published on January 3, 2025 on the Financial Times’ Chinese website