• Xuming Hu

    I'm an assistant professor of the AI Thrust at The Hong Kong University of Science and Technology (Guangzhou).

    I am also being affiliated with the Department of Computer Science and Engineering at The Hong Kong University of Science and Technology (Clear Water Bay).

    Slide 1

  • Xuming Hu

    My research interests lie in the fields of Natural Language Processing and Large Language Models.

    Details
    Slide 2

Welcome to my personal website

Our lab is hiring motivated and capable Ph.D. and MPhil students interested in Natural Language Processing.

About Me

Introduction

Hi! My name is Xuming Hu (胡旭明). I’m an Assistant Professor of the AI Thrust at The Hong Kong University of Science and Technology (Guangzhou). I am also being affiliated with the Department of Computer Science and Engineering at The Hong Kong University of Science and Technology (Clear Water Bay).


Education

I received my Ph.D. at the School of Software in Tsinghua University, where I am advised by Prof. Philip S. Yu (ACM/IEEE Fellow). I was a Visiting Scholar at CUHK MISC Lab from May 2022 to August 2023 under the supervision of Prof. Irwin King (IEEE Fellow).

Prospective Students

We sincerely welcome interested students! Please refer to this section for more detail.
[Updated on June 30, 2024] Thanks for your interest in joining our group! We have 3-4 open positions for Ph.D. students in 2025 Spring and Fall in AI Thrust. For MPhil students, we have 4-5 open positions in 2024 Fall in AI Thrust, please contact me after passing the interview with the school's Red Bird MPhil committee.
selfie

Dr. Xuming Hu

Assistant professor

News

Research Outline

My research interests lie in the fields of natural language processing and deep learning. In particular, I am interested in large language models and focusing on:
  • Exploring Trustworthy LLMs,
  • Studying Multimodal LLMs,
  • Exploring Efficient LLMs,
  • Delving into the "AI for Science" initiative.


My list of publications can be found on Google Scholar

Exploring
Trustworthy LLMs

Studying
Multimodal LLMs

Exploring
Efficient LLMs

AI for
Science


Exploring trustworthy LLMs aims to ensure the content generated by these models is sufficiently truthful. We attempt to mitigate the hallucination phenomenon in LLMs through methods such as retrieval augmented and the use of watermarks in LLMs.
Multimodal LLMs can transcend the boundaries of text and venture into the realm of vision, opening new perspectives for the development of embodied intelligence and sparking imagination about the future application potential of LLMs.
The high cost of training and reasoning of LLMs and the high computing power required have hindered the development of LLMs. How to accelerate the reasoning speed of LLMs, and reduce the parameter scale of LLMs are important research directions.
I explore the real-world applications of LLMs, such as in biology, materials, healthcare, recommendation systems, and social networks, etc., committed to making LLMs better serve society.

Services

  • Editorial Services
    ACL Rolling Review
  • Conference Area Chair
    ACL 2024, NAACL 2024, EACL 2024
  • Conference Program Committee Member
    ACL 2022-2023, EMNLP 2021-2023, NAACL 2022, AAAI 2022-2024, SIGKDD 2023, SIGIR 2023-2024, WWW 2024
  • Journal Reviewer
    IEEE TKDE, IEEE TNNLS, IEEE/ACM TASLP

Selected Publications

A few selected publications are listed for each research direction. See Google Scholar for a full list of publications.

Trusted Knowledge Exploration

HIE

Do Large Language Models Know about Facts?

HGTM

An Unforgeable Publicly Verifiable Watermark for Large Language Models

HGTM

A Semantic Invariant Robust Watermark for Large Language Models

ICLR 2024(Spotlight)

Xuming Hu, Junzhe Chen, Xiaochuan Li, Yufei Guo, Lijie Wen, Philip S. Yu, Zhijiang Guo

Pinocchio is a benchmark assessing large language models' (LLMs) factual knowledge across diverse domains and languages, through 20K questions. It reveals LLMs' limitations in handling factual information and spurious correlations, emphasizing challenges in achieving trustworthy artificial intelligence.

ICLR 2024

Aiwei Liu, Leyi Pan, Xuming Hu, Shu’ang Li, Lijie Wen, Irwin King, Philip S. Yu

Recent advancements in text watermarking for LLMs aim to mitigate issues like fake news and copyright infringement. Traditional watermark detection methods, reliant on a secret key, are vulnerable to security risks. To overcome this, a new unforgeable watermark algorithm has been developed, employing separate neural networks for watermark generation and detection, while sharing token embedding parameters for efficiency.

ICLR 2024

Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, Lijie Wen

To address the security and counterfeiting issues in current text watermarking for LLMs, we employs distinct neural networks for watermark generation and detection, sharing token embedding parameters for efficiency. This approach ensures high detection accuracy and complicates forgery attempts, offering enhanced security and computational efficiency with fewer parameters.
GraphRNN

Read it Twice: Towards Faithfully Interpretable Fact Verification by Revisiting Evidence

HGTM

MR2: A Benchmark for Multimodal Retrieval-Augmented Rumor Detection in Social Media

HGTM

CHEF: A Pilot Chinese Dataset for Evidence-Based Fact-Checking

SIGIR 2023

Xuming Hu, Zhaochen Hong, Zhijiang Guo, Lijie Wen, Philip S. Yu

ReRead is a fact verification model designed to enhance the accuracy of real-world fact verification tasks. It retrieves evidence from source documents, focusing on obtaining evidence that is both faithful (reflecting the model's decision-making process) and plausible (convincing to humans).

SIGIR 2023

Xuming Hu, Zhijiang Guo, Junzhe Chen, Lijie Wen, Philip S. Yu

MR2 is a multimodal, multilingual dataset for rumor detection, addressing the evolving nature of misinformation on social media, which increasingly intertwines text and images. It offers a platform for developing advanced rumor detection systems capable of retrieving and reasoning over internet-sourced evidence from both text and image modalities. This dataset provides a challenging testbed for evaluating such systems.

NAACL 2022

Xuming Hu, Zhijiang Guo, Guanyu Wu, Aiwei Liu, Lijie Wen, Philip S. Yu

CHEF is the first Chinese Evidence-based Fact-checking dataset, featuring 10K real-world claims across various domains like politics and public health, with annotated evidence from the Internet. It aims to address the scarcity of non-English tools in automated fact-checking, particularly for Chinese.

Structured Knowledge Extraction

MolGroup

Reading Broadly to Open Your Mind: Improving Open Relation Extraction with Search Documents under Self-supervisions

DeSCo

Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis

DeSCo

Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction

IEEE TKDE 2024

Xuming Hu , Zhaochen Hong, Chenwei Zhang, Aiwei Liu, Shiao Meng, Lijie Wen, Irwin King, Philip S. Yu

Web-SelfORE is a self-supervised framework for open-domain relation extraction, utilizing a pretrained language model to analyze web documents and extract relational features. It enhances relation classification through adaptive clustering and self-supervised signals, showing superior performance on four public datasets compared to existing baselines.

ACL 2023

Xuming Hu, Zhijiang Guo, Zhiyang Teng, Irwin King, Philip S. Yu

This research enhances multimodal relation extraction (MRE) by retrieving both textual and visual evidence from object, sentence, and image levels. A novel approach is developed to synthesize information across these levels for improved reasoning between modalities.

ACM MM 2023

Xuming Hu, Junzhe Chen, Aiwei Liu, Shiao Meng, Lijie Wen, Philip S. Yu

PROMU is a novel approach to multimodal entity and relation extraction from text, leveraging unlabeled image-caption pairs for pre-training. It proposes unique objectives for aligning entities and relations with objects in images using soft pseudo-labels, enhancing extraction capabilities.
GNNExplainer

Think Rationally about What You See: Continuous Rationale Extraction for Relation Extraction

NeuralExe

SelfLRE: Self-refining Representation Learning for Low-resource Relation Extraction

IDGNN

Scene Graph Modification as Incremental Structure Expanding

SIGIR 2023

Xuming Hu, Zhaochen Hong, Chenwei Zhang, Irwin King, Philip S. Yu

RE2 is a rationale extraction framework for enhancing relation extraction by identifying relevant content and filtering out noise in sentences. It applies continuity and sparsity principles with an optimizable binary mask for token selection, ensuring semantic coherence. Demonstrated to surpass baselines in experiments on four datasets, RE2 effectively adjusts rationales in relation to given labels.

SIGIR 2023

Xuming Hu, Junzhe Chen, Shiao Meng, Lijie Wen, Philip S. Yu

SelfLRE is a novel low-resource relation extraction architecture, blending self-training and self-ensembling learning to enhance task-specific representations on unlabeled data.

COLING 2022

Xuming Hu, Zhijiang Guo, Yu Fu, Lijie Wen, Philip S. Yu

ISE is a scene graph modification method that incrementally expands existing graphs based on natural language queries, maintaining unaltered structures. It efficiently iterates between node and edge predictions, showing significant improvements over previous models.
GraphSAGE

HiURE: Hierarchical Exemplar Contrastive Learning for Unsupervised Relation Extraction

GraphRNN

Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction

GNNExplainer

Semi-supervised Relation Extraction via Incremental Meta Self-Training

NAACL 2022

Shuliang Liu*, Xuming Hu*, Chenwei Zhang, Shu’ang Li, Lijie Wen, Philip S. Yu

HiURE is a novel contrastive learning framework for unsupervised relation extraction, overcoming common issues in existing methods. It employs cross hierarchy attention to derive hierarchical signals from relational features and uses exemplar-wise contrastive learning for optimizing sentence relation representation.

EMNLP 2021

Xuming Hu, Chenwei Zhang, Yawen Yang, Xiaohe Li, Li Lin, Lijie Wen, Philip S. Yu

GradLRE is a method developed for low-resource relation extraction (LRE), addressing the challenges of pseudo label generation and feedback loops in learning paradigms. It employs a Gradient Imitation Reinforcement Learning approach to align pseudo label data with the gradient descent of labeled data.

EMNLP 2021 (Findings)

Xuming Hu, Chenwei Zhang, Fukun Ma, Chenyao Liu, Lijie Wen, Philip S. Yu

MetaSRE is a semi-supervised relation extraction method designed to minimize reliance on large-scale annotations. It features a Relation Label Generation Network that evaluates pseudo labels through meta-learning, based on the performance of a Relation Classification Network.
GraphSAGE

SelfORE: Self-supervised Relational Feature Learning for Open Relation Extraction

EMNLP 2020 (Oral)

Xuming Hu, Chenwei Zhang, Yusong Xu, Lijie Wen, Philip S. Yu

SelfORE is a self-supervised framework for open relation extraction, utilizing a pretrained language model for adaptive clustering and bootstrapping self-supervised signals in relation classification.

Multimodal Knowledge Learning

DeSCo

Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction

DeSCo

Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis

HGTM

MR2: A Benchmark for Multimodal Retrieval-Augmented Rumor Detection in Social Media

MM 2023

Xuming Hu, Junzhe Chen, Aiwei Liu, Shiao Meng, Lijie Wen, Philip S. Yu

PROMU is a novel approach to multimodal entity and relation extraction from text, leveraging unlabeled image-caption pairs for pre-training. It proposes unique objectives for aligning entities and relations with objects in images using soft pseudo-labels, enhancing extraction capabilities.

ACL 2023

Xuming Hu, Zhaochen Hong, Zhijiang Guo, Lijie Wen, Philip S. Yu

This research enhances multimodal relation extraction (MRE) by retrieving both textual and visual evidence from object, sentence, and image levels. A novel approach is developed to synthesize information across these levels for improved reasoning between modalities.

SIGIR 2023

Xuming Hu, Chenwei Zhang, Irwin King, Philip S. Yu

MR2 is a multimodal, multilingual dataset for rumor detection, addressing the evolving nature of misinformation on social media, which increasingly intertwines text and images. It offers a platform for developing advanced rumor detection systems capable of retrieving and reasoning over internet-sourced evidence from both text and image modalities. This dataset provides a challenging testbed for evaluating such systems.

To Prospective Students

For prospective students, I appreciate reading the following before reaching out to me through email. To make it easier for me to identify the applications, use "PhD (or Research Assistant , Visiting Student) Application" as your title.

PhDs

When reaching out to me, in addition to your CV, it would be best to demonstrate the following in your email.
  • Applicants do not need to have a degree in computer science, but they should possess good coding skills and a basic understanding of natural language processing. The research direction of the applicants does not need to align with mine, as long as they have sufficient interest in large language models.
  • Prospective students are encouraged to visit our laboratory in advance to gain a better understanding of how our lab operates. This will facilitate a more informed decision based on mutual selection principles.
  • Students who have only one publication as a (co-)first author, demonstrating their ability to develop ideas, implement, analyze, and write papers, are often considered more favorably than those who have participated in numerous publications without leading these projects.

Note:

I am aware that many students without a first-author publication record are interested in applying to our lab. We welcome these students to participate in our publication-oriented projects or lead one influential open-source project. We are looking for self-motivated students. I will guide you in advancing these projects, and strong performance will significantly enhance your chances of a successful application.

Research Assistants/Visiting Students

I welcome research assistants, visiting students, and interns at all levels. Students are required to demonstrate a strong interest and good background knowledge in large language models. While prior research experience is encouraged, it is not mandatory. All positions for research assistants, visiting students, and internships can be remote. Research assistant positions will be compensated according to the applicant's background.

Contact

Location

I'm currently located at No.1 Du Xue Rd, Nansha District, Guangzhou.

Email

You could reach me via email. Show Email
I will try my best to respond if the schedule permits, unless I'm overwhelmed by emails.

Web template borrow from Rex (Zhitao) Ying