© Wuhan University 2024

0 Introduction

Named Entity Recognition (NER) stands as a cornerstone in natural language processing and represents a fundamental undertaking. The principal objective revolves around the discernment of entity spans nestled within sentences and their subsequent categorization into precise classes. These classes encompass a spectrum of designations, notably encompassing but not limited to Person, Organization, and Location. As a traditional sequence labeling task, NER provides essential technical support for downstream applications such as information extraction, knowledge graphs, and text summarization.

The NER task has undergone several significant evolutions since its inception. In the early stages, the rule-based and dictionary-based approaches gained considerable traction. This method relies too heavily on domain experts to formulate rules and templates that may must be revised when dealing with complex linguistic expressions and diverse inputs. As machine learning has progressed, statistically based methods have emerged. As a quintessential statistical approach, conditional random fields ^[1] have demonstrated the capability to address intricate sequence annotation tasks by modeling the interdependencies among markers. However, despite this progress, these methods must be improved when effectively identifying intricate patterns. As the volume of trainable data grows and computer arithmetic capabilities improve, existing approaches have yielded promising results through deep learning. The researchers constructed the model using a more complex network structure, significantly improving model performance. In addition, existing supervised and unsupervised methods rely too heavily on the amount of annotated data. However, in real-world scenarios, NER systems frequently encounter the need to rapidly adapt to new entity types not encountered during training. This adaptation is typically accomplished through fine-tuning the original model, thereby enabling the system to perform effectively in the new domain.

Researchers have proposed few-shot learning to establish innovative concepts with a limited number of instances. In this approach, the model is initially trained within a richly-resource domain and transposed to a scarce-resource domain for specific missions. The model must quickly adapt to the data distribution within the target domain, relying on a sparse set of annotated data. Currently, few-shot learning is typically trained using the N-way K-shot pattern, where N represents the number of classes, and K represents the number of samples per class. Figure 1 illustrates an example of 2-way 1-shot instances in the target domain. Two samples of the target domain with labels, each containing only one entity type, were given. The objective is to recognize entities within the query example.

Fig. 1 A 2-way 1-shot example in the target domain

Currently, few-shot NER methods can be broadly categorized into two main types. One-stage methods classify individual words in a sentence directly by analyzing the feature distribution of the constructed classes. Fritzler et al^[2] represented class prototypes by averaging tokens with the same label and categorizing them based on the distance from the prototype. Yang et al^[3] utilized the transfer matrix instead of retraining the conditional random fields (CRF) model of the target domain and classified it by the k nearest neighbor (kNN) algorithm. Figure 2 shows the traditional method based on kNN. The value of k significantly influences the classification decision. Das et al^[4] optimized the distribution distance between tokens of the same category through contrastive learning and utilized Gaussian distribution embeddings to differentiate labeling categories. Unlike the one-stage approaches, the two-stage method places more emphasis on the recognition of entity spans, and most of this work is based on a prototypical network^[5]. They assume that each entity type belongs to a prototype for training and uses the kNN method for classification. Wang et al ^[6] formulated the classification problem as a span-level matching problem and decomposed it into a series of span processes. Ma et al ^[7] utilized meta-learning to train the span detector, aiming to discover a universal parameter initialization that can swiftly adapt to new entity classes. Wang et al^[8] introduced a global boundary matrix and adjusted span representations through prototypical learning. Li et al^[9] take different combinations of type names and support samples as contrast and use type-aware filtering strategies to remove spans that are far from the target domain.

Fig. 2 The traditional classification method based on kNN

Despite remarkable advancements, current methods continue to grapple with challenges when confronted with few-shot NER. First, as with other sequence labeling issues, entity categories can be notably influenced by neighboring words, culminating in what is widely acknowledged as the short-range dependency issue. In practical terms, entity tokens seldom appear in isolation but manifest consecutively. There are also smoothing-based ^[10] methods used to address model overconfidence by spreading the probability of the span matrix over the span of neighboring entities. In pursuit of new solutions, we intend to capture the spatial intricacies of the boundary matrix via a multiscale convolutional approach and assign different convolutional kernel weights according to the actual situation. Subsequently, we aim to merge the results before and after convolution using a residual network^[11]. This strategy is devised to discern and delineate a greater number of neighboring entities, thus enhancing the model's accuracy in NER. Referring to previous models, the selection of prototypes typically involves averaging samples belonging to the same class, assuming equal contribution from all samples to the prototype. However, in practical scenarios, different sample points contribute to the prototype to varying extents, thus requiring the allocation of distinct weights to each sample point.

In summary, in this work, we design a two-stage framework. In the first stage, we pass through a biaffine layer to generate the entity boundary matrix, which aids in determining the position of the entities in the sentence. To extract span matrix spatial features, we use multiscale convolution to construct the spatial relations of the fractional matrix, and the label smoothing effect is also achieved, which can better identify nested entities. In the second stage, we improve the prototypical network and assign different weights to different samples based on the KL divergence between distributions.

Our contributions can be summarized as follows:

1) We propose a novel, robust framework to tackle the problem of NER in resource-constrained scenarios.

2) We utilize multiscale convolution for feature extraction on the spatial dimension of the bounding matrix and use a weighted prototypical network for categorizing.

3) The experimental results validate the framework's effectiveness in few-shot settings. Compared with the benchmark models, the F1 score of our framework shows a good improvement in different settings.

1 Related Work

1.1 Meta-Learning

Researchers have proposed the concept of few-shot learning to drive the application of machine learning in scenarios with extremely scarce sample data ^[12]. Meta-learning, a popular paradigm for few-shot learning, aims to discover an optimal set of parameters that enable the model to rapidly adapt to new tasks. Finn et al^[13] redefined the gradient descent algorithm and designed a model-agnostic meta-learner. Li et al^[14] concurrently trained initial parameters update direction and step size based on the foundation of model-agnostic meta-learning (MAML). Jiang et al^[15] introduced an attention-based meta-learning approach for unknown tasks and applied it to the field of NLP. Subsequently, meta-learning has been widely applied to address problems with limited data, such as machine translation^[16,17] and text classification^[18-20].

1.2 Few-Shot NER

Hou et al^[21] introduced a collapsed dependency transfer mechanism into CRF to transfer abstract label dependency patterns as transition scores. Ji et al^[22] constructed a dispersed and distributed prototype-enhanced entity-level prototypical network. Chen et al^[23] employed limited labeled samples for class-incremental learning and generated synthetic data for pre-existing classes using a source domain model. Wang et al^[24] transformed data representation from a high-resource to a low-resource domain through data augmentation ^[25]. Zhou et al^[26] utilized the high-quality augmented data generated by the model to provide rich knowledge of entity regularities. Zhang et al^[27] utilized prompt templates containing entity category information to construct labeling prototypes, enhancing the model's suitability for migration.

2 Method

Figure 3 illustrates the framework diagram of our approach. The model is first trained to generate a span matrix on the support set and then classified it using class prototypes. We first introduce the preliminaries. Then, we discuss how to obtain a boundary matrix with multiscale convolution and use a weight prototypical network to classification.

Fig.3 The framework of our proposed

2.1 Preliminaries

In this stage, we formulate a few-shot named entity recognition as a span-based sequence labeling task. Given an input sequence $X=\{x_i{\}}_{i=\mathrm{1}}^L$ of length L, we aim to identify all entity spans $M=\{{\left(s_j,e_j\right)}_{j=\mathrm{1}}^{L^{\text{'}}}\}$ and classify them into corresponding labels $Y=\{y_t{\}}_{t=\mathrm{1}}^n$, where, $x_i$ is the $i$-th token, $s_j/e_j$ denotes the start$/$end position for the $j$-th span, $L^{\text{'}}$ is the number of spans in the sentence, and $y_t$ is the $t$-th entity type in the label set $Y$. We use standard N-way K-shot settings and divide data in the source domain as training episodes ${\epsilon }_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}=\left\{(S_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}},Q_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}},T_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}})\right\}$, where $S_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}=(X_s,M_s,Y_s)$ denotes the support set, $Q_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}=(X_Q,M_Q,Y_Q)$ denotes the query set, and $T_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}=Y_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}\bigcup O$ is the corresponding type set. We use a similar method to construct the target domain data for the testing process to validate the model's performance on the novel domain. Given some novel episodes ${\epsilon }_{\mathrm{n}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}}=\left\{(S_{\mathrm{n}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}},Q_{\mathrm{n}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}},T_{\mathrm{n}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}})\right\}$, where $S_{\mathrm{n}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}},Q_{\mathrm{n}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}}$ represent the support and query sets in the novel domain, $T_{\mathrm{n}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}}$is the novel-type set. We expect to use a few support sets $S_{\mathrm{n}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}}$ to fine-tune the model and make predictions on the query set $Q_{\mathrm{n}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}}$. In general, $T_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}\bigcap T_{\mathrm{n}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{l}}=\mathrm{\varnothing }$.

2.2 Entity Span Extractor

As a classic two-stage approach, we only extract all candidate entity spans from the sentences without classifying them in this stage. Given an input sequence $X=\{x_i{\}}_{i=\mathrm{1}}^L$ from the support set $S_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$, we first utilize a pre-trained model to encode the input tokens into well-initialized embeddings $H=\{h_i{\}}_{i=\mathrm{1}}^L$.

$[h_{\mathrm{1}},h_{\mathrm{2}},\cdots ,h_L]=\mathrm{P}\mathrm{L}\mathrm{M}([x_{\mathrm{1}},x_{\mathrm{2}},\cdots ,x_L])$(1)

where $H\in {\mathbb{R}}^{L\times h^{\text{'}}}$ denotes the hidden layer output of pre-trained encoder, $h\text{'}$ denotes the hidden size. After obtaining the contextual representation, we use two separate feedforward neural networks to create different representations $h_j^s/h_j^e$ for the start$/$end positions of the $j$-th span and then adopt a Biaffine Layer ^[10,28] to get the predicted score matrix:

$P_x=h_j^s{W}_a{h}_j^e+W_b(h_j^s\oplus h_j^e)+b_m$(2)

where, $W_a,W_b$ are the trainable parameters, $b_m$ denotes the bias.

Considering that the labels in the support set are visible, we use a global boundary matrix to represent the ground truth of the training process.

${\mathbf{\Omega }}_{s_j,e_j}=\{\begin{array}{cc}\mathrm{1},& s_j\le e_j\wedge (s_j,e_j)\in M\\ \mathrm{0},& s_j\le e_j\wedge (s_j,e_j)\notin M\\ -\mathrm{i}\mathrm{n}\mathrm{f},& s_j>e_j\end{array}$(3)

where $s_j/e_j$ denotes the start/end position for the $j$-th span, ${\mathbf{\Omega }}_{s_j,e_j}$ is the score of the span ($s_j,e_j$), $M$ denotes the spans in a sentence that belong to entity types.

Since neighboring cells in the span matrix affect each other, so we use CNN with three-dimensional convolutional kernels for spatial modeling. Considering the effect of distance on span labels, we assign different weighting factors to these convolutions.

$C_{x\mathrm{1}}=\mathrm{G}\mathrm{e}\mathrm{L}\mathrm{U}(\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{\nu }\mathrm{2}\mathrm{d}(P_x)))$(4)

$C_x=C_{x\mathrm{1}}{\lambda }_{\mathrm{1}}+C_{x\mathrm{2}}{\lambda }_{\mathrm{2}}+C_{x\mathrm{3}}{\lambda }_{\mathrm{3}}$(5)

where $\{{\lambda }_{\mathrm{1}},{\lambda }_{\mathrm{2}},{\lambda }_{\mathrm{3}}\}$ represent the proportion of results for three different scales of convolution,$C_x$ indicates the final summed result.

Considering that most of the words in the sentence belong to nonentities and the categories are imbalanced among other entities, we followed Wang et al^[8] to use the span-based cross-entropy loss function to constrain the boundary information on each training support set. The aim is to encourage the model to be more focused on hard-to-classify samples during training by reducing the weights of easy-to-classify samples.

$L_{\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{n}}=\mathrm{l}\mathrm{o}\mathrm{g}(\mathrm{1}+\sum _{\mathrm{1}\le e_j\le s_j\le L}\mathrm{e}\mathrm{x}\mathrm{p}({(-\mathrm{1})}^{{\mathit{\Omega }}_{s_j,e_j}}(P_x+C_x)))$(6)

3.3 Span Classification

In this phase, our objective is to categorize the spans generated earlier. The traditional prototypical network averages all samples belonging to the same category to obtain the prototype^{[5, 29]}. Considering that different sample points have different degrees of contribution to the class prototype, we design a new prototypical calculation method. Given a test set $S=\{s_t{\}}_{t=\mathrm{1}}^T$, where $s_t$ represents a collection of all samples belonging to the same class, $x_i$ represents one of the samples. We measure the difference in distribution between samples $x_i$ and the $s_t$ using KL divergence, where the weight of sample x can be measured by the distribution changes when the sample is not present in the test set.

$D_{\mathrm{K}\mathrm{L}}(x_i)=D_{\mathrm{K}\mathrm{L}}[s_t|\left|s_t-x_i\right|]$(7)

We use the KL divergence as the weight of the sample x. When the KL divergence between all samples $s_t$ and the sample distribution without $x_i$ is smaller, it proves that the sample point contributes less to the prototype, and the corresponding weight is smaller.

$W\left(x_i\right)=D_{\mathrm{K}\mathrm{L}}\left(x_i\right)$(8)

Class prototypes can be calculated by the product of weights and sample points as follows:

$c_k=\frac{{\displaystyle \sum _{i=\mathrm{1}}^{|s_k|}}W(x_i)f_{\varphi }(x_i)}{{\displaystyle \sum _{i=\mathrm{1}}^{|s_k|}}W(x_i)}$(9)

where $c_k$ represents the prototype of class $k$, and $f_{\varphi }(x_i)$ describes the sample features mapped to a high-dimensional space.

Finally, we optimize the model by the cross-entropy loss function.

$L_{\mathrm{d}\mathrm{i}\mathrm{s}}=-\mathrm{l}\mathrm{o}\mathrm{g}\frac{\mathrm{1}}{T}{\displaystyle \sum _{i=\mathrm{1}}^T}\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(-d\left(f_{\varphi }\left(\widehat{x}\right),c_k\right)\right)}{{\sum }_k \mathrm{e}\mathrm{x}\mathrm{p}\left(-d\left(f_{\varphi }\left(\widehat{x}\right),c_k\right)\right)}$(10)

where $\widehat{x}$ indicates a new sample to be tested.

3 Experiments

In this section, we present a comparison of our method with the existing few-shot NER framework. Detailed descriptions of the training settings and the final results are provided in the subsequent sections.

3.1 Settings

1) Datasets

To evaluate the generalization effect of the model in different domains, we conduct experiments on several public NER datasets, and split them into two groups. Table 1 presents the summary statistics of the datasets.

Few-NERD^[30] is a novel NER dataset created using data from Wikipedia and designed for few-shot learning scenarios. Unlike previous datasets, it is annotated with a hierarchy of 8 coarse-grained and 66 fine-grained entity types. To validate the impact at different entity granularities, the researchers further divided the data into two categories, i.e., Inter and Intra.

Cross-NER contains four datasets from different fields, including the CoNLL-03 ^[31] dataset from the news domain, the WNUT-17 ^[32] dataset from the social domain, the OntoNotes ^[33] dataset from the general domain, and the GUM ^[34] dataset from the Wiki domain.

2) Hyperparameters

We used the BERT-base ^[35] as the backbone encoder to initialize the word vector. The Baffine decoder with the affine layers of hidden size 150 and dropout rate 0.2. The learning rate was searched between 2E-5 and 5E-6 on the randomly initialized weights. We chose AdamW ^[36] as our optimizer with a linear warm-up in the first 10% steps and a weight decay of 0.1. The batch size is set to 8, and the max sequence length is set to 128. We have chosen {3,5,7} as the convolution kernel size of the boundary matrix, and the corresponding weights of the three types of convolutions are {0.6,0.3,0.1}. We chose PyTorch as our development environment with version 1.8, and the model was trained on an RTX 3090 GPU.

3) Baselines

We compared existing competitive few-shot NER models, such as ProtoBERT^[5], Matching Network^[37], StructShot and NNShot^[3], ESD^[6], CONTaiNER^[4], L-TapNet+CDT^[21], DecomMeta^[7], SpanProto^[8], and TadNER^[9].

3.2 Main Results

Table 2 compares our model with other baseline models on the Few-NERD dataset.

1) Our model significantly outperforms TadNER in both Inter and Intra tasks. Notably, the performance in the Inter task surpasses that in the Intra task, indicating that Few-shot NER presents more significant challenges under coarse-grained conditions.

2) Across all experimental results, the performance of 1-2 shots are worse than that of 5-10 shots, mainly because fewer samples are more accessible to selection bias. The model will show a good classification effect when the selected sample points are closer to the real class prototype. However, this uncertainty of the sample point makes it difficult for the model to find that point in most cases.

3) All span-based methods outperform token-based methods in our experiments.

Table 3 displays the model's performance on Cross-NER. The results indicate that our model also performs well in cross-domain data and exhibits a 1.35% and 1.48% improvement compared to the baseline. This underscores the strong adaptability of our approach.

Figure 4 shows the impact of the number of fine-tuning steps on the F1 score. It can be observed that the model already performed well without fine-tuning. As the fine-tuning steps increase, the model's performance continues to improve, which indicates that our model has strong domain transfer capabilities.

Fig. 4 The effectiveness of fine-tuning

3.3 Ablation Study

To verify the role of each module in the model, we design the following ablation experiment.

1) w/o. Multiscale Convolution, where we remove the multidimensional convolution module and directly use the span range matrix generated by the biaffine module for subsequent work.

2) w/o. Entity Span Extractor, where we do not extract entity spans but employ a traditional token-based prototypical network to train the model.

3) w/o. WeightProto Learning, where we use the kNN algorithm to classify the candidate span.

As depicted in Table 4, each component positively contributes to the model's performance. The removal of the multiscale Convolution module leads to a 2.57% decrease in the model's F1 score, underscoring the significance of spatial characterization within the boundary matrix. Furthermore, the span-based model surpasses the token-based approach in terms of efficacy, aligning with the comparative effectiveness observed across various domains. Finally, we opted for a weight-based prototype model. During the initialization phase, we embed instances randomly and assign different weights to multiple instances through model training. Experimental results demonstrate that our approach yields promising outcomes.

3.4 Visualization

Considering that the above experimental results cannot visualize the distribution of each entity class after model training, we use t-distributed Stochastic Neighbor Embedding (t-SNE)^[38] to downsize the high-dimensional vectors. It is evident from Fig. 5 that our method makes the distribution of spans belonging to the same entity class more concentrated and the class spacing clearer, which also reflects the superiority of our framework.

Fig. 5 t-SNE visualization of our framework on the Few-NERD dataset with 5-way 5-10-shot settings

4 Conclusion

We have introduced a comprehensive framework with the aim of addressing the challenge of identifying a limited set of named entities within a particular domain. Our empirical evaluations indicate that the two-stage methodology demonstrates superior performance compared to prevailing one-stage techniques. To thoroughly explore the spatial correlations among neighboring spans, we employ a multiscale convolution mechanism to facilitate the rationalization of spatial information within the entity span matrix. This information is subsequently integrated with the original data through a residual module, thereby enhancing the model's capacity to discern short-range dependencies. Considering that different samples have different degrees of contribution to the prototype, we propose an improved prototype calculation method to measure the importance of each sample by the KL divergence of the sample distribution. Extensive experimentation validates the efficacy of our proposed method by substantially outperforming the baseline.

References

All Tables

All Figures

	Fig. 1 A 2-way 1-shot example in the target domain
In the text

	Fig. 2 The traditional classification method based on kNN
In the text

	Fig.3 The framework of our proposed
In the text

	Fig. 4 The effectiveness of fine-tuning
In the text

	Fig. 5 t-SNE visualization of our framework on the Few-NERD dataset with 5-way 5-10-shot settings
In the text