In the era of Generative AI and Large Language Models (LLMs), it is easy to default to prompting a billion-parameter model for every Natural Language Processing (NLP) task. However, when it comes to structured prediction tasks like Named Entity Recognition (NER), modern production systems often require structural constraints, low inference latency, and zero hallucination.

This is where Conditional Random Fields (CRF) shine. As a discriminative undirected graphical model, CRF offers an elegant mathematical framework to model sequential dependencies.

In this article, we will bridge the gap between theory and practice: exploring the core mathematics behind CRFs and walking through a hands-on implementation using the classic CRF++ toolkit.

---

1. Why CRF Still Matters in the Age of LLMs

Standard classification models assume that data instances are independent and identically distributed (i.i.d.). However, language is inherently sequential. In NER, predicting a token's label depends heavily on its neighbors.

While a classic Softmax layer outputs the probability of each label independently, a CRF layer models the joint probability of the entire label sequence globally.

The Trade-offs: LLMs vs. CRFs in Production

• Sequence Constraints: LLMs can fail to adhere to structural formats (e.g., generating an I-PER tag without a preceding B-PER tag in BIO tagging). CRFs enforce strict transition constraints via a learned transition matrix.
• Efficiency: A CRF-based model can process thousands of sentences per second on a single CPU core, costing a fraction of an LLM API call.
• Deterministic Boundaries: For domain-specific NER (e.g., medical or legal texts), CRFs offer explicit control over feature engineering, ensuring predictable and reliable boundaries.

---

2. The Mathematics Behind CRF

CRF is a discriminative model that directly models the conditional probability $P(\mathbf{y}|\mathbf{x})$, where $\mathbf{x}$ is the input sequence (words) and $\mathbf{y}$ is the output sequence (labels).

Given a sentence $\mathbf{x}$, the conditional probability of a label sequence $\mathbf{y}$ is defined as:

$$P(\mathbf{y}|\mathbf{x}) = \frac{1}{Z(\mathbf{x})} \exp \left( \sum_{i=1}^{n} \sum_{j} \lambda_j f_j(\mathbf{y}_{i-1}, \mathbf{y}_i, \mathbf{x}, i) \right)$$

Where:

• $f_j(\mathbf{y}_{i-1}, \mathbf{y}_i, \mathbf{x}, i)$ is a user-defined feature function that scores the combination of the current label, the previous label, and the input sequence at position $i$.
• $\lambda_j$ is the weight of the $j$-th feature function, learned during training.
• $Z(\mathbf{x})$ is the Partition Function (normalization factor) that guarantees the probabilities over all possible label sequences sum up to 1:

$$Z(\mathbf{x}) = \sum_{\mathbf{y}'} \exp \left( \sum_{i=1}^{n} \sum_{j} \lambda_j f_j(\mathbf{y}'_{i-1}, \mathbf{y}'_i, \mathbf{x}, i) \right)$$

Feature Functions: The Core Mechanism

CRF allows us to inject domain knowledge using two types of feature functions:

1. State Features (Transition from Input to State): $f(y_i, \mathbf{x}, i)$ — e.g., "If the current word $x_i$ is capitalized and ends with '-stein', how likely is $y_i$ to be 'B-PER'?"
2. Transition Features (State to State): $f(y_{i-1}, y_i, \mathbf{x}, i)$ — e.g., "How likely is a 'B-PER' tag to be followed by an 'I-PER' tag?"

---

Basic Usage of CRF++

1. Downloading the Toolkit

• Linux Version (with source code) & Windows Version: You can download them from the Official CRF++ SourceForge Page. The Windows version does not require installation; it can be used directly via the command line after extraction.

2. Installation Steps on Linux

In a Linux environment, after extracting the package and entering the directory, you need root privileges to execute the following commands in sequence:

./configure
make
su
make install

---

3. Training Corpus Format

1. Columns and Rows: The corpus must contain at least two columns. Columns are separated by spaces or tabs. Every row (except for empty lines) must have the exact same number of columns.
2. Sentence Separation: Sentences are separated by an empty line.
3. Example (with two columns of features):

太 Sd N
短 Sa N
而 Bu N
已 Eu N
。 Sw N

---

4. Feature Selection and Template Writing

CRF++ locates features using relative positions in the format of %x[row, col] (both row and column indices start from 0).

1. Feature Positioning Example

Suppose the current row is the row for "京" (Jing) in "北京市" (Beijing City):

“  Sw  N
北  Bns B-LOC
京  Mns I-LOC  <-- Current Row (0)
市  Ens I-LOC
首  Bn  N

• %x[-1,0] represents the 1st column of the previous row, which is "北".
• %x[0,1] represents the 2nd column of the current row, which is "Mns".
• %x[-1,0]/%x[0,0] represents the combination of the 1st column of the previous row and the current row, which is "北/京".

2. Creating Templates

Templates are mainly divided into Unigram (templates starting with U) and Bigram (templates starting with B). Note that "Uni/Bi" here refers to the combination of output tags, not the features themselves.

• Template File Example:

# Unigram
U00:%x[-2,0]
U01:%x[-1,0]
U02:%x[0,0]

Note: Rows starting with # are comments and will be ignored by the system.

---

5. Training and Decoding Commands

1. Model Training

Use the crf_learn command to train your model:

crf_learn <template_file> <training_corpus> <generated_model_file>

• Meanings of Training Output Parameters:
• iter: The current iteration number.
• terr: Tag Error Rate.
• serr: Sentence Error Rate.
• obj: The current value of the objective function. The training is complete when this value converges.

2. Model Prediction / Decoding

Use the crf_test command for prediction, and you can use the > redirect operator to save the results to a file:

crf_test -m <model_file> <test_file> > <output_path>

*Example: crf_test -m model test.txt > result.txt*

---

6. Using the CoNLL 2000 Evaluation Tool

You can use the CoNLL 2000 script to evaluate the model's Precision, Recall, and F1-score.

• Data Requirements: The test file needs to include the gold-standard answers. After decoding with crf_test, the predicted results will be appended as the last column. The evaluation tool will then compare the second-to-last column (the answer) with the last column (the prediction).
• Running Command:

perl conlleval.pl < <evaluation_file>

Note: Before using this evaluation tool, you must convert all tabs in the evaluation file into spaces, otherwise the tool may throw an error.

---

Named Entity Recognition (NER) Using Conditional Random Fields (CRF)

I. Experimental Environment

• a) Software: Windows XP Pro SP3, Visual Studio 2008 & .NET 2005 (Dotnet2.0), CRF++, Perl
• b) Hardware: CPU: CM420, RAM: 2GB DDR533, HDD: 160GB 8M SATA Fujitsu

---

II. Experimental Process

Unless specified otherwise, the following results are obtained by splitting the provided training corpus into a 7:3 ratio for training and evaluation according to the assignment requirements.

a) Direct Application of CRF

The format of the provided corpus perfectly matches the requirements of Conditional Random Fields, so the CRF model is applied directly for training and testing. (The files for this experiment are in the test1.rar package).

1. Convert document encoding to UTF-8 (CRF++ throws an error when using UTF-16).
2. Define the template as follows:

# Unigram
U00:%x[-2,0]
U01:%x[-1,0]
U02:%x[0,0]
U03:%x[1,0]
U04:%x[2,0]
U10:%x[-1,0]/%x[0,0]
U11:%x[0,0]/%x[1,0]

3. Train and learn features using CRF++ (Relevant information below):

• Command: crf_learn template_file train_file model
• Where template_file is the template file and train_file is the training corpus (both need to be prepared in advance); model is the file generated by CRF++ based on the template and training corpus, which is used for decoding.

i. The template_file Format

1. The basic format of a template is %x[row,col], which is used to specify a token in the input data.

• row determines the relative row offset from the current token.
• col determines the absolute column index. (Refer to the layout below)

	col 0	col 1	col 2
row -2	疆 (Jiang)	Ens	I-LOC
row -1	总 (Zong)	Bn	N
row 0	统 (Tong)	En	N	Current Row
row 1	阿 (A)	Bns	B-PER
row 2	利 (Li)	Mns	I-PER

Template	Represented Feature
U00:%x[-2,0]	疆
U01:%x[-1,0]	总
U02:%x[0,0]	统
U03:%x[1,0]	阿
U04:%x[2,0]	利
U10:%x[-1,0]/%x[0,0]	总/统
U11:%x[0,0]/%x[1,0]	统/阿

2. Types of Feature Templates

• a) Unigram Template: Starts with the letter U. When a template is prefixed with U, CRF++ automatically generates a set of feature functions. The total number of feature functions generated by a model is $L \times N$, where $L$ is the number of output classes and $N$ is the number of unique strings expanded based on the given template.

• b) Bigram Template: Starts with the letter B. It is used to describe bigram features. The system will automatically generate combinations of the current output token and the previous output token. The total number of distinct features generated is $L \times L \times N$, where $L$ is the number of output classes and $N$ is the number of unique features produced by this template.

• c) Difference Between the Two Templates: Note that Unigram/Bigram refers to the Unigram/Bigrams of the output tokens, not the features!

• Unigram: $\lvert\text{output tag}\rvert \times \lvert\text{all possible strings expanded from the template}\rvert$

• Bigram: $\lvert\text{output tag}\rvert \times \lvert\text{output tag}\rvert \times \lvert\text{all possible strings expanded from the template}\rvert$

• b) Training Log Sample: iter=88 terr=0.01365 serr=0.23876 obj=67066.17413 diff=0.00006 Where: iter is the number of iterations; terr is the token error rate; serr is the sentence error rate; obj is the current objective value (training terminates when it converges); diff is the relative change from the previous objective value.

4. Done! 2706.41 s (Execution time on Computer 1).
5. Testing on the Test Corpus:

• a) Command: crf_test -m model_file test_file > result_file Where model_file is the generated model file, test_file is the corpus to be tested, and > result_file is the redirection statement to output the screen stream directly into result_file.
• b) The decoding speed of CRF++ is very fast, especially when writing directly to a file. However, due to feature selection issues, the precision and recall rates are not high.
• c) The results are evaluated using the conlleval.pl script (the code is located in the root directory of the submission package). The evaluation command is: perl conlleval.pl < output.txt, where output.txt is the file to be evaluated. A Perl interpreter is required. The detailed results are as follows:

Entity	Precision	Recall	FB1	Tokens Count
LOC	63.67%	72.93%	67.98	5623	382251.5
ORG	21.26%	35.90%	26.71	4491	119954.6
PER	65.90%	65.06%	65.47	2554	167210.4
Macro Average	53.39%	Micro Average	52.84%

ii. Expanding the Feature Set

Since very few features were selected previously, it was hypothesized that incorporating more valid features would improve performance. Thus, the template was updated as follows (relevant data files for this experiment are in the test2.rar package):

1. Template 2:

# Unigram
U00:%x[-2,0]
U01:%x[-1,0]
U02:%x[0,0]
U03:%x[1,0]
U04:%x[2,0]
U5:%x[-2,0]/%x[-1,0]
U6:%x[-1,0]/%x[0,0]
U7:%x[0,0]/%x[1,0]
U8:%x[1,0]/%x[2,0]

2. Experimental Data:

• a) Training Process: iter=94 terr=0.00571 serr=0.12313 obj=53321.45523 diff=0.00000 Done! 2915.53 s
• b) Test Results:

Entity	Precision	Recall	FB1	Tokens Count
LOC	66.86%	74.31%	70.39	5456	384047.8
ORG	26.95%	41.02%	32.53	4048	131681.4
PER	68.29%	65.67%	66.96	2488	166596.5
Macro Average	56.63%	Micro Average	56.90%

Analysis: While there is noticeable improvement, the scores remain significantly low.

---

b) Rule-Based Post-Processing for Optimization

i. Error Analysis

By analyzing the errors (detailed in the files starting with error in each package), the main errors can be categorized into the following types:

1. When characters within the same predicted entity have conflicting types, the type with the higher character frequency wins. If the counts are equal, it defaults to LOC in most cases.
2. The starting character of an entity must follow the B-??? format.
3. The boundary tokens (start and end) of entities follow specific patterns (e.g., delimited by stop words, verbs, etc.).
4. Words directly following certain fixed entities should be in the B-??? format (e.g., after province names).
5. Entities with tiny gaps between them might be merged into a single entity.
6. ...etc.

ii. Optimization Results

Based on these characteristics, I planned to test each rule sequentially to optimize the results. Due to time constraints, only four or five rules were evaluated. The first two rules (Rule 1 and Rule 2) proved to be the most effective; combining them improved the performance by about 12%. Applying these corrections to the test2 outputs yielded:

Entity	Precision	Recall	FB1	Tokens Count
LOC	79.40%	76.43%	77.89	4966	386801.7
ORG	53.86%	52.63%	53.24	3457	184050.7
PER	80.88%	67.09%	73.34	2327	170662.2
Macro Average	68.16%	Micro Average	68.98%

Analysis: Although the F-score ($FB1$) increased dramatically, the overall performance is still not ideal.

---

c) Word Segmentation and POS Tagging Prior to CRF Learning

i. Intent

It became clear that focusing solely on character-level features was insufficient. Therefore, I attempted to leverage word segmentation and Part-of-Speech (POS) tagging information. Since the original task did not provide this data, a tool was used to segment and tag the text first (the segmentation tool can be found in the root directory of the attachment package).

ii. Feature Representation

After word segmentation and tagging, the character features are structured as follows:

Character	POS & Segmentation Tag	Entity Label
：	Sw	N
印	Bns	B-LOC
度	Ens	I-LOC
首	Bd	N
先	Ed	N

iii. Template Customization

A new template was established specifically targeting these multi-column features.

iv. Training and Testing

Using the new template for training, the model was decoded and evaluated via conlleval, yielding the following results: iter=226 terr=0.00935 serr=0.17661 act=2913330 obj=42785.69115 diff=0.00009 Done! 4502.97 s

Entity	Precision	Recall	FB1	Tokens Count
LOC	82.05%	89.97%	85.83	20309	1743121
ORG	48.36%	65.12%	55.50	13818	766899
PER	91.52%	93.15%	92.33	9189	848420.4
Macro Average	77.89%	Micro Average	77.53%

v. Further Rule Optimization

Applying the previously built post-processing rules to these new results brought the final performance to:

Entity	Precision	Recall	FB1	Tokens Count
LOC	90.34%	90.37%	90.36	18878	1705816
ORG	70.47%	71.54%	71.00	12474	885654
PER	94.85%	92.70%	93.76	8954	839527
Macro Average	85.04%	Micro Average	85.12%

Based on this optimal setup, the model was trained on Test_utf16.ner to finally generate finalAnswer.txt.

---

III. Experimental Results Comparison Table

ID	Strategy Used	Result (F-Score)	Method Improvement	Performance Gain	Notes
1	Character-based CRF (1)	~53%	-	-
2	Character-based CRF (2)	~56.7%	Used richer feature context.	~3.7%	Features strongly impact the outcome, but due to hardware and time limits, more features couldn't be added to verify.
3	Character CRF + Rules	~68.5%	Manually added rules for post-processing.	~11.8%	Rules successfully compensate for machine learning limits. Tried various rules (and altered execution order).
4	Segmentation + POS + CRF	~77.7%	Paradigm shift in feature representation.	~9.2%	Introducing the concept of "words" is clearly effective.
5	Segmentation + POS + CRF + Rules	~85.1%	Introduced rules on top of strategy 4.	~7.4%	Certain drawbacks of ML methods do not change regardless of condition changes.

---

IV. Future Work

• a) Explore additional rules to minimize the inherent flaws of pure machine learning methods.
• b) Try treating word segmentation and POS tagging as completely separate attributes to observe their distinct impacts on the results.
• c) Improve the accuracy of the baseline word segmentation and POS tagging tools to achieve better downstream NER performance.

---

V. Key Precautions

• a) Encoding formats can prevent certain files from being processed correctly; stay alert to formatting errors if crashes occur.
• b) Different programs require different delimiters (mostly spaces vs. tabs). Pay close attention to whether your file delimiters meet the program specifications.
• c) The small utility scripts developed during the experiment do not include user manuals, but their interfaces are simple and clean, making them easy to master.

Developed Tools Inventory

• Felomeng.BackFormation: Converts between the standard corpus format and the word segmentation/tagging format. It also includes functions to merge two types of tags or delete segmentation info.
• Felomeng.ErrorExtractor: An error extraction tool that pulls errors from output files (containing ground truth labels) to facilitate experimental analysis.
• Felomeng.NERRules: Originally featured four functions. Since the first three proved ineffective during testing, its primary function now is to optimize output via rule-based corrections on top of machine learning predictions.

---

Postscript: In reality, the final performance is heavily dependent on how the training and testing datasets are partitioned. I adopted a strict split of the first 70% for training and the remaining 30% for testing. By subsequently refining the data selection methodology, the accuracy can surpass 92%. Anyone interested is encouraged to experiment with different ways of extracting the training and testing corpora.