Add chunk eval op #5016

guoshengCS · 2017-10-23T13:37:06Z

resolves #4749

… add-ChunkEvalOp

qingqing01 · 2017-10-30T05:28:17Z

paddle/operators/chunk_eval_op.cc

+    ctx->SetOutputDim("F1-Score", {1});
+  }
+
+  framework::DataType IndicateDataType(


IndicateDataType is a protected member function.

protected: framework::DataType IndicateDataType(...) { }

qingqing01 · 2017-10-30T08:53:20Z

paddle/operators/chunk_eval_op.h

+      tag_single = -1;
+    } else {
+      PADDLE_THROW("Unknown chunk scheme.");
+    }


Do we need to define a struct for these arguments and put these arguments initialization code to another member function？

qingqing01 · 2017-10-30T09:03:11Z

paddle/operators/chunk_eval_op.cc

+Chunk evaluator is used to evaluate segment labelling accuracy for a
+sequence. It calculates precision, recall and F1 scores for the chunk detection.
+To use chunk evaluator, several concepts need to be clarified firstly.
+[Chunk type] is the type of the whole chunk and a chunk consists of one or several words.  (For example in NER, ORG for organization name, PER for person name etc.)


Add an empty line before line 81 and 82.

Give the full name for the NER.

Rewrite the doc.

qingqing01 · 2017-10-30T09:08:23Z

paddle/operators/chunk_eval_op.cc

+    IOBES    Four labels for chunk type X, B-X for chunk begining, I-X for chunk inside, E-X for chunk end and S-X for single word chunk.
+
+To make it clear, let's illustrate by an NER example.
+Assuming that there are three named entity types including ORG, PER and LOC which are called 'chunk type' here,


Explain the LOC here?

Rewrite the doc.

qingqing01 · 2017-10-30T09:11:38Z

paddle/operators/chunk_eval_op.cc

+
+    tagType = label % numTagType
+    chunkType = label / numTagType
+    otherChunkType = numChunkTypes


The numTagType and numChunkTypes here is clear, but better to explain them again.

Rewrite the doc.

qingqing01 · 2017-10-30T09:20:52Z

paddle/operators/chunk_eval_op.h

+                 tag_end, tag_single, excluded_chunk_types);
+    }
+    *precision_data =
+        !num_output_segments ? 0 : (T)num_correct / num_output_segments;


(T) num_correct -> static_cast<T>

wanghaoshuang · 2017-10-31T10:50:33Z

paddle/operators/chunk_eval_op.cc

+Chunk evaluator is used to evaluate segment labelling accuracy for a
+sequence. It calculates precision, recall and F1 scores for the chunk detection.
+To use chunk evaluator, several concepts need to be clarified firstly.
+[Chunk type] is the type of the whole chunk and a chunk consists of one or several words.  (For example in NER, ORG for organization name, PER for person name etc.)


It's necessary that we explain meaning of 'chunk' before 'chunk type'
[chunk] is a subset of the tokens in a sentence. a yellow dog is a chunk of sentence I have a yellow dog.. And chunk of sentence can be noun phrase, person name, organization name and so on.

Rewrite the doc.

wanghaoshuang · 2017-10-31T10:52:26Z

paddle/operators/chunk_eval_op.cc

+Chunk evaluator is used to evaluate segment labelling accuracy for a
+sequence. It calculates precision, recall and F1 scores for the chunk detection.
+To use chunk evaluator, several concepts need to be clarified firstly.
+[Chunk type] is the type of the whole chunk and a chunk consists of one or several words.  (For example in NER, ORG for organization name, PER for person name etc.)


the whole chunk -> a chunk?

Rewrite the doc.

wanghaoshuang · 2017-10-31T10:57:17Z

paddle/operators/chunk_eval_op.cc

+sequence. It calculates precision, recall and F1 scores for the chunk detection.
+To use chunk evaluator, several concepts need to be clarified firstly.
+[Chunk type] is the type of the whole chunk and a chunk consists of one or several words.  (For example in NER, ORG for organization name, PER for person name etc.)
+[Tag type] indicates the position of a word in a chunk. (B for begin, I for inside, E for end, S for single)


O for outside

Rewrite the doc.

wanghaoshuang · 2017-10-31T11:13:11Z

paddle/operators/chunk_eval_op.cc

+"IOB" so tagType has two values: 0 for B and 1 for I.
+Here we will use I-LOC to explain the above mapping rules in detail.
+For I-LOC, the label id is 5, so we can get tagType=1 and chunkType=2, which means I-LOC is a part of NER chunk LOC
+and the tag is I.


How about giving an example here?

Steven B-PER 2
Paul I-PER 3
Jobs I-PER 3
works O 6
for O 6
Baidu B-ORG 0
Inc. I-ORG 1
at O 6
Beijing B-LOC 4
of I-LOC 5
China I-LOC 5

Rewrite the doc.

wanghaoshuang · 2017-11-01T01:57:34Z

paddle/operators/chunk_eval_op.h

+
+  void EvalOneSeq(const int* output, const int* label, int length,
+                  std::vector<Segment>& output_segments,
+                  std::vector<Segment>& label_segments,


output_segments and label_segments are not used outside of EvalOneSeq. So why not difine them in EvalOneSeq and remove them from arguments list?

lcy-seso

The codes in this PR LGTM (from the original chunk evaluator). But the documentation needs to refine. I think we can merge the codes and ask someone who is familiar with sequence tagging task and good at English writing for help to refine the doc.

lcy-seso · 2017-11-01T02:05:06Z

paddle/operators/chunk_eval_op.cc

+    AddInput("Label", "(Tensor, default: Tensor<int>) Labels of the data.");
+    AddOutput(
+        "Precision",
+        "(float) The precision ratio of the predictions on current data.");


The evaluated precision (called positive predictive value) of chunks on the given mini-batch.

lcy-seso · 2017-11-01T02:05:31Z

paddle/operators/chunk_eval_op.cc

+        "Precision",
+        "(float) The precision ratio of the predictions on current data.");
+    AddOutput("Recall",
+              "(float) The recall ratio of the predictions on current data.");


The evaluated recall (true positive rate or sensitivity) of chunks on the given mini-batch.

I think we should tell the users such an evaluation is performed on the mini-batch, not on the data tested up to now. But, once we change this, and make sure to update the doc.

lcy-seso · 2017-11-01T02:10:53Z

paddle/operators/chunk_eval_op.cc

+    AddOutput("Recall",
+              "(float) The recall ratio of the predictions on current data.");
+    AddOutput("F1-Score",
+              "(float) The F1-Score of the predictions on current data.");


The evaluated F1-Score on the given mini-batch.

lcy-seso · 2017-11-01T03:53:06Z

paddle/operators/chunk_eval_op.cc

+                   framework::OpAttrChecker *op_checker)
+      : OpProtoAndCheckerMaker(proto, op_checker) {
+    AddInput("Inference",
+             "(Tensor, default: Tensor<int>) Predictions from the network.");


Add a "." after (Tensor, default: Tensor). The same below.

(Tensor, default: Tensor<int>). Predictions from the network.

lcy-seso · 2017-11-01T03:56:37Z

paddle/operators/chunk_eval_op.cc

+      : OpProtoAndCheckerMaker(proto, op_checker) {
+    AddInput("Inference",
+             "(Tensor, default: Tensor<int>) Predictions from the network.");
+    AddInput("Label", "(Tensor, default: Tensor<int>) Labels of the data.");


The true tag sequences.

lcy-seso · 2017-11-01T03:59:00Z

paddle/operators/chunk_eval_op.cc

+              "(float) The F1-Score of the predictions on current data.");
+    AddAttr<int>("num_chunk_types", "(int) The number of chunk type.");
+    AddAttr<std::string>("chunk_scheme",
+                         "(string, default IOB) The label scheme.")


The labeling scheme indicating how to encode the chunks, including IOB, x, x, x, (all the supported schemes.) It is better to add a reference here to explain how these schemes label chunks.

lcy-seso · 2017-11-01T04:02:15Z

paddle/operators/chunk_eval_op.cc

+        "excluded_chunk_types",
+        "(list<int>) A list<int> indicating chunk types not to be counted.")
+        .SetDefault(std::vector<int>{});
+    AddComment(R"DOC(


Here, I think it will much better to explain what is chunk first. For example, maybe like this.

Chunks are about character spans. In the sequence tagging problem, chunks are sequences of tokens (words or other units) and tags (tag labels, categories).

Rewrite the doc.

lcy-seso · 2017-11-01T04:05:43Z

paddle/operators/chunk_eval_op.cc

+        .SetDefault(std::vector<int>{});
+    AddComment(R"DOC(
+Chunk evaluator is used to evaluate segment labelling accuracy for a
+sequence. It calculates precision, recall and F1 scores for the chunk detection.


the chunk detection --> chunks the model predicts.

Rewrite the doc.

lcy-seso · 2017-11-01T05:03:43Z

paddle/operators/chunk_eval_op.cc

+    AddComment(R"DOC(
+Chunk evaluator is used to evaluate segment labelling accuracy for a
+sequence. It calculates precision, recall and F1 scores for the chunk detection.
+To use chunk evaluator, several concepts need to be clarified firstly.


we first introduce some related concepts.

Rewrite the doc.

lcy-seso · 2017-11-01T05:18:17Z

paddle/operators/chunk_eval_op.cc

+        .SetDefault("IOB");
+    AddAttr<std::vector<int>>(
+        "excluded_chunk_types",
+        "(list<int>) A list<int> indicating chunk types not to be counted.")


indicating chunk types that are not counted.

This explanation is hard to understand for users.

Done. Add see below for details.

… add-ChunkEvalOp

guoshengCS added 2 commits October 23, 2017 21:17

Add chunk_eval_op

bb9d68d

Merge branch 'develop' of https://github.com/PaddlePaddle/paddle into…

4b84f07

… add-ChunkEvalOp

guoshengCS added the OpPorting label Oct 23, 2017

qingqing01 requested review from qingqing01, wanghaoshuang and lcy-seso October 24, 2017 02:42

qingqing01 reviewed Oct 30, 2017

View reviewed changes

wanghaoshuang reviewed Oct 31, 2017

View reviewed changes

wanghaoshuang reviewed Nov 1, 2017

View reviewed changes

lcy-seso reviewed Nov 1, 2017

View reviewed changes

guoshengCS added 2 commits November 6, 2017 17:10

Merge branch 'develop' of https://github.com/PaddlePaddle/paddle into…

ece1d57

… add-ChunkEvalOp

Refine ChunkEvalOp by following comments and rewrite the doc

c8dcd9a

qingqing01 approved these changes Nov 9, 2017

View reviewed changes

qingqing01 merged commit aa34067 into PaddlePaddle:develop Nov 10, 2017

Add chunk eval op #5016

Add chunk eval op #5016

Conversation

guoshengCS commented Oct 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Nov 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Nov 1, 2017 •

edited

Loading