Pytorch: some characters cannot be found in 'vocab' #1029

khawar-islam · 2022-08-25T01:03:17Z

khawar-islam
Aug 25, 2022

Hello @frgfm I am also training a recognition model for korean language and i made three changes but still there is some problem like above.

First changed [Added korean language]

VOCABS: Dict[str, str] = {
    'digits': string.digits,
    'ascii_letters': string.ascii_letters,
    'punctuation': string.punctuation,
    'currency': '£€¥¢฿',
    'ancient_greek': 'αβγδεζηθικλμνξοπρστυφχψωΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ',
    'arabic_letters': 'ءآأؤإئابةتثجحخدذرزسشصضطظعغـفقكلمنهوىي',
    'persian_letters': 'پچڢڤگ',
    'hindi_digits': '٠١٢٣٤٥٦٧٨٩',
    'arabic_diacritics': 'ًٌٍَُِّْ',
    'arabic_punctuation': '؟؛«»—',
    'korean':'ㄱㄴㄷㄹㅁㅂㅅㅇㅈㅊㅋㅌㅍㅎㅏㅑㅓㅕㅗㅛㅜㅠㅡㅣ'
}

VOCABS['latin'] = VOCABS['digits'] + VOCABS['ascii_letters'] + VOCABS['punctuation']
VOCABS['english'] = VOCABS['latin'] + '°' + VOCABS['currency']
VOCABS['legacy_french'] = VOCABS['latin'] + '°' + 'àâéèêëîïôùûçÀÂÉÈËÎÏÔÙÛÇ' + VOCABS['currency']
VOCABS['french'] = VOCABS['english'] + 'àâéèêëîïôùûüçÀÂÉÈÊËÎÏÔÙÛÜÇ'
VOCABS['portuguese'] = VOCABS['english'] + 'áàâãéêíïóôõúüçÁÀÂÃÉÊÍÏÓÔÕÚÜÇ'
VOCABS['spanish'] = VOCABS['english'] + 'áéíóúüñÁÉÍÓÚÜÑ' + '¡¿'
VOCABS['german'] = VOCABS['english'] + 'äöüßÄÖÜẞ'
VOCABS['arabic'] = (VOCABS['digits'] + VOCABS['hindi_digits'] + VOCABS['arabic_letters'] + VOCABS['persian_letters'] +
                    VOCABS['arabic_diacritics'] + VOCABS['arabic_punctuation'] + VOCABS['punctuation'])
VOCABS['czech'] = VOCABS['english'] + 'áčďéěíňóřšťúůýžÁČĎÉĚÍŇÓŘŠŤÚŮÝŽ'
VOCABS['vietnamese'] = (VOCABS['english'] +
                        'áàảạãăắằẳẵặâấầẩẫậéèẻẽẹêếềểễệóòỏõọôốồổộỗơớờởợỡúùủũụưứừửữựiíìỉĩịýỳỷỹỵ' +
                        'ÁÀẢẠÃĂẮẰẲẴẶÂẤẦẨẪẬÉÈẺẼẸÊẾỀỂỄỆÓÒỎÕỌÔỐỒỔỘỖƠỚỜỞỢỠÚÙỦŨỤƯỨỪỬỮỰIÍÌỈĨỊÝỲỶỸỴ')
VOCABS['korean'] = (VOCABS['digits'] + VOCABS['english'] + VOCABS['korean'])

Second changed [french to korean]

parser.add_argument('--vocab', type=str, default="korean", help='Vocab to be used for training')

Training start

python3 references/recognition/train_pytorch.py master --train_path /media/cvpr/CM_1/CM/dataset_KR/dataset/train --val_path /media/cvpr/CM_1/CM/dataset_KR/dataset/val --epochs 5 --batch_size 16

After following thread, I wrote two lines

from doctr.datasets.utils import encode_sequences
encode_sequences(["korean"], vocab='ㄱㄴㄷㄹㅁㅂㅅㅇㅈㅊㅋㅌㅍㅎㅏㅑㅓㅕㅗㅛㅜㅠㅡㅣ')

Error

Traceback (most recent call last):
  File "/media/cvpr/CM_1/CM/dataset_KR/doctr/doctr/datasets/utils.py", line 76, in encode_string
    return list(map(vocab.index, input_string))
ValueError: substring not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 2, in <module>
    encode_sequences(["korean"], vocab='ㄱㄴㄷㄹㅁㅂㅅㅇㅈㅊㅋㅌㅍㅎㅏㅑㅓㅕㅗㅛㅜㅠㅡㅣ')
  File "/media/cvpr/CM_1/CM/dataset_KR/doctr/doctr/datasets/utils.py", line 151, in encode_sequences
    for idx, seq in enumerate(map(partial(encode_string, vocab=vocab), sequences)):
  File "/media/cvpr/CM_1/CM/dataset_KR/doctr/doctr/datasets/utils.py", line 78, in encode_string
    raise ValueError("some characters cannot be found in 'vocab'")
ValueError: some characters cannot be found in 'vocab'

felixdittrich92 · 2022-08-25T07:15:49Z

felixdittrich92
Aug 25, 2022
Maintainer

Hi @khawar-islam 👋,

vocabs.py:

# Copyright (C) 2021-2022, Mindee.

# This program is licensed under the Apache License version 2.
# See LICENSE or go to <https://www.apache.org/licenses/LICENSE-2.0.txt> for full license details.

import string
from typing import Dict

__all__ = ['VOCABS']


VOCABS: Dict[str, str] = {
    'digits': string.digits,
    'ascii_letters': string.ascii_letters,
    'punctuation': string.punctuation,
    'currency': '£€¥¢฿',
    'ancient_greek': 'αβγδεζηθικλμνξοπρστυφχψωΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ',
    'arabic_letters': 'ءآأؤإئابةتثجحخدذرزسشصضطظعغـفقكلمنهوىي',
    'persian_letters': 'پچڢڤگ',
    'hindi_digits': '٠١٢٣٤٥٦٧٨٩',
    'arabic_diacritics': 'ًٌٍَُِّْ',
    'arabic_punctuation': '؟؛«»—'
}

VOCABS['latin'] = VOCABS['digits'] + VOCABS['ascii_letters'] + VOCABS['punctuation']
VOCABS['english'] = VOCABS['latin'] + '°' + VOCABS['currency']
VOCABS['legacy_french'] = VOCABS['latin'] + '°' + 'àâéèêëîïôùûçÀÂÉÈËÎÏÔÙÛÇ' + VOCABS['currency']
VOCABS['french'] = VOCABS['english'] + 'àâéèêëîïôùûüçÀÂÉÈÊËÎÏÔÙÛÜÇ'
VOCABS['portuguese'] = VOCABS['english'] + 'áàâãéêíïóôõúüçÁÀÂÃÉÊÍÏÓÔÕÚÜÇ'
VOCABS['spanish'] = VOCABS['english'] + 'áéíóúüñÁÉÍÓÚÜÑ' + '¡¿'
VOCABS['german'] = VOCABS['english'] + 'äöüßÄÖÜẞ'
VOCABS['arabic'] = (VOCABS['digits'] + VOCABS['hindi_digits'] + VOCABS['arabic_letters'] + VOCABS['persian_letters'] +
                    VOCABS['arabic_diacritics'] + VOCABS['arabic_punctuation'] + VOCABS['punctuation'])
VOCABS['czech'] = VOCABS['english'] + 'áčďéěíňóřšťúůýžÁČĎÉĚÍŇÓŘŠŤÚŮÝŽ'
VOCABS['vietnamese'] = (VOCABS['english'] +
                        'áàảạãăắằẳẵặâấầẩẫậéèẻẽẹêếềểễệóòỏõọôốồổộỗơớờởợỡúùủũụưứừửữựiíìỉĩịýỳỷỹỵ' +
                        'ÁÀẢẠÃĂẮẰẲẴẶÂẤẦẨẪẬÉÈẺẼẸÊẾỀỂỄỆÓÒỎÕỌÔỐỒỔỘỖƠỚỜỞỢỠÚÙỦŨỤƯỨỪỬỮỰIÍÌỈĨỊÝỲỶỸỴ')
VOCABS['korean'] = (VOCABS['digits'] + VOCABS['punctuation'] +
                    VOCABS['currency'] + 'ㄱㄴㄷㄹㅁㅂㅅㅇㅈㅊㅋㅌㅍㅎㅏㅑㅓㅕㅗㅛㅜㅠㅡㅣ')

test.py

from doctr.datasets.utils import encode_sequences
from doctr.datasets import VOCABS
x = encode_sequences(["ㅌㅍㅎㅏㅑㅓㅕ", "12345"], vocab=VOCABS['korean'])
print(x)

[[58 59 60 61 62 63 64 -1]
 [ 1  2  3  4  5 -1 -1 -1]]

works without problems :)

you have passed the string korean to encode but provided a vocab which contains only korean characters this can not work 😅

32 replies

khawar-islam Sep 2, 2022
Author

@felixdittrich92 I think 20 epochs will take almost 5 days. Please check below link

https://wandb.ai/khawar512/text-recognition/runs/r4p7vrb2

felixdittrich92 Sep 2, 2022
Maintainer

@khawar-islam have you changed the project privacy to public ? I can't see it

Otherwise post a screenshot that i can have a look at the first charts

khawar-islam Sep 2, 2022
Author

@felixdittrich92 You can check it now

https://wandb.ai/khawar512/text-recognition

felixdittrich92 Sep 2, 2022
Maintainer

that the log if you ran the command i posted + updated fonts ?

and doctr is the newest version main ? pip install -e git+https://github.com/mindee/doctr.git#egg=python-doctr[torch] --upgrade

khawar-islam Sep 5, 2022
Author

@felixdittrich92 Yes, updated command with fonts, please have a look

python3 train_pytorch.py master --epochs 20 --batch_size 16 --font "ko/NanumBarunGothicBold.ttf,ko/나눔손글씨 따악단단.ttf,ko/NanumBarunGothicLight.ttf,ko/나눔손글씨 딸에게 엄마가.ttf,ko/NanumBarunGothic.ttf,ko/나눔손글씨 또박또박.ttf,ko/NanumBarunGothicUltraLight.ttf,ko/나눔손글씨 마고체.ttf,ko/NanumBarunpenB.ttf,ko/ 나눔손글씨 맛있는체.ttf,ko/NanumBarunpenR.ttf,ko/나눔손글씨 몽돌.ttf,ko/NanumBrush.ttf,ko/나눔손글씨 무궁화.ttf,ko/NanumGothicBold.ttf,ko/나눔손글씨 무진장체.ttf,ko/NanumGothicExtraBold.ttf,ko/나눔손글씨 미니 손글씨.ttf,ko/NanumGothicLight.ttf,ko/나눔손글씨 미래나무.ttf,ko/NanumGothic.ttf,ko/나눔손글씨 바른정신.ttf,ko/NanumMyeongjoBold.ttf,ko/나눔손글씨 바른히피.ttf,ko/NanumMyeongjoExtraBold.ttf,ko/나눔손글 씨 반짝반짝 별.ttf,ko/NanumMyeongjo.ttf,ko/나눔손글씨 배은혜체.ttf,ko/NanumPen.ttf,ko/나눔손글씨 백의의 천사.ttf,ko/NanumSquareB.ttf,ko/ 나눔손글씨 버드나무.ttf,ko/NanumSquareEB.ttf,ko/나눔손글씨 범솜체.ttf,ko/NanumSquareL.ttf,ko/나눔손글씨 부장님 눈치체.ttf,ko/NanumSquareRoundB.ttf,ko/나눔손글씨 북극성.ttf,ko/NanumSquareRoundEB.ttf,ko/나눔손글씨 비상체.ttf,ko/NanumSquareRoundL.ttf,ko/나눔손글씨 빵구니맘 손 글씨.ttf,ko/NanumSquareRoundR.ttf,ko/나눔손글씨 사랑해 아들.ttf,ko/NanumSquareR.ttf,ko/나눔손글씨 상해찬미체.ttf,ko/나눔손글씨 끄트머리체.ttf,ko/나눔손글씨 성실체.ttf,ko/나눔손글씨 강부장님체.ttf,ko/나눔손글씨 세계적인 한글.ttf,ko/나눔손글씨 느릿느릿체.ttf,ko/나눔손글씨 세 아체.ttf,ko/나눔손글씨 노력하는 동희.ttf,ko/나눔손글씨 아기사랑체.ttf,ko/나눔손글씨 나무정원.ttf,ko/나눔손글씨 아름드리 꽃나무.ttf,ko/나 눔손글씨 기쁨밝음.ttf,ko/나눔손글씨 아빠글씨.ttf,ko/나눔손글씨 금은보화.ttf,ko/나눔손글씨 아빠의 연애편지.ttf,ko/나눔손글씨 고려글꼴.ttf,ko/나눔손글씨 아인맘 손글씨.ttf,ko/나눔손글씨 김유이체.ttf,ko/나눔손글씨 아줌마 자유.ttf,ko/나눔손글씨 대광유리.ttf,ko/나눔손글씨 안쌍체.ttf,ko/나눔손글씨 달의궤도.ttf,ko/나눔손글씨 암스테르담.ttf,ko/나눔손글씨 가람연꽃.ttf,ko/나눔손글씨 야근하는 김주임.ttf,ko/나눔손글씨 다채사랑.ttf,ko/나눔손글씨 야채장수 백금례.ttf,ko/나눔손글씨 규리의 일기.ttf,ko/나눔손글씨 엄마사랑.ttf,ko/나눔손글씨 강인한 위로.ttf,ko/나눔손글씨 엉겅퀴체.ttf,ko/나눔손글씨 곰신체.ttf,ko/나눔손글씨 여름글씨.ttf,ko/나눔손글씨 갈맷글.ttf,ko/나눔손글씨 연지체.ttf,ko/나눔손글씨 다진체.ttf,ko/나눔손글씨 열아홉의 반짝임.ttf,ko/나눔손글씨 다행체.ttf,ko/나눔손글씨 열일체.ttf,ko/나눔손글씨 꽃내음.ttf,ko/나눔손글씨 예당체.ttf,ko/나눔손글씨 나는 이겨낸다.ttf,ko/나눔손글씨 예쁜 민경체.ttf,ko/나눔손글씨 고딕 아니고 고딩.ttf,ko/나눔손글씨 옥비체.ttf,ko/나 눔손글씨 다시 시작해.ttf,ko/나눔손글씨 와일드.ttf,ko/나눔손글씨 나의 아내 손글씨.ttf,ko/나눔손글씨 우리딸 손글씨.ttf,ko/나눔손글씨 고려글꼴.ttf,ko/나눔손글씨 의미있는 한글.ttf,ko/나눔손글씨 달의궤도.ttf,ko/나눔손글씨 자부심지우.ttf,ko/나눔손글씨 따뜻한 작별.ttf" --wb --max-chars 45 --wd 0.1

I have clone master repository almost two weeks ago but i didn't install through pip.

felixdittrich92 · 2022-09-05T05:37:11Z

felixdittrich92
Sep 5, 2022
Maintainer

Hi @khawar-islam yeah let me know the results 👍

1 reply

khawar-islam Sep 5, 2022
Author

Hi @felixdittrich92 below is the training and validation loss log.

Validation loss decreased inf --> 6.92626: saving state...
Epoch 1/20 - Validation loss: 6.92626 (Exact: 0.00% | Partial: 0.00%)
Epoch 4/20 - Validation loss: 6.72172 (Exact: 0.00% | Partial: 0.00%)
Epoch 2/20 - Validation loss: 6.92296 (Exact: 0.00% | Partial: 0.00%)
Epoch 3/20 : |--------------------------------------------------------------------| 0.01% [9/67062 00:01<2:31:54 Training loss: 6.92805]]
Epoch 3/20 - Validation loss: 6.71868 (Exact: 0.00% | Partial: 0.00%)
Epoch 4/20 : |-----------------------------------------------------------------------------| 0.00% [0/67062 00:00<?]
Epoch 6/20 - Validation loss: 6.71504 (Exact: 0.00% | Partial: 0.00%)
Epoch 5/20 - Validation loss: 6.71285 (Exact: 0.00% | Partial: 0.00%)
Epoch 6/20 : |--------------------------------------------------------------------| 0.02% [13/67062 00:01<2:09:07 Training loss: 6.77116]
Epoch 7/20 - Validation loss: 6.71719 (Exact: 0.00% | Partial: 0.00%)
Epoch 8/20 - Validation loss: 6.72013 (Exact: 0.00% | Partial: 0.00%)
Epoch 10/20 - Validation loss: 6.73127 (Exact: 0.00% | Partial: 0.00%)
Epoch 9/20 - Validation loss: 6.7091 (Exact: 0.00% | Partial: 0.00%)
Epoch 10/20 : |-------------------------------------------------------------------| 0.01% [4/67062 00:00<4:13:04 Training loss: 6.64411]]
Epoch 11/20 - Validation loss: 6.71524 (Exact: 0.00% | Partial: 0.00%)
Epoch 12/20 - Validation loss: 6.71217 (Exact: 0.00% | Partial: 0.00%)
Validation loss decreased 6.7091 --> 6.70775: saving state...
Epoch 13/20 - Validation loss: 6.70775 (Exact: 0.00% | Partial: 0.00%)
Epoch 14/20 - Validation loss: 6.70795 (Exact: 0.00% | Partial: 0.00%)
Epoch 15/20 - Validation loss: 6.71767 (Exact: 0.00% | Partial: 0.00%)
Epoch 16/20 - Validation loss: 6.71232 (Exact: 0.00% | Partial: 0.00%)
Epoch 18/20 - Validation loss: 6.71014 (Exact: 0.00% | Partial: 0.00%)
Epoch 17/20 - Validation loss: 6.70741 (Exact: 0.00% | Partial: 0.00%)
Epoch 18/20 : |-----------------------------------------------------------------------------| 0.00% [0/67062 00:00<?]
Epoch 19/20 - Validation loss: 6.7126 (Exact: 0.00% | Partial: 0.00%)
Epoch 20/20 - Validation loss: 6.71125 (Exact: 0.00% | Partial: 0.00%)

felixdittrich92 · 2022-09-05T06:03:33Z

felixdittrich92
Sep 5, 2022
Maintainer

Hi @khawar-islam if the Images with your fonts generated correctly that result is not possible ... Are you sure you have the latest changes from main branch pulled ?

69 replies

khawar-islam Oct 12, 2022
Author

Hello @felixdittrich92, Actually the validation is similar to training data and just hangul characters are different. Yes, when i tested on sample data that contains handwritten Hangul, receipts, and many more than model collapse. In some extent the prediction of some Hangul characters are correct. Like if you see the recognized character in yellow box, the first hangul is correct.

felixdittrich92 Oct 12, 2022
Maintainer

Ok but this are good news we go in the correct direction. 👍
One request could you try to pad the input as described in #1084 before the inference and share the result again?
About the recognition if the padding does not increase the detection/recognition results we need more "different" data.
But in a first step let's test this 👍

khawar-islam Oct 13, 2022
Author

Hello @felixdittrich92 I added padding on image but it does not bring any accuracy improvement. It means that we have to generate more data. Am i right?

Experiment with --weight_decay 0.1

Validation loss decreased inf --> 4.35542: saving state...                                                                                   
Epoch 1/5 - Validation loss: 4.35542 (Exact: 0.02% | Partial: 0.02%)
Validation loss decreased 4.35542 --> 4.28319: saving state...                                                                               
Epoch 2/5 - Validation loss: 4.28319 (Exact: 0.03% | Partial: 0.03%)
Validation loss decreased 4.28319 --> 4.28195: saving state...                                                                               
Epoch 3/5 - Validation loss: 4.28195 (Exact: 0.03% | Partial: 0.03%)
Validation loss decreased 4.28195 --> 4.27768: saving state...                                                                               
Epoch 4/5 - Validation loss: 4.27768 (Exact: 0.03% | Partial: 0.03%)
Epoch 5/5 - Validation loss: 4.2838 (Exact: 0.03% | Partial: 0.03%)                                                                          
wandb: Waiting for W&B process to finish... (success).

felixdittrich92 Oct 13, 2022
Maintainer

Hi @khawar-islam yep keep a look at: https://github.com/clovaai/synthtiger looks like a much better way to create a dataset i trained MASTER on SynthText (toy run) which is simliar to the images you can create with the linked repo a while ago and it works very well for english

khawar-islam Oct 13, 2022
Author

Thank you @felixdittrich92 Sure! I will create a new dataset and before that i will share some samples with you.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pytorch: some characters cannot be found in 'vocab' #1029

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 102 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Pytorch: some characters cannot be found in 'vocab' #1029

khawar-islam Aug 25, 2022

Replies: 3 comments · 102 replies

felixdittrich92 Aug 25, 2022 Maintainer

khawar-islam Sep 2, 2022 Author

felixdittrich92 Sep 2, 2022 Maintainer

khawar-islam Sep 2, 2022 Author

felixdittrich92 Sep 2, 2022 Maintainer

khawar-islam Sep 5, 2022 Author

felixdittrich92 Sep 5, 2022 Maintainer

khawar-islam Sep 5, 2022 Author

felixdittrich92 Sep 5, 2022 Maintainer

khawar-islam Oct 12, 2022 Author

felixdittrich92 Oct 12, 2022 Maintainer

khawar-islam Oct 13, 2022 Author

felixdittrich92 Oct 13, 2022 Maintainer

khawar-islam Oct 13, 2022 Author

khawar-islam
Aug 25, 2022

Replies: 3 comments 102 replies

felixdittrich92
Aug 25, 2022
Maintainer

khawar-islam Sep 2, 2022
Author

felixdittrich92 Sep 2, 2022
Maintainer

khawar-islam Sep 2, 2022
Author

felixdittrich92 Sep 2, 2022
Maintainer

khawar-islam Sep 5, 2022
Author

felixdittrich92
Sep 5, 2022
Maintainer

khawar-islam Sep 5, 2022
Author

felixdittrich92
Sep 5, 2022
Maintainer

khawar-islam Oct 12, 2022
Author

felixdittrich92 Oct 12, 2022
Maintainer

khawar-islam Oct 13, 2022
Author

felixdittrich92 Oct 13, 2022
Maintainer

khawar-islam Oct 13, 2022
Author