Refatorar testes para não utilizar argumento `temp_dir` (problemas com versão pacote click) #38

gabrielbdornas · 2021-07-06T13:42:19Z

Devido ao erro apresentado no issue #35 foi necessário passar a versão do pacote click de 8.0.1 para 7.1.2. O problema encontrado foi uma incompatibilidade de versões entre a utilizada no dpckan e versões as necessárias para funcionamento do pacote frictionless e suas dependências.
As versões abaixo de 8 do click não apresentam o parâmetro "temp_dir" para a função isolated_filesystem () o que fez os testes do pacote pararem de funcionar, conforme erro descrito:

======================================================================
ERROR: test_production_env (test_dataset_without_datapackage_file.TestDatasetWithoutDatapackageFile)
Testing error for lack of datapackage file inside dataset home directory (--env homologacao)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gabrielbdornas/code/gabrielbdornas/dados-mg/dpckan/dpckan/tests/test_dataset_without_datapackage_file.py", line 30, in test_production_env
    with runner.isolated_filesystem(temp_dir=get_file_path()):
  File "/usr/lib/python3.9/contextlib.py", line 244, in helper
    return _GeneratorContextManager(func, args, kwds)
  File "/usr/lib/python3.9/contextlib.py", line 87, in __init__
    self.gen = func(*args, **kwds)
TypeError: isolated_filesystem() got an unexpected keyword argument 'temp_dir'

----------------------------------------------------------------------
Ran 14 tests in 0.092s

FAILED (errors=14)
make: *** [Makefile:25: tests] Error 1

Necessário refazer a estrutura dos testes para retirar este parâmetro ou acompanhar a atualização da biblioteca click nos pacotes instalados juntamente com frictionless (para tanto acompanhar esse issue)

The text was updated successfully, but these errors were encountered:

Issue #50 standardized CLI interface commands, that's why we must update its calls inside tests files. It's not the main objective of #38 but to put tests working again these must be corrected. Resolves: Chunks of #38 and related to #50

One of the main objectives of dpckan 2º iteration (#15) is the code refactor process. Because of "readability counts" we're trying to follow the PEP 8 style guide for python code or the small_snack_case patter. Function lerDadosJsonMapeado no longer exists so its importations in tests files must be removed Function load_complete_datapackage is imported but needs some tests yet. Resolves: Chunks of #38

As shown in #38, click package version 7.1.2 doesn't have 'temp_dir' argument in isolated_filesystem() function. New click tests documentation: https://click.palletsprojects.com/en/8.0.x/testing/ Old click tests documentation: https://click.palletsprojects.com/en/7.x/testing/ We need to remove this argument and run tests and see the results This commit also changes the old "public" function to the new "create". As said in commit a3158b5, Issue #50 standardized CLI interface commands, that's why we must update its calls inside tests files. It's not the main objective of #38 but to put tests working again these must be corrected. Resolves: Chunks of #38 and related to #50

One of the main objectives of dpckan 2º iteration (#15) is the code refactor process. Because of "readability counts" we're trying to follow the PEP 8 style guide for python code or the small_snack_case patter. Function lerDadosJsonMapeado no longer exists so its importations in tests files must be removed Function load_complete_datapackage is imported but needs some tests yet. Part the clean was made in f07668f but this two was forgotten Resolves: Chunks of #38

fjuniorr · 2021-08-06T21:04:37Z

@gabrielbdornas quando for corrigir isso, atualiza também a versão do frictionless.py pra evitar o erro

ModuleNotFoundError: No module named 'click._bashcomplete'

que ainda está acontecendo em uma instalação vanilla do frictionless 4.12.2.

tiangolo · 2021-08-30T09:19:02Z

I just released Typer 0.4.0, which is compatible with both Click 7 and Click 8. 🎉

Upgrading Typer should handle the issue.

See #38

Include ipdb package to debug code See #38

See #38

fjuniorr · 2021-09-08T14:38:15Z

@gabrielbdornas estou recebendo os erros abaixo ao rodar os testes. Tem um palpite do que pode ser?

$ make tests
Realizando Testes
....EEEE......
======================================================================
ERROR: test_homologa_env (test_dataset_publish_dataset_successfully.TestDatasetPublishDatasetSuccessfully)
Testing dataset publication sucessfully homologacao environment
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/fjunior/Local/cge/dados-mg/dpckan/dpckan/tests/test_dataset_publish_dataset_successfully.py", line 33, in test_homologa_env
    delete_dataset(ckan_instance, dataset_name)
  File "/Users/fjunior/Local/cge/dados-mg/dpckan/dpckan/functions.py", line 68, in delete_dataset
    ckan_instance.action.package_delete(id = dataset_name)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/ckanapi/common.py", line 51, in action
    return self._ckan.call_action(name, data_dict=kwargs)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/ckanapi/remoteckan.py", line 85, in call_action
    status, response = self._request_fn(url, data, headers, files, requests_kwargs)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/ckanapi/remoteckan.py", line 89, in _request_fn
    r = self.session.post(url, data=data, headers=headers, files=files,
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/requests/sessions.py", line 528, in request
    prep = self.prepare_request(req)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/requests/sessions.py", line 456, in prepare_request
    p.prepare(
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/requests/models.py", line 316, in prepare
    self.prepare_url(url, params)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/requests/models.py", line 390, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'dados.mg.gov.br/api/action/package_delete': No schema supplied. Perhaps you meant http://dados.mg.gov.br/api/action/package_delete?

======================================================================
ERROR: test_production_env (test_dataset_publish_dataset_successfully.TestDatasetPublishDatasetSuccessfully)
Testing dataset publication sucessfully production environment
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/fjunior/Local/cge/dados-mg/dpckan/dpckan/tests/test_dataset_publish_dataset_successfully.py", line 55, in test_production_env
    delete_dataset(ckan_instance, dataset_name)
  File "/Users/fjunior/Local/cge/dados-mg/dpckan/dpckan/functions.py", line 68, in delete_dataset
    ckan_instance.action.package_delete(id = dataset_name)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/ckanapi/common.py", line 51, in action
    return self._ckan.call_action(name, data_dict=kwargs)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/ckanapi/remoteckan.py", line 78, in call_action
    url = self.address.rstrip('/') + '/' + url
AttributeError: 'NoneType' object has no attribute 'rstrip'

======================================================================
ERROR: test_homologa_env (test_dataset_publish_dataset_with_schema_successfully.TestDatasetPublishDatasetWithSchemaSuccessfully)
Testing dataset publication sucessfully homologacao environment
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/fjunior/Local/cge/dados-mg/dpckan/dpckan/tests/test_dataset_publish_dataset_with_schema_successfully.py", line 33, in test_homologa_env
    delete_dataset(ckan_instance, dataset_name)
  File "/Users/fjunior/Local/cge/dados-mg/dpckan/dpckan/functions.py", line 68, in delete_dataset
    ckan_instance.action.package_delete(id = dataset_name)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/ckanapi/common.py", line 51, in action
    return self._ckan.call_action(name, data_dict=kwargs)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/ckanapi/remoteckan.py", line 85, in call_action
    status, response = self._request_fn(url, data, headers, files, requests_kwargs)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/ckanapi/remoteckan.py", line 89, in _request_fn
    r = self.session.post(url, data=data, headers=headers, files=files,
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/requests/sessions.py", line 528, in request
    prep = self.prepare_request(req)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/requests/sessions.py", line 456, in prepare_request
    p.prepare(
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/requests/models.py", line 316, in prepare
    self.prepare_url(url, params)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/requests/models.py", line 390, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'dados.mg.gov.br/api/action/package_delete': No schema supplied. Perhaps you meant http://dados.mg.gov.br/api/action/package_delete?

======================================================================
ERROR: test_production_env (test_dataset_publish_dataset_with_schema_successfully.TestDatasetPublishDatasetWithSchemaSuccessfully)
Testing dataset publication sucessfully production environment
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/fjunior/Local/cge/dados-mg/dpckan/dpckan/tests/test_dataset_publish_dataset_with_schema_successfully.py", line 55, in test_production_env
    delete_dataset(ckan_instance, dataset_name)
  File "/Users/fjunior/Local/cge/dados-mg/dpckan/dpckan/functions.py", line 68, in delete_dataset
    ckan_instance.action.package_delete(id = dataset_name)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/ckanapi/common.py", line 51, in action
    return self._ckan.call_action(name, data_dict=kwargs)
  File "/Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/ckanapi/remoteckan.py", line 78, in call_action
    url = self.address.rstrip('/') + '/' + url
AttributeError: 'NoneType' object has no attribute 'rstrip'

----------------------------------------------------------------------
Ran 14 tests in 9.283s

FAILED (errors=4)
make: *** [tests] Error 1

fjuniorr · 2021-09-08T17:18:02Z

tldr

Os testes passaram quando eu criei as variáveis de ambiente CKAN_HOST, CKAN_KEY, CKAN_HOST_PRODUCAO, CKAN_KEY_PRODUCAO diretamente no meu ~/.bash_profile.

histórico

Inicialmente eu estava com valores incorretos no arquivo .env na raiz do meu projeto e achei que esse era o problema. No entanto, mesmo criando as 4 variáveis corretamente os testes ainda não acessam o valor dessas variáveis. Imagino que os testes são inicializados de forma independente do que o "resto" do dpckan.

Além disso, mesmo depois de configurar tudo corretamente recebi algumas mensagens de erro que não consegui reproduzir (deixando aqui pra posteridade)

$ make tests
Realizando Testes
...../Users/fjunior/.virtualenvs/dpckan/lib/python3.9/site-packages/urllib3/connectionpool.py:938: ResourceWarning: unclosed <ssl.SSLSocket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('192.168.0.140', 50027), raddr=('200.198.34.165', 443)>
  conn.set_cert(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
.........
----------------------------------------------------------------------
Ran 14 tests in 122.786s

OK

Sobre esses testes eu fico com as seguintes impressões

Os testes no ambiente de homologação e produção aumentam pouco a "cobertura dos testes" no sentido que a lógica do teste é exatamente igual
A necessidade de acesso a instâncias do CKAN mg.gov.br é problemática do ponto de vista de desenvolvedores externos (talvez usar https://demo.ckan.org/)
Precisamos de estudar um pouco mais sobre testes de clientes que consumem uma API e mocking

See #38

gabrielbdornas · 2021-10-26T14:36:01Z

@fjuniorr removi os testes para ambientes de produção neste commit, conforme sugerido no final deste comentário

gabrielbdornas mentioned this issue Jul 9, 2021

38/tests refactoring #54

Merged

fjuniorr changed the title ~~Problemas com versão pacote click~~ Refatorar testes para não utilizar argumento temp_dir (problemas com versão pacote click) Aug 6, 2021

fjuniorr assigned gabrielbdornas Aug 30, 2021

gabrielbdornas added a commit that referenced this issue Sep 3, 2021

Update click, frictionless and pip versions

0992a74

See #38

gabrielbdornas mentioned this issue Sep 3, 2021

Dataset create revision #84

Merged

gabrielbdornas added a commit that referenced this issue Sep 4, 2021

Fix test problems

8a8bf3d

Include ipdb package to debug code See #38

gabrielbdornas added a commit that referenced this issue Sep 4, 2021

Comment test to fix publish successfully

fd97441

See #38

gabrielbdornas added a commit that referenced this issue Sep 4, 2021

Fix pulish dataset successfully test

f14d57f

See #38

gabrielbdornas added a commit that referenced this issue Sep 5, 2021

Fix pulish dataset production successfully test

f64d311

See #38

gabrielbdornas added a commit that referenced this issue Sep 5, 2021

Fix pulish dataset production successfully test

eda6408

See #38

gabrielbdornas added a commit that referenced this issue Sep 5, 2021

Including temp_dir

c78edcd

See #38

gabrielbdornas added a commit that referenced this issue Sep 5, 2021

Fix pulish dataset without name test

da38c03

See #38

gabrielbdornas added a commit that referenced this issue Sep 5, 2021

Fix pulish dataset without resource test

9cab94f

See #38

gabrielbdornas added a commit that referenced this issue Sep 5, 2021

Fix pulish dataset with schema test

0aabb01

See #38

gabrielbdornas added a commit that referenced this issue Sep 5, 2021

Fix pulish dataset with datapackage error test

4a7baab

See #38

gabrielbdornas added a commit that referenced this issue Sep 5, 2021

Fix pulish dataset without datapackage file test

4d64117

See #38

gabrielbdornas closed this as completed Sep 5, 2021

gabrielbdornas added a commit that referenced this issue Sep 7, 2021

Correct delete function

8a5f280

See #38

fjuniorr reopened this Sep 8, 2021

fjuniorr closed this as completed Sep 8, 2021

gabrielbdornas added a commit that referenced this issue Sep 9, 2021

Improve tests

1c8f69d

See #38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refatorar testes para não utilizar argumento `temp_dir` (problemas com versão pacote click) #38

Refatorar testes para não utilizar argumento `temp_dir` (problemas com versão pacote click) #38

gabrielbdornas commented Jul 6, 2021 •

edited

Loading

fjuniorr commented Aug 6, 2021

tiangolo commented Aug 30, 2021

fjuniorr commented Sep 8, 2021

fjuniorr commented Sep 8, 2021

gabrielbdornas commented Oct 26, 2021

Refatorar testes para não utilizar argumento temp_dir (problemas com versão pacote click) #38

Refatorar testes para não utilizar argumento temp_dir (problemas com versão pacote click) #38

Comments

gabrielbdornas commented Jul 6, 2021 • edited Loading

fjuniorr commented Aug 6, 2021

tiangolo commented Aug 30, 2021

fjuniorr commented Sep 8, 2021

fjuniorr commented Sep 8, 2021

tldr

histórico

gabrielbdornas commented Oct 26, 2021

Refatorar testes para não utilizar argumento `temp_dir` (problemas com versão pacote click) #38

Refatorar testes para não utilizar argumento `temp_dir` (problemas com versão pacote click) #38

gabrielbdornas commented Jul 6, 2021 •

edited

Loading