Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Robustness of Data Load (empty data) #112

Closed
dgedgedge opened this issue Feb 19, 2021 · 2 comments
Closed

Improve Robustness of Data Load (empty data) #112

dgedgedge opened this issue Feb 19, 2021 · 2 comments
Assignees
Milestone

Comments

@dgedgedge
Copy link

dgedgedge commented Feb 19, 2021

I try to load the following data : https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/vaccinations/vaccinations.csv

It's a csv file with multiple data for multiple countries.
When I try to load the data to have it as an internal data source, Colums 1,3,10 with analytics:load I get the following errors.
For me it is due to the fact that some data in this columns is empty.
I suggest to ptotect this behaviour by testing forhand that the data is a number before trying to insert it to tha database...

Note: Seeing this error I would ask if you did think to protect the code against SQL injection (from the data source) ?

occ analytics:load 12

In AbstractMySQLDriver.php line 106:
                                                                                                                                                                                                                                       
  An exception occurred while executing 'INSERT INTO `oc_analytics_facts` (`user_id`, `dataset`, `dimension1`, `dimension2`, `value`, `timestamp`) VALUES(?, ?, ?, ?, ?, ?)' with params ["xx", 25, "Albania", "2021-01-11", false  
  , 1613720978]:                                                                                                                                                                                                                       
                                                                                                                                                                                                                                       
  SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect decimal value: '' for column `nextcloud`.`oc_analytics_facts`.`value` at row 1                                                                                              
                                                                                                                                                                                                                                       

In PDOStatement.php line 129:
                                                                                                                                           
  SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect decimal value: '' for column `nextcloud`.`oc_analytics_facts`.`value` at row 1  
                                                                                                                                           

In PDOStatement.php line 127:
                                                                                                                                           
  SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect decimal value: '' for column `nextcloud`.`oc_analytics_facts`.`value` at row 1  
                                                                                                                                           

Note:
Nextcloud 20.0.7
Analytics 3.3.2
Debian GNU/Linux 10 (buster)
Mariadb...

@Rello
Copy link
Owner

Rello commented Feb 19, 2021

Hi
Thank you for the report. Yes, there is still a lot of hardening to be done when the basics are implemented.
will get back...

@Rello Rello self-assigned this Feb 23, 2021
@Rello Rello added this to the 3.4.0 milestone Feb 23, 2021
@Rello
Copy link
Owner

Rello commented Feb 23, 2021

Bildschirmfoto 2021-02-23 um 10 25 57

Catch errors not to break everything - just skip incorrect records

@Rello Rello added the pending release part of the next release version label Feb 23, 2021
@Rello Rello removed the pending release part of the next release version label Mar 13, 2021
@Rello Rello closed this as completed Mar 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants