Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update column types in SQL storage models #15

Closed

Conversation

TheophileDiot
Copy link
Contributor

@TheophileDiot TheophileDiot commented Feb 8, 2024

Description

This pull request addresses compatibility issues with MySQL databases caused by the TEXT column type. MySQL requires specific sizes in schema definitions, which were not being met by the TEXT type. To resolve this, the column types in SQL storage models are updated to String(512).

Changes Made

Modified Files

  • src/cscapi/sql_storage.py: Updated column types from TEXT to String(512) to meet MySQL schema requirements.

Copy link
Collaborator

@rr404 rr404 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK to adjust string size, but lets do it more precisely.
Also we need to find a TEXT alternative for LONG strings, some MUST accept unlimited sizes like scenarios

password = Column(TEXT)
scenarios = Column(TEXT)
machine_id = Column(String(512), unique=True)
token = Column(String(512))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[critical] (!) token are currently over 1000char long !

@rr404
Copy link
Collaborator

rr404 commented Feb 8, 2024

MySQL supports the TEXT type, would it be a python sqlalchemy issue ?

Two things:

  • OK to change to sized string to be more optimal (and non breaking, example "token" >1088)
  • But we need to keep unsized strings for some fields like "scenarios"

@TheophileDiot is your mysql DB really not suporting TEXT type ? or is it when python tries to convert the sqlalchemy type TEXT ?

@rr404
Copy link
Collaborator

rr404 commented Feb 8, 2024

@julienloizelet Lets make a list or cleverly thought string size that leave room for evolution where needed while being precise when we know it wont change
Example:

  • MachineDBModel.machine_id, we could go up to 256 and be future proof i guess
  • MachineDBModel.scenarios need to stay unlimited
  • ...
  • DecisionDBModel.origin we know it's never gonna be much, even if we one day have the list name in there it won't go over 64
  • DecisionDBModel.duration we could probably size it to 32 ir less what are the possible longest values?
  • DecisionDBModel.until is it a timestamp or date? it's probably something that has and will keep a fixed size
  • ....

@rr404 rr404 marked this pull request as draft February 8, 2024 16:26
@julienloizelet
Copy link
Collaborator

julienloizelet commented Feb 9, 2024

Hi @TheophileDiot and @rr404 ,

I've tested with a mysql 5.7 database.

TEXT is supported but we have an error on the machine_id field because it is an index:

BLOB/TEXT column 'machine_id' used in key specification without a key length.

See https://stackoverflow.com/a/1827099/7497291 for some details.

If we change TEXT to String(256) for machine_id, it works great.

(Tested also with mariadb 10.8 and postgresql 11)

I will create a PR to fix that.

Other improvements (like setting accurate length for all fields) can be done later, I guess.

Thanks

@rr404
Copy link
Collaborator

rr404 commented Feb 9, 2024

Closing this PR because it will be fixed by #17

@TheophileDiot TheophileDiot deleted the mysql-model-fix branch February 14, 2024 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants