Manage dependencies

[utkarsharma2]

1. Added conditional imports to manage import errors due potential package absence.
2. Updated pyproject.toml to move packages to optional packages. Choose Postgres as default db for time being till we have support for sqlite.

Resolves #82

[utkarsharma2]

@dimberman @tatiana - Do you guys have any suggestion to test this with UT?

[uranusjr]

If a user needs Postgres support, they’d install astro[postgres] and pick up the Postgres provider from the extra, so this line shouldn’t be needed.

[utkarsharma2]

So Postgre is default DB until we have support for sqlite.

[tatiana]

What do you think if we gave a more helpful message, telling the users how to install the missing packages?

[utkarsharma2]

Yes, I agree, I think we can point out the command to run in error message itself.
Like then they can do pip install astro-project[aws] if package related to AWS was missing.

[tatiana]

That would be awesome, @utkarsharma2 !

[tatiana]

it may be worth renaming gcs to gcp

[utkarsharma2]

I'll update this.
FYI - @dimberman

[tatiana]

@utkarsharma2 it's great to see progress in this stream!

1. Docs: we probably want to update the docs on how the installation changed, explaining how users installing Astro with pip can benefit from this feature. It's straightforward if we are installing locally, having the pyproject.toml, but it may be worth documenting to users downloading from PyPI using pip install astro-projects[snowflake] etc.

2. Tests: I agree we need to find a way of testing things before merging this feature.
   A strategy could be to mock the value of some_module.__import__ so it raises ModuleNotFoundError, have you tried this?

[kaxil]

@uranusjr Does all keyword work with flit -- pypa/flit#488

I wanted to use flit for https://github.com/astronomer/astronomer-providers/ but the following two issues made me think otherwise:

• license metadata is checked but then discarded pypa/flit#525 (Minor, yet annoying)
• pip install packagename[all] support pypa/flit#488

[uranusjr]

The keys here can be arbitrary, everything should work. (Well, if it is a valid extra name.)

[uranusjr]

Flit currently only puts license information in trove classifiers. The license field is pending standardisation. (If we really want to we can probably extend Flit to add that field to the wheel, but the value wouldn’t be very useful due to its lack of standardisation.)

The second issue you linked is requesting a “magic” extra that’s added automatically without declaring. IMO that’s not that good an idea, but manually declaring all should work just fine.

[tatiana]

Thank you very much, @kaxil & @uranusjr , I always learn a lot from your comments!

[tatiana]

It looks great, thank you very much @utkarsharma2 !