-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[DOP-13252] Improve MySQL documentation
- Loading branch information
Showing
9 changed files
with
716 additions
and
35 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
Improve MySQL documentation: | ||
* Add "Types" section describing mapping between MySQL and Spark types | ||
* Add "Prerequisites" section describing different aspects of connecting to MySQL | ||
* Separate documentation of ``DBReader`` and ``MySQL.sql`` | ||
* Add examples for ``MySQL.fetch`` and ``MySQL.execute`` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
.. _mysql-prerequisites: | ||
|
||
Prerequisites | ||
============= | ||
|
||
Version Compatibility | ||
--------------------- | ||
|
||
* MySQL server versions: 5.7, 8.0 | ||
* Spark versions: 2.3.x - 3.5.x | ||
* Java versions: 8 - 20 | ||
|
||
See `official documentation <https://dev.mysql.com/doc/relnotes/connector-j/en/news-8-0-33.html>`_. | ||
|
||
Installing PySpark | ||
------------------ | ||
|
||
To use MySQL connector you should have PySpark installed (or injected to ``sys.path``) | ||
BEFORE creating the connector instance. | ||
|
||
See :ref:`install-spark` installation instruction for more details. | ||
|
||
Connecting to MySQL | ||
----------------------- | ||
|
||
Connection host | ||
~~~~~~~~~~~~~~~ | ||
|
||
It is possible to connect to MySQL by using either DNS name of host or it's IP address. | ||
|
||
If you're using MySQL cluster, it is currently possible to connect only to **one specific node**. | ||
Connecting to multiple nodes to perform load balancing, as well as automatic failover to new master/replica are not supported. | ||
|
||
Connection port | ||
~~~~~~~~~~~~~~~ | ||
|
||
Connection is usually performed to port 3306. Port may differ for different MySQL instances. | ||
Please ask your MySQL administrator to provide required information. | ||
|
||
Required grants | ||
~~~~~~~~~~~~~~~ | ||
|
||
Ask your MySQL cluster administrator to set following grants for a user, | ||
used for creating a connection: | ||
|
||
.. tabs:: | ||
|
||
.. code-tab:: sql Read + Write | ||
|
||
-- allow external tables in the same schema as target table | ||
GRANT CREATE ON myschema.* TO username@'192.168.1.%'; | ||
|
||
-- allow read & write access to specific table | ||
GRANT SELECT, INSERT ON myschema.mytable TO username@'192.168.1.%'; | ||
|
||
.. code-tab:: sql Read only | ||
|
||
-- allow read access to specific table | ||
GRANT SELECT ON myschema.mytable TO username@'192.168.1.%'; | ||
|
||
In example above ``'192.168.1.%''`` is a network subnet ``192.168.1.0 - 192.168.1.255`` | ||
where Spark driver and executors are running. To allow connecting user from any IP, use ``'%'`` (not secure!). | ||
|
||
More details can be found in `official documentation <https://dev.mysql.com/doc/refman/en/grant.html>`_. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
.. _clickhouse-sql: | ||
|
||
Reading from Clickhouse using ``Clickhouse.sql`` | ||
================================================ | ||
|
||
.. warning:: | ||
|
||
Please take into account :ref:`clickhouse-types` | ||
|
||
:obj:`Clickhouse.sql <onetl.connection.db_connection.clickhouse.connection.Clickhouse.sql>` allows passing custom SQL query, | ||
but does not support incremental strategies. | ||
|
||
Method also accepts :obj:`JDBCReadOptions <onetl.connection.db_connection.jdbc.options.JDBCReadOptions>`. | ||
|
||
Syntax support | ||
-------------- | ||
|
||
Only queries with the following syntax are supported: | ||
|
||
* ``SELECT ...`` | ||
* ``WITH alias AS (...) SELECT ...`` | ||
|
||
Queries like ``SHOW ...`` are not supported. | ||
|
||
This method also does not support multiple queries in the same operation, like ``SET ...; SELECT ...;``. | ||
|
||
Examples | ||
-------- | ||
|
||
.. code-block:: python | ||
from onetl.connection import Clickhouse | ||
clickhouse = Clickhouse(...) | ||
df = clickhouse.sql( | ||
""" | ||
SELECT | ||
id, | ||
key, | ||
CAST(value AS text) value, | ||
updated_at | ||
FROM | ||
some.mytable | ||
WHERE | ||
key = 'something' | ||
""", | ||
options=Clickhouse.ReadOptions(partition_column="id", num_partitions=10), | ||
) | ||
References | ||
---------- | ||
|
||
.. currentmodule:: onetl.connection.db_connection.clickhouse.connection | ||
|
||
.. automethod:: Clickhouse.sql |
Oops, something went wrong.