Skip to content
This repository has been archived by the owner on Aug 16, 2022. It is now read-only.

Intro, basic queries, delete, cli, odbc #202

Merged
merged 54 commits into from
May 14, 2020

Conversation

ashwinkumar12345
Copy link
Contributor

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

ashwinkumar12345 added 2 commits March 29, 2020 01:50
```sql
SELECT *
FROM accounts
WHERE _id = 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I missing something, or is Amber indexed above with an ID of 1 in your _bulk statement?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, changed "_id = 1".


```json
PUT accounts/_bulk?refresh
{"index":{"_id":"1"}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where's this data from? Just curious. If there's a reference here, I'm not catching it.

Copy link
Contributor Author

@ashwinkumar12345 ashwinkumar12345 Apr 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


### Search and aggregate data

Use the `SELECT` clause along with `FROM`, `WHERE`, `GROUP BY`, `HAVING`, `ORDER BY`, and `LIMIT` to search and aggregate data.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clause, along with


Use the `SELECT` clause along with `FROM`, `WHERE`, `GROUP BY`, `HAVING`, `ORDER BY`, and `LIMIT` to search and aggregate data.

Among these clauses, `SELECT` and `FROM` are required, as they specify which fields to retrieve and which indices to retrieve them from. All other clauses are optional and you can use them according to your requirements.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are optional. Use them according to your needs.

@@ -56,8 +56,375 @@ When you return data in CSV or raw format, each row corresponds to a *document*,

## User interfaces
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's mention right at the top that, y'know, we have a UI now, it's good for use cases X and Y, and include a screenshot.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashwinkumar12345 Could you help change the GET request above to the correspondent POST request? Because we're removing GET endpoint very soon due to security concern.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the confusion but a POST request should look like the sample call here: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/sql-support.html.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed.

Comment on lines 18 to 20
```sql
GET https://<host>:<port>/_opendistro/_sql?sql=select * from my-index limit 50
POST _opendistro/_sql?sql=select * from my-index limit 50
```
Copy link
Contributor

@dai-chen dai-chen Apr 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GET endpoint is removed already due to security concern in opendistro-for-elasticsearch/sql#414. And POST request requires SQL query present in request body rather than URL parameter. Could you help remove it from the documentation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed. Please review.

@aetter aetter added the upcoming release Don't merge until the version or feature is available label Apr 30, 2020

- `endpoint`: You do not need to specify an option, anything that follows the launch command `odfesql` is considered as the endpoint. If you do not provide an endpoint, by default, the SQL CLI connects to http://localhost:9200.
- `-u/-w`: Supports username and password for HTTP basic authentication, such as with the security plugin or fine-grained access control for Amazon Elasticsearch Service.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing an auth option here --aws-auth.
"--aws-auth: turn on to use AWS sigV4 authentication. It can be configured by AWS CLI aws configure command. If provided with this option, ODFE SQL CLI will try to retrieve local aws configuration to authenticate and connect"
Basically, this option is only needed when user connects to Amazon Elasticsearch endpoint

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be --aws-auth.

@ashwinkumar12345 ashwinkumar12345 changed the title Intro, basic queries, delete, cli Intro, basic queries, delete, cli, odbc May 11, 2020
"size": 5,
"status": 200
}
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add the following to make cursor protocol complete.


To fetch subsequent pages, use the cursor from last response

 
>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opendistro/_sql -d '{
   "cursor": "d:eyJhIjp7fSwicyI6IkRYRjFaWEo1UVc1a1JtVjBZMmdCQUFBQUFBQUFBQU1XZWpkdFRFRkZUMlpTZEZkeFdsWnJkRlZoYnpaeVVRPT0iLCJjIjpbeyJuYW1lIjoiZmlyc3RuYW1lIiwidHlwZSI6InRleHQifSx7Im5hbWUiOiJsYXN0bmFtZSIsInR5cGUiOiJ0ZXh0In1dLCJmIjo1LCJpIjoiYWNjb3VudHMiLCJsIjo5NTF9"
}'

The result will only have fetch_size number of datarows and cursor. The last page will have only datarows and no cursor. The datarows can have more than fetch_size number of records in case the nested fields are flattened.

{
   "cursor": "d:eyJhIjp7fSwicyI6IkRYRjFaWEo1UVc1a1JtVjBZMmdCQUFBQUFBQUFBQU1XZWpkdFRFRkZUMlpTZEZkeFdsWnJkRlZoYnpaeVVRPT0iLCJjIjpbeyJuYW1lIjoiZmlyc3RuYW1lIiwidHlwZSI6InRleHQifSx7Im5hbWUiOiJsYXN0bmFtZSIsInR5cGUiOiJ0ZXh0In1dLCJmIjo1LCJpIjoiYWNjb3VudHMabcde12345",
  "datarows": [
    [
      "Abbas",
      "Hussain"
    ],
    [
      "Chen",
      "Dai"
    ],
    [
      "Anirudha",
      "Jadhav"
    ],
    [
      "Peng",
      "Huo"
    ],
    [
      "John",
      "Doe"
    ]
  ]
}

The cursor context is automatically cleared on the last page. To explicitly clear the cursor context use _opendistro/_sql/close endpoint.

>> curl -H 'Content-Type: application/json' -X POST localhost:9200/_opendistro/_sql/close -d '{
   "cursor": "d:eyJhIjp7fSwicyI6IkRYRjFaWEo1UVc1a1JtVjBZMmdCQUFBQUFBQUFBQU1XZWpkdFRFRkZUMlpTZEZkeFdsWnJkRlZoYnpaeVVRPT0iLCJjIjpbeyJuYW1lIjoiZmlyc3RuYW1lIiwidHlwZSI6InRleHQifSx7Im5hbWUiOiJsYXN0bmFtZSIsInR5cGUiOiJ0ZXh0In1dLCJmIjo1LCJpIjoiYWNjb3VudHMiLCJsIjo5NTF9"
}'

Response:

{"succeeded":true}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

@aetter aetter merged commit 5aea96f into opendistro:master May 14, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
sql upcoming release Don't merge until the version or feature is available
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants