Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated and renamed spark-client.md to spark-metadata-client.md #8042

Closed
wants to merge 1 commit into from
Closed

Updated and renamed spark-client.md to spark-metadata-client.md #8042

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions docs/reference/spark-client.md → docs/reference/spark-metadata-client.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
title: Spark Client
description: The lakeFS Spark client performs operations on lakeFS committed metadata stored in the object store.
title: Spark Metadata Client
description: The lakeFS Spark client performs operations on lakeFS committed Metadata stored in the object store.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why captitalize it everywhere?

parent: Reference
---


# lakeFS Spark Metadata Client

Utilize the power of Spark to interact with the metadata on lakeFS. Possible use cases include:
Utilize the power of Spark to interact with the Metadata on lakeFS. Possible use cases include:

* Creating a DataFrame for listing the objects in a specific commit or branch.
* Computing changes between two commits.
Expand All @@ -16,10 +16,10 @@ Utilize the power of Spark to interact with the metadata on lakeFS. Possible use

## Getting Started

Please note that Spark 2 is no longer supported with the lakeFS metadata client.
Please note that Spark 2 is no longer supported with the lakeFS Metadata client.
{: .note }

The Spark metadata client is compiled for Spark 3.1.2 with Hadoop 3.2.1, but
The Spark Metadata client is compiled for Spark 3.1.2 with Hadoop 3.2.1, but
can work for other Spark versions and higher Hadoop versions.

<div class="tabs">
Expand Down Expand Up @@ -48,12 +48,12 @@ Include this assembled jar (an "Überjar") from S3, from

## Configuration

1. To read metadata from lakeFS, the client should be configured with your lakeFS endpoint and credentials, using the following Hadoop configurations:
1. To read Metadata from lakeFS, the client should be configured with your lakeFS endpoint and credentials, using the following Hadoop configurations:

| Configuration | Description |
|--------------------------------------|--------------------------------------------------------------|
| `spark.hadoop.lakefs.api.url` | lakeFS API endpoint, e.g: `http://lakefs.example.com/api/v1` |
| `spark.hadoop.lakefs.api.access_key` | The access key to use for fetching metadata from lakeFS |
| `spark.hadoop.lakefs.api.access_key` | The access key to use for fetching Metadata from lakeFS |
| `spark.hadoop.lakefs.api.secret_key` | Corresponding lakeFS secret key |

1. The client will also directly interact with your storage using Hadoop FileSystem.
Expand Down Expand Up @@ -102,7 +102,7 @@ Include this assembled jar (an "Überjar") from S3, from
*/
```

1. Run SQL queries on your metadata:
1. Run SQL queries on your Metadata:

```scala
df.createOrReplaceTempView("files")
Expand All @@ -117,7 +117,7 @@ Include this assembled jar (an "Überjar") from S3, from
*/
```

1. Search by user metadata:
1. Search by user Metadata:

```scala
import io.treeverse.clients.LakeFSContext
Expand All @@ -127,12 +127,12 @@ Include this assembled jar (an "Überjar") from S3, from

val key = "SomeKey"
val searchedValue = "val3"
df.select("key", "user_metadata")
df.select("key", "user_Metadata")
.filter(_.getMap[String, String](1).toMap.get(s"X-Amz-Meta-${key}").getOrElse("") == searchedValue)
.show()
/* output example:
+---------+-----------------------------------------------------+
|key |user_metadata |
|key |user_Metadata |
+---------+-----------------------------------------------------+
|file1.txt|{X-Amz-Meta-SomeKey -> val3, X-Amz-Meta-Tag -> blue} |
|file8.txt|{X-Amz-Meta-SomeKey -> val3, X-Amz-Meta-Tag -> green}|
Expand Down
Loading