Skip to content
This repository has been archived by the owner on Jul 20, 2023. It is now read-only.

Commit

Permalink
feat(samples): Add POSIX & Manifest samples (#67)
Browse files Browse the repository at this point in the history
* feat(samples): Add POSIX samples

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* Fix: `fs.promises` import

* fix: clean-up created transfer job

* chore: Add (safe) debug log

* fix: misc bugs

* chore: remove debug log

* feat: add POSIX to POSIX sample

- Also, update tests and comments

* chore: typo & clean-up

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* refactor: styling

* feat: Add POSIX Download sample

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* fix: Add ending '/'

* style: shorten variable

* feat: Transfer Manifest request

* fix: typo

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* fix: `replaceAll` (not available in Node 12) -> `replace`

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
  • Loading branch information
danielbankhead and gcf-owl-bot[bot] authored Jun 14, 2022
1 parent 795658b commit 1056c9f
Show file tree
Hide file tree
Showing 10 changed files with 822 additions and 0 deletions.
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,11 @@ Samples are in the [`samples/`](https://github.com/googleapis/nodejs-storage-tra
| Aws-request | [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/aws-request.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/aws-request.js,samples/README.md) |
| Check-latest-transfer-operation | [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/check-latest-transfer-operation.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/check-latest-transfer-operation.js,samples/README.md) |
| Get-transfer-job-with-retries | [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/get-transfer-job-with-retries.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/get-transfer-job-with-retries.js,samples/README.md) |
| Manifest-request | [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/manifest-request.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/manifest-request.js,samples/README.md) |
| Nearline-request | [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/nearline-request.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/nearline-request.js,samples/README.md) |
| Posix-download | [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/posix-download.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/posix-download.js,samples/README.md) |
| Posix-request | [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/posix-request.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/posix-request.js,samples/README.md) |
| Posix-to-posix-request | [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/posix-to-posix-request.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/posix-to-posix-request.js,samples/README.md) |
| Quickstart | [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/quickstart.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/quickstart.js,samples/README.md) |
| Transfer-check | [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/transfer-check.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/transfer-check.js,samples/README.md) |

Expand Down
72 changes: 72 additions & 0 deletions samples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,11 @@
* [Aws-request](#aws-request)
* [Check-latest-transfer-operation](#check-latest-transfer-operation)
* [Get-transfer-job-with-retries](#get-transfer-job-with-retries)
* [Manifest-request](#manifest-request)
* [Nearline-request](#nearline-request)
* [Posix-download](#posix-download)
* [Posix-request](#posix-request)
* [Posix-to-posix-request](#posix-to-posix-request)
* [Quickstart](#quickstart)
* [Transfer-check](#transfer-check)

Expand Down Expand Up @@ -85,6 +89,23 @@ __Usage:__



### Manifest-request

View the [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/manifest-request.js).

[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/manifest-request.js,samples/README.md)

__Usage:__


`node samples/manifest-request.js`


-----




### Nearline-request

View the [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/nearline-request.js).
Expand All @@ -102,6 +123,57 @@ __Usage:__



### Posix-download

View the [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/posix-download.js).

[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/posix-download.js,samples/README.md)

__Usage:__


`node samples/posix-download.js`


-----




### Posix-request

View the [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/posix-request.js).

[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/posix-request.js,samples/README.md)

__Usage:__


`node samples/posix-request.js`


-----




### Posix-to-posix-request

View the [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/posix-to-posix-request.js).

[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage-transfer&page=editor&open_in_editor=samples/posix-to-posix-request.js,samples/README.md)

__Usage:__


`node samples/posix-to-posix-request.js`


-----




### Quickstart

View the [source code](https://github.com/googleapis/nodejs-storage-transfer/blob/main/samples/quickstart.js).
Expand Down
99 changes: 99 additions & 0 deletions samples/manifest-request.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

'use strict';

async function main(
projectId = 'my-project',
sourceAgentPoolName = '',
rootDirectory = '',
gcsSinkBucket = '',
manifestLocation = ''
) {
// [START storagetransfer_manifest_request]

// Imports the Google Cloud client library
const {
StorageTransferServiceClient,
} = require('@google-cloud/storage-transfer');

/**
* TODO(developer): Uncomment the following lines before running the sample.
*/
// Your project id
// const projectId = 'my-project'

// The agent pool associated with the POSIX data source. Defaults to the default agent
// const sourceAgentPoolName = 'projects/my-project/agentPools/transfer_service_default'

// The root directory path on the source filesystem
// const rootDirectory = '/directory/to/transfer/source',

// The ID of the GCS bucket to transfer data to
// const gcsSinkBucket = 'my-sink-bucket'

// Transfer manifest location. Must be a
// const manifestLocation = 'gs://my-bucket/sample_manifest.csv'

// Creates a client
const client = new StorageTransferServiceClient();

/**
* Creates a request to transfer from the local file system to the sink bucket
*/
async function transferDirectory() {
const createRequest = {
transferJob: {
projectId,
transferSpec: {
sourceAgentPoolName,
posixDataSource: {
rootDirectory,
},
gcsDataSink: {bucketName: gcsSinkBucket},
transferManifest: {
location: manifestLocation,
},
},
status: 'ENABLED',
},
};

// Runs the request and creates the job
const [transferJob] = await client.createTransferJob(createRequest);

const runRequest = {
jobName: transferJob.name,
projectId: projectId,
};

await client.runTransferJob(runRequest);

console.log(
`Created and ran a transfer job from '${rootDirectory}' to '${gcsSinkBucket}' using manifest \`${manifestLocation}\` with name ${transferJob.name}`
);
}

transferDirectory();
// [END storagetransfer_manifest_request]
}

main(...process.argv.slice(2));

process.on('unhandledRejection', err => {
console.error(err.message);
process.exitCode = 1;
});
99 changes: 99 additions & 0 deletions samples/posix-download.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

'use strict';

async function main(
projectId = 'my-project',
sinkAgentPoolName = '',
gcsSourceBucket = '',
gcsSourcePath = '',
rootDirectory = ''
) {
// [START storagetransfer_download_to_posix]

// Imports the Google Cloud client library
const {
StorageTransferServiceClient,
} = require('@google-cloud/storage-transfer');

/**
* TODO(developer): Uncomment the following lines before running the sample.
*/
// Your project id
// const projectId = 'my-project'

// The agent pool associated with the POSIX data sink. Defaults to the default agent
// const sinkAgentPoolName = 'projects/my-project/agentPools/transfer_service_default'

// Google Cloud Storage source bucket name
// const gcsSourceBucket = 'my-gcs-source-bucket'

// An optional path on the Google Cloud Storage bucket to download from
// const gcsSourcePath = 'foo/bar/'

// The root directory path on the source filesystem
// const rootDirectory = '/directory/to/transfer/source',

// Creates a client
const client = new StorageTransferServiceClient();

/**
* Creates a request to transfer from the local file system to the sink bucket
*/
async function transferDirectory() {
const createRequest = {
transferJob: {
projectId,
transferSpec: {
sinkAgentPoolName,
gcsDataSource: {
bucketName: gcsSourceBucket,
path: gcsSourcePath,
},
posixDataSink: {
rootDirectory,
},
},
status: 'ENABLED',
},
};

// Runs the request and creates the job
const [transferJob] = await client.createTransferJob(createRequest);

const runRequest = {
jobName: transferJob.name,
projectId: projectId,
};

await client.runTransferJob(runRequest);

console.log(
`Downloading from '${gcsSourceBucket}' (path: \`${gcsSourcePath}\`) to '${rootDirectory}' with name ${transferJob.name}`
);
}

transferDirectory();
// [END storagetransfer_download_to_posix]
}

main(...process.argv.slice(2));

process.on('unhandledRejection', err => {
console.error(err.message);
process.exitCode = 1;
});
92 changes: 92 additions & 0 deletions samples/posix-request.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

'use strict';

async function main(
projectId = 'my-project',
sourceAgentPoolName = '',
rootDirectory = '',
gcsSinkBucket = ''
) {
// [START storagetransfer_transfer_from_posix]

// Imports the Google Cloud client library
const {
StorageTransferServiceClient,
} = require('@google-cloud/storage-transfer');

/**
* TODO(developer): Uncomment the following lines before running the sample.
*/
// Your project id
// const projectId = 'my-project'

// The agent pool associated with the POSIX data source. Defaults to the default agent
// const sourceAgentPoolName = 'projects/my-project/agentPools/transfer_service_default'

// The root directory path on the source filesystem
// const rootDirectory = '/directory/to/transfer/source',

// The ID of the GCS bucket to transfer data to
// const gcsSinkBucket = 'my-sink-bucket'

// Creates a client
const client = new StorageTransferServiceClient();

/**
* Creates a request to transfer from the local file system to the sink bucket
*/
async function transferDirectory() {
const createRequest = {
transferJob: {
projectId,
transferSpec: {
sourceAgentPoolName,
posixDataSource: {
rootDirectory,
},
gcsDataSink: {bucketName: gcsSinkBucket},
},
status: 'ENABLED',
},
};

// Runs the request and creates the job
const [transferJob] = await client.createTransferJob(createRequest);

const runRequest = {
jobName: transferJob.name,
projectId: projectId,
};

await client.runTransferJob(runRequest);

console.log(
`Created and ran a transfer job from '${rootDirectory}' to '${gcsSinkBucket}' with name ${transferJob.name}`
);
}

transferDirectory();
// [END storagetransfer_transfer_from_posix]
}

main(...process.argv.slice(2));

process.on('unhandledRejection', err => {
console.error(err.message);
process.exitCode = 1;
});
Loading

0 comments on commit 1056c9f

Please sign in to comment.