Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add use compress parallel streaming_load demo #3

Open
wubx opened this issue Sep 16, 2022 · 1 comment
Open

Add use compress parallel streaming_load demo #3

wubx opened this issue Sep 16, 2022 · 1 comment

Comments

@wubx
Copy link
Contributor

wubx commented Sep 16, 2022

No description provided.

@PsiACE
Copy link

PsiACE commented Sep 19, 2022

We may need to load some data in a specific compression format, so a new request header needs to be set up to specify the compression format, such as gzip.

put.setHeader("compression", "gzip");

If you are considering using your own generated data for validation, something like cn.hutool.core.util.ZipUtil can help you compress the generated data. In this example, you can replace this line of code to get a tsv compressed with gzip.

// old, without gzip
// loader.sendData(client, loadUrl, loadData.getBytes());
// new, with gzip
loader.sendData(client, loadUrl, ZipUtil.gzip(loadData.getBytes()));

To import cn.hutool, you need to edit pom.xml to add this dependency.

<dependency>
  <groupId>cn.hutool</groupId>
  <artifactId>hutool-all</artifactId>
  <version>5.8.6</version>
</dependency>

And, of course, import it in java.

import cn.hutool.core.util.ZipUtil;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants