Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

change the e2e test maven to use a fixed maven version #95

Merged
merged 1 commit into from
May 23, 2022
Merged

change the e2e test maven to use a fixed maven version #95

merged 1 commit into from
May 23, 2022

Conversation

imaffe
Copy link

@imaffe imaffe commented May 23, 2022

What is the purpose of the change

(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)

Brief change log

(for example:)

  • The TaskInfo is stored in the blob store on job creation time as a persistent artifact
  • Deployments RPC transmits only the blob storage reference
  • TaskManagers retrieve the TaskInfo from the blob cache

Verifying this change

Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (100MB)
  • Extended integration test for recovery after master (JobManager) failure
  • Added test that validates that TaskInfo is transferred only once across recoveries
  • Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@imaffe imaffe merged commit f7d1135 into develop May 23, 2022
@delete-merged-branch delete-merged-branch bot deleted the develop-0523-modify-daily-e2e branch May 23, 2022 14:30
imaffe added a commit that referenced this pull request Jun 21, 2022
@imaffe
change the e2e test maven to use a fixed maven version (#95)
2f179f0
imaffe added a commit that referenced this pull request Jun 21, 2022
@imaffe
change the e2e test maven to use a fixed maven version (#95)
a580d91
imaffe added a commit that referenced this pull request Jun 21, 2022
@imaffe
Revert "change the e2e test maven to use a fixed maven version (#95)"
ee52eab 
This reverts commit a580d91.
imaffe added a commit that referenced this pull request Nov 11, 2022
@imaffe
Include all commits from develop (1.15)
b05e808 
89: setup flink PR checkt pipeline (#90)

change the e2e test maven to use a fixed maven version (#95)

fix a typo in daily e2e test (#96)

Introduce SQL Connector (#92)

* [FLINK-25686][Connector/Pulsar]: Add schema evolution support for pulsar source connector

* [FLINK-25686] fix according to review

* [SQL Connector]: add source and sink SQL connector

* [FLINK-25686] fix a test and run PR

98: add PulsarCatalog and tests (#107)

* 98: add PulsarCatalog and tests

104: add sql-gateway dependencies json (#110)

104: add protobuf jar dependency and declare as not provided (#114)

[FLINK-27881][Connector/Pulsar] The key(String) in PulsarMessageBuilder returns null.

[FLINK-27399][Connector/Pulsar] Modify start cursor and stop cursor, unified the behaviors.

[FLINK-27399][Connector/Pulsar] Create the initial subscription instead seek every time. This should fix the wrong position setting.

[FLINK-28082][Connector/Pulsar] Add end-to-end encryption support for Pulsar connector.

[FLINK-28083][Connector/Pulsar] PulsarSource work with object-reusing DeserializationSchema.

[FLINK-26182][Connector/pulsar] Extract common logic from Pulsar source test tools.

[FLINK-26182][Connector/pulsar] Create a e2e tests for Pulsar sink connector based on connector testing framework.

[FLINK-27231][Connector/Pulsar] Bump pulsar to 2.10.0

[FLINK-27611][Connector/Pulsar] Fix ConcurrentModificationException during Pulsar checkpoint notification.

[FLINK-27400][Connector/Pulsar] Never subscribe the Pulsar internal system name by using regex.

[FLINK-28084][Connector/Pulsar] Remove the reconsume later logic for Pulsar unordered reader.

[FLINK-28085][Connector/Pulsar] Close the finished splits properly for Pulsar reader.

[FLINK-28085][Connector/Pulsar] Close the pending transaction before close the reader.

Fix the compile error in code.

[FLINK-28351][Connector/Pulsar] Add dynamic sink topic support for Pulsar (#129)

feat: Add open method for dynamic topic extractor. (#130)

fix: always delete topic even when there are subscribers

121: add new tests for documentation
100: timestamp should be larger than 0

143: create default database is not exist

136: support value.format config

[FLINK-26931][Connector/Pulsar] Make the producer name and consumer name unique for each instance. (#152)

[FLINK-28609][Connector/Pulsar] PulsarSchema didn't get properly serialized.

[FLINK-28609][Connector/Pulsar] Drop useless end-to-end test classes.

[FLINK-25686][Connector/Pulsar]: Add schema evolution support for pulsar source connector.

137: use a random default subscription name in Pulsar SQL Connector

138: update the documentation

Create new CODEOWNERS file

[FLINK-28820][Connector/Pulsar] Drop MailboxExecutor, improve the writing performance for PulsarSink.

[FLINK-28820][Connector/Pulsar] Deprecated unused message writing queue length option.

[FLINK-28870][Connector/Pulsar] Add fetch time option for Pulsar Source. Avoid hanging on small message income rates.

[FLINK-28505][Connector/Pulsar] Support topic auto creation for Pulsar sink.

[FLINK-28870][Connector/Pulsar] Increase fetch time for tests.

[FLINK-28870][Connector/Pulsar] Increase fetch time for tests.

163: support unbounded stop cursor in SQL Connector

176: add functional testing for StopCursor

flink-183: add pulsar client config guides in the doc

flink-173: add releationship between apache/flink and streamnative/flink

drop extra empty line

Apply suggestions from code review

Co-authored-by: Huanli Meng <48120384+Huanli-Meng@users.noreply.github.com>

170: add properties metadata testing and documentation

Backport FLINK-27399 and support new start cursor and stop cursor.

Backport FLINK-27917 and drop consumer seek method. (#204)

[FLINK-28960][Connector/Pulsar] Add jaxb-api back to flink-connector-pulsar (#205)

PulsarSchema: fix the byte array serialization issues. (#207)

166: support upsert pulsar

115: add keySerialization open() method

[FLINK-28934][Connector/pulsar] Better split assignment for different Pulsar's subscriptions. (#220)

127: fix EOF exception in tests

156: introduce basic protobuf native format

Add common request helper for PulsarAdmin (#222)

chore: add example on how to enable jsr310 with avro schema.

fix: event time should exceed zero.

fix: Make Key_Shared subscription works, add new KeyHash RangeGenerator. (#226)

103: skip rat check

feat: support Key_Shared subscription in old ways. (#232)

221: add some description on avro schema

Add auth builder. (#244)

fix: remove pulsar.consumer.autoUpdatePartitionsIntervalSeconds.

Fix: wrong batch size assertion. (#247)

146: add changelogITCase for Pulsar SQL Connector

add e2e tests

feat: crash the taskmanager when meet errors.

[FLINK-29709][Connector/Pulsar] Bump the Pulsar to latest 2.10.2 (#250)

Move message release logic to correct space. (#251)

fix: make sure pulsar schema could serialize and deserialize in 2.10.2 (#252)

Cut Develop for 1.15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@imaffe