Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/main' into node-role-remote-se…
Browse files Browse the repository at this point in the history
…archer

Signed-off-by: Tianli Feng <ftianli@amazon.com>

# Conflicts:
#	CHANGELOG.md
  • Loading branch information
Tianli Feng committed Oct 6, 2022
2 parents 4c65296 + 3186935 commit 017b202
Show file tree
Hide file tree
Showing 18 changed files with 372 additions and 21 deletions.
4 changes: 4 additions & 0 deletions .github/workflows/gradle-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ on:
- 'dependabot/**'
pull_request_target:
types: [opened, synchronize, reopened]
workflow_run:
workflows: ["Gradle Precommit"]
types:
- completed

jobs:
gradle-check:
Expand Down
9 changes: 6 additions & 3 deletions .github/workflows/precommit.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
name: Gradle Precommit
on: [pull_request]

jobs:
precommit:
runs-on: ubuntu-latest
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
steps:
- uses: actions/checkout@v2
- name: Set up JDK 11
Expand All @@ -13,4 +16,4 @@ jobs:
distribution: adopt
- name: Run Gradle
run: |
./gradlew precommit --parallel
./gradlew javadoc precommit --parallel
22 changes: 21 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
# CHANGELOG

Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)

## [Unreleased]

### Added

- Add support for s390x architecture ([#4001](https://github.com/opensearch-project/OpenSearch/pull/4001))
- Github workflow for changelog verification ([#4085](https://github.com/opensearch-project/OpenSearch/pull/4085))
- Point in time rest layer changes for create and delete PIT API ([#4064](https://github.com/opensearch-project/OpenSearch/pull/4064))
Expand All @@ -21,6 +24,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- [Segment Replication] Add snapshot and restore tests for segment replication feature ([#3993](https://github.com/opensearch-project/OpenSearch/pull/3993))
- Added missing javadocs for `:example-plugins` modules ([#4540](https://github.com/opensearch-project/OpenSearch/pull/4540))
- Add missing Javadoc tag descriptions in unit tests ([#4629](https://github.com/opensearch-project/OpenSearch/pull/4629))
- Add getter for path field in NestedQueryBuilder ([#4636](https://github.com/opensearch-project/OpenSearch/pull/4636))
- Added precommit support for windows ([#4676](https://github.com/opensearch-project/OpenSearch/pull/4676))
- Added release notes for 1.3.6 ([#4681](https://github.com/opensearch-project/OpenSearch/pull/4681))
- Added precommit support for MacOS ([#4682](https://github.com/opensearch-project/OpenSearch/pull/4682))
- Recommission API changes for service layer ([#4320](https://github.com/opensearch-project/OpenSearch/pull/4320))
- Add a new node role which can provide search capability for remote shard ([#4689](https://github.com/opensearch-project/OpenSearch/pull/4689))

### Dependencies
Expand All @@ -43,8 +51,10 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Bumps `reactor-core` from 3.4.18 to 3.4.23 ([#4548](https://github.com/opensearch-project/OpenSearch/pull/4548))
- Bumps `jempbox` from 1.8.16 to 1.8.17 ([#4550](https://github.com/opensearch-project/OpenSearch/pull/4550))
- Bumps `hadoop-hdfs` from 3.3.3 to 3.3.4 ([#4644](https://github.com/opensearch-project/OpenSearch/pull/4644))
- Bumps `jna` from 5.11.0 to 5.12.1 ([#4656](https://github.com/opensearch-project/OpenSearch/pull/4656))

### Changed

- Dependency updates (httpcore, mockito, slf4j, httpasyncclient, commons-codec) ([#4308](https://github.com/opensearch-project/OpenSearch/pull/4308))
- Use RemoteSegmentStoreDirectory instead of RemoteDirectory ([#4240](https://github.com/opensearch-project/OpenSearch/pull/4240))
- Plugin ZIP publication groupId value is configurable ([#4156](https://github.com/opensearch-project/OpenSearch/pull/4156))
Expand All @@ -59,8 +69,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Further simplification of the ZIP publication implementation ([#4360](https://github.com/opensearch-project/OpenSearch/pull/4360))
- Relax visibility of the HTTP_CHANNEL_KEY and HTTP_SERVER_CHANNEL_KEY to make it possible for the plugins to access associated Netty4HttpChannel / Netty4HttpServerChannel instance ([#4638](https://github.com/opensearch-project/OpenSearch/pull/4638))
- Load the deprecated master role in a dedicated method instead of in setAdditionalRoles() ([#4582](https://github.com/opensearch-project/OpenSearch/pull/4582))
- Include Windows OS in Bootstrap initializeNatives() check for definitelyRunningAsRoot() ([#4656](https://github.com/opensearch-project/OpenSearch/pull/4656))
- Add APIs (GET/PUT) to decommission awareness attribute ([#4261](https://github.com/opensearch-project/OpenSearch/pull/4261))
- Improve Gradle pre-commit checks to pre-empt Jenkins build ([#4660](https://github.com/opensearch-project/OpenSearch/pull/4660))
- Update to Apache Lucene 9.4.0 ([#4661](https://github.com/opensearch-project/OpenSearch/pull/4661))
- Backport Apache Lucene version change for 2.4.0 ([#4677](https://github.com/opensearch-project/OpenSearch/pull/4677))

### Deprecated

Expand All @@ -69,6 +82,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Unused object and import within TransportClusterAllocationExplainAction ([#4639](https://github.com/opensearch-project/OpenSearch/pull/4639))

### Fixed

- `opensearch-service.bat start` and `opensearch-service.bat manager` failing to run ([#4289](https://github.com/opensearch-project/OpenSearch/pull/4289))
- PR reference to checkout code for changelog verifier ([#4296](https://github.com/opensearch-project/OpenSearch/pull/4296))
- `opensearch.bat` and `opensearch-service.bat install` failing to run, missing logs directory ([#4305](https://github.com/opensearch-project/OpenSearch/pull/4305))
Expand All @@ -93,6 +107,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Fixed the ignore_malformed setting to also ignore objects ([#4494](https://github.com/opensearch-project/OpenSearch/pull/4494))
- [Segment Replication] Ignore lock file when testing cleanupAndPreserveLatestCommitPoint ([#4544](https://github.com/opensearch-project/OpenSearch/pull/4544))
- Updated jackson to 2.13.4 and snakeyml to 1.32 ([#4556](https://github.com/opensearch-project/OpenSearch/pull/4556))
- Fixing PIT flaky tests ([4632](https://github.com/opensearch-project/OpenSearch/pull/4632))
- Fixed day of year defaulting for round up parser ([#4627](https://github.com/opensearch-project/OpenSearch/pull/4627))
- Fixed the SnapshotsInProgress error during index deletion ([#4570](https://github.com/opensearch-project/OpenSearch/pull/4570))
- [Segment Replication] Adding check to make sure checkpoint is not processed when a shard's shard routing is primary ([#4630](https://github.com/opensearch-project/OpenSearch/pull/4630))
Expand All @@ -102,12 +117,17 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- [Segment Replication] Adding check to make sure checkpoint is not processed when a shard's shard routing is primary ([#4630](https://github.com/opensearch-project/OpenSearch/pull/4630))
- [Bug]: Fixed invalid location of JDK dependency for arm64 architecture([#4613](https://github.com/opensearch-project/OpenSearch/pull/4613))
- [Bug]: Alias filter lost after rollover ([#4499](https://github.com/opensearch-project/OpenSearch/pull/4499))
- Attempt to fix Github workflow for Gradle Check job ([#4679](https://github.com/opensearch-project/OpenSearch/pull/4679))
- Fix flaky DecommissionControllerTests.testTimesOut ([4683](https://github.com/opensearch-project/OpenSearch/pull/4683))

### Security

- CVE-2022-25857 org.yaml:snakeyaml DOS vulnerability ([#4341](https://github.com/opensearch-project/OpenSearch/pull/4341))

## [2.x]

### Added

- Github workflow for changelog verification ([#4085](https://github.com/opensearch-project/OpenSearch/pull/4085))
- Label configuration for dependabot PRs ([#4348](https://github.com/opensearch-project/OpenSearch/pull/4348))
- Added RestLayer Changes for PIT stats ([#4217](https://github.com/opensearch-project/OpenSearch/pull/4217))
Expand All @@ -123,11 +143,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
### Removed

### Fixed

- PR reference to checkout code for changelog verifier ([#4296](https://github.com/opensearch-project/OpenSearch/pull/4296))
- Commit workflow for dependabot changelog helper ([#4331](https://github.com/opensearch-project/OpenSearch/pull/4331))

### Security


[Unreleased]: https://github.com/opensearch-project/OpenSearch/compare/2.2.0...HEAD
[2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.2.0...2.x
2 changes: 1 addition & 1 deletion buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ dependencies {
api 'com.netflix.nebula:gradle-info-plugin:11.3.3'
api 'org.apache.rat:apache-rat:0.13'
api 'commons-io:commons-io:2.7'
api "net.java.dev.jna:jna:5.11.0"
api "net.java.dev.jna:jna:5.12.1"
api 'gradle.plugin.com.github.johnrengelman:shadow:7.1.2'
api 'org.jdom:jdom2:2.0.6.1'
api 'org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.7.10'
Expand Down
2 changes: 1 addition & 1 deletion buildSrc/version.properties
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ slf4j = 1.7.36
asm = 9.3

# when updating the JNA version, also update the version in buildSrc/build.gradle
jna = 5.5.0
jna = 5.12.1

netty = 4.1.79.Final
joda = 2.10.13
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;

Expand Down Expand Up @@ -71,7 +72,7 @@ public void testCreateAndDeletePit() throws IOException {
assertTrue(deletePitResponse.getDeletePitResults().get(0).getPitId().equals(createPitResponse.getId()));
}

public void testDeleteAllAndListAllPits() throws IOException {
public void testDeleteAllAndListAllPits() throws IOException, InterruptedException {
CreatePitRequest pitRequest = new CreatePitRequest(new TimeValue(1, TimeUnit.DAYS), true, "index");
CreatePitResponse pitResponse = execute(pitRequest, highLevelClient()::createPit, highLevelClient()::createPitAsync);
CreatePitResponse pitResponse1 = execute(pitRequest, highLevelClient()::createPit, highLevelClient()::createPitAsync);
Expand All @@ -90,16 +91,19 @@ public void testDeleteAllAndListAllPits() throws IOException {
List<String> pits = getAllPitResponse.getPitInfos().stream().map(r -> r.getPitId()).collect(Collectors.toList());
assertTrue(pits.contains(pitResponse.getId()));
assertTrue(pits.contains(pitResponse1.getId()));
CountDownLatch countDownLatch = new CountDownLatch(1);
ActionListener<DeletePitResponse> deletePitListener = new ActionListener<>() {
@Override
public void onResponse(DeletePitResponse response) {
countDownLatch.countDown();
for (DeletePitInfo deletePitInfo : response.getDeletePitResults()) {
assertTrue(deletePitInfo.isSuccessful());
}
}

@Override
public void onFailure(Exception e) {
countDownLatch.countDown();
if (!(e instanceof OpenSearchStatusException)) {
throw new AssertionError("Delete all failed");
}
Expand All @@ -123,6 +127,7 @@ public void onFailure(Exception e) {
};
highLevelClient().getAllPitsAsync(RequestOptions.DEFAULT, getPitsListener);
highLevelClient().deleteAllPitsAsync(RequestOptions.DEFAULT, deletePitListener);
assertTrue(countDownLatch.await(10, TimeUnit.SECONDS));
// validate no pits case
getAllPitResponse = highLevelClient().getAllPits(RequestOptions.DEFAULT);
assertTrue(getAllPitResponse.getPitInfos().size() == 0);
Expand Down
10 changes: 10 additions & 0 deletions release-notes/opensearch.release-notes-1.3.6.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
## 2022-10-04 Version 1.3.6 Release Notes

### Upgrades
* Update to Netty 4.1.80.Final ([#4379](https://github.com/opensearch-project/OpenSearch/pull/4379))
* Revert to Netty 4.1.79.Final ([#4433](https://github.com/opensearch-project/OpenSearch/pull/4433))
* Bumps jackson and snakeyaml dependencies ([#4600](https://github.com/opensearch-project/OpenSearch/pull/4600))
* Bump commons-configuration2 from 2.7 to 2.8.0 in /plugins/repository-hdfs ([#4646](https://github.com/opensearch-project/OpenSearch/pull/4646))

### Bug Fixes
* Set analyzer to regex query string search ([#4220](https://github.com/opensearch-project/OpenSearch/pull/4220))
1 change: 1 addition & 0 deletions server/licenses/jna-5.12.1.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
b1e93a735caea94f503e95e6fe79bf9cdc1e985d
1 change: 0 additions & 1 deletion server/licenses/jna-5.5.0.jar.sha1

This file was deleted.

2 changes: 1 addition & 1 deletion server/src/main/java/org/opensearch/Version.java
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ public class Version implements Comparable<Version>, ToXContentFragment {
public static final Version V_2_2_2 = new Version(2020299, org.apache.lucene.util.Version.LUCENE_9_3_0);
public static final Version V_2_3_0 = new Version(2030099, org.apache.lucene.util.Version.LUCENE_9_3_0);
public static final Version V_2_3_1 = new Version(2030199, org.apache.lucene.util.Version.LUCENE_9_3_0);
public static final Version V_2_4_0 = new Version(2040099, org.apache.lucene.util.Version.LUCENE_9_3_0);
public static final Version V_2_4_0 = new Version(2040099, org.apache.lucene.util.Version.LUCENE_9_4_0);
public static final Version V_3_0_0 = new Version(3000099, org.apache.lucene.util.Version.LUCENE_9_4_0);
public static final Version CURRENT = V_3_0_0;

Expand Down
124 changes: 124 additions & 0 deletions server/src/main/java/org/opensearch/bootstrap/JNAAdvapi32Library.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.bootstrap;

import com.sun.jna.Native;
import com.sun.jna.Pointer;
import com.sun.jna.ptr.IntByReference;
import com.sun.jna.ptr.PointerByReference;
import com.sun.jna.Structure;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.lucene.util.Constants;

import java.util.List;

/**
* Library for Windows/Advapi32
*
* @opensearch.internal
*/
final class JNAAdvapi32Library {

private static final Logger logger = LogManager.getLogger(JNAAdvapi32Library.class);

private static final class Holder {
private static final JNAAdvapi32Library instance = new JNAAdvapi32Library();
}

private JNAAdvapi32Library() {
if (Constants.WINDOWS) {
try {
Native.register("advapi32");
logger.debug("windows/Advapi32 library loaded");
} catch (NoClassDefFoundError e) {
logger.warn("JNA not found. native methods and handlers will be disabled.");
} catch (UnsatisfiedLinkError e) {
logger.warn("unable to link Windows/Advapi32 library. native methods and handlers will be disabled.");
}
}
}

static JNAAdvapi32Library getInstance() {
return Holder.instance;
}

/**
* Access right required to query an access token.
* Used by {@link #OpenProcessToken(Pointer, int, PointerByReference)}.
*
* https://learn.microsoft.com/en-us/windows/win32/secauthz/access-rights-for-access-token-objects
*/
public static final int TOKEN_QUERY = 0x0008;

/**
* TOKEN_INFORMATION_CLASS enumeration value that specifies the type of information being assigned to or retrieved from an access token.
* Used by {@link #GetTokenInformation(Pointer, int, Structure, int, IntByReference)}.
*
* https://learn.microsoft.com/en-us/windows/win32/api/winnt/ne-winnt-token_information_class
*/
public static final int TOKEN_ELEVATION = 0x14;

/**
* Native call to the Advapi32 API to open the access token associated with a process.
*
* @param processHandle Handle to the process whose access token is opened.
* The process must have the PROCESS_QUERY_INFORMATION access permission.
* @param desiredAccess Specifies an access mask that specifies the requested types of access to the access token.
* These requested access types are compared with the discretionary access control list (DACL) of the token to determine which accesses are granted or denied.
* @param tokenHandle Pointer to a handle that identifies the newly opened access token when the function returns.
* @return If the function succeeds, the return value is true.
* If the function fails, the return value is false.
* To get extended error information, call GetLastError.
*/
native boolean OpenProcessToken(Pointer processHandle, int desiredAccess, PointerByReference tokenHandle);

/**
* Retrieves a specified type of information about an access token.
* The calling process must have appropriate access rights to obtain the information.
*
* @param tokenHandle Handle to an access token from which information is retrieved.
* If TokenInformationClass specifies TokenSource, the handle must have TOKEN_QUERY_SOURCE access.
* For all other TokenInformationClass values, the handle must have TOKEN_QUERY access.
* @param tokenInformationClass Specifies a value from the TOKEN_INFORMATION_CLASS enumerated type to identify the type of information the function retrieves.
* @param tokenInformation Pointer to a buffer the function fills with the requested information.
* The structure put into this buffer depends upon the type of information specified by the TokenInformationClass parameter.
* @param tokenInformationLength Specifies the size, in bytes, of the buffer pointed to by the TokenInformation parameter.
* If TokenInformation is NULL, this parameter must be zero.
* @param returnLength Pointer to a variable that receives the number of bytes needed for the buffer pointed to by the TokenInformation parameter.
* If this value is larger than the value specified in the TokenInformationLength parameter, the function fails and stores no data in the buffer.
* @return If the function succeeds, the return value is true.
* If the function fails, the return value is zero.
* To get extended error information, call GetLastError.
*/
native boolean GetTokenInformation(
Pointer tokenHandle,
int tokenInformationClass,
Structure tokenInformation,
int tokenInformationLength,
IntByReference returnLength
);

/**
* The TOKEN_ELEVATION structure indicates whether a token has elevated privileges.
*
* https://learn.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-token_elevation
*/
public static class TokenElevation extends Structure {
/**
* A nonzero value if the token has elevated privileges; otherwise, a zero value.
*/
public int TokenIsElevated;

@Override
protected List<String> getFieldOrder() {
return List.of("TokenIsElevated");
}
}
}
Loading

0 comments on commit 017b202

Please sign in to comment.