Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the hazelcast discovery to use the service account and https. #2

Merged
merged 1 commit into from
May 19, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

<groupId>com.github.pires</groupId>
<artifactId>hazelcast-kubernetes-bootstrapper</artifactId>
<version>0.2-SNAPSHOT</version>
<version>0.3-SNAPSHOT</version>
<packaging>jar</packaging>

<name>Hazelcast Kubernetes Bootstrapper</name>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,21 @@
import com.hazelcast.core.Hazelcast;
import java.io.IOException;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.security.cert.X509Certificate;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.util.List;
import java.util.UUID;
import java.util.concurrent.CopyOnWriteArrayList;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSession;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.CommandLineRunner;
Expand All @@ -43,46 +55,70 @@ public class HazelcastDiscoveryController implements CommandLineRunner {

@JsonIgnoreProperties(ignoreUnknown = true)
static class Address {

public String IP;
}

@JsonIgnoreProperties(ignoreUnknown = true)
static class Subset {

public List<Address> addresses;
}

@JsonIgnoreProperties(ignoreUnknown = true)
static class Endpoints {

public List<Subset> subsets;
}

private static String getServiceAccountToken() throws IOException {
String file = "/var/run/secrets/kubernetes.io/serviceaccount/token";
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this file provisioned?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's automatically included in all pods now. See:

kubernetes/kubernetes#7101

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this present in 0.17.0? Cause

015-05-20T11:41:15.444282980Z 2015-05-20 11:41:15.439  INFO 12 --- [           main] c.g.p.h.HazelcastDiscoveryController     : Asking k8s registry at https://kubernetes.default.cluster.local..
2015-05-20T11:41:15.447430811Z 2015-05-20 11:41:15.447  WARN 12 --- [           main] c.g.p.h.HazelcastDiscoveryController     : Request to Kubernetes API failed
2015-05-20T11:41:15.447455987Z
2015-05-20T11:41:15.447459540Z java.nio.file.NoSuchFileException: /var/run/secrets/kubernetes.io/serviceaccount/token
2015-05-20T11:41:15.447462767Z  at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
2015-05-20T11:41:15.447465764Z  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
2015-05-20T11:41:15.447468341Z  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
2015-05-20T11:41:15.447470841Z  at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
2015-05-20T11:41:15.447473430Z  at java.nio.file.Files.newByteChannel(Files.java:361)
2015-05-20T11:41:15.447476757Z  at java.nio.file.Files.newByteChannel(Files.java:407)
2015-05-20T11:41:15.447479365Z  at java.nio.file.Files.readAllBytes(Files.java:3152)
2015-05-20T11:41:15.447481931Z  at com.github.pires.hazelcast.HazelcastDiscoveryController.getServiceAccountToken(HazelcastDiscoveryController.java:73)
2015-05-20T11:41:15.447484868Z  at com.github.pires.hazelcast.HazelcastDiscoveryController.run(HazelcastDiscoveryController.java:107)
2015-05-20T11:41:15.447487676Z  at org.springframework.boot.SpringApplication.runCommandLineRunners(SpringApplication.java:672)
2015-05-20T11:41:15.447490031Z  at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:690)
2015-05-20T11:41:15.447492460Z  at org.springframework.boot.SpringApplication.run(SpringApplication.java:321)
2015-05-20T11:41:15.447494917Z  at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:139)
2015-05-20T11:41:15.447497353Z  at com.github.pires.hazelcast.Application.main(Application.java:27)
2015-05-20T11:41:15.447500453Z  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2015-05-20T11:41:15.447502947Z  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2015-05-20T11:41:15.447505408Z  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2015-05-20T11:41:15.447517981Z  at java.lang.reflect.Method.invoke(Method.java:497)
2015-05-20T11:41:15.447520811Z  at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
2015-05-20T11:41:15.447523966Z

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't. 0.17.1 has just been tagged. Let me give it a try.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's in 0.17.1

Brendan
On May 21, 2015 1:48 PM, "Paulo Pires" notifications@github.com wrote:

In
src/main/java/com/github/pires/hazelcast/HazelcastDiscoveryController.java
#2 (comment)
:

 public List<Subset> subsets;

}

  • private static String getServiceAccountToken() throws IOException {
  • String file = "/var/run/secrets/kubernetes.io/serviceaccount/token";

It wasn't. 0.17.1 has just been tagged. Let me give it a try.


Reply to this email directly or view it on GitHub
https://github.com/pires/hazelcast-kubernetes-bootstrapper/pull/2/files#r30827448
.

return new String(Files.readAllBytes(Paths.get(file)));
}

private static String getEnvOrDefault(String var, String def) {
final String val = System.getenv(var);
return (val == null || val.isEmpty())
? def
: val;
}

// TODO: Load the CA cert when it is available on all platforms.
private static TrustManager[] trustAll = new TrustManager[] {
new X509TrustManager() {
public void checkServerTrusted(X509Certificate[] certs, String authType) {}
public void checkClientTrusted(X509Certificate[] certs, String authType) {}
public X509Certificate[] getAcceptedIssuers() { return null; }
}
};
private static HostnameVerifier trustAllHosts = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
};

@Override
public void run(String... args) {
final String hostName = getEnvOrDefault("KUBERNETES_RO_SERVICE_HOST",
"localhost");
final String hostPort = getEnvOrDefault("KUBERNETES_RO_SERVICE_PORT",
"8080");
String serviceName = getEnvOrDefault("HAZELCAST_SERVICE", "hazelcast");
String path = "/api/v1beta3/namespaces/default/endpoints/";
final String host = "http://" + hostName + ":" + hostPort;
final String host = "https://kubernetes.default.cluster.local";
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last time I checked kubernetes.local was the default domain. Has this changed?

Also, what happens if the user specifies a different domain?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it was changed in:

kubernetes/kubernetes#8105

I suppose we could customize it, but given the auth stuff, it's going to be fairly complicated to get it all working correctly.

log.info("Asking k8s registry at {}..", host);

final List<String> hazelcastEndpoints = new CopyOnWriteArrayList<>();

try {
String token = getServiceAccountToken();

SSLContext ctx = SSLContext.getInstance("SSL");
ctx.init(null, trustAll, new SecureRandom());

URL url = new URL(host + path + serviceName);
HttpsURLConnection conn = (HttpsURLConnection)url.openConnection();
// TODO: remove this when and replace with CA cert loading, when the CA is propogated
// to all nodes on all platforms.
conn.setSSLSocketFactory(ctx.getSocketFactory());
conn.setHostnameVerifier(trustAllHosts);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO to remove once a CA is available

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

conn.addRequestProperty("Authorization", "Bearer " + token);

ObjectMapper mapper = new ObjectMapper();
Endpoints endpoints = mapper.readValue(url, Endpoints.class);
Endpoints endpoints = mapper.readValue(conn.getInputStream(), Endpoints.class);
if (endpoints != null) {
// Here is a problem point, endpoints.endpoints can be null in first node cases.
if (endpoints.subsets != null && !endpoints.subsets.isEmpty()) {
Expand All @@ -92,7 +128,7 @@ public void run(String... args) {
});
}
}
} catch (IOException ex) {
} catch (IOException | NoSuchAlgorithmException | KeyManagementException ex) {
log.warn("Request to Kubernetes API failed", ex);
}

Expand Down