Support TLS for Elasticsearch and Additional YAML for Kibana #187

russcam · 2018-05-16T05:51:36Z

This PR adds

support for encrypting communication with Elasticsearch on the HTTP and Transport layer.
support for passing additional kibana YAML configuration to be appended to /etc/kibana/kibana.yml config file

The template has supported TLS to Elasticsearch from a public IP address through the use of Application Gateway for some time. With such a configuration however, offload happens on the Application Gateway, and inter-node communication on the Transport layer is still unencrypted.

This PR allows PKCS#12 archive certs to be passed base64 encoded to be used for TLS of HTTP and Transport layers. For Elasticsearch 5.x which does not support PKCS#12 format, the archive is converted to PEM format. If a certificate is passed for the HTTP layer, Kibana is also configured to use this certificate.

Some important notes:

Elasticsearch verificationMode is set to certificate which verifies that the provided certificate is signed by a trusted authority (CA), but does not perform hostname verification.
Kibana 5.3.0+ verificationMode is set to certificate for the Elasticsearch HTTP cert for 5.3.0+which verifies that the provided certificate is signed by a trusted authority (CA), but does not perform hostname verification. Kibana communicates with Elasticsearch through the network IP address of the internal load balancer.
Kibana < 5.3.0, certificate verification is disabled as there is not an option to perform certificate verification only. Additionally, if the key has a passphrase, this is removed, as these versions of Kibana do not support passing a passphrase for the key.

Mpdreamz · 2018-05-29T07:34:54Z

src/scripts/kibana-install.sh

+        echo "elasticsearch.ssl.certificate: /etc/kibana/ssl/elasticsearch-http.crt" >> $KIBANA_CONF
+        # A user may provide a certificate that would fail full verification mode,
+        # so default to certificate mode which verifies that the provided certificate is signed
+        # by a trusted authority (CA), but does not perform any hostname verification.


I think the template should take an optional CA certificate elasticsearchCaCertificate and use certgen to generate the elasticsearch node's HTTP and Transport certificates. This way we do not have to lower the verification mode.

Reusing the same HTTP/Transport certificate for multiple nodes is not something we should promote.

To do that, you'd need to have the key for the CA, which is unlikely to be the case if they're using an existing CA.
For transport our recommendation is to have a cluster-specific CA. But for HTTP, there's a good chance they'd want to use their own CA so that their HTTP clients are already configured to trust it.

tvernum · 2018-06-04T08:49:05Z

Using PKCS#12 for trust is little tricky. If the .p12 came from certutil, then it will work fine. But if it came from somewhere else, then it probably won't work as a truststore - the entries that are needed to store a CA in PKCS#12 are an Oracle extension.
I don't know what the expectation is around where users would get their PKCS#12 files from, but given the popularity of PFX within the Microsoft world, it seems reasonably likely that this could become an issue.

There's way around it - you could work out that the p12 doesn't have CA entries and add them, or fail. Or we could add a way to provide CAs separately. I'm just not sure if it's going to be a problem for Azure.

This commit adds support for securing HTTP and Transport layers with PKCS#12 archive certificates. This allows easier configuration for a cluster that has a commercial license.

This commit adds certificate verification mode for TLS for Http and Transport layer. Since a user can supply a certificate that would fail full verification mode, default to certificate mode.

This commit always passes the password for a PKCS#12 archive cert to openssl, to convert to PEM format, even when the password is an empty string. Use the insecure flag when calling localhost over HTTPS with curl. Since the subject name in certificate used to secure the HTTP layer may not match host name localhost (it's likely to be tied to a public domain name), the --cacert flag for curl cannot be used.

This commit sets elasticsearch.ssl.verify: false for Kibana 5.2.0 and older. Kibana connects to Elasticsearch through an internal load balancer IP address which will likely fail verification which will also verify hostname, so disable verification by default for these versions.

This commit adds detail to specify that the Kibana cert and key are in PEM format. Support PKCS#12 archive in future would be useful.

This commit updates integration tests to use the parameters-file argument to pass the parameters to azure cli. Since parameters can now contain base 64 encoded certificates, the input can be longer than the maximum characters allowed in Windows (8192). Add generated certificates for integration tests.

This commit adds a --test parameter to be able to filter the integration tests that should be run. The parameter value is a regular expression string.

Include additional parameters in README. Add notes for additional arguments for npm run test

This commit adds the certificate CA to requests made in integration tests

This commit changes the integration test that tests the yaml configuration parameters to whitelist any indices starting with a dot; the .security index is too restrictive for the indices created by alerting/watcher.

This commit removes the hostname verification check used by the node's request module. Since all certs used are self-signed, hostname verification will fail. Tests still verify the CA.

This commit adds a --nodestroy parameter that when passed, does not delete resource groups after integration tests

russcam · 2018-07-06T07:31:25Z

@Mpdreamz @elastic/es-security This PR is now ready to review again.

Some important points:

Kibana communicates with the Elasticsearch through the internal IP address of an internal (layer 4) load balancer
Elasticsearch can be accessed from outside of the virtual network either by
- an external (layer 4) load balancer with a public IP and name on an azure sub domain. Only port 9200 is open.
or
- Application Gateway (layer 7) load balancer with a public IP and name on an azure sub domain. Application Gateway performs SSL offload, and can optionally support end-to-end TLS by re-encrypting traffic from Application Gateway to the backend pool. Only port 9200 is open.

The following changes have been made:

A PKCS#12 archive can be passed with the CA to generate a certificate for each node in the cluster to secure the Transport layer. Each certificate generated will contain Subject Alternative Name of DNS: $HOSTNAME and IP: internal host IP address (e.g. 10.0.0.7). Elasticsearch is configured with full verification mode on the Transport layer.
A PKCS#12 archive can be passed with the certificate used to secure the HTTP layer.

When Kibana is installed, if the archive contains the CA cert, then Kibana will be configured with certificate verification when communicating with Elasticsearch. If no CA cert can be extracted, Kibana will be configured with no verification.

When Application Gateway is configured for external access, the public key from the PKCS#12 archive is added to a whitelist on Application Gateway to allow it to communicate with resources secured with this certificate on the backend pool. From research, it seems that the cert for the HTTP layer must contain a Subject Alternative Name DNS entry that matches the Subject CN.
A PKCS#12 archive can be passed with the CA to generate a certificate for each node in the cluster to secure the HTTP layer.

This option is offered as an alternative to point 2. Each certificate generated will contain Subject Alternative Name of DNS: $HOSTNAME, IP: internal host IP address (e.g. 10.0.0.7) and IP: internal load balancer IP address.

When Kibana is installed, it will be configured with full verification when communicating with Elasticsearch, since the internal load balancer IP address is included in each generated certificate.

This option will not work with Application Gateway because there is not an easy/nice way of providing the public key from each generated cert to the whitelist of Application Gateway. Additionally, Application Gateway is limited to the number of certificates that can be whitelisted (up to 10, default of 5).

Versions of Elasticsearch and Kibana less than 5.3.0 have been removed from the template as they are either EOL or very soon to be EOL.

The changes of most pertinence are the handling and configuration of certs in elasticsearch-ubuntu.install.sh and kibana-install.sh

russcam · 2018-07-06T07:32:13Z

Since last review, I've rebased from master to include the changes related to managed disks.

Mpdreamz

LGTM

Do we need esTransportSecurity though wether esTransportCaCertBlob is specified or not could toggle whether to self sign or not.

Same for esHttpSecurity do we want to support No and no esHttpCertBlob?

Mpdreamz · 2018-07-09T11:34:42Z

src/scripts/elasticsearch-ubuntu-install.sh

+
+      if [[ -f $BIN_DIR/elasticsearch-certutil || -f $BIN_DIR/x-pack/certutil ]]; then
+          local CERTUTIL=$BIN_DIR/elasticsearch-certutil
+          if [[ ! -f $CERTUTIL ]]; then


👍 was looking for version checks but this is better.

jkakavas · 2018-07-09T15:07:58Z

README.md

@@ -146,13 +190,16 @@ posts for further information
    </td><td><code>Standard_A2</code></td></tr>

  <tr><td>kibanaCertBlob</td><td>string</td>
-    <td>A Base-64 encoded form of the certificate (.crt) to secure HTTPS communication between the browser and Kibana.</td><td><code>""</code></td></tr>
+    <td>A Base-64 encoded form of the certificate (.crt) in PEM format to secure HTTPS communication between the browser and Kibana.</td><td><code>""</code></td></tr>


A PEM certificate will always be a base64 encoded DER certificate so Base-64 encoded form sounds superfluous I think

I agree that PEM will already be base 64, but I'd like to keep the description here for consistency, and think it's a good idea to keep the process for including certs consistent across all the cert parameters i.e. as a consumer of the template, if I need to add certs in whatever format, I know that I always need to base 64 encode the contents and pass that as the parameter.

I'll add an example to the README of how to pass certificates to the template, to make it clear.

i.e. as a consumer of the template, if I need to add certs in whatever format

Then maybe it's worth stating that we support other formats? Indeed Elasticsearch supports reading DER encoded certificates. Then again, as a consumer, if I have a DER certificate and I need to Base64 encode it in order to use it in the template, I am effectively converting it to a PEM certificate, so in essence we do only support PEM certificates.

Just a comment though, not a request for an additional change. I'd write this differently, but I don't have a so strong opinion on how it should be written here.

Perhaps it would be more clear if I remove the in PEM format part, and leave as before? This way, whether providing a PEM cert or DER cert, so long as it's base 64 encoded (I appreciate that PEM is already base 64 😃), the certificate will work.

jkakavas · 2018-07-09T15:08:17Z

README.md


  <tr><td>kibanaKeyBlob</td><td>securestring</td>
-    <td>A Base-64 encoded form of the private key (.key) to secure HTTPS communication between the browser and Kibana.</td><td><code>""</code></td></tr>
+    <td>A Base-64 encoded form of the private key (.key) in PEM format to secure HTTPS communication between the browser and Kibana.</td><td><code>""</code></td></tr>


Again PEM keys will always be Base64 encoded.

Similar to #187 (comment)

I'll add an example to the README of how to pass certificates to the template, to make it clear.

jkakavas · 2018-07-09T15:15:12Z

README.md

+    </td><td><code>No</code></td></tr>
+
+  <tr><td>esHttpCertBlob</td><td>string</td>
+    <td>A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) certificate to secure communication for HTTP layer to Elasticsearch. <strong>X-Pack plugin must be installed</strong>


I assume you don't want to pass the PKCS#12 Keystores as Base64 encoded strings but rather pass the file directly as a param right ? This goes for the rest of the settings below.
Rephrase to something like: A PKCS#12 archive (.p12/.pfx) containing the key and certificate needed to secure communication for the HTTP layer of Elasticsearch ?

The PKCS#12 archives need to be base 64 encoded as all parameters are passed over HTTP to the template, whether the template is part of a Marketplace offering or hosted publicly.

If we end up exposing the ability for users to upload certificate files in the Azure Marketplace UI, The file upload control will base 64 encode content to pass it to the template

Just re-read this comment. I'll update the description to make it more clear.

My point here is that the way it is written now , it looks like we're asking users to take a PKCS#12 archive, find a way to represent its bytes in Base64 format and use that string here. I'm not familiar with how Azure templates work but I assumed there is an easy way to add the actual binary file here instead of its Base64 string representation.

If we do need a Base64 string with the encoded contents of the PKCS#12 archive, we could add an example of how to get them ( in bash or powershell ) ?

Also I still think that the PKCS#12 archive certificate part sounds strange as there are two nouns. If the containing the key and certificate I suggested above sounds too long, maybe consider dropping the certificate from the original description to keep it simple ?

we're asking users to take a PKCS#12 archive, find a way to represent its bytes in Base64 format and use that string here. I'm not familiar with how Azure templates work but I assumed there is an easy way to add the actual binary file here instead of its Base64 string representation.

Yes, that's what we're asking users to do, and it's the way the contents of files are typically provided when deploying Azure resources.

Also I still think that the PKCS#12 archive certificate part sounds strange as there are two nouns. If the containing the key and certificate I suggested above sounds too long, maybe consider dropping the certificate from the original description to keep it simple ?

I agree, I'll update to your suggestion "containing the key and certificate" 👍

I agree, I'll update to your suggestion "containing the key and certificate" +1

I'd update that here (in the Parameters table) too.

jkakavas · 2018-07-09T15:16:55Z

README.md

+
+  <tr><td>esHttpCertPassword</td><td>securestring</td>
+    <td>The password for the PKCS#12 archive (.p12/.pfx) certificate to secure communication for HTTP layer to Elasticsearch. Optional as the archive may not be encrypted. <br /><br />
+    If using <code>esHttpCaCertBlob</code>, this password will be used to encrypt the private key of generated certificates.


I would argue in favor of adding an extra parameter to be used for encrypting the generated private keys below, and use this only as the password of the esHttpCertBlob if any.

I think the description is incorrect here; when a CA cert is passed to generate certs for the HTTP layer, esHttpCertPassword can be used as the passphrase for the generated PKCS#12 archives. I'll update the description to make this clearer.

jkakavas · 2018-07-09T15:17:20Z

README.md

+    <td>Either <code>Yes</code> or <code>No</code> to configure SSL/TLS for the HTTP layer of Elasticsearch. <strong>X-Pack plugin must be installed and either <code>esHttpCertBlob</code> or <code>esHttpCaCertBlob</code> provided</strong></strong>
+    </td><td><code>No</code></td></tr>
+
+  <tr><td>esHttpCertBlob</td><td>string</td>


Maybe archive instead of blob for the rest of this ?

I'd like to stick to using the parameter names for this. The *Blob suffix has been used in the template for encoded parameters, so would like to stick to this for consistency.

Re-read this comment. I think renaming the PKCS#12 archive parameters would make sense e.g.

esHttpCaCertBlob -> esHttpCaArchiveBlob

However, the Application Gateway parameter for providing a PKCS#12 archive that has been in the template for over a year is called appGatewayCertBlob, so think *CertBlob would make for a more consistent naming approach, even though the parameter name is less/not technically correct.

jkakavas · 2018-07-09T15:23:51Z

README.md

+
+**_and_**
+
+* supply the public key in CER format for the certificate within the PKCS#12 archive


.cer is just a suffix, there is no CER format .openssl pkcs12 -in http_cert.p12 -out public_key.cer -clcerts -nokeys will export the certificate in PEM format.

Also this is not just the public key, but rather the certificate.

I agree 😄 I was sticking to the same terminology that Microsoft has used in their documentation for enabling end-to-end encryption with Application Gateway (see Note at point 7). I'll update to indicate this is the certificate

jkakavas · 2018-07-09T15:24:36Z

README.md

    <strong>Required when selecting <code>gateway</code> for load balancing.</strong>
    </td><td><code>""</code></td></tr>

+  <tr><td>appGatewayEsHttpCertPublicKey</td><td>securestring</td>
+    <td>The Base-64 encoded public key (.cer) for the certificate used to secure the HTTP layer of Elasticsearch. Used by the Application Gateway to whitelist certificates used by the backend pool. Required when using <code>esHttpCertBlob</code> to secure the HTTP layer of Elasticsearch and selecting <code>gateway</code> for load balancing. <strong>X-Pack plugin must be installed</strong>


The Base-64 encoded public key (.cer) for the certificate == The certificate

Will update 👍

jkakavas · 2018-07-09T15:26:53Z

README.md

+passphrase can be passed with `esTransportCertPassword` to encrypt the generated certificate
+on each node.
+
+A new CA PKCS#12 archive can be generated using [Elastic's certutil command](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html). The simplest command to generate a CA cert is


The expectation here is that users will download Elasticsearch on their own and run certutil? It doesn't sound very streamlined experience, but I don't have any good alternative options to suggest.

I'll make it clear that this is just a suggestion for how to create a CA cert, but that they may already have one, or could generate one in other ways e.g. through PowerShell

russcam · 2018-07-10T01:23:08Z

Do we need esTransportSecurity though wether esTransportCaCertBlob is specified or not could toggle whether to self sign or not.

Same for esHttpSecurity do we want to support No and no esHttpCertBlob?

@Mpdreamz I agree that they are superfluous. We could configure/generate/persist certificates on node VMs independently of enabling HTTP or Transport TLS i.e. be able to pass certs, to set them up on the VMs, but pass esHttpSecurity = No and esTransportSecurity = No to not configure TLS. Thoughts on enabling this with the parameters, or removing them altogether?

This commit renames the mainTemplate appGatewayEsHttpCertPublicKey to simply appGatewayEsHttpCertBlob, to indicate that the parameter is the public key and should be provided Base-64 encoded.

This commit addresses the PR comments to improve the descriptions for each of the parameters, and include a section on how to pass certificate related parameters.

jkakavas · 2018-07-10T04:05:08Z

README.md

+    </td><td><code>No</code></td></tr>
+
+  <tr><td>esHttpCertBlob</td><td>string</td>
+    <td>A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) certificate to secure communication for HTTP layer to Elasticsearch. <strong>X-Pack plugin must be installed</strong>


My point here is that the way it is written now , it looks like we're asking users to take a PKCS#12 archive, find a way to represent its bytes in Base64 format and use that string here. I'm not familiar with how Azure templates work but I assumed there is an easy way to add the actual binary file here instead of its Base64 string representation.

If we do need a Base64 string with the encoded contents of the PKCS#12 archive, we could add an example of how to get them ( in bash or powershell ) ?

Also I still think that the PKCS#12 archive certificate part sounds strange as there are two nouns. If the containing the key and certificate I suggested above sounds too long, maybe consider dropping the certificate from the original description to keep it simple ?

jkakavas · 2018-07-10T04:09:26Z

README.md

+* **TLS for communication with Elasticsearch via the HTTP layer through an external load balancer**
+* **TLS for communication with Elasticsearch via the HTTP layer through Application Gateway**
+* **TLS for communication between Elasticsearch nodes via the Transport layer**
+* **TLS for communication beween the browser and Kibana**


This gets configured below https://github.com/elastic/azure-marketplace/pull/187/files#diff-04c6e90faac2675aa89e2176d2eec7d8R192

It's optional in the template to configure TLS, but really want to point out at the top of the README that you should be configuring this!

jkakavas · 2018-07-10T04:24:40Z

README.md

@@ -146,13 +190,16 @@ posts for further information
    </td><td><code>Standard_A2</code></td></tr>

  <tr><td>kibanaCertBlob</td><td>string</td>
-    <td>A Base-64 encoded form of the certificate (.crt) to secure HTTPS communication between the browser and Kibana.</td><td><code>""</code></td></tr>
+    <td>A Base-64 encoded form of the certificate (.crt) in PEM format to secure HTTPS communication between the browser and Kibana.</td><td><code>""</code></td></tr>


i.e. as a consumer of the template, if I need to add certs in whatever format

Then maybe it's worth stating that we support other formats? Indeed Elasticsearch supports reading DER encoded certificates. Then again, as a consumer, if I have a DER certificate and I need to Base64 encode it in order to use it in the template, I am effectively converting it to a PEM certificate, so in essence we do only support PEM certificates.

Just a comment though, not a request for an additional change. I'd write this differently, but I don't have a so strong opinion on how it should be written here.

jkakavas · 2018-07-10T04:27:20Z

README.md

+
+**_and_**
+
+* supply the public certificate in CER format from the PKCS#12 archive


russcam · 2018-07-10T05:11:18Z

@jkakavas Really appreciate your time in looking at this. I think I've now addressed all of your comments, if you have a chance to take another look.

jkakavas

LGTM, thanks for the discussion!

Mpdreamz · 2018-07-10T07:03:18Z

@russcam what would be the downsides of removing them?

I'm +1 to not supporting No and no certblob. E.g if you pass one we'll use that otherwise we use self genned ones.

russcam · 2018-07-10T07:15:16Z

@russcam what would be the downsides of removing them?

No downside to removing them now; happy to do so as it's two less parameters 😄

russcam requested review from Mpdreamz and tvernum May 16, 2018 05:51

russcam mentioned this pull request May 21, 2018

Enable SAML realm configuration for SSO with AAD #188

Merged

Mpdreamz suggested changes May 29, 2018

View reviewed changes

russcam force-pushed the feature/tls branch 3 times, most recently from 0d506b5 to edf5160 Compare July 4, 2018 09:43

russcam added 22 commits July 5, 2018 09:38

Update links

4cd22fd

Add support for additional kibana.yml configuration

530b602

Add support for HTTP and Transport layer encryption

fd91dbf

This commit adds support for securing HTTP and Transport layers with PKCS#12 archive certificates. This allows easier configuration for a cluster that has a commercial license.

update links

d4cebf8

Add certificate verification mode

fcf1c66

This commit adds certificate verification mode for TLS for Http and Transport layer. Since a user can supply a certificate that would fail full verification mode, default to certificate mode.

Add TLS related parameters to parameters files

3c9c96a

Add support for Elasticsearch HTTP cert arguments

129ef1e

Use HTTPS for localhost calls when HTTP certificate supplied

477b201

Pipe cert passwords to openssl through stdin

c8c9df1

Support TLS for HTTP and Transport layer for 5.x clusters

c55bb77

Additional log messages

1b55de6

Add details for Kibana cert and key format

55720cb

This commit adds detail to specify that the Kibana cert and key are in PEM format. Support PKCS#12 archive in future would be useful.

Add --test parameter to filter integration tests

bf30f75

This commit adds a --test parameter to be able to filter the integration tests that should be run. The parameter value is a regular expression string.

Update README

529c200

Include additional parameters in README. Add notes for additional arguments for npm run test

Update README for configuring TLS

6e41fee

Pass cert when making requests in integration tests

f7febc6

This commit adds the certificate CA to requests made in integration tests

Add integration tests for additional yaml

f1e8790

Double escape quotes in additionalYaml parameters

d8a2a77

Whitelist indices starting with dot (.)

5203535

This commit changes the integration test that tests the yaml configuration parameters to whitelist any indices starting with a dot; the .security index is too restrictive for the indices created by alerting/watcher.

russcam added 5 commits July 5, 2018 15:53

Update README with details for SSL/TLS

fd934d4

Generate certs with SAN DNS entries

1d09203

Add pkcs12 with password for integration tests

f995ec4

Remove hostname verification in integration tests

35c3337

This commit removes the hostname verification check used by the node's request module. Since all certs used are self-signed, hostname verification will fail. Tests still verify the CA.

Check certs are CAs

48ffb41

russcam mentioned this pull request Jul 6, 2018

Kibana fails to start if password contains '!' character #197

Closed

russcam added 3 commits July 6, 2018 16:00

Use correct cert for Loadbalancer

2d731ea

Support --nodestroy parameter

bde3436

This commit adds a --nodestroy parameter that when passed, does not delete resource groups after integration tests

Update README

5169972

Mpdreamz approved these changes Jul 9, 2018

View reviewed changes

jkakavas suggested changes Jul 9, 2018

View reviewed changes

russcam added 2 commits July 10, 2018 12:44

Rename appGatewayEsHttpCertPublicKey and associated properties

8736137

This commit renames the mainTemplate appGatewayEsHttpCertPublicKey to simply appGatewayEsHttpCertBlob, to indicate that the parameter is the public key and should be provided Base-64 encoded.

Address PR comments

c954053

This commit addresses the PR comments to improve the descriptions for each of the parameters, and include a section on how to pass certificate related parameters.

jkakavas reviewed Jul 10, 2018

View reviewed changes

Address PR comments

b2fd3c8

jkakavas approved these changes Jul 10, 2018

View reviewed changes

russcam added 5 commits July 11, 2018 10:45

Remove separate esHttpSecurity and esTransportSecurity parameters

45969c5

Remove remaining references to esTransportSecurity

1433fd5

Include password for HTTP cert

baf4ce3

Simplify determining if --nodestroy is passed

8ff135d

Update links

38daff3

russcam merged commit d7e38b6 into master Jul 11, 2018

russcam deleted the feature/tls branch July 13, 2018 10:28


		_and_

		* supply the public key in CER format for the certificate within the PKCS#12 archive


		_and_

		* supply the public certificate in CER format from the PKCS#12 archive

Support TLS for Elasticsearch and Additional YAML for Kibana #187

Support TLS for Elasticsearch and Additional YAML for Kibana #187

Conversation

russcam commented May 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvernum commented Jun 4, 2018

russcam commented Jul 6, 2018 • edited Loading

russcam commented Jul 6, 2018

Mpdreamz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

russcam Jul 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

russcam Jul 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

russcam commented Jul 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

russcam commented Jul 10, 2018

jkakavas left a comment

Choose a reason for hiding this comment

Mpdreamz commented Jul 10, 2018

russcam commented Jul 10, 2018

russcam commented Jul 6, 2018 •

edited

Loading

russcam Jul 10, 2018 •

edited

Loading

russcam Jul 10, 2018 •

edited

Loading