pine64 Support #93

ThomDietrich · 2017-02-11T16:58:56Z

I'm in the process of building an openHABian pine64 image.
@kaikreuzer @dmcwhinnie

Progress so far:

with Add user configuration, replace static user, catch all RPi specifics, cleanup #90, Decouple raspbian-ua-netinst and extend build script #91 and Replace Oracle Java by zulu embedded #92 I've paved the way for the openHABian config tool to be compatible with other platforms (Pine64 tests still needed)
I've decided to go with Ubuntu Base Image by longsleep
I'm currently in the process of hacking and building this image to start the unattended openhabian setup just like it is done on the RPi
After the build succeeds and the first time setup is working, further improvements and bug fixing is needed in openhabian-config (first tests show a few failing setup steps)
FireMotD needs improvement too.

Good news: A first test did already set up and install a working openHAB!

ThomDietrich · 2017-02-11T17:16:58Z

First build successful. Flashing now...

dmcwhinnie · 2017-02-12T00:13:14Z

Purchased a new SD card this week, so ready to test when you are...

ThomDietrich · 2017-02-12T16:02:25Z

The image is almost finished. The next step will be modifications and improvements inside the openhabian setup. That's easier. @dmcwhinnie I'd be happy to provide you with a first image in a few hours.

dmcwhinnie · 2017-02-12T21:42:54Z

great! May not get to it today, but will get it in the next coupe days.

…

On Sun, Feb 12, 2017 at 3:17 PM, Thomas Dietrich ***@***.***> wrote: @dmcwhinnie <https://github.com/dmcwhinnie> It took longer than I thought but here it is: https://drive.google.com/open?id= 0B67AyLNQlS3dMndLSkY0TnlDYXM — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#93 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMefoRSincnAitq8YqsGejwviwZqjXXzks5rb3bzgaJpZM4L-Oho> .

kaikreuzer · 2017-02-12T22:11:57Z

Started flashing my SD card :-)

One drawback to the RPi net-installer is that the extracted image is 4GB, so initial flashing takes much longer.

ThomDietrich · 2017-02-12T22:14:47Z

Indeed :) One does appreciate that after a while.

Sadly there was a small error in the image, you will not have much fun with it. Give me a few seconds.
It would be great to have a few testers in the next few days.

kaikreuzer · 2017-02-12T22:15:36Z

Ok, cancelling the flashing...
Will be happy to test any further updates!

ThomDietrich · 2017-02-13T00:50:59Z

Finally here we go! https://drive.google.com/drive/folders/0B67AyLNQlS3dYzJEbGtBOUdmTTg?usp=sharing
The setup process seems to be stable now. It will be a bit simpler as soon as the pine image build project accepted three PRs of mine. Okay, currently there are two problems:

The first boot seems to need a bit longer to provide network connectivity, the second boot will then successfully start and finish the openHABian setup. After that you'll have (most obvious) the openHABian login screen and everything else, just like with the RPi. check the progress: grep rc.local /var/log/syslog.
Zulu Embedded doesn't work 😢 I've tracked it down to the absence of /lib/ld-linux-armhf.so.3, which was already discussed here, which I didn'T want to go through now. It's late enough and I hope @kaikreuzer knows the solutions to that little problem :)

Happy Hacking!

dmcwhinnie · 2017-02-13T03:16:22Z

First install looks good! To fix the armhf library issue, these steps need to be added.

# Add armhf architecture
dpkg --add-architecture armhf
#verify
dpkg --print-foreign-architectures
#update source
apt-get update
#Install required armhf binaries
apt-get install libc6:armhf libncurses5:armhf libstdc++6:armhf

Once I did that, I was able to start openHAB. Reboot started everything automatically. Note, I can't find the source I found that information, so it may be possible that we don't need all 3 libraries for Java to work. All I have in my notes is to install all 3. 😊

I also did some basic tests with samba, z-wave addon installed, and detected the zwave module shipped with the original IOT Pine 64 shipments.

kaikreuzer · 2017-02-13T08:59:23Z

Err, stupid question: What is the default user to log in with?

ThomDietrich · 2017-02-13T11:38:37Z

I found a solution to the first problem (not in the image yet, so you will have to reboot once) and can confirm that the steps as described by @dmcwhinnie do the job (Thanks!).

@kaikreuzer ubuntu:ubuntu ;D

kaikreuzer · 2017-02-13T11:42:00Z

Ah :-) Needs to be changed 😎

kaikreuzer · 2017-02-13T13:04:00Z

Just FTR: Whatever I execute, I see many of these warnings:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = "C"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

ThomDietrich · 2017-02-13T13:04:44Z

I had longer discussions about that before. till now we decided to stay with the default password (e.g. pi:raspberry) but ask the user to change it: https://github.com/openhab/openhabian/blob/pine64-build/openhabian-setup.sh#L1006-L1042 - This routine is called whenever a user exits sudo openhabian-config.

ThomDietrich · 2017-02-13T13:05:32Z

Ah, setting locale was one of the last things I automated yesterday. Didn't ensure it's running smoothly... https://github.com/openhab/openhabian/blob/pine64-build/openhabian-setup.sh#L149-L158

kaikreuzer · 2017-02-13T13:08:03Z

till now we decided to stay with the default password

I was also hoping for a different user, like e.g. "openhab" instead of "ubuntu". But I see that this won't work for e.g. the netinstaller that isn't under our control. And nothing to urgent for sure. But on the long run, it would be neat to have the same user on any openHABian installation and not different once depending on whether you run it on pi or pine.

kaikreuzer · 2017-02-13T13:28:48Z

Regarding Zulu JDK, all I needed to make it work was:

dpkg --add-architecture armhf
apt-get update
apt-get install libc6:armhf

@dmcwhinnie Are libncurses5:armhf& libstdc++6:armhf really required as well?

@ThomDietrich I also found the solution for the font issue! It is as simple as installing fontconfig:armhf!

kaikreuzer · 2017-02-13T13:34:22Z

Configuring the Z-Wave board as /dev/ttyS2 also directly worked without any further tweaking - so overall, I agree with @dmcwhinnie: Great work so far! 👍

dmcwhinnie · 2017-02-13T13:37:01Z

@kaikreuzer Are libncurses5:armhf& libstdc++6:armhf really required as well?

I never confirmed all 3 were required. I had found the information somewhere, in the past, which said all 3 were needed. Once I had it working, I never went back to fully confirm if all 3 were needed. I suspect only the 1 is needed, and your setup seems to confirm.

ThomDietrich · 2017-02-13T13:52:56Z

But I see that this won't work for e.g. the netinstaller that isn't under our control.
same user on any openHABian installation

Oh but it is 😃 I just decided to set it to pi/ubuntu for now. You see, in the beginning of openHABian I wanted a "hassle-free" start, which includes being able to log in with the password RPi users with little or no Linux experience know: raspberry. Now with possibly more and more platforms supported I'd agree, one username:password would be nice.
Are you sure about the openhab user? That's the system user openHAB is running under. It's not a good practice to user system users as interactive/sudo users... How about "openhabian" or "hab" or "enthusiast"

kaikreuzer · 2017-02-13T14:04:57Z

Are you sure about the openhab user?

Well, it would be in line with our docs for manual setup.

kaikreuzer · 2017-02-13T14:06:33Z

Or do you mean we should actually have two users? "openhab" for running the service and then an admin/sudo user for manual logins?

ThomDietrich · 2017-02-13T14:07:48Z

@kaikreuzer could you please provide me with one example command to provoke the locale warning...

Regarding Zulu JDK

Thanks. Turns out all I missed in my late nighter yesterday was the apt update 😅

I also found the solution for the font issue! It is as simple as installing fontconfig:armhf!

Amazing. Do I still need to confirm this on the RPi?

Configuring the Z-Wave board as /dev/ttyS2 also directly worked without any further tweaking

You mean inside openHAB? No changes on system side needed? Perfect!

I never went back to fully confirm if all 3 were needed

Okay then, I'll add the one and it's good that all three of them are mentioned here in case of later problems.

I agree with @dmcwhinnie: Great work so far! 👍

Gracias :)

To summarize: I'll build a new image this evening. It will include:

first boot fix
libc6:armhf for Zulu
fontconfig:armhf for charts
corrected locale settings

I'll also work on a fix for FireMotD to detect the pine64 platform.

It would also be nice to have the on-board LED blink after the setup was successfully finished (just as with the RPi - @kaikreuzer @dmcwhinnie the pine I got from Kai (Thanks again!) is a prototype and doesn't have a status LED. Meaning this would be up to you to figure out. If something like this is not easily available, We can ignore that for now. It's just nice to have and not necessary.

ThomDietrich · 2017-02-13T14:21:16Z

Well, it would be in line with our docs for manual setup.

sudo adduser --system --no-create-home --group --disabled-login openhab

This is a system user without (ssh) login ;)

Or do you mean we should actually have two users? "openhab" for running the service and then an admin/sudo user for manual logins?

Yup, that's how something like this is normally done. Linux users like "openhab" or "www-data" should be reasonably restricted. The end user should have his own interactive user with unrelated privileges. The fact that some users give openhab sudo is another story 💩 😉

kaikreuzer · 2017-02-13T14:27:58Z

This is a system user without (ssh) login ;)

Ok, then I would go for "openhabian".

could you please provide me with one example command to provoke the locale warning...

It e.g. happens when doing apt-get installs like

sudo apt-get install fontconfig:armhf

Do I still need to confirm this on the RPi?

It has already been confirmed to be working here. What might be good is to check the minimal requirement. It might be enough to install libfontconfig1:armhf - we might not need the "full" fontconfig. Likewise, I had installed ttf-dejavu for my first tests, so I am not sure whether this is required or not.

You mean inside openHAB? No changes on system side needed? Perfect!

Yes, inside openHAB.

ThomDietrich · 2017-02-13T19:18:49Z

Ok, then I would go for "openhabian".

The shell will look like this:

[20:15:28] openhabian@openHABianPine64:~$

Maybe we can come up with something nicer?

Building new image, including libc6:armhf libfontconfig1:armhf and the first boot fix.

dmcwhinnie · 2017-02-13T21:47:52Z

For the status LED, looks like it is doable based on this
https://gist.github.com/pfeerick/00b04df3ce65979158b117b4e1f7c168
and
http://forum.pine64.org/showthread.php?tid=2234

The issue I see is that this is a script running in a loop, so would have to spawn something and keep it running in the background. Not quite as clean as the PI for causing it to heartbeat.

I should add, that this is for the "Power" LED, I believe the board also as spot for an additional LED, but mine doe not have that one. So if that is what you are thinking, I don't have that either.

kaikreuzer · 2017-02-13T22:34:49Z

Maybe we can come up with something nicer?

Maybe some shorter hostname? openHABPine? PineHAB?

ThomDietrich · 2017-02-14T01:25:05Z

Here we go. I've uploaded a new image to test. Changes since yesterday:

Changed username:password to openhabian:openhabian
Added libc6:armhf for Zulu - tested!
Added libfontconfig1:armhf for charts - tested!
Improved boot up readiness test
Removed two steps from the build that were merged in the used project today
Make sure openHAB is started after being installed
Checked locale setting: everything seems to be in order.

The bootup time from power to openHAB Initial Setup screen was around 12 minutes btw.

https://drive.google.com/file/d/0B67AyLNQlS3dVjYwWlAxOV9CVTg/view?usp=sharing

I'm satisfied with the base now. @kaikreuzer @dmcwhinnie @umiddelb please give it a full test spin! 😉 Things still on the TODO list:

FireMotD interface & platform info
The menu needs to be rid of the RPi specific entries (can not be executed)
Mosquitto (besides others?) is not installed successfully.
Maybe other options and programs will fail or need setting up

Regarding the username:password: I would like to use something less boring and maybe a bit cheerful. How do you like enthusiast:smarthome or smart:home? Whoever is reading this, please give your vote ;)

@umiddelb I've read your mail today but as you can see I'm currently a bit busy. I'll answer you soon!

@kaikreuzer: While testing the chart, I encountered two usability issues. 1) Basic UI doesn't show the "Choose your sitemap" link list, where did it go? 2) the difference between Configuration and Preferences is unclear/un-intuitive. Also I was confused that the pre-configured sitemap for Basic UI is still called "_default". Might be time to call it "default"? We don't have to discuss these in detail here, if you think it's worth to look into I can open new issues.

dmcwhinnie · 2017-02-14T01:43:06Z

I like smart:home. With my typing skills I'll never get enthusiast right each time....

I'll let you know how the testing goes...

ThomDietrich · 2017-02-16T00:21:35Z

FireMotD problems are fixed. OutsideIT/FireMotD#34, OutsideIT/FireMotD#35
Looking into the "zulu thing" now @kaikreuzer

ThomDietrich · 2017-02-16T02:52:49Z

I'll send a mail with my findings tomorrow. The short version: Läuft

ThomDietrich · 2017-02-16T15:39:19Z

The End Is Nigh.

Zulu Embedded is now installed from repository (should work on Pine, Pi and Debian) (Charts are working)
FireMotD PRs are merged. The login screen is now clean and correct
Wifi setups now also covers the Pine, the firmware is only installed on RPi @dmcwhinnie
A PR for documentation is available

I will do one last important test this evening: RPi compatibility. If this is successful, I'm ready to merge and upload new releases. After a few days I'll mark this point with a new version number.

Open todos: validate pine64_platform_scripts execution, modify file permissions, check Mosquitto install, rearrange menu entries, change username:password.

Any comments?

kaikreuzer · 2017-02-17T08:58:54Z

Let me know once you have a new image available for testing!

ThomDietrich · 2017-02-17T11:08:11Z

There you go 😉 https://drive.google.com/open?id=0B67AyLNQlS3dam9HQVpDMU9VaGs

kaikreuzer · 2017-02-17T11:10:39Z

Downloading ⏳

ThomDietrich · 2017-02-17T11:12:29Z

In my eyes it's finished, All improvements that might still be needed are unrelated to the base image.
Last tow changes would be switching from development branch to master and setting the default password

kaikreuzer · 2017-02-17T14:03:21Z

FYI, when running apt-get upgrade I still see the locale warnings:

Get:1 https://openhab.jfrog.io/openhab/openhab-linuxpkg unstable/main arm64 openhab2 all 2.1.0~20170216144237-1 [53.2 MB]
Fetched 53.2 MB in 8s (6345 kB/s)                                                                                                                                                                                                            
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = "C"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = "C"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

ThomDietrich · 2017-02-17T14:13:11Z

But why...

My system:

$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

These are the commands supposed to be executed:

# List available locales
cat /usr/share/i18n/SUPPORTED
# Generate locales, set default
/usr/sbin/locale-gen en_US.UTF-8 de_DE.UTF-8
/usr/sbin/update-locale LANG=en_US.UTF-8

kaikreuzer · 2017-02-17T14:18:46Z

Does this help?

[15:17:44] ubuntu@openHABianPine64:~$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE=UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

ThomDietrich · 2017-02-17T14:24:42Z

I fear you'll have to investigate. This doesn't make sense to me and I'm not seeing these issues on my* Pine.

What happens after LC_CTYPE="en_US.UTF-8"?

kaikreuzer · 2017-02-17T15:07:47Z

Everything fine then:

[16:06:58] ubuntu@openHABianPine64:~$ LC_CTYPE="en_US.UTF-8"
[16:07:04] ubuntu@openHABianPine64:~$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

ThomDietrich · 2017-02-17T15:11:06Z

https://askubuntu.com/questions/599808/cannot-set-lc-ctype-to-default-locale-no-such-file-or-directory

This same problem (LC_CTYPE=UTF-8, which is wrong) can happen when you login over ssh from a Mac to a linux box, and your terminal automatically sets environment variables. There's a checkbox for that. Uncheck it, and you're good to go. In iTerm it's in the profile->Terminal Tab.

At least an explanation...

ThomDietrich · 2017-02-17T15:14:25Z

Could you please try the following and reconnect/reboot:

/usr/sbin/update-locale LC_CTYPE="en_US.UTF-8" LC_ALL="en_US.UTF-8" LANG="en_US.UTF-8"

kaikreuzer · 2017-02-17T16:31:15Z

That solves the issue!

ThomDietrich · 2017-02-17T16:58:00Z

Amazing ✨ Thanks Kai ;)

I'll do the last changes today and then release both new images. I'll do announcement in the openHABian thread as usual.

kaikreuzer · 2017-02-17T17:10:36Z

Besides the locale, everything looks really good!
I tried a lot of stuff, including charts, Z-Wave, etc - all working like a charm.
Awesome work, @ThomDietrich!

dmcwhinnie · 2017-02-17T18:04:57Z

Yay! Good to hear! I think this will close out these 3 issues too.
57,58,66

umiddelb · 2017-02-18T11:31:45Z

I've tried the latest image (openhabianpine64-2017021801-git088eb4e.img). Concerning the locale issue, there is still a mismatch between the default setting (en_GB.UTF-8) and the locales to be generated by default (en_US.UTF-8). Either you switch to en_US.UTF-8 or just enable en_GB.UTF-8 in /etc/locale.gen.

ThomDietrich · 2017-02-18T11:51:51Z

@umiddelb I'm afraid I do not understand. There is no en_GB

https://github.com/openhab/openhabian/blob/master/build-pine64-image/openhabian.pine64.conf

ThomDietrich · 2017-02-18T12:30:20Z

@AngelosF you might want to take a look at this https://github.com/openhab/openhabian/blob/master/openhabian-setup.sh#L237-L247

umiddelb · 2017-02-18T13:03:19Z

@umiddelb I'm afraid I do not understand. There is no en_GB

I see. en_GB.UTF-8 is my default locale on the box I use to login to the Pine64.
I've noticed that the file /etc/default/locale is empty on the Pine64, so sshd obviously takes the locale settings from the system where I come from

After

sudo update-locale LC_ALL="en_US.UTF-8" LANG="en_US.UTF-8" LC_MESSAGES=POSIX

the locale setting in pinned to en_US.UTF-8 on the Pine64.

ThomDietrich · 2017-02-18T13:45:37Z

Hm okay... that's exactly what I'm doing. See #93 (comment)

Just tested on my device:

[14:44:37] openhabian@openHABianPine64:~$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8
[14:44:41] openhabian@openHABianPine64:~$ cat /etc/default/locale
#  File generated by update-locale
LANGUAGE=en_US.UTF-8
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8

umiddelb · 2017-02-18T13:55:11Z

May be I've taken the wrong image, /etc/default/locale is empty in openhabianpine64-2017021801-git088eb4e.img

ThomDietrich · 2017-02-18T14:26:21Z

Nope that's the one. Should be filled after executing "Basic Setup" once. However that's part of the initial setup... Could you open a new issue please!?

umiddelb · 2017-02-18T17:59:55Z

The first-boot.sh failed last time for unknown reason (I got a notification on the console, but I didn't retain first-boot.log, unfortunately). This time first-boot.sh completes the setup without any error.

ThomDietrich · 2017-02-18T18:23:16Z

But then... you should have seen this message: http://docs.openhab.org/installation/openhabian.html#faq-successful ?? (Just realized, that the tip in the middle is only valid for the RPi)

umiddelb · 2017-02-18T20:52:06Z

Hm, when I logged in via ssh, I've seen the usual Ubuntu banner, not the customized one. And I'm pretty sure that I didn't get the extensive error message on the console.

ThomDietrich mentioned this issue Feb 16, 2017

Add Pine64 build process, Zulu Java, many improvements #97

Merged

ThomDietrich closed this as completed in #97 Feb 17, 2017

pine64 Support #93

pine64 Support #93

Comments

ThomDietrich commented Feb 11, 2017

ThomDietrich commented Feb 11, 2017 • edited Loading

dmcwhinnie commented Feb 12, 2017

ThomDietrich commented Feb 12, 2017

dmcwhinnie commented Feb 12, 2017 via email

kaikreuzer commented Feb 12, 2017

ThomDietrich commented Feb 12, 2017

kaikreuzer commented Feb 12, 2017

ThomDietrich commented Feb 13, 2017 • edited Loading

dmcwhinnie commented Feb 13, 2017 • edited Loading

kaikreuzer commented Feb 13, 2017

ThomDietrich commented Feb 13, 2017

kaikreuzer commented Feb 13, 2017

kaikreuzer commented Feb 13, 2017

ThomDietrich commented Feb 13, 2017

ThomDietrich commented Feb 13, 2017 • edited Loading

kaikreuzer commented Feb 13, 2017

kaikreuzer commented Feb 13, 2017

kaikreuzer commented Feb 13, 2017

dmcwhinnie commented Feb 13, 2017

ThomDietrich commented Feb 13, 2017

kaikreuzer commented Feb 13, 2017

kaikreuzer commented Feb 13, 2017

ThomDietrich commented Feb 13, 2017 • edited Loading

ThomDietrich commented Feb 13, 2017 • edited Loading

kaikreuzer commented Feb 13, 2017

ThomDietrich commented Feb 13, 2017

dmcwhinnie commented Feb 13, 2017 • edited Loading

kaikreuzer commented Feb 13, 2017

ThomDietrich commented Feb 14, 2017 • edited Loading

dmcwhinnie commented Feb 14, 2017

ThomDietrich commented Feb 16, 2017

ThomDietrich commented Feb 16, 2017

ThomDietrich commented Feb 16, 2017 • edited Loading

kaikreuzer commented Feb 17, 2017

ThomDietrich commented Feb 17, 2017

kaikreuzer commented Feb 17, 2017

ThomDietrich commented Feb 17, 2017 • edited Loading

kaikreuzer commented Feb 17, 2017

ThomDietrich commented Feb 17, 2017

kaikreuzer commented Feb 17, 2017

ThomDietrich commented Feb 17, 2017

kaikreuzer commented Feb 17, 2017

ThomDietrich commented Feb 17, 2017

ThomDietrich commented Feb 17, 2017

kaikreuzer commented Feb 17, 2017

ThomDietrich commented Feb 17, 2017

kaikreuzer commented Feb 17, 2017

dmcwhinnie commented Feb 17, 2017

umiddelb commented Feb 18, 2017

ThomDietrich commented Feb 18, 2017

ThomDietrich commented Feb 18, 2017 • edited Loading

umiddelb commented Feb 18, 2017

ThomDietrich commented Feb 18, 2017

umiddelb commented Feb 18, 2017

ThomDietrich commented Feb 18, 2017

umiddelb commented Feb 18, 2017

ThomDietrich commented Feb 18, 2017 • edited Loading

umiddelb commented Feb 18, 2017

ThomDietrich commented Feb 11, 2017 •

edited

Loading

ThomDietrich commented Feb 13, 2017 •

edited

Loading

dmcwhinnie commented Feb 13, 2017 •

edited

Loading

ThomDietrich commented Feb 13, 2017 •

edited

Loading

ThomDietrich commented Feb 13, 2017 •

edited

Loading

ThomDietrich commented Feb 13, 2017 •

edited

Loading

dmcwhinnie commented Feb 13, 2017 •

edited

Loading

ThomDietrich commented Feb 14, 2017 •

edited

Loading

ThomDietrich commented Feb 16, 2017 •

edited

Loading

ThomDietrich commented Feb 17, 2017 •

edited

Loading

ThomDietrich commented Feb 18, 2017 •

edited

Loading

ThomDietrich commented Feb 18, 2017 •

edited

Loading