Skip to content

HADOOP-19709. [JDK17] Add debian:12 and debian:13 as a build platform with JDK-17 as default #8001

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: trunk
Choose a base branch
from
+353 −3
Open

HADOOP-19709. [JDK17] Add debian:12 and debian:13 as a build platform with JDK-17 as default #8001

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion BUILDING.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ On Linux / Mac:

$ ./start-build-env.sh [OS platform]

- [OS Platform] One of [rockylinux_8, debian_11, ubuntu_20, ubuntu_24, windows_10].
- [OS Platform] One of [rockylinux_8, debian_11, debian_12, debian_13, ubuntu_20, ubuntu_24, windows_10].
Default is 'ubuntu_20'.
Note: Currently only default ('ubuntu_20') is supported on arm machine

Expand Down
111 changes: 111 additions & 0 deletions dev-support/docker/Dockerfile_debian_12
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Dockerfile for installing the necessary dependencies for building Hadoop.
# See BUILDING.txt.

FROM debian:12

WORKDIR /root

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

#####
# Disable suggests/recommends
#####
RUN echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/10disableextras
RUN echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/10disableextras

ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_TERSE=true

######
# Platform package dependency resolver
######
COPY pkg-resolver pkg-resolver
RUN chmod a+x pkg-resolver/*.sh pkg-resolver/*.py \
&& chmod a+r pkg-resolver/*.json

######
# Install packages from apt
######
# hadolint ignore=DL3008,SC2046
RUN apt-get -q update
RUN apt-get -q install -y --no-install-recommends wget apt-transport-https gpg gpg-agent gawk ca-certificates
RUN apt-get -q install -y --no-install-recommends python3
RUN echo "deb https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" > /etc/apt/sources.list.d/adoptium.list
RUN wget -q -O - https://packages.adoptium.net/artifactory/api/gpg/key/public > /etc/apt/trusted.gpg.d/adoptium.asc
RUN apt-get -q update
RUN apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py debian:12)
RUN apt-get clean
RUN update-java-alternatives -s temurin-17-jdk-amd64
RUN rm -rf /var/lib/apt/lists/*

# Enable the locale
RUN sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g' /etc/locale.gen
RUN locale-gen
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'
ENV PYTHONIOENCODING=utf-8

######
# Set env vars required to build Hadoop
######
ENV MAVEN_HOME=/opt/maven
ENV PATH="${PATH}:${MAVEN_HOME}/bin"
# JAVA_HOME must be set in Maven >= 3.5.0 (MNG-6003)
ENV JAVA_HOME=/usr/lib/jvm/temurin-17-jdk-amd64

#######
# Set env vars for SpotBugs 4.2.2
#######
ENV SPOTBUGS_HOME=/opt/spotbugs

#######
# Set env vars for Google Protobuf 3.21.12
#######
ENV PROTOBUF_HOME=/opt/protobuf
ENV PATH="${PATH}:/opt/protobuf/bin"

###
# Avoid out of memory errors in builds
###
ENV MAVEN_OPTS="-Xms256m -Xmx3072m"

# Skip gpg verification when downloading Yetus via yetus-wrapper
ENV HADOOP_SKIP_YETUS_VERIFICATION=true

####
# Install packages
####
RUN pkg-resolver/install-maven.sh debian:12
RUN pkg-resolver/install-cmake.sh debian:12
RUN pkg-resolver/install-spotbugs.sh debian:12
RUN pkg-resolver/install-boost.sh debian:12
RUN pkg-resolver/install-protobuf.sh debian:12
RUN pkg-resolver/install-hadolint.sh debian:12
RUN pkg-resolver/install-intel-isa-l.sh debian:12

###
# Everything past this point is either not needed for testing or breaks Yetus.
# So tell Yetus not to read the rest of the file:
# YETUS CUT HERE
###

# Add a welcome message and environment checks.
COPY hadoop_env_checks.sh /root/hadoop_env_checks.sh
RUN chmod 755 /root/hadoop_env_checks.sh
# hadolint ignore=SC2016
RUN echo '${HOME}/hadoop_env_checks.sh' >> /root/.bashrc
110 changes: 110 additions & 0 deletions dev-support/docker/Dockerfile_debian_13
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Dockerfile for installing the necessary dependencies for building Hadoop.
# See BUILDING.txt.

FROM debian:13

WORKDIR /root

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

#####
# Disable suggests/recommends
#####
RUN echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/10disableextras
RUN echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/10disableextras

ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_TERSE=true

######
# Platform package dependency resolver
######
COPY pkg-resolver pkg-resolver
RUN chmod a+x pkg-resolver/*.sh pkg-resolver/*.py \
&& chmod a+r pkg-resolver/*.json

######
# Install packages from apt
######
# hadolint ignore=DL3008,SC2046
RUN apt-get -q update
RUN apt-get -q install -y --no-install-recommends wget apt-transport-https gpg gpg-agent gawk ca-certificates
RUN apt-get -q install -y --no-install-recommends python3
RUN echo "deb https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" > /etc/apt/sources.list.d/adoptium.list
RUN wget -q -O - https://packages.adoptium.net/artifactory/api/gpg/key/public > /etc/apt/trusted.gpg.d/adoptium.asc
RUN apt-get -q update
RUN apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py debian:13)
RUN apt-get clean
RUN update-java-alternatives -s temurin-17-jdk-amd64
RUN rm -rf /var/lib/apt/lists/*
Comment on lines +46 to +55
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

each RUN produces one image layer, you should concat those shell commands by && instead


RUN sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g' /etc/locale.gen
RUN locale-gen
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'
ENV PYTHONIOENCODING=utf-8

######
# Set env vars required to build Hadoop
######
ENV MAVEN_HOME=/opt/maven
ENV PATH="${PATH}:${MAVEN_HOME}/bin"
# JAVA_HOME must be set in Maven >= 3.5.0 (MNG-6003)
ENV JAVA_HOME=/usr/lib/jvm/temurin-17-jdk-amd64

#######
# Set env vars for SpotBugs 4.2.2
#######
ENV SPOTBUGS_HOME=/opt/spotbugs

#######
# Set env vars for Google Protobuf 3.21.12
#######
ENV PROTOBUF_HOME=/opt/protobuf
ENV PATH="${PATH}:/opt/protobuf/bin"

###
# Avoid out of memory errors in builds
###
ENV MAVEN_OPTS="-Xms256m -Xmx3072m"

# Skip gpg verification when downloading Yetus via yetus-wrapper
ENV HADOOP_SKIP_YETUS_VERIFICATION=true

####
# Install packages
####
RUN pkg-resolver/install-maven.sh debian:13
RUN pkg-resolver/install-cmake.sh debian:13
RUN pkg-resolver/install-spotbugs.sh debian:13
RUN pkg-resolver/install-boost.sh debian:13
RUN pkg-resolver/install-protobuf.sh debian:13
RUN pkg-resolver/install-hadolint.sh debian:13
RUN pkg-resolver/install-intel-isa-l.sh debian:13

###
# Everything past this point is either not needed for testing or breaks Yetus.
# So tell Yetus not to read the rest of the file:
# YETUS CUT HERE
###

# Add a welcome message and environment checks.
COPY hadoop_env_checks.sh /root/hadoop_env_checks.sh
RUN chmod 755 /root/hadoop_env_checks.sh
# hadolint ignore=SC2016
RUN echo '${HOME}/hadoop_env_checks.sh' >> /root/.bashrc
Loading