本文介绍了使用Service Catalog和OC命令部署OpenShift应用、部署基本概念和流程、扩展存储、清理OpenShift对象等。以Angular 6集成Spring Boot 2,Spring Security,JWT和CORS中的Spring Boot和Angular项目为例，详细讲解了S2I和Pipeline两种部署方式。
OKD版本3.11，Spring Boot项目源码heroes-api，Angular项目源码heroes-web。
初识OpenShift部署
Service Catalog
OpenShift初始安装中含有一些样例APP供大家学习使用。其中有Apache HTTP Server和Apache HTTP Server(httpd)，这两者有什么区别？分别点击进入可以发现：
Apache HTTP Server使用template(template名字为httpd-example)部署方式。
Apache HTTP Server(httpd)使用builder image(image stream名字为httpd)部署方式。
Service Catalog样例使用了template和builder image(image+source)两种部署方式。进入Application Console中的openshift项目可查看template和image。
查看template，点击Resources -> Other Resources -> Template：
查看Image Stream，点击Builds -> Images：
其他部署方式
在Service Catalog中，除从Catalog直接选择Item外，还提供了其他三种方式：
Deploy Image可以直接从image或image stream部署应用：
Import YAML / JSON 用来从YAML或JSON创建资源，比如image stream、template：
Select from Project 从指定的Project中选择template来部署应用：
部署Apache HTTP Server
Apache HTTP Server的两种部署方式本质上是相同的，Build策略均为S2I(Source-to-Image)，使用S2I构建的Docker镜像来部署应用。Source均使用Apache HTTP Server (httpd) S2I Sample Application，Docker基础镜像(builder image)均使用Apache HTTP Server Container Image。httpd-example template定义了整体部署流程并实现了参数化。
以下是httpd-example template中BuildConfig部分的定义：
- apiVersion: v1
kind: BuildConfig
metadata:
annotations:
description: Defines how to build the application
template.alpha.openshift.io/wait-for-ready: 'true'
name: '${NAME}'
spec:
output:
to:
kind: ImageStreamTag
name: '${NAME}:latest'
source:
contextDir: '${CONTEXT_DIR}'
git:
ref: '${SOURCE_REPOSITORY_REF}'
uri: '${SOURCE_REPOSITORY_URL}'
type: Git
strategy:
sourceStrategy:
from:
kind: ImageStreamTag
name: 'httpd:2.4'
namespace: '${NAMESPACE}'
type: Source
triggers:
- type: ImageChange
- type: ConfigChange
- github:
secret: '${GITHUB_WEBHOOK_SECRET}'
type: GitHub
- generic:
secret: '${GENERIC_WEBHOOK_SECRET}'
type: Generic
参数定义及默认值：
parameters:
- description: The name assigned to all of the frontend objects defined in this template.
displayName: Name
name: NAME
required: true
value: httpd-example
- description: The OpenShift Namespace where the ImageStream resides.
displayName: Namespace
name: NAMESPACE
required: true
value: openshift
- description: Maximum amount of memory the container can use.
displayName: Memory Limit
name: MEMORY_LIMIT
required: true
value: 512Mi
- description: The URL of the repository with your application source code.
displayName: Git Repository URL
name: SOURCE_REPOSITORY_URL
required: true
value: 'https://github.com/openshift/httpd-ex.git'
- description: >-
Set this to a branch name, tag or other ref of your repository if you are
not using the default branch.
displayName: Git Reference
name: SOURCE_REPOSITORY_REF
- description: >-
Set this to the relative path to your project if it is not in the root of
your repository.
displayName: Context Directory
name: CONTEXT_DIR
- description: >-
The exposed hostname that will route to the httpd service, if left blank a
value will be defaulted.
displayName: Application Hostname
name: APPLICATION_DOMAIN
...
我们先使用builder image方式部署Apache，来了解一下部署的整体流程：
点击"advanced options"，可以设置git branch、context、secret，自定义Route、Build Configuration、Deployment Configuration、Resource Limits等。此处填完基本内容后直接点击Create，创建App，然后从成功页面进入Project Overview：
部署过程中自动创建Service、Route、Build、Deployment、Image。进入Application Console的Applications和Builds可以查看详细信息，其中会创建3个pod：httpd-1-build、http-1-deploy、httpd-1-xxxxx，部署完毕后http-1-deploy会自动删除。
部署成功后，测试访问Apache Server(Route定义的Hostname)，页面如下：
下面解释一下涉及到的基本概念。
基本概念
Service(Kubernetes Service)
内部load balancer，用在OpenShift内部网络中，可使用Service ClusterIP或Hostname访问。
apiVersion: v1
kind: Service
metadata:
annotations:
openshift.io/generated-by: OpenShiftWebConsole
creationTimestamp: '2019-03-26T02:12:50Z'
labels:
app: httpd
name: httpd
namespace: my-project
resourceVersion: '3004428'
selfLink: /api/v1/namespaces/my-project/services/httpd
uid: a81c759f-4f6c-11e9-9a7d-02fa2ffc40e6
spec:
clusterIP: 172.30.225.159
ports:
- name: 8080-tcp
port: 8080
protocol: TCP
targetPort: 8080
selector:
deploymentconfig: httpd
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
其中，selector定义了查找container(pod)进行负载均衡的标签。
Route
定义一个hostname来公开Service，以便外部客户可以访问Service，默认hostname为：[app-name]-[project-name].[openshift_master_default_subdomain]。
Build
构建App Image，使用S2I时即从builder image和Source Code来构建App Image。默认builder image和build配置变化时会重新build。
查看Builds -> httpd -> #1 的YAML文本，可以了解Build流程为FetchInputs -> Assemble -> CommitContainer -> PushImage：
...
status:
completionTimestamp: '2019-03-26T02:13:30Z'
config:
kind: BuildConfig
name: httpd
namespace: my-project
duration: 40000000000
output:
to:
imageDigest: 'sha256:5c1f20f20baaa796f4518d11ded13c6fac33e7a377774cfec77aa1e6e6a7cbb2'
outputDockerImageReference: 'docker-registry.default.svc:5000/my-project/httpd:latest'
phase: Complete
stages:
- durationMilliseconds: 3434
name: FetchInputs
startTime: '2019-03-26T02:12:56Z'
steps:
- durationMilliseconds: 3434
name: FetchGitSource
startTime: '2019-03-26T02:12:56Z'
- durationMilliseconds: 2127
name: CommitContainer
startTime: '2019-03-26T02:13:11Z'
steps:
- durationMilliseconds: 2127
name: CommitContainer
startTime: '2019-03-26T02:13:11Z'
- durationMilliseconds: 3426
name: Assemble
startTime: '2019-03-26T02:13:10Z'
steps:
- durationMilliseconds: 3426
name: AssembleBuildScripts
startTime: '2019-03-26T02:13:10Z'
- durationMilliseconds: 16143
name: PushImage
startTime: '2019-03-26T02:13:14Z'
steps:
- durationMilliseconds: 16143
name: PushImage
startTime: '2019-03-26T02:13:14Z'
startTimestamp: '2019-03-26T02:12:50Z'
Build Strategy
OpenShift支持Source-to-Image、Docker、Pipeline、Custom四种Build Strategy。
strategy:
sourceStrategy:
from:
kind: "ImageStreamTag"
name: "builder-image:latest"
strategy:
dockerStrategy:
from:
kind: "ImageStreamTag"
name: "debian:latest"
spec:
source:
git:
uri: "https://github.com/openshift/ruby-hello-world"
strategy:
jenkinsPipelineStrategy:
jenkinsfilePath: some/repo/dir/filename
strategy:
customStrategy:
from:
kind: "DockerImage"
name: "openshift/sti-image-builder"
Deployment
部署App Image，Deployment包含三种对象：DeploymentConfig、ReplicationController、Pod。DeploymentConfig包含部署策略、image配置、环境变量等，ReplicationController包含复制相关信息。App Image和deployment配置变化时会自动重新Deploy。
进入Deployments -> httpd -> #1，编辑Replicas或调节pods数可以增删pod：
Deployment Strategy
修改或升级App，即重新部署应用时的部署方式。部署配置(DeploymentConfig)支持三种策略：Rolling、Recreate、Custom。通过修改Route可以实现蓝/绿部署、A/B部署。
- Rolling 默认策略，当新版本Pod状态变为Ready后才scale down老版本Pod，可能同时存在新老版本的Pod
- Recreate 先终止所有Pod(Scale down the previous deployment to zero)再部署新Pod
- Custom 自定义部署行为
ImageStream
OpenShift管理容器镜像的方式，其中定义了dockerImageReference，ImageStream tag定义了同docker image各版本的映射关系。Build成功后会自动创建ImageStream。
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
annotations:
openshift.io/generated-by: OpenShiftWebConsole
creationTimestamp: '2019-03-26T02:12:50Z'
generation: 1
labels:
app: httpd
name: httpd
namespace: my-project
resourceVersion: '3004571'
selfLink: /apis/image.openshift.io/v1/namespaces/my-project/imagestreams/httpd
uid: a81b14bf-4f6c-11e9-9a7d-02fa2ffc40e6
spec:
lookupPolicy:
local: false
status:
dockerImageRepository: 'docker-registry.default.svc:5000/my-project/httpd'
tags:
- items:
- created: '2019-03-26T02:13:30Z'
dockerImageReference: >-
docker-registry.default.svc:5000/my-project/httpd@sha256:5c1f20f20baaa796f4518d11ded13c6fac33e7a377774cfec77aa1e6e6a7cbb2
generation: 1
image: >-
sha256:5c1f20f20baaa796f4518d11ded13c6fac33e7a377774cfec77aa1e6e6a7cbb2
tag: latest
Template
定义整体部署流程并实现参数化，包含Service、Route、ImageStream、BuildConfig、DeploymentConfig、parameters等部分。
了解了以上基本概念就很容易理解httpd-example template了，您可以自己部署测试，此处不再赘述。
OC Tool
使用oc new-app部署应用
继续之前，先将以前创建的测试project删除或新建一个project。
$ oc delete project my-project
$ oc new-project my-project
在Service Catalog一节我们提到了部署应用的三种方式：template、builder image(image+source)、image，对应的命令如下：
$ oc new-app httpd-example -p APPLICATION_DOMAIN=httpd-example.apps.itrunner.org
$ oc new-app openshift/httpd:2.4~https://github.com/openshift/httpd-ex.git --name=httpd-ex
$ oc new-app my-project/httpd-ex --name=httpd
说明：
- image+source的语法为[image]~[source]
- 第三种方式使用的image为第二种方式中生成的
- 后面两种方式不会自动创建Route，需要手工创建：
$ oc expose service httpd-ex --name httpd-ex --hostname=httpd-ex.apps.itrunner.org
$ oc expose service httpd --name httpd --hostname=httpd.apps.itrunner.org
从JSON/YAML创建资源：
$ oc create -f <filename> -n <project>
使用oc命令还可以直接从source code创建应用，可以使用本地或远程source code：
$ oc new-app /path/to/source/code
$ oc new-app https://github.com/sclorg/cakephp-ex
可以指定子目录：
$ oc new-app https://github.com/sclorg/s2i-ruby-container.git --context-dir=2.0/test/puma-test-app
可以指定branch：
$ oc new-app https://github.com/openshift/ruby-hello-world.git#beta4
OpenShift自动检测代码根目录或指定目录，如果存在Dockerfile则使用Docker build策略，如果存在Jenkinsfile则使用Pipeline build策略，否则使用Source build策略(S2I)。
下面的例子使用了Source build策略：
$ oc new-app https://github.com/sclorg/cakephp-ex
使用Source build策略时，new-app通过检测根目录或指定目录的文件来确定language builder：
| Language |
Files |
| jee |
pom.xml |
| nodejs |
app.json, package .json |
| perl |
cpanfile, index.pl |
| php |
composer.json, index.php |
| python |
requirements.txt, setup.py |
| ruby |
Gemfile, Rakefile, config.ru |
| scala |
build.sbt |
| golang |
Godeps, main.go |
然后根据语言，从OpenShift Server或Docker Hub Registry中查找与语言匹配的image。
也可以指定策略，如下：
$ oc new-app /home/user/code/myapp --strategy=docker
查看template和image stream
查看所有template和image stream：
$ oc new-app --list
单独查看template或image stream：
$ oc get templates -n openshift
$ oc get imagestreams -n openshift
查看httpd-example template详细信息：
$ oc describe template httpd-example -n openshift
查看httpd image stream详细信息：
$ oc describe imagestream httpd -n openshift
查看httpd-example template的YAML定义：
$ oc new-app --search --template=httpd-example --output=yaml
从所有template、image stream、docker image中查找"httpd"：
$ oc new-app --search httpd
再谈Route
前面的例子Route使用的协议均为http，如何启用https呢?
使用Web Console时，编辑route启用Secure route即可：
TLS Termination有三种类型：edge、passthrough、reencrypt
- edge 访问route使用https协议，route到内部网络为非加密的，如未配置证书则使用默认证书。
- reencrypt 全部访问路径均是加密的
- passthrough 加密通信直接发送到目标，route不需提供TLS Termination。
使用oc命令创建route：
$ oc create route edge httpd-ex --service httpd-ex --hostname httpd-ex.apps.itrunner.org --path / --insecure-policy Redirect -n my-project
S2I
Source-to-Image (S2I)是一个框架，可以容易地将应用程序源代码作为输入生成一个新的docker image。
使用S2I构建image，在装配过程中可以执行大量复杂的操作，所有操作仅创建一个新的layer，加速了处理过程。S2I使得软件开发工程师不必关心docker image的制作，仅负责编写assemble、run等脚本，也可以防止开发工程师在image构建过程中执行任意yum安装等不适宜的操作。S2I简化了docker image的制作。
S2I需要以下三个基本要素：
- builder image S2I以此image为基础来构建新的image
- Sources
- S2I Scripts
在构建过程中，S2I先获取sources和scripts，将其打包为tar文件后放入builder image中。在执行assemble script前，S2I解压tar文件到io.openshift.s2i.destination指定的目录，默认目录为/tmp（分别解压到/tmp/src、/tmp/scripts目录）。
S2I Scripts
S2I Scripts可以位于以下位置，优先级从高到低：
- 在BuildConfig中指定位置
strategy:
sourceStrategy:
from:
kind: "ImageStreamTag"
name: "builder-image:latest"
scripts: "http://somehost.com/scripts_directory"
- 应用程序源码的.s2i/bin目录
- 在builder image的Dockerfile中定义位置（在样例Apache HTTP Server中使用了这种方式 ）
LABEL io.openshift.s2i.scripts-url="image:///usr/libexec/s2i"
io.openshift.s2i.scripts-url和BuildConfig定义位置均可采用以下形式：
- image:///path_to_scripts_dir - image的绝对路径
- file:///path_to_scripts_dir - host绝对或相对路径
- http(s)://path_to_scripts_dir - URL
S2I Scripts：
| Script |
Description |
| assemble (required) |
获取源码、编译、打包。在增量编译时，如果定义了save-artifacts，先恢复artifact |
| run (required) |
运行应用 |
| save-artifacts (optional) |
收集依赖以便加速后续编译，比如.m2 |
| usage (optional) |
显示image使用帮助信息 |
| test/run (optional) |
检查image能否正常工作 |
Example assemble script
#!/bin/bash
# restore build artifacts
if [ "$(ls /tmp/artifacts/ 2>/dev/null)" ]; then
mv /tmp/artifacts/* $HOME/.
fi
# move the application source
mv /tmp/s2i/src $HOME/src
# build application artifacts
pushd ${HOME}
make all
# install the artifacts
make install
popd
Example run script
#!/bin/bash
# run the application
/opt/application/run.sh
Example save-artifacts script
#!/bin/bash
# Besides the tar command, all other output to standard out must
# be surpressed. Otherwise, the tar stream will be corrupted.
pushd ${HOME} >/dev/null
if [ -d deps ]; then
# all deps contents to tar stream
tar cf - deps
fi
popd >/dev/null
注意：save-artifacts只能有tar stream输出，不能含有其它任何输出。
Example usage script
#!/bin/bash
# inform the user how to use the image
cat <<EOF
This is a S2I sample builder image, to use it, install
https://github.com/openshift/source-to-image
EOF
S2I Tool
仅为学习S2I的基本知识和S2I Tool的使用，部署OpenShift应用时是不必安装的。
安装S2I Tool
下载S2I后解压安装：
# tar -xvf release.tar.gz .
# cp /path/to/s2i /usr/local/bin
S2I命令
create 创建生成builder image的基础目录结构
build 构建新的image
rebuild 重建image
usage 显示image usage信息
version 显示s2i version
completion Generate completion for the s2i command (bash or zsh)
示例
- 使用s2i create创建目录结构
s2i create语法：
s2i create <imageName> <destination> [flags]
$ s2i create ruby-centos7 ruby-centos7
执行以上命令生成的目录结构如下：
ruby-centos7
├── Dockerfile
├── Makefile
├── README.md
├── s2i
│ └── bin
│ ├── assemble
│ ├── run
│ ├── save-artifacts
│ └── usage
└── test
├── run
└── test-app
└── index.html
- 构建builder image
$ cd ruby-centos7
$ make build
- 构建App image
s2i build语法：
s2i build <source> <image> [<tag>] [flags]
# cd ruby-centos7
# s2i build test/test-app/ ruby-centos7 ruby-app
- 运行image
# docker run --rm -d -p 8080:8080 --name ruby-app ruby-app
增量构建与save-artifacts
Maven、Angular等项目编译时需要下载其他依赖，为提高编译速度，避免重复下载，S2I支持增量build。增量build依赖以前构建的image，image名字必须相同，且image中必须含有save-artifacts script。
增量build流程如下：
- S2I从以前构建的app image创建一个新的docker container
- S2I运行container中的save-artifacts将依赖打包为tar文件输出到stdout
- S2I下载source与artifacts一起打包作为新image的输入
- S2I启动新container、运行assemble ...
artifacts默认解压到/tmp/artifacts目录中。
启用增量构建
- 使用s2i命令时增加参数--incremental=true：
# s2i build test/test-app/ ruby-centos7 ruby-app --incremental=true
- 在OpenShift Web Console中修改BuildConfig，增加incremental: true，然后点击Start Build。
strategy:
type: "Source"
sourceStrategy:
from:
kind: "ImageStreamTag"
name: "incremental-image:latest"
incremental: true
部署Spring Boot App
Builder Image
Dockerfile
# heroes-api-centos7
FROM centos:latest
RUN yum -y update && yum clean all
# Set the labels that are used for OpenShift to describe the builder image.
LABEL maintainer="Sun Jingchuan <jason@163.com>" \
io.k8s.description="Heroes API" \
io.k8s.display-name="Heroes API" \
io.openshift.expose-services="8080:http" \
io.openshift.tags="spring-boot,heroes-api" \
# this label tells s2i where to find its mandatory scripts(run, assemble, save-artifacts)
# io.openshift.s2i.scripts-url="image:///usr/libexec/s2i" \
io.openshift.s2i.scripts-url="image:///tmp/scripts" \
io.openshift.s2i.destination="/tmp"
ENV JAVA_HOME=/usr/lib/jdk1.8.0_202 \
MAVEN_HOME=/usr/lib/apache-maven-3.6.0 \
APP_ROOT=/opt/heroes
ENV PATH=${JAVA_HOME}/bin:${MAVEN_HOME}/bin:${APP_ROOT}/bin:${PATH} HOME=${APP_ROOT}
# Include jdk and maven in lib
COPY lib /usr/lib
COPY bin ${APP_ROOT}/bin
# Copy the S2I scripts to /usr/libexec/s2i
# COPY .s2i/bin /usr/libexec/s2i
RUN chmod -R u+x ${APP_ROOT}/bin && \
chgrp -R 0 ${APP_ROOT} && \
chmod -R g=u ${APP_ROOT} /etc/passwd
USER 10001
WORKDIR ${APP_ROOT}
ENTRYPOINT [ "uid_entrypoint" ]
EXPOSE 8080
# Inform the user how to run this image.
# CMD ["/usr/libexec/s2i/usage"]
说明：
- lib目录中存放了JDK和Maven
- 使用源码中的S2I Scripts，builder image中不提供，但需定义io.openshift.s2i.scripts-url
- 默认，为了安全，OpenShift中只能使用User ID，不能使用用户名，不能使用docker默认用户root，因此要授予目录适当权限。
- 源码bin目录中存放了用户创建脚本uid_entrypoint，内容如下：
#!/bin/bash
if ! whoami &> /dev/null; then
if [ -w /etc/passwd ]; then
echo "${USER_NAME:-default}:x:$(id -u):0:${USER_NAME:-default} user:${HOME}:/sbin/nologin" >> /etc/passwd
fi
fi
exec "$@"
编译builder image并上传到Registry
# docker build -t heroes-api-centos7:v1.0.0 .
# docker tag heroes-api-centos7:v1.0.0 registry.itrunner.org/heroes-api-centos7:v1.0.0
# docker push registry.itrunner.org/heroes-api-centos7:v1.0.0
import image
$ oc import-image heroes-api-centos7:v1.0.0 -n heroes --confirm --insecure --from='registry.itrunner.org/heroes-api-centos7:v1.0.0'
导入命令中指定了参数-n heroes，image会导入heroes项目中。成功导入后可在项目的Builds -> Images中查看image，但在Service Catalog中仍不可见，需要修改Image Stream定义，增加annotations：
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
annotations:
openshift.io/image.dockerRepositoryCheck: '2019-03-27T08:40:27Z'
creationTimestamp: '2019-03-27T08:39:50Z'
generation: 1
name: heroes-api-centos7
namespace: heroes
resourceVersion: '3337267'
selfLink: /apis/image.openshift.io/v1/namespaces/heroes/imagestreams/heroes-api-centos7
uid: e280929e-506b-11e9-a2ec-0288bf58ecc2
spec:
lookupPolicy:
local: false
tags:
- annotations:
description: build heroes-api on CentOS 7
iconClass: icon-spring
openshift.io/display-name: Heroes API
openshift.io/provider-display-name: itrunner
sampleRepo: 'https://github.com/sunjc/heroes-api.git'
supports: itrunner
tags: 'builder,java'
version: '1.0.0'
from:
kind: DockerImage
name: 'registry.itrunner.org/heroes-api-centos7:v1.0.0'
...
为了在Service Catalog中显示，tags中必须含有“builder”。只有导入openshift项目的image才是全局可见的，否则仅在本项目Catalog可见。如未显示请刷新页面。
S2I Scripts
- assemble
#!/bin/bash -e
# restore build artifacts
if [ -d /tmp/artifacts/.m2 ]; then
echo "restore build artifacts"
mv /tmp/artifacts/.m2 $HOME/.
fi
# move the application source
mv /tmp/src $HOME/src
# build the application artifacts
pushd $HOME/src
mvn clean package -Pdev -Dmaven.test.skip=true
popd
# move the artifacts
mv $HOME/src/target/heroes-api-1.0.0.jar $HOME/
rm -rf $HOME/src
- run
#!/bin/bash
java -jar $HOME/heroes-api-1.0.0.jar
- save-artifacts
#!/bin/bash
# Besides the tar command, all other output to standard out must be surpressed. Otherwise, the tar stream will be corrupted.
pushd ${HOME} >/dev/null
if [ -d .m2 ]; then
# all .m2 contents to tar stream
tar cf - .m2
fi
popd >/dev/null
- usage
#!/bin/bash
# inform the user how to use the image
cat <<EOF
This is a S2I sample builder image, to use it, install
https://github.com/openshift/source-to-image
EOF
部署
$ oc new-app heroes/heroes-api-centos7:v1.0.0~https://github.com/sunjc/heroes-api.git --name=heroes-api
$ oc expose service heroes-api --name heroes-api --hostname heroes.apps.itrunner.org --path /api -n heroes
或
$ oc create route edge heroes-api --service heroes-api --hostname heroes.apps.itrunner.org --path /api --insecure-policy Redirect -n heroes
说明：
- 如果使用私有git仓库，new-app需要增加参数--source-secret=yoursecret。Secret可在Resources -> Secrets中配置。
- route的path配置为/api，不是根目录，这样可以防止外部用户访问api-docs、swagger-ui
- 为了内部用户可以访问api-docs、swagger-ui，给service增加externalIPs参数，指定一个或多个OpenShift Node地址，如下：
spec:
clusterIP: 172.30.80.170
externalIPs:
- 10.188.12.116
ports:
- name: 8080-tcp
port: 8080
protocol: TCP
targetPort: 8080
部署Angular App
Builder Image
Dockerfile
# heroes-web-centos7
FROM centos/httpd:latest
RUN yum -y update && \
curl -sL https://rpm.nodesource.com/setup_10.x | bash - && yum -y install nodejs && \
yum clean all && npm install -g @angular/cli@latest
# Set the labels that are used for OpenShift to describe the builder image.
LABEL maintainer="Sun Jingchuan <jason@163.com>" \
io.k8s.description="Heroes Web" \
io.k8s.display-name="Heroes Web" \
io.openshift.expose-services="8080:http" \
io.openshift.tags="angular,heroes-web" \
# this label tells s2i where to find its mandatory scripts(run, assemble, save-artifacts)
# io.openshift.s2i.scripts-url="image:///usr/libexec/s2i" \
io.openshift.s2i.scripts-url="image:///tmp/scripts" \
io.openshift.s2i.destination="/tmp"
ENV APP_ROOT=/opt/heroes
ENV PATH=${APP_ROOT}/bin:${PATH} HOME=${APP_ROOT} HTTPD_MAIN_CONF_PATH=/etc/httpd/conf
COPY bin ${APP_ROOT}/bin
# Copy the S2I scripts to /usr/libexec/s2i
# COPY .s2i/bin /usr/libexec/s2i
RUN chmod -R u+x ${APP_ROOT}/bin && \
chgrp -R 0 ${APP_ROOT} && \
chmod -R g=u ${APP_ROOT} /etc/passwd /var/www/html /run/httpd && \
chown -R root:root /run/httpd /etc/httpd && \
sed -i -e "s/^User apache/User default/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
sed -i -e "s/^Group apache/Group root/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
sed -i -e "s/^Listen 80/Listen 8080/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
sed -ri " s!^(\s*CustomLog)\s+\S+!\1 |/usr/bin/cat!g; s!^(\s*ErrorLog)\s+\S+!\1 |/usr/bin/cat!g;" ${HTTPD_MAIN_CONF_PATH}/httpd.conf
USER 10001
WORKDIR ${APP_ROOT}
ENTRYPOINT [ "uid_entrypoint" ]
EXPOSE 8080
# Inform the user how to run this image.
# CMD ["/usr/libexec/s2i/usage"]
说明：
- 需要安装nodejs、angular/cli
- httpd相关目录要授予适当权限，需要修改httpd用户、组、日志输出、监听端口
修改用户后，不能再使用80端口，否则会报错：permission denied: ah00072: make_sock: could not bind to address
编译builder image并上传到Registry
# docker build -t heroes-web-centos7:v1.0.0 .
# docker tag heroes-web-centos7:v1.0.0 registry.itrunner.org/heroes-web-centos7:v1.0.0
# docker push registry.itrunner.org/heroes-web-centos7:v1.0.0
import image
$ oc import-image heroes-web-centos7:v1.0.0 -n heroes --confirm --insecure --from='registry.itrunner.org/heroes-web-centos7:v1.0.0'
编辑ImageStream
$ oc edit is/heroes-web-centos7
增加如下annotations:
tags:
- annotations:
description: build heroes-web on CentOS 7
iconClass: icon-angularjs
openshift.io/display-name: Heroes Web
openshift.io/provider-display-name: itrunner
sampleRepo: 'https://github.com/sunjc/heroes-web.git'
supports: itrunner
tags: builder,javascript
version: 1.0.0
from:
kind: DockerImage
name: 'registry.itrunner.org/heroes-web-centos7:v1.0.0'
S2I Scripts
- assemble
#!/bin/bash -e
# move the application source
mv /tmp/src $HOME/src
# restore build artifacts
if [ "$(ls /tmp/artifacts/ 2>/dev/null)" ]; then
mv /tmp/artifacts/* $HOME/src
fi
# build the application artifacts
pushd $HOME/src
npm install
ng build --prod --base-href=/heroes/
# Install the artifacts
mv dist /var/www/html/heroes
mv node_modules $HOME/node_modules
popd
rm -rf $HOME/src
- run
#!/bin/bash
# run the application
exec httpd -D FOREGROUND $@
- save-artifacts
#!/bin/bash
# Besides the tar command, all other output to standard out must be surpressed. Otherwise, the tar stream will be corrupted.
pushd ${HOME} >/dev/null
if [ -d node_modules ]; then
# all node_modules contents to tar stream
tar cf - node_modules
fi
popd >/dev/null
- usage
#!/bin/bash
# inform the user how to use the image
cat <<EOF
This is a S2I sample builder image, to use it, install
https://github.com/openshift/source-to-image
EOF
部署
$ oc new-app heroes/heroes-web-centos7:v1.0.0~https://github.com/sunjc/heroes-web.git --name=heroes-web
$ oc expose service heroes-web --name heroes-web --hostname heroes.apps.itrunner.org --path /heroes --port 8080-tcp -n heroes
或
$ oc create route edge heroes-web --service heroes-web --hostname heroes.apps.itrunner.org --path /heroes \
--insecure-policy Redirect --port 8080-tcp -n heroes
Image管理
Internal Registry
前面的例子，我们使用了私有Docker Registry "registry.itrunner.org"，先将base image上传到私有Registry，然后再导入到OpenShift中。我们也可以直接使用OpenShift内部的Registry(安装在default项目中)，执行push、pull等操作。
为了访问Internal Registry必须先执行docker login登录Registry，需使用openshift用户名(或email)，使用有效的openshift token作为密码。
- 登录openshift获取token
$ oc login https://openshift.itrunner.org:8443 -u jason --certificate-authority=/path/to/cert.crt
$ oc whoami -t
- 登录Internal Registry
# docker login -u <user_name> -e <email_address> -p <token_value> <registry_server>:<port>
如在openshift节点内访问Internal Registry，registry_server可以使用service hostname "docker-registry.default.svc"，port为5000；如在外部访问，registry_server则需使用route hostname，比如"docker-registry-default.apps.itrunner.org" ，docker-registry route的TLS Termination需配置为Re-encrypt。
# docker login -u jason -p xxxxxxxxxxx docker-registry-default.apps.itrunner.org
- Push Image
# docker tag heroes-web-centos7:v1.0.0 docker-registry-default.apps.itrunner.org/heroes/heroes-web-centos7:latest
# docker push docker-registry-default.apps.itrunner.org/heroes/heroes-web-centos7:latest
注意：tag格式必须为project/name
push成功后，openshift将自动创建image stream。
- Pull Image
# docker pull docker-registry-default.apps.itrunner.org/heroes/heroes-web-centos7:latest
- 查看Repository
$ curl -u jason:xxxxxxxxxx -kv https://docker-registry-default.apps.itrunner.org/v2/_catalog?n=100
用户必须有list权限：
$ oc adm policy add-cluster-role-to-user registry-viewer user
- 查看image stream
$ oc get is -n heroes
NAME DOCKER REPO TAGS UPDATED
heroes-api docker-registry.default.svc:5000/heroes/heroes-api latest 7 days ago
heroes-api-centos7 docker-registry.default.svc:5000/heroes/heroes-api-centos7 v1.0.0 7 days ago
heroes-web docker-registry.default.svc:5000/heroes/heroes-web latest 7 days ago
heroes-web-centos7 docker-registry.default.svc:5000/heroes/heroes-web-centos7 latest,v1.0.0 19 hours ago
Image Stream
Image Stream是OpenShift管理容器镜像的方式，其中定义了dockerImageReference，利用tag定义了同docker image各版本的映射关系。Image Stream本身不包含image data，image stream元数据与其他集群信息一起存储在etcd实例中。
image stream更新后可以触发build或deployment，如docker image已更新，但未更新image stream则不会触发build或deployment。
image可以源自OpenShift Internal Registry、外部Registry(比如registry.redhat.io、docker.io、私有Registry)、OpenShift集群中的image stream。
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
annotations:
openshift.io/image.dockerRepositoryCheck: '2019-04-09T05:59:17Z'
creationTimestamp: '2019-04-01T08:02:37Z'
generation: 2
name: heroes-web-centos7
namespace: heroes
resourceVersion: '2145022'
selfLink: >-
/apis/image.openshift.io/v1/namespaces/heroes/imagestreams/heroes-web-centos7
uid: 83deac8c-5454-11e9-af2b-02f23e935364
spec:
lookupPolicy:
local: false
tags:
- annotations:
description: >-
build heroes-web on CentOS 7. WARNING: By selecting this tag, your
application will automatically update to use the latest version.
iconClass: icon-angularjs
openshift.io/display-name: Heroes Web (Latest)
openshift.io/provider-display-name: itrunner
sampleRepo: 'https://github.com/sunjc/heroes-web.git'
supports: itrunner
tags: 'builder,javascript'
from:
kind: DockerImage
name: 'docker-registry.default.svc:5000/heroes/heroes-web-centos7:latest'
generation: 2
importPolicy: {}
name: latest
referencePolicy:
type: Source
- annotations:
description: build heroes-web on CentOS 7
iconClass: icon-angularjs
openshift.io/display-name: Heroes Web 1.0.0
openshift.io/provider-display-name: itrunner
sampleRepo: 'https://github.com/sunjc/heroes-web.git'
supports: itrunner
tags: 'builder,javascript'
version: 1.0.0
from:
kind: DockerImage
name: 'registry.itrunner.org/heroes-web-centos7:v1.0.0'
generation: 1
importPolicy:
insecure: true
name: v1.0.0
referencePolicy:
type: Source
status:
dockerImageRepository: 'docker-registry.default.svc:5000/heroes/heroes-web-centos7'
tags:
- items:
- created: '2019-04-08T07:37:11Z'
dockerImageReference: >-
docker-registry.default.svc:5000/heroes/heroes-web-centos7@sha256:7e4126ec9ec0d4158d962936a38f255806731d33d6fe03b29d95d82759823fcd
generation: 2
image: >-
sha256:7e4126ec9ec0d4158d962936a38f255806731d33d6fe03b29d95d82759823fcd
tag: latest
- items:
- created: '2019-04-01T08:02:37Z'
dockerImageReference: >-
registry.itrunner.org/heroes-web-centos7@sha256:7e4126ec9ec0d4158d962936a38f255806731d33d6fe03b29d95d82759823fcd
generation: 1
image: >-
sha256:7e4126ec9ec0d4158d962936a38f255806731d33d6fe03b29d95d82759823fcd
tag: v1.0.0
Reference Policy
当使用从外部Registry导入的image时，"引用策略" 允许指定从何处提取image。有两个选项：Local和Source：
- Local 从OpenShift Internal Registry提取image
- Source 直接从外部Registry提取image
查询Image Stream信息
$ oc describe is/heroes-web-centos7
$ oc describe istag/heroes-web-centos7:latest
为外部Image添加tag
$ oc tag docker.io/openshift/base-centos7:latest base-centos7:latest
为Image Stream附加tag
给现有tag附加一个tag：
$ oc tag heroes-api-centos7:v1.0.0 heroes-api-centos7:latest
更新tag
$ oc tag heroes-api-centos7:v1.0.1 heroes-api-centos7:latest
说明：与附加tag不同，两个tag都应存在
删除tag
$ oc tag -d heroes-api-centos7:v1.0.0
或
$ oc delete istag/heroes-api-centos7:v1.0.0
删除Image Stream
$ oc delete is base-centos7
定期更新tag
可以使用--scheduled：
$ oc tag docker.io/python:3.6.0 python:3.6 --scheduled
也可以在tag定义中设置importPolicy.scheduled为true：
apiVersion: v1
kind: ImageStream
metadata:
name: python
spec:
tags:
- from:
kind: DockerImage
name: docker.io/python:3.6.0
name: latest
importPolicy:
scheduled: true
周期默认为15分钟。
Binary Build
Binary Build主要应用于测试和Jenkins pipeline场景。开发人员提交源码前如果想先测试一下，可使用本地源码来构建App Image。Binary Build不能自动触发，只能手动执行。
Binary Build使用oc start-build命令，需要提供BuildConfig或存在的build，支持从几种以下source来构建App Image：
- --from-file 比如Dockerfile
- --from-dir 本地目录，start-build打包此目录并上传到openshift
- --from-archive tar、tar.gz、zip等
- --from-repo 本地源码目录，start-build打包最近commit的代码并上传到openshift
From a directory
$ oc start-build heroes-web --from-dir="." --follow
或
$ oc start-build --from-build=heroes-web-1 --from-dir="." --follow
From a Git repository
$ git commit -m "My changes"
$ oc start-build heroes-web --from-repo="." --follow
Jenkins Pipeline
...
stage("Build Image") {
steps {
dir('heroes-web/dist') {
sh 'oc start-build heroes-web --from-dir . --follow'
}
}
}
...
Pipeline部署
Jenkins是广泛应用的CI工具，大多数工程师都有使用经验，更习惯使用Jenkins部署应用。使用Jenkins Pipeline部署OpenShift应用，不影响原有流程，可以正常执行测试、代码质量检查、编译打包等操作，只需在部署时调用oc start-build。
安装Jenkins
OpenShift提供了两个Jenkins Template：jenkins-ephemeral、jenkins-persistent，一种使用瞬时存储，一种使用持久存储，两者均使用jenkins image stream(docker.io/openshift/jenkins-2-centos7:v3.11)。jenkins image安装了OpenShift Client、OpenShift Login、OpenShift Sync、Kubernetes、Kubernetes Credentialst等插件。安装后既可以在OpenShift中运行Job，也可以在Jenkins中运行Job。
jenkins-persistent采用动态存储配置，PVC默认名称为jenkins(JENKINS_SERVICE_NAME)：
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: '${JENKINS_SERVICE_NAME}'
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: '${VOLUME_CAPACITY}'
storageClassName: glusterfs-storage-block
- apiVersion: v1
kind: DeploymentConfig
metadata:
annotations:
template.alpha.openshift.io/wait-for-ready: 'true'
name: '${JENKINS_SERVICE_NAME}'
spec:
...
template:
metadata:
labels:
name: '${JENKINS_SERVICE_NAME}'
spec:
containers:
- capabilities: {}
...
volumeMounts:
- mountPath: /var/lib/jenkins
name: '${JENKINS_SERVICE_NAME}-data'
...
volumes:
- name: '${JENKINS_SERVICE_NAME}-data'
persistentVolumeClaim:
claimName: '${JENKINS_SERVICE_NAME}'
如使用GlusterFS需要先修改Template，指定storageClassName：
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: '${JENKINS_SERVICE_NAME}'
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: '${VOLUME_CAPACITY}'
storageClassName: glusterfs-storage-block
下面使用jenkins-persistent来安装，命令如下：
$ oc project heroes
$ oc new-app jenkins-persistent -p VOLUME_CAPACITY=2Gi -p MEMORY_LIMIT=2Gi
--> Deploying template "openshift/jenkins-persistent" to project heroes
Jenkins
---------
Jenkins service, with persistent storage.
NOTE: You must have persistent volumes available in your cluster to use this template.
A Jenkins service has been created in your project. Log into Jenkins with your OpenShift account. The tutorial at https://github.com/openshift/origin/blob/master/examples/jenkins/README.md contains more information about using this template.
* With parameters:
* Jenkins Service Name=jenkins
* Jenkins JNLP Service Name=jenkins-jnlp
* Enable OAuth in Jenkins=true
* Memory Limit=2Gi
* Volume Capacity=2Gi
* Jenkins ImageStream Namespace=openshift
* Disable memory intensive administrative monitors=false
* Jenkins ImageStreamTag=jenkins:2
* Fatal Error Log File=false
--> Creating resources ...
route.route.openshift.io "jenkins" created
persistentvolumeclaim "jenkins" created
deploymentconfig.apps.openshift.io "jenkins" created
serviceaccount "jenkins" created
rolebinding.authorization.openshift.io "jenkins_edit" created
service "jenkins-jnlp" created
service "jenkins" created
--> Success
Access your application via route 'jenkins-heroes.apps.itrunner.org'
Run 'oc status' to view your app.
注意：如MEMORY_LIMIT配置低，则Jenkins Master节点架构为Linux (i386)。
查看存储
安装成功后，可以从安装Jenkins的node查看存储：
$ oc get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
jenkins-1-hw5q5 1/1 Running 0 5m 10.131.0.26 app2.itrunner.org <none>
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jenkins Bound pvc-bf3ff63d-6f1c-11e9-9dd9-02ef509f23d0 2Gi RWO glusterfs-storage 5m
# mount | grep pvc-bf3ff63d-6f1c-11e9-9dd9-02ef509f23d0
10.188.12.116:vol_0e157791c95b65a94011aed789d2037b on /var/lib/origin/openshift.local.volumes/pods/c12d7625-6f1c-11e9-ad9d-02499a450338/volumes/kubernetes.io~glusterfs/63d-6f1c-11e9-9dd9-02ef509f23d0 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
# cd /var/lib/origin/openshift.local.volumes/pods/c12d7625-6f1c-11e9-ad9d-02499a450338/volumes/kubernetes.io~glusterfs/63d-6f1c-11e9-9dd9-02ef509f23d0
扩展存储
当原配置的存储容量不满足需求时，可以扩展存储。
- StorageClass的属性allowVolumeExpansion必须设为true
$ oc edit sc/glusterfs-storage
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
creationTimestamp: 2019-04-28T02:58:06Z
name: glusterfs-storage
resourceVersion: "1903911"
selfLink: /apis/storage.k8s.io/v1/storageclasses/glusterfs-storage
uid: 723320e6-6961-11e9-b13d-02947d98b66e
parameters:
resturl: http://heketi-storage.app-storage.svc:8080
restuser: admin
secretName: heketi-storage-admin-secret
secretNamespace: app-storage
provisioner: kubernetes.io/glusterfs
reclaimPolicy: Delete
volumeBindingMode: Immediate
查看glusterfs-storage：
$ oc describe sc glusterfs-storage
Name: glusterfs-storage
IsDefaultClass: No
Annotations: <none>
Provisioner: kubernetes.io/glusterfs
Parameters: resturl=http://heketi-storage.app-storage.svc:8080,restuser=admin,secretName=heketi-storage-admin-secret,secretNamespace=app-storage
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
- 更新PVC spec.resources.requests.storage
$ oc edit pvc/jenkins
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
openshift.io/generated-by: OpenShiftNewApp
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs
creationTimestamp: 2019-05-05T10:01:26Z
finalizers:
- kubernetes.io/pvc-protection
labels:
app: jenkins-persistent
template: jenkins-persistent-template
name: jenkins
namespace: heroes
resourceVersion: "1904277"
selfLink: /api/v1/namespaces/heroes/persistentvolumeclaims/jenkins
uid: bf3ff63d-6f1c-11e9-9dd9-02ef509f23d0
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
storageClassName: glusterfs-storage
volumeName: pvc-bf3ff63d-6f1c-11e9-9dd9-02ef509f23d0
...
安装/更新插件
使用openshift用户登录jenkins。
进入系统管理 -> 插件管理，安装/更新插件。为了编译Angular需要安装NodeJS插件（另一种选择，可以使用node agent）。
配置全局工具
- JDK
- Maven
- NodeJS
首次调用时，Jenkins会自动安装全局工具，pipeline配置如下：
tools {
jdk 'jdk8'
maven 'maven-3.6'
}
tools {
nodejs 'nodejs-10.15'
}
创建配置文件
下面仍以部署Spring Boot和Angular工程为例介绍Pipeline的使用，开始之前请重建heroes project。
部署Spring Boot App
- 构建Builder Image
Jenkins负责编译、打包，因此Builder Image不再需要Maven，修改如下：
Dockerfile
# jdk8-centos7
FROM centos:latest
RUN yum -y update && yum clean all
# Set the labels that are used for OpenShift to describe the builder image.
LABEL maintainer="Sun Jingchuan <jason@163.com>" \
io.k8s.description="Oracle JDK 1.8.0_202 based on CentOS 7" \
io.k8s.display-name="Oracle JDK 1.8.0_202" \
io.openshift.expose-services="8080:http" \
io.openshift.tags="jdk8"
ENV JAVA_HOME=/usr/lib/jdk1.8.0_202 \
APP_ROOT=/opt/app-root
ENV PATH=${JAVA_HOME}/bin:${APP_ROOT}/bin:${PATH} HOME=${APP_ROOT}
COPY lib/jdk1.8.0_202 ${JAVA_HOME}
COPY bin ${APP_ROOT}/bin
RUN chmod -R u+x ${APP_ROOT}/bin && \
chgrp -R 0 ${APP_ROOT} && \
chmod -R g=u ${APP_ROOT} /etc/passwd
USER 10001
WORKDIR ${APP_ROOT}
ENTRYPOINT [ "uid_entrypoint" ]
EXPOSE 8080
编译、上传builder image
这次我们上传到OpenShift Docker Registry，注意push前要先执行docker login。
# docker build -t jdk8-centos7 .
# docker tag jdk8-centos7:latest docker-registry-default.apps.itrunner.org/heroes/jdk8-centos7:latest
# docker push docker-registry-default.apps.itrunner.org/heroes/jdk8-centos7:latest
- 创建App Image Dockerfile
App Image以builder image为基础，仅需拷贝Jenkins编译好的jar，内容如下：
Dockerfile.app
# heroes-api
FROM heroes/jdk8-centos7:latest
COPY heroes-api-1.0.0.jar $HOME
CMD java -jar $HOME/heroes-api-1.0.0.jar
- 从Dockerfile创建BuildConfig
$ cat Dockerfile.app | oc new-build -D - --name heroes-api
执行后生成BuildConfig的部分内容如下：
spec:
failedBuildsHistoryLimit: 5
nodeSelector: null
output:
to:
kind: ImageStreamTag
name: 'heroes-api:latest'
postCommit: {}
resources: {}
runPolicy: Serial
source:
dockerfile: "FROM heroes/jdk8-centos7:latest\r\n\r\nCOPY heroes-api-1.0.0.jar $HOME\r\n\r\nCMD java -jar $HOME/heroes-api-1.0.0.jar"
type: Dockerfile
strategy:
dockerStrategy:
from:
kind: ImageStreamTag
name: 'jdk8-centos7:latest'
type: Docker
会自启动build，但此时没有jar导致build失败，pipeline调用时才会传入jar。
- 创建Pipeline BuildConfig
pipeline.yml
apiVersion: v1
kind: BuildConfig
metadata:
name: heroes-api-pipeline
spec:
strategy:
jenkinsPipelineStrategy:
jenkinsfile: |-
pipeline {
agent any
tools {
jdk 'jdk8'
maven 'maven-3.6'
}
stages {
stage("Clone Source") {
steps {
checkout([$class: 'GitSCM',
branches: [[name: '*/master']],
extensions: [
[$class: 'RelativeTargetDirectory', relativeTargetDir: 'heroes-api']
],
userRemoteConfigs: [[url: 'https://github.com/sunjc/heroes-api.git']]
])
}
}
stage("Build Backend") {
steps {
dir('heroes-api') {
sh 'mvn clean package -Pdev -Dmaven.test.skip=true'
}
}
}
stage("Build Image") {
steps {
dir('heroes-api/target') {
sh 'oc start-build heroes-api --from-dir . --follow'
}
}
}
}
}
type: JenkinsPipeline
Pipeline可以内嵌在BuildConfig内，也可以引用git中的Jenkinsfile（推荐）:
apiVersion: v1
kind: BuildConfig
metadata:
name: heroes-api-pipeline
spec:
source:
git:
uri: "https://github.com/sunjc/heroes-api"
strategy:
jenkinsPipelineStrategy:
jenkinsfilePath: Jenkinsfile
type: JenkinsPipeline
创建pipeline：
$ oc create -f ./pipeline.yml
如果在项目中没有安装Jenkins，创建pipeline后将自动部署jenkins-ephemeral。
- 启动pipeline
$ oc start-build heroes-api-pipeline
也可以在jenkins或Application Console -> Builds -> Pipelines中启动。
- 部署App
$ oc new-app heroes-api
$ oc create route edge heroes-api --service heroes-api --hostname heroes.apps.itrunner.org --path /api --insecure-policy Redirect
部署Angular App
- 构建Builder Image
Dockerfile
# httpd-centos7
FROM centos/httpd:latest
RUN yum -y update && yum clean all
LABEL maintainer="Sun Jingchuan <jason@163.com>" \
io.k8s.description="Apache Httpd 2.4" \
io.k8s.display-name="Apache Httpd 2.4" \
io.openshift.expose-services="8080:http" \
io.openshift.tags="httpd"
ENV APP_ROOT=/opt/app-root
ENV PATH=${APP_ROOT}/bin:${PATH} HOME=${APP_ROOT} HTTPD_MAIN_CONF_PATH=/etc/httpd/conf
COPY bin ${APP_ROOT}/bin
COPY .s2i/bin/run ${APP_ROOT}/bin/run
RUN chmod -R u+x ${APP_ROOT}/bin && \
chgrp -R 0 ${APP_ROOT} && \
chmod -R g=u ${APP_ROOT} /etc/passwd /var/www/html /run/httpd && \
chown -R root:root /run/httpd /etc/httpd && \
sed -i -e "s/^User apache/User default/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
sed -i -e "s/^Group apache/Group root/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
sed -i -e "s/^Listen 80/Listen 8080/" ${HTTPD_MAIN_CONF_PATH}/httpd.conf && \
sed -ri " s!^(\s*CustomLog)\s+\S+!\1 |/usr/bin/cat!g; s!^(\s*ErrorLog)\s+\S+!\1 |/usr/bin/cat!g;" ${HTTPD_MAIN_CONF_PATH}/httpd.conf
USER 10001
WORKDIR ${APP_ROOT}
ENTRYPOINT [ "uid_entrypoint" ]
EXPOSE 8080
编译、上传Builder Image
# docker build -t httpd-centos7 .
# docker tag httpd-centos7:latest docker-registry-default.apps.itrunner.org/heroes/httpd-centos7:latest
# docker push docker-registry-default.apps.itrunner.org/heroes/httpd-centos7:latest
- 创建App Image Dockerfile
Dockerfile.app
# heroes-web
FROM heroes/httpd-centos7:latest
COPY heroes /var/www/html/heroes
CMD $HOME/bin/run
- 从Dockerfile创建BuildConfig
$ cat Dockerfile.app | oc new-build -D - --name heroes-web
执行后将启动build，此时没有传入内容导致build失败。
- 创建Pipeline BuildConfig
pipeline.yml
apiVersion: v1
kind: BuildConfig
metadata:
name: heroes-web-pipeline
spec:
source:
git:
uri: "https://github.com/sunjc/heroes-web"
strategy:
jenkinsPipelineStrategy:
jenkinsfilePath: Jenkinsfile
type: JenkinsPipeline
Jenkinsfile
pipeline {
agent any
tools {
nodejs 'nodejs-10.15'
}
stages {
stage("Clone Source") {
steps {
checkout([$class: 'GitSCM',
branches: [[name: '*/master']],
extensions: [
[$class: 'RelativeTargetDirectory', relativeTargetDir: 'heroes-web']
],
userRemoteConfigs: [[url: 'https://github.com/sunjc/heroes-web']]
])
}
}
stage("Build Angular") {
steps {
dir('heroes-web') {
sh 'npm install'
sh 'ng config -g cli.warnings.versionMismatch false'
sh 'ng build --prod --base-href=/heroes/'
}
}
}
stage("Build Image") {
steps {
dir('heroes-web/dist') {
sh 'oc start-build heroes-web --from-dir . --follow'
}
}
}
}
}
创建pipeline：
$ oc create -f ./pipeline.yml
- 启动pipeline
$ oc start-build heroes-web-pipeline
- 部署App
$ oc new-app heroes-web
$ oc create route edge heroes-web --service heroes-web --hostname heroes.apps.itrunner.org --path /heroes \
--insecure-policy Redirect --port 8080-tcp -n heroes
OpenShift DSL
OpenShift Jenkins image安装了OpenShift Jenkins Client插件，支持使用OpenShift Domain Specific Language(DSL)定义pipeline。
def templatePath = 'https://raw.githubusercontent.com/openshift/nodejs-ex/master/openshift/templates/nodejs-mongodb.json'
def templateName = 'nodejs-mongodb-example'
pipeline {
agent {
node {
label 'nodejs'
}
}
options {
timeout(time: 20, unit: 'MINUTES')
}
stages {
stage('preamble') {
steps {
script {
openshift.withCluster() {
openshift.withProject() {
echo "Using project: ${openshift.project()}"
}
}
}
}
}
stage('cleanup') {
steps {
script {
openshift.withCluster() {
openshift.withProject() {
openshift.selector("all", [ template : templateName ]).delete()
if (openshift.selector("secrets", templateName).exists()) {
openshift.selector("secrets", templateName).delete()
}
}
}
}
}
}
stage('create') {
steps {
script {
openshift.withCluster() {
openshift.withProject() {
openshift.newApp(templatePath)
}
}
}
}
}
stage('build') {
steps {
script {
openshift.withCluster() {
openshift.withProject() {
def builds = openshift.selector("bc", templateName).related('builds')
timeout(5) {
builds.untilEach(1) {
return (it.object().status.phase == "Complete")
}
}
}
}
}
}
}
stage('deploy') {
steps {
script {
openshift.withCluster() {
openshift.withProject() {
def rm = openshift.selector("dc", templateName).rollout().latest()
timeout(5) {
openshift.selector("dc", templateName).related('pods').untilEach(1) {
return (it.object().status.phase == "Running")
}
}
}
}
}
}
}
stage('tag') {
steps {
script {
openshift.withCluster() {
openshift.withProject() {
openshift.tag("${templateName}:latest", "${templateName}-staging:latest")
}
}
}
}
}
}
}
具体请参考OpenShift Pipeline Builds
清理对象
随着不断的构建、部署应用，build、deployment、image等对象会逐渐增多，管理员应定期清理不再需要的对象。
OpenShift提供了oc adm prune命令来清理对象，支持auth、groups、builds、deployments、images等类型。
$ oc adm prune <object_type> <options>
Pruning builds
$ oc adm prune builds [<options>]
| Option |
Description |
| --confirm |
Indicate that pruning should occur, instead of performing a dry-run |
| --orphans |
Prune all builds whose build config no longer exists, status is complete, failed, error, or canceled |
| --keep-complete=N |
Per build config, keep the last N builds whose status is complete. (default 5) |
| --keep-failed=N |
Per build config, keep the last N builds whose status is failed, error, or canceled (default 1) |
| --keep-younger-than=duration |
Do not prune any object that is younger than duration relative to the current time. (default 60m) |
$ oc adm prune builds --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m --confirm
Pruning deployments
$ oc adm prune deployments [<options>]
| Option |
Description |
| --confirm |
Indicate that pruning should occur, instead of performing a dry-run |
| --orphans |
Prune all deployments whose deployment config no longer exists, status is complete or failed, and replica count is zero |
| --keep-complete=N |
Per deployment config, keep the last N deployments whose status is complete and replica count is zero. (default 5) |
| --keep-failed=N |
Per deployment config, keep the last N deployments whose status is failed and replica count is zero. (default 1) |
| --keep-younger-than=duration |
Do not prune any object that is younger than duration relative to the current time. (default 60m) Valid units of measurement include nanoseconds (ns), microseconds (us), milliseconds (ms), seconds (s), minutes (m), and hours (h) |
$ oc adm prune deployments --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m --confirm
Pruning images
$ oc adm prune images [<options>]
system:admin用户不能执行清理image的操作，必须使用普通用户登录，且用户必须有system:image-pruner角色。
$ oc login https://openshift.itrunner.org:8443 --token=xxxx
| Option |
Description |
| --all |
Include images that were not pushed to the registry, but have been mirrored by pullthrough. This is on by default. To limit the pruning to images that were pushed to the integrated registry, pass --all=false |
| --certificate-authority |
The path to a certificate authority file to use when communicating with the OKD-managed registries. Defaults to the certificate authority data from the current user’s configuration file. If provided, secure connection will be initiated |
| --confirm |
Indicate that pruning should occur, instead of performing a dry-run. This requires a valid route to the integrated container image registry. If this command is run outside of the cluster network, the route needs to be provided using --registry-url |
| --force-insecure |
Use caution with this option. Allow an insecure connection to the Docker registry that is hosted via HTTP or has an invalid HTTPS certificate |
| --keep-tag-revisions=N |
For each image stream, keep up to at most N image revisions per tag. (default 3) |
| --keep-younger-than=duration |
Do not prune any image that is younger than duration relative to the current time. Do not prune any image that is referenced by any other object that is younger than duration relative to the current time. (default 60m) |
| --prune-over-size-limit |
Prune each image that exceeds the smallest limit defined in the same project. This flag cannot be combined with --keep-tag-revisions nor --keep-younger-than |
| --registry-url |
The address to use when contacting the registry. The command will attempt to use a cluster-internal URL determined from managed images and image streams. In case it fails (the registry cannot be resolved or reached), an alternative route that works needs to be provided using this flag. The registry host name may be prefixed by https:// or http:// which will enforce particular connection protocol |
| --prune-registry |
In conjunction with the conditions stipulated by the other options, this option controls whether the data in the registry corresponding to the OKD Image API Objects is pruned. By default, image pruning processes both the Image API Objects and corresponding data in the registry. This options is useful when you are only concerned with removing etcd content, possibly to reduce the number of image objects, but are not concerned with cleaning up registry storage; or intend to do that separately by Hard Pruning the Registry, possibly during an appropriate maintenance window for the registry |
使用--keep-younger-than清理image，不会清理以下情况的image：
- 创建时间在--keep-younger-than内的所有Pod
- 创建时间在--keep-younger-than内的所有ImageStream
- 正在运行的Pod
- 状态为pending的Pod
- 所有replication controllers
- 所有deployment configurations
- 所有build configurations
- 所有builds
使用--prune-over-size-limit清理超过指定Limit的image，不会清理以下情况的image：
- 正在运行的Pod
- 状态为pending的Pod
- 所有replication controllers
- 所有deployment configurations
- 所有build configurations
- 所有builds
示例：
$ oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm
$ oc adm prune images --prune-over-size-limit --confirm
清理image后不会更新registry cache，为了清理cache可以重新部署registry：
$ oc rollout latest dc/docker-registry -n default
docker prune
docker相关prune命令：
# docker container prune
# docker image prune
# docker volume prune
慎用，会删除openshift/origin-docker-builder、openshift/origin-deployer。
推荐文档
潘晓华Michael - OpenShift
参考文档
OKD Latest Documentation
source-to-image