install
openjdk
https://jdk.java.net/22/
$ cat jenkins.sysusers
u jenkins - "Jenkins CI" /var/lib/jenkins
g jenkins -
$ sudo systemd-sysusers $PWD/jenkins.sysusers
Creating group jenkins with gid 970.
Creating user jenkins (Jenkins CI) with uid 970 and gid 970.
$ sudo install -dm 750 -o jenkins -g jenkins /var/lib/jenkins
$ cat jenkins.tmpfiles
d /var/cache/jenkins 0755 jenkins jenkins -
$ sudo cp jenkins.tmpfiles /usr/lib/tmpfiles.d/jenkins.conf
$ sudo install -dm 755 -o jenkins -g jenkins /var/cache/jenkins
$ cat /etc/default/jenkins
JAVA=/opt/jdk-21.0.2/bin/java
JAVA_ARGS=-Xmx4096m
JAVA_OPTS=
JENKINS_USER=jenkins
JENKINS_HOME=/var/lib/jenkins
JENKINS_WAR=/usr/share/jenkins/jenkins.war
JENKINS_WEBROOT=--webroot=/var/cache/jenkins
JENKINS_PORT=--httpPort=8090
JENKINS_OPTS=
JENKINS_COMMAND_LINE="$JAVA $JAVA_ARGS $JAVA_OPTS -jar $JENKINS_WAR $JENKINS_WEBROOT $JENKINS_PORT $JENKINS_OPTS"
$ cat /lib/systemd/system/jenkins.service
[Unit]
Description=Extendable continuous integration server
After=network.target
[Service]
User=jenkins
Type=exec
EnvironmentFile=/etc/default/jenkins
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=jenkins
ExecStart=/bin/sh -c 'eval $JENKINS_COMMAND_LINE'
[Install]
WantedBy=multi-user.target
setup
登录 jenkins，搜索并安装 gitea, kubernetes 插件。
K8s 配置
K8s 集群配置私有 registry：
$ sudo vi /etc/rancher/k3s/registries.yaml
mirrors:
192.168.1.71:5000:
endpoint:
- "http://192.168.1.71:5000"
$ sudo systemctl restart k3s
K8s 集群创建 jenkins 用户（为了简单起见，创建具有管理员权限的用户）：
kubectl apply -f - <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
name: devops
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: devops
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: jenkins
namespace: devops
annotations:
kubernetes.io/service-account.name: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: jenkins
namespace: devops
EOF
添加 K8s 认证信息
$ kubectl get secrets -n devops jenkins -o jsonpath='{.data.token}' | base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6IjhOanRwejJlUE9EVkFQUzBSemhSbkRNbFRlM2FrVW9YMktpT2RiWFp2d2MifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZXZvcHMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiamVua2lucyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJqZW5raW5zIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiOWQ1MDUyMWQtNWU2Mi00OGUyLWI2YWItNWRiMDQwMjRlYzU4Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRldm9wczpqZW5raW5zIn0.SbBYb5XPgZaeEAWFdf1IqWVQ5eBGkgUBdjwn3UrO2b5296LOhMGaSfRY2fijdvhNkHXlIpw7j_DjFp2I_KXUm2jMsV5gNLXrQQ-CaUcFWHSDXbj7jTmaNnC2gdxTdR_uWNlLtgtiruMXOKdLEj_6oZ6YRThjbMstnFcICKjXcB56yvDV2iwaAs9ywJ6cr3gvxKqOUHR8oDwkJ728xPqDhT6dkAXPz48vN3aVzS4LRaIOnFqI2exWqYBAYJQsRvMukEs_Eicv-qk7yfGBkyxYRBqQjbG24Nt7BBiDxiYHbixI3uHi--RTPm4dJZ1WbMcTWMIyxnEw4vCnhpGsuYVtmA
添加 K8s 集群
Gitea 配置
添加 Gitea 认证信息
配置 Gita 服务器信息
Jenkins 工程配置
为每个 Gitea 的 Orgnization（也就是 group）创建 Organization Folder 类型的 Jenkins 工程
下图中的 Owner 是 Orgnization（也就是 group）名：
分支相关的配置：
git vs checkout vs multibranch
The
gitstep is a simplified shorthand for a subset of the more powerfulcheckoutstep. Thecheckoutstep is the preferred SCM checkout method. It provides significantly more functionality than thegitstep.
git
https://www.jenkins.io/doc/pipeline/steps/git/#git-git
The git plugin includes a multibranch provider for Jenkins Multibranch Pipelines and for Jenkins Organization Folders. The git plugin multibranch provider is a “base implementation” that uses command line git. Users should prefer the multibranch implementation for their git provider when one is available. Multibranch implementations for specific git providers can use REST API calls to improve the Jenkins experience and add additional capabilities. Multibranch implementations are available for GitHub, Bitbucket, GitLab, Gitea, and Tuleap.
Multibranch Pipelines
https://github.com/jenkinsci/git-plugin/tree/git-5.2.1?tab=readme-ov-file#multibranch-pipelines
The Multibranch Pipeline project type enables you to implement different Jenkinsfiles for different branches of the same project. In a Multibranch Pipeline project, Jenkins automatically discovers, manages and executes Pipelines for branches which contain a Jenkinsfile in source control.
Organization Folders enable Jenkins to monitor an entire GitHub Organization, Bitbucket Team/Project, GitLab organization, or Gitea organization and automatically create new Multibranch Pipelines for repositories which contain branches and pull requests containing a Jenkinsfile.
Branches and Pull Requests
https://www.jenkins.io/doc/book/pipeline/multibranch/
SCM API
https://plugins.jenkins.io/scm-api/
Jenkinsfile
podTemplate(
podRetention: always(),
activeDeadlineSeconds: 86400, // 1 day
containers: [
containerTemplate(
name: 'cross-builder',
image: '192.168.1.71:5000/ubuntu-23.04-cross-builder:latest',
ttyEnabled: true,
command: 'cat',
)
]
) {
node(POD_LABEL) {
// JENKINS-30600
stage('checkout') {
// scm is a GitSCM instance
checkout scm
}
container('cross-builder') {
stage('build') {
withEnv(['PATH+EXTRA=/opt/go/bin:/opt/node-v20.11.1-linux-arm64/bin']) {
sh '''#!/bin/bash
set -exo pipefail
git submodule update --init
export SASS_BINARY_PATH=/opt/node-sass.node
make xcubed-linux-amd64 ASSET=0
'''
}
}
}
}
}
podRetentionControls the behavior of keeping agent pods. Can be ’never()’, ‘onFailure()’, ‘always()’, or ‘default()’ - if empty will default to deleting the pod after activeDeadlineSeconds has passed.activeDeadlineSecondsIf podRetention is set to never() or onFailure(), the pod is deleted after this deadline is passed.
debug
K8s debug pod
$ kubectl describe po -n devops xcube-xcubed-pr-4-16-3rjqz-2hgzh-8xrh1
Name: xcube-xcubed-pr-4-16-3rjqz-2hgzh-8xrh1
Namespace: devops
Labels: jenkins=slave
jenkins/label=xcube_xcubed_PR-4_16-3rjqz
jenkins/label-digest=f6665f4e3eb5ee61be7285ba49664bfabfc93661
Containers:
cross-builder:
jnlp:
$ kubectl debug -it -n devops xcube-xcubed-pr-4-16-3rjqz-2hgzh-8xrh1 --target cross-builder --image 192.168.1.71:5000/cross-builder -- /bin/bash
$ kubectl exec -it -n devops xcube-xcubed-pr-4-16-3rjqz-2hgzh-8xrh1 -c cross-builder -- /bin/bash
debug log
org.csanchez.jenkins.plugins.kubernetes
org.jenkinsci.plugin.gitea
org.jenkinsci.plugins.durabletask.BourneShellScript
Viewing logs
https://www.jenkins.io/doc/book/system-administration/viewing-logs/
plugin build
$ mvn package -DskipTests
然后选择构建出来的 hpi 插件进行安装：
checkout step
checkout step 代码实现如下：
https://github.com/jenkinsci/workflow-scm-step-plugin/blob/workflow-scm-step-2.13/src/main/java/org/jenkinsci/plugins/workflow/steps/scm/GenericSCMStep.java
checkout 需要传递一个 SCM 实例。
例如 checkout scmGit() 实际上就是创建 GitSCM 实例：
https://github.com/jenkinsci/git-plugin/blob/git-5.2.1/src/main/java/hudson/plugins/git/GitSCM.java#L1641
而在 gitea 插件创建的 Organization Folder 项目（同样也是 Multibranch 项目）中，scm 实例由如下代码创建（无需显式创建）：
https://github.com/jenkinsci/workflow-multibranch-plugin/blob/workflow-multibranch-2.26.1/src/main/java/org/jenkinsci/plugins/workflow/multibranch/SCMVar.java#L58
https://github.com/jenkinsci/gitea-plugin/blob/gitea-1.3.0/src/main/java/org/jenkinsci/plugin/gitea/GiteaSCMBuilder.java#L258
If you’re in a multibranch pipeline environment, you can simply use the ‘scm’ variable, which is injected into the build by Jenkins
How do I obtain reference to my SCM object in Jenkins Pipeline?
https://stackoverflow.com/questions/38999442/how-do-i-obtain-reference-to-my-scm-object-in-jenkins-pipeline
Global Variable Reference
https://www.jenkins.io/doc/book/pipeline/getting-started/#global-variable-reference
container step
container step 代码实现如下：
https://github.com/jenkinsci/kubernetes-plugin/blob/4203.v1dd44f5b_1cf9/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java
主要逻辑是 doLaunch 通过 exec 在 pod 中创建 shell 进程，doExec 通过 stdin 将需要执行的命令注入到 shell 进程中运行。
如下日志是设置 org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS 为 false（默认值）时的日志：
Executing command: "nohup" "sh" "-c" "(cp '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-35aa4531/script.sh' '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-35aa4531/script.sh.copy'; { while [ -d '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-35aa4531' -a \! -f '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-35aa4531/jenkins-result.txt' ]; do touch '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-35aa4531/jenkins-log.txt'; sleep 3; done } & jsc=durable-48159556e8d1494972178539085aaf743734a1ee6ad4fcf2681cf1af11017b91; JENKINS_SERVER_COOKIE=\$jsc '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-35aa4531/script.sh.copy' > '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-35aa4531/jenkins-log.txt' 2>&1; echo \$? > '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-35aa4531/jenkins-result.txt.tmp'; mv '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-35aa4531/jenkins-result.txt.tmp' '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-35aa4531/jenkins-result.txt'; wait) >&- 2>&- &"
如下日志是设置 org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS 为 true 时的日志：
Executing command: "nohup" "sh" "-c" "cp '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-8d98a68b/script.sh' '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-8d98a68b/script.sh.copy'; { while [ -d '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-8d98a68b' -a \! -f '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-8d98a68b/jenkins-result.txt' ]; do touch '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-8d98a68b/jenkins-log.txt'; sleep 3; done } & jsc=durable-48159556e8d1494972178539085aaf743734a1ee6ad4fcf2681cf1af11017b91; JENKINS_SERVER_COOKIE=\$jsc '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-8d98a68b/script.sh.copy' > '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-8d98a68b/jenkins-log.txt' 2>&1; echo \$? > '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-8d98a68b/jenkins-result.txt.tmp'; mv '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-8d98a68b/jenkins-result.txt.tmp' '/home/jenkins/agent/workspace/xcube_xcubed_PR-4@tmp/durable-8d98a68b/jenkins-result.txt'; wait"
sh step
sh step 代码实现如下：
// https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/workflow-durable-task-step-2.40/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/ShellStep.java
@Override protected DurableTask task() {
return new BourneShellScript(script);
}
@Override public StepExecution start(StepContext context) throws Exception {
String path = context.get(EnvVars.class).get("PATH");
if (path != null && path.contains("$PATH")) {
context.get(TaskListener.class).getLogger().println("Warning: JENKINS-41339 probably bogus PATH=" + path + "; perhaps you meant to use ‘PATH+EXTRA=/something/bin’?");
}
return super.start(context);
}
// https://github.com/jenkinsci/durable-task-plugin/blob/durable-task-1.39/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java
private List<String> scriptLauncherCmd() {
String scriptPathCopy = scriptPath + ".copy"; // copy file to protect against "Text file busy", see JENKINS-70874
cmdString = String.format("cp '%s' '%s'; { while [ -d '%s' -a \\! -f '%s' ]; do touch '%s'; sleep 3; done } & jsc=%s; %s=$jsc %s '%s' > '%s' 2>&1; echo $? > '%s.tmp'; mv '%s.tmp' '%s'; wait",
scriptPath,
scriptPathCopy,
controlDir,
resultFile,
logFile,
cookieValue,
cookieVariable,
interpreter,
scriptPathCopy,
logFile,
resultFile, resultFile, resultFile);}
List<String> cmd = new ArrayList<>();
if (os != OsType.DARWIN) { // JENKINS-25848
cmd.add("nohup");
}
if (LAUNCH_DIAGNOSTICS) {
cmd.addAll(Arrays.asList("sh", "-c", cmdString));
} else {
// JENKINS-58290: launch in the background. Also close stdout/err so docker-exec and the like do not wait.
cmd.addAll(Arrays.asList("sh", "-c", "(" + cmdString + ") >&- 2>&- &"));
}
return cmd;
}
显然，上面 K8s 插件日志中记录的命令行由 sh step 生成（包括命令开始和结束的 nohup, &）。
此外，注意 shell 脚本执行的差异：
$ cat x.sh
#!/bin/bash -i
echo $-
$ sh ./x.sh
$ sh -c './x.sh'
$ sh -c '(./x.sh)'
前一条命令直接使用 /bin/sh 作为解释器运行，后两条命令使用 shebang 中定义的解释器运行。
设置 sh 插件属性：
How to add Java arguments to Jenkins?
https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-controllers/how-to-add-java-arguments-to-jenkins
tty
根据前面的日志，sh step 生成的命令行可以简化如下：
nohup sh -c "(tty) &"
在 LAUNCH_DIAGNOSTICS 为 true 时可以简化如下：
nohup sh -c "tty"
由于 nohup 的存在，所有的 tty 都会输出 not a tty，然而，即使去掉 nohup，sh -c "(tty) &" 输出也是 not a tty，只有去掉 nohup 同时 LAUNCH_DIAGNOSTICS 为 true 时才会输出关联的 tty。
最最重要的是，kubernetes-plugin 在 exec 时根本就没有调用 withTTY() 接口：
// https://github.com/jenkinsci/kubernetes-plugin/blob/4203.v1dd44f5b_1cf9/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L470
ExecWatch watch = getClient()
.pods()
.inNamespace(getNamespace())
.withName(getPodName())
.inContainer(containerName)
.redirectingInput(STDIN_BUFFER_SIZE) // JENKINS-50429
.writingOutput(stream)
.writingError(stream)
.usingListener(new ExecListener() {})
.exec(sh);
综上，sh step 里的 shell 脚本是没有关联 tty 的，所以如下的 jenkins 脚本：
container('cross-builder') {
stage('checkout') {
git url: 'http://git.xcube.com/xcube/xcubed.git', branch: 'master'
}
stage('build') {
sh '''#!/bin/bash -i
set -exo pipefail
'''
}
}
会出现警告：
bash: cannot set terminal process group (19): Inappropriate ioctl for device
bash: no job control in this shell
nohup breaks logname and tty commands
https://lists.gnu.org/archive/html/bug-coreutils/2008-02/msg00278.html
对于 Fedora, CentOS 系发行版而言，~/.bash_profile 默认会执行 ~/.bashrc：
$ cat ~/.bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
因此如果要加载 $PATH 环境变量，可以使用 #!/bin/bash -l。
但是 Ubuntu 没有类似的处理，而且会显式判断 PS1 变量：
$ cat ~/.bashrc
# If not running interactively, don't do anything
[ -z "$PS1" ] && return
因此对于基于 Ubuntu 的镜像，$PATH 环境变量的处理会比较麻烦，要么在容器镜像的 bash profile 中增加类似 Fedora 的处理，要么忍受 bash -i 的警告（不影响 bash 运行，bash 调用 initialize_job_control 时未处理返回值），要么在 pipeline 中显式增加 withEnv 处理：
container('cross-builder') {
stage('build') {
withEnv(['PATH+EXTRA=/opt/go/bin:/opt/rust/bin:/opt/node/bin']) {
sh '''#!/bin/bash
set -exo pipefail
echo $PATH
'''
}
}
}
注意 EXTRA 不是关键字，只是一个名字而已。
withEnv: Set environment variables
https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/#withenv-set-environment-variables
references
Setting Up a Jenkins Pipeline with Gitea for Packer Image Builds: A Step-by-Step Guide
https://tcude.net/setting-up-jenkins-pipeline-with-gitea-for-packer-image-builds-guide/
Kubernetes
https://plugins.jenkins.io/kubernetes/
The Kubernetes Plugin offers different methods to authenticate to a remote Kubernetes cluster:
- Secret Text Credentials: using a Service Account token
- Secret File Credentials: using a KUBECONFIG file
- Username/Password Credentials: using a username and password
- Certificate Credentials: using client certificates
Kubernetes Plugin: Authenticate with a ServiceAccount to a remote cluster
https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-controllers/kubernetes-plugin-authenitcation-to-a-remote-kubernetes-cluster
stage('build') {
sh '''#!/bin/bash -iexo pipefail
echo $PWD
make
'''
}
-o pipefail
If set, the return value of a pipeline is the value of the last (rightmost) command
to exit with a non-zero status, or zero if all commands in the pipeline exit
successfully. This option is disabled by default.
Builds do not source ~/.bashrc or ~/.bash_profile
https://www.reddit.com/r/jenkinsci/comments/djcp72/builds_do_not_source_bashrc_or_bash_profile/
Is the .bashrc executed in linux when #/bin/bash is included in a script?
https://stackoverflow.com/questions/39214152/is-the-bashrc-executed-in-linux-when-bin-bash-is-included-in-a-script
Bash Startup Files
https://www.gnu.org/software/bash/manual/bash.html#Bash-Startup-Files
最后修改于 2024-05-05