Jenkins

From Omnia
Jump to navigation Jump to search

Jenkins

Jenkins - http://jenkins-ci.org/

Jenkins monitors executions of repeated jobs, such as building a software project or jobs run by cron. Among those things, current Jenkins focuses on the following two jobs:

  • Building/testing software projects continuously, just like CruiseControl or DamageControl. In a nutshell, Jenkins provides an easy-to-use so-called continuous integration system, making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build. The automated, continuous build increases the productivity.
  • Monitoring executions of externally-run jobs, such as cron jobs and procmail jobs, even those that are run on a remote machine. For example, with cron, all you receive is regular e-mails that capture the output, and it is up to you to look at them diligently and notice when it broke. Jenkins keeps those outputs and makes it easy for you to notice when something is wrong.

Installation

Get latest: 

wget http://mirrors.jenkins-ci.org/war/latest/jenkins.war

Easy Start

Easy installation: Just java -jar jenkins.war, or deploy it in a servlet container. No additional install, no database. 

java -jar jenkins.war

Tomcat

Install tomcat6 to /opt/jenkins.

Jenkins WAR

Installation to ROOT: 

cd /opt/jenkins/webapps
## latest
# wget http://mirrors.jenkins-ci.org/war/latest/jenkins.war -O ROOT.war
## stable lts
wget http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war -O ROOT.war

Restart tomcat and the application should deploy to ROOT.

Startup

The easiest way to execute Jenkins is through the built in Winstone servlet container. You can execute Jenkins like this: 

$ java -jar jenkins.war

Of course, you probably want to send the output of Jenkins to a log file, and if you're on Unix, you probably want to use nohup: 

$ nohup java -jar jenkins.war > $LOGFILE 2>&1

Command Line Parameter:

  • --httpPort=$HTTP_PORT - Runs Jenkins listener on port $HTTP_PORT using standard http protocol. The default is port 8080. To disable (because you're using https), use port -1.

Source: Starting and Accessing Jenkins - Jenkins - Jenkins Wiki - https://wiki.jenkins-ci.org/display/JENKINS/Starting+and+Accessing+Jenkins

Configuration

Jenkins configuration files are stored in ~/.jenkins 

rm -rf ~/.jenkins

Jenkins configuration manager: 

http://[SERVER:PORT]/configure

JENKINS_HOME directory

JENKINS_HOME directory - Administering Jenkins - https://wiki.jenkins-ci.org/display/JENKINS/Administering+Jenkins 

JENKINS_HOME
 +- config.xml     (jenkins root configuration)
 +- *.xml          (other site-wide configuration files)
 +- userContent    (files in this directory will be served under your http://server/userContent/)
 +- fingerprints   (stores fingerprint records)
 +- plugins        (stores plugins)
 +- jobs
     +- [JOBNAME]      (sub directory for each job)
         +- config.xml     (job configuration file)
         +- workspace      (working directory for the version control system)
         +- latest         (symbolic link to the last successful build)
         +- builds
             +- [BUILD_ID]     (for each build)
                 +- build.xml      (build result summary)
                 +- log            (log file)
                 +- changelog.xml  (change log

LDAP

Jenkins configuration manager: 

http://[SERVER:PORT]/configure
  • Check "Enable security" - note: looks like this has moved to "Configure Global Security"
  • Access Control - Security Realm - LDAP
    • Server: ldap.oeey.com
    • root DN: dc=oeey,dc=com
    • User search filter: uid={0}
  • Authorization
    • Anyone can do anything

Verify you can login, then change the Authorization:

  • Check "Enable security"
  • Authorization
    • Logged-in users can do anything

Or really locked down:

  • Check "Enable security"
  • Authorization
    • Matrix-based security # really locked down
    • (OR) Project-based Matrix Authorization Strategy # really locked down

NOTE: Add user/group 'authenticated' to select any authenticated user for permissions.

Set Unstable on Exit Code

Modern Jenkins versions (since 2.26, October 2016) solved this: it's just an advanced option for the Execute shell build step!

2ZK2N.png

You can just choose and set an arbitrary exit value; if it matches, the build will be unstable. Just pick a value which is unlikely to be launched by a real process in your build.

ref: https://stackoverflow.com/questions/8148122/how-to-mark-a-build-unstable-in-jenkins-when-running-shell-scripts

Plugins

Enable:

  • -

Add:

  • "Python" - Adds the ability to execute python scripts as build steps. Other than that, this plugin works pretty much like the standard shell script support. - https://plugins.jenkins.io/python/
  • "Node and Label parameter" - The plugin can configure additional parameters for a job. These new parameter types are 'Node' and 'Label'. This is specially useful if you want to execute the job on different nodes without changing the configuration. - https://plugins.jenkins.io/nodelabelparameter/

---


Suggested Add:

Enable:

Disable:

Don't disable: (or cause failure)

  • JUnit Plugin

Other useful plugins:

  • JobConfigHistory Plugin - Job history plugin for hudson.
  • Parameterized Trigger Plugin
  • JIRA Plugin - This plugin integrates Jenkins to Atlassian JIRA.
  • Config AutoRefresh Plugin - The Config AutoRefresh Plugin provides a way to configure the auto-refresh rate from the Jenkins UI.
  • Build Pipeline Plugin - Add a new view plugin to hudson
  • NodeLabel Parameter Plugin
  • Email-ext plugin - This plugin is a replacement for Jenkins's email publisher
  • OfflineOnFailure - This plugin allows you to take nodes offline immediately when a job reports FAILURE.
  • Jenkins Mercurial plugin - This plugin integrates Mercurial SCM with Hudson. It includes repository browsing support for hg serve/hgweb, Google Code, and Bitbucket. Features include guaranteed clean builds, named branch support, Forest extension support, module lists, Mercurial tool installation, and automatic caching.
  • thinBackup - Backups the most important global and job specific configuration files.
  • Backup plugin - Backup or restore your Hudson configuration files
  • Show Build Parameters plugin - This plugin shows the parameter values on the main build page
  • Jenkins Throttle Concurrent Builds Plug-in - This plugin allows for throttling the number of concurrent builds of a project running per node or globally.

Descriptions:

  • Build-timeout Plugin - This plugin allows builds to be automatically terminated after the specified amount of time has elapsed.
  • Python Plugin - Adds the ability to execute python scripts as build steps.
  • Nested View Plugin - View type to allow grouping job views into multiple levels instead of one big list of tabs.

Restart Jenkins

Jenkins can be restarted, without having to restart tomcat from the Plugin UpdateCenter: 

https://jenkins-server/updateCenter/

Or even easier: 

https://jenkins-server/restart

Build Triggers - Build Periodically

	
This field follows the syntax of cron (with minor differences). Specifically, each line consists of 5 fields separated by TAB or whitespace:
MINUTE HOUR DOM MONTH DOW
MINUTE	Minutes within the hour (0-59)
HOUR	The hour of the day (0-23)
DOM	The day of the month (1-31)
MONTH	The month (1-12)
DOW	The day of the week (0-7) where 0 and 7 are Sunday.
To specify multiple values for one field, the following operators are available. In the order of precedence,

'*' can be used to specify all valid values.
'M-N' can be used to specify a range, such as "1-5"
'M-N/X' or '*/X' can be used to specify skips of X's value through the range, such as "*/15" in the MINUTE field for "0,15,30,45" and "1-6/2" for "1,3,5"
'A,B,...,Z' can be used to specify multiple values, such as "0,30" or "1,3,5"
Empty lines and lines that start with '#' will be ignored as comments.

In addition, '@yearly', '@annually', '@monthly', '@weekly', '@daily', '@midnight', and '@hourly' are supported.

Examples	
# every minute
* * * * *
# every 5 mins past the hour 
5 * * * *

Trigger Build by URL

Builds in Jenkins can be triggered periodically (on a schedule, specified in configuration), or when source changes in the project have been detected, or they can be automatically triggered by requesting the URL: 

http://YOURHOST/jenkins/job/PROJECTNAME/build

Source: https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-Configuringautomaticbuilds

Builds by e-mail (sendmail)

If you have the root account of your system and you are using sendmail, I found it the easiest to tweak /etc/aliases and add the following entry: 

jenkins-foo: "|/bin/wget -o /dev/null http://YOURHOST/jenkins/job/PROJECTNAME/build"

and then run "newaliases" command to let sendmail know of the change. Whenever someone sends an e-mail to "jenkins-foo@yoursystem", this will trigger a new build. See this for more details about configuring sendmail.

Source: https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-Buildsbyemail%28sendmail%29


User Content

Location: 

/root/.jenkins/userContent

readme.txt: 

Files in this directory will be served under your http://server/hudson/userContent/

Fake Fail Node

fake-fail 

#!/bin/bash

for i in `seq 1 10` ; do
  echo "FAKE MANAGER - FORCE FAILURE"
done
exit 1

Shell: (medium)

  • Just change /bin/sh to point to the fake-fail script (this can startup failures however)
[esx-fail] $ /bin/sh -xe /jenkins/tmp/hudson4954560349301610012.sh
  • Maybe the best option, sh wrapper:

sh wrapper: 

#!/bin/bash

#echo "Executing /bin/sh Wrapper..."
if [ "$USER" = "fio" ] ; then
  /home/fio/bin/fake-fail
else
  /bin/bash "$@"
fi

Python: (easy!)

  • Just change the path to have 'python' link to the fake-fail script.
[esx-fail] $ python /jenkins/tmp/hudson8514675783666558740.py

Change Shell

Go to:

  • Configure System > Shell > Shell executable

Does not appear to be a way to change this per node.


If you start your job with #!/bin/bash Jenkins will use that instead of the default (plus -xe).

References:

init script

#!/bin/sh

# chkconfig: 2345 90 10
# description: Jenkins Server

# Source function library.
. /etc/init.d/functions

NAME=jenkins
PIDFILE=/var/run/$NAME.pid
LOGFILE=/opt/jenkins/jenkins.log
COMMAND=/opt/java/bin/java
COMMAND_ARGS="-jar /opt/jenkins/jenkins.war --httpPort=-1 --httpsPort=443 --httpsKeyStore=/opt/jenkins/keystore --httpsKeyStorePassword=oeey"

export JENKINS_HOME=/data/jenkins

start() {
    if [ -e $PIDFILE ] ; then
        echo -n "Already running!"
        failure
        return 1
    fi
    nohup $COMMAND $COMMAND_ARGS > $LOGFILE 2>&1 &
    COMMAND_PID=$!
    kill -0 $COMMAND_PID
    if [ $? -eq 0 ] ; then
        echo $COMMAND_PID > $PIDFILE
        success
    else
        failure
    fi
}

stop() {
    if [ -e $PIDFILE ] ; then
        kill $( cat $PIDFILE )
        if [ $? -ne 0 ] ; then
            echo -n "Unable to kill!"
            failure
            return 1
        fi
        rm $PIDFILE
        success
    else
        echo -n "Not running!"
        failure
    fi
}

case $1 in
    start)
        echo -n "Starting $NAME: "
        start
        echo ""
    ;;
    stop)
        echo -n "Stopping $NAME: "
        stop
        echo ""
    ;;
    restart)
        echo -n "Restarting $NAME: "
        stop
        sleep 1
        start
        echo ""
    ;;
    *)
        echo "usage: $NAME {start|stop|restart}"
        exit 1
    ;;
esac

exit 0

Modified from Source: https://wiki.jenkins-ci.org/display/JENKINS/Starting+and+Accessing+Jenkins

Reverse Proxy

Test: 

curl -iL http://your.reverse.proxy/jenkins/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test

Apache Reverse Proxy: 

NameVirtualHost *:80
<VirtualHost *:80>
    ServerName jenkins.myserver.com

    ProxyPass         /  http://localhost:8080/ nocanon
    ProxyPassReverse  /  http://localhost:8080/
    ProxyRequests     Off
    AllowEncodedSlashes NoDecode
   
    # Local reverse proxy authorization override
    # Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
    <Proxy http://localhost:8080/jenkins*>
        Order deny,allow
        Allow from all
    </Proxy>

    ServerAdmin admin@jenkins.myserver.com
    ErrorLog logs/jenkins.myserver.com-error_log
    CustomLog logs/jenkins.myserver.com-access_log common
</VirtualHost>

References:

Proxy Pass

Similar to forwarding, but takes care of redirects: 

  ProxyPass         /mirror/foo/ http://foo.com/
  ProxyPassReverse  /mirror/foo/ http://foo.com/

Suppose the local server has address http://wibble.org/; then 

  ProxyPass         /mirror/foo/ http://foo.com/
  ProxyPassReverse  /mirror/foo/ http://foo.com/

will not only cause a local request for the <http://wibble.org/mirror/foo/bar> to be internally converted into a proxy request to <http://foo.com/bar> (the functionality ProxyPass provides here). It also takes care of redirects the server foo.com sends: when http://foo.com/bar is redirected by him to http://foo.com/quux Apache adjusts this to http://wibble.org/mirror/foo/quux before forwarding the HTTP redirect response to the client. 

<VirtualHost *:80>
    ServerName wiki.oeey.com

    #Forwarding Traffic
    ProxyPass         / http://localhost:8080/
    ProxyPassReverse  / http://localhost:8080/

    ServerAdmin website@oeey.com
    ErrorLog logs/wiki-error_log
    CustomLog logs/wiki-access_log common
</VirtualHost>

Proxy Pass to Jenkins: [1] 

NameVirtualHost *:80
<VirtualHost *:80>
    ServerName jenkins.oeey.com

    #Forwarding Traffic
    #RewriteEngine on
    #RewriteRule ^(/.*)  http://localhost:8080/$1   [P]

    ProxyPass         /  http://localhost:8080/ nocanon
    ProxyPassReverse  /  http://localhost:8080/
    ProxyRequests     Off
    ProxyPreserveHost On
    AllowEncodedSlashes NoDecode

    # Local reverse proxy authorization override
    # Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
    <Proxy http://localhost:8080/*>
        Order deny,allow
        Allow from all
    </Proxy>

    ServerAdmin admin@oeey.com
    ErrorLog logs/jenkins.oeey.com-error_log
    CustomLog logs/jenkins.oeey.com-access_log common
</VirtualHost>

Proxy Pass HTTPS

Proxy Pass to Jenkins: 

NameVirtualHost *:443
<VirtualHost *:443>
    ServerName jenkins.oeey.com

    # Enable SSL Support
    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/localhost.crt
    SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

    # Forwarding Traffic
    # RewriteEngine on
    # RewriteRule ^(/.*)  http://localhost:8080/$1   [P]

    ProxyPass         /  http://localhost:8080/ nocanon
    ProxyPassReverse  /  http://localhost:8080/
    ProxyPassReverse  /  http://jenkins.oeey.com/
    ProxyRequests     Off
    AllowEncodedSlashes NoDecode

    # ProxyPreserveHost On    # this doesn't seem to be required?
    # RequestHeader set X-Forwarded-Proto "http"   # should this be https?

    # https://stackoverflow.com/questions/42267602/why-does-jenkins-say-my-reverse-proxy-setup-is-broken
    # avoid the "WARNING h.d.ReverseProxySetupMonitor#getTestForReverseProxySetup: http://... vs. https://..."
    # avoid the "it appears that your reverse proxy set up is broken"
    RequestHeader set X-Forwarded-Proto https

    # NOTE: Only needed if your configuration denies this by default...
    # Local reverse proxy authorization override
    # Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
    #<Proxy http://localhost:8080/jenkins*>
    #    Order deny,allow
    #    Allow from all
    #</Proxy>

    ServerAdmin admin@oeey.com
    ErrorLog logs/jenkins.oeey.com-error_log
    CustomLog logs/jenkins.oeey.com-access_log common
</VirtualHost>

Optimizations

7 Ways to Optimize Jenkins/Hudson White Paper

  1. Back Up and Restore - Just Do It!
  2. Plan for Disk Usage Growth Up Front - Prepare for Disk Usage Growth
  3. Use Native Packages - Use OS-specific installation packages over the default war-based installation
  4. Take Advantage of Distributed Builds - Do Distributed Builds with Nodes
  5. Use Labels - Labels are simply tags you can assign to nodes to describe their capacities
  6. Use a Memorable URL for Jenkins - Invest in an Easy-to-Remember URL
  7. Prevent Build Record Sprawl - Keep only a subset of build records around with "Discard Old Builds"

Source: 7 Ways to Optimize Jenkins/Hudson White Paper http://di388e0fcqllf.cloudfront.net/whitepapers/7WaysToOptimizeJenkins.pdf

Seven Habits of Highly Effective Jenkins Users

  1. HABIT 1: MAKE YOUR MASTER STABLE AND RESTORABLE
  2. HABIT 2: BREAK UP THE BLOAT
    • Multiple Masters - split up masters by team, function, access, etc.
    • Break up your jobs - modularization and reuse are good techniques
    • Tools for breaking up your jobs: parameterized trigger + conditional build step, copy artifact, promoted builds, build pipeline plugin, workflow plugin
  3. HABIT 3: AUTOMATE JENKINS TASKS!
    • Script console and Scriptler
    • Generate jobs programmatically (Jenkins REST API and CLI)
  4. HABIT 4: TEND YOUR PLUGIN GARDEN
    • Do you really need that plugin?
    • Disable plugins you don't use
    • Plugins can cause instability
    • Clean up old plugins and their data
    • My Essential plugins:
      • Job Config History
      • Static analysis plugins
      • xUnit
      • Parameterized Trigger and Conditional Build Step
      • Tool Environment
      • EnvInject
      • Rebuild
      • Build Timeout
  5. HABIT 5: INTEGRATE WITH OTHER TOOLS AND SERVICES
    • REST API
    • Source Control
    • Jira
    • Artifactory
  6. HABIT 6: MAKE YOUR SLAVES FUNGIBLE
    • "Fungibility is the property of a good or a commodity whose individual units are capable of mutual substitution"
    • A fungible slave is a slave you can replace easily with another slave
    • Make slaves identical (replication, puppet/chef/ansible, etc)
    • Make slaves reusable and flexible (general purpose)
    • Use the cloud
  7. HABIT 7: JOIN THE COMMUNITY
    • Write plugins, open JIRAs, fix bugs, get on mailing lists, help others

Source: Seven Habits of Highly Effective Jenkins Users (2014 edition!) - http://www.slideshare.net/andrewbayer/seven-habits-of-highly-effective-jenkins-users-2014-edition

Startup Options

[root@jenkins jenkins]# java -jar /opt/jenkins/jenkins.war --help
Running from: /opt/jenkins/jenkins.war
webroot: $user.home/.jenkins
Jenkins Continuous Integration Engine 1.452
Usage: java -jar jenkins.war [--option=value] [--option=value]

Options:
   --daemon                 = fork into background and run as daemon (Unix only)
   --config                 = load configuration properties from here. Default is ./winstone.properties
   --prefix                 = add this prefix to all URLs (eg http://localhost:8080/prefix/resource). Default is none
   --commonLibFolder        = folder for additional jar files. Default is ./lib

   --logfile                = redirect log messages to this file
   --logThrowingLineNo      = show the line no that logged the message (slow). Default is false
   --logThrowingThread      = show the thread that logged the message. Default is false
   --debug                  = set the level of debug msgs (1-9). Default is 5 (INFO level)

   --httpPort               = set the http listening port. -1 to disable, Default is 8080
   --httpListenAddress      = set the http listening address. Default is all interfaces
   --httpDoHostnameLookups  = enable host name lookups on incoming http connections (true/false). Default is false
   --httpsPort              = set the https listening port. -1 to disable, Default is disabled
                              if neither --httpsCertificate nor --httpsKeyStore are specified,
                              https is run with one-time self-signed certificate.
   --httpsListenAddress     = set the https listening address. Default is all interfaces
   --httpsDoHostnameLookups = enable host name lookups on incoming https connections (true/false). Default is false
   --httpsKeyStore          = the location of the SSL KeyStore file.
   --httpsKeyStorePassword  = the password for the SSL KeyStore file. Default is null
   --httpsCertificate       = the location of the PEM-encoded SSL certificate file.
                              (the one that starts with '-----BEGIN CERTIFICATE-----')
                              must be used with --httpsPrivateKey.
   --httpsPrivateKey        = the location of the PEM-encoded SSL private key.
                              (the one that starts with '-----BEGIN RSA PRIVATE KEY-----')
   --httpsKeyManagerType    = the SSL KeyManagerFactory type (eg SunX509, IbmX509). Default is SunX509
   --ajp13Port              = set the ajp13 listening port. -1 to disable, Default is 8009
   --ajp13ListenAddress     = set the ajp13 listening address. Default is all interfaces
   --controlPort            = set the shutdown/control port. -1 to disable, Default disabled

   --handlerCountStartup    = set the no of worker threads to spawn at startup. Default is 5
   --handlerCountMax        = set the max no of worker threads to allow. Default is 300
   --handlerCountMaxIdle    = set the max no of idle worker threads to allow. Default is 50

   --simulateModUniqueId    = simulate the apache mod_unique_id function. Default is false
   --useSavedSessions       = enables session persistence (true/false). Default is false
   --mimeTypes=ARG          = define additional MIME type mappings. ARG would be EXT=MIMETYPE:EXT=MIMETYPE:...
                              (e.g., xls=application/vnd.ms-excel:wmf=application/x-msmetafile)
   --maxParamCount=N        = set the max number of parameters allowed in a form submission to protect
                              against hash DoS attack (oCERT #2011-003). Default is 10000.
   --usage / --help         = show this message
   --version                = show the version and quit

Security options:
   --realmClassName               = Set the realm class to use for user authentication. Defaults to ArgumentsRealm class

   --argumentsRealm.passwd.<user> = Password for user <user>. Only valid for the ArgumentsRealm realm class
   --argumentsRealm.roles.<user>  = Roles for user <user> (comma separated). Only valid for the ArgumentsRealm realm class

   --fileRealm.configFile         = File containing users/passwds/roles. Only valid for the FileRealm realm class

Access logging:
   --accessLoggerClassName        = Set the access logger class to use for user authentication. Defaults to disabled
   --simpleAccessLogger.format    = The log format to use. Supports combined/common/resin/custom (SimpleAccessLogger only)
   --simpleAccessLogger.file      = The location pattern for the log file(SimpleAccessLogger only)

Add Color to Console Output

AnsiColor plugins gives you opportunity to color monochromatic Jenkins console output.

1. Install AnsiColor plugin On Jenkins: Manage Jenkins > Manage Plugins > Available > search and install 'Ansi Color'

2. Configure your build/job Under Build Environment section check Color ANSI Console Output and select xterm ref: https://blog.mphomphego.co.za/blog/2017/04/13/jenkins-add-color-to-console-output.html 

set +x
info() {
echo "\033[1;33m[Info]    \033[0m $1"
}


error() {
echo "\033[1;31m[Error]   \033[0m $1"
}

success() {
echo "\033[1;32m[Success] \033[0m $1"
}

info "This is information message"
error "Houston we have a problem"
success "Great!!!"

echo "Foreground colors"
echo "\033[31m Red \033[0m"
echo "\033[32m Green \033[0m"
echo "\033[33m Yellow \033[0m"
echo "\033[34m Blue \033[0m"
echo "\033[35m Magneta \033[0m"
echo "\033[36m Cyan \033[0m"
echo "Background colors"
echo "\033[41m Red \033[0m"
echo "\033[42m Green \033[0m"
echo "\033[43m Yellow \033[0m"
echo "\033[44m Blue \033[0m"
echo "\033[45m Magneta \033[0m"
echo "\033[46m Cyan \033[0m"
echo "Different combinations"
echo "\033[1;31m Red \033[0m"
echo "\033[1;4;37;42m Green \033[0m"
echo "\033[1;43m Yellow \033[0m"
set -x

API

Jenkins CLI

Jenkins CLI - Jenkins - Jenkins Wiki - https://wiki.jenkins.io/display/JENKINS/Jenkins+CLI

Download: 

https://jenkins.example.com/jnlpJars/jenkins-cli.jar

Sample (Get Node Details): 

java -jar jenkins-cli.jar -http -auth myuser:mypassword -s https://jenkins.oeey.com:8443 get-node my-node-01

Create Node

https://jenkins.oeey.com/cli/command/create-node

Python

See Python/Jenkins

"Python Plugin" - Adds the ability to execute python scripts as build steps. - https://wiki.jenkins-ci.org/display/JENKINS/Python+Plugin

Add node environment variable (node properties): 

name=PYTHONUNBUFFERED
value=1

Jenkins Python API

Python API - http://python-jenkins.readthedocs.io/en/latest/api.html 

pip install python-jenkins

http://python-jenkins.readthedocs.io/en/latest/ 

"""  Jenkins Python API Utilities."""
import argparse
import datetime
import glob
import json
import multiprocessing
import os
import pdb
import re
import subprocess
import sys
import tempfile
import time
import urllib3
from copy import deepcopy
from operator import itemgetter

import jenkins
from jira import JIRA

urllib3.disable_warnings()


# Jenkins Python API Utilities.
if __name__ == '__main__':

   # Get user parameters.
    parser = argparse.ArgumentParser(description="Jenkins")
    parser.add_argument('--jenkins-user', metavar='USER', type=str, default="jenkins",
                        help='The Jenkins user ID.')
    parser.add_argument('--jenkins-password', metavar='PASSWORD', type=str, default="password",
                        help='The Jenkins password.')
    parser.add_argument('--jenkins-url', metavar='URL', type=str, default="https://jenkins.oeey.com:8443",
                        help='The Jenkins URL.')
    args = parser.parse_args()

   # Set up jenkins.
    jenkins_server = jenkins.Jenkins(args.jenkins_url, username=args.jenkins_user, password=args.jenkins_password)

   # Delete server nodes.
    nodes = jenkins_server.get_nodes()
    for node in nodes:
        #print node
        # delete nodes with "old-" in name
        if "old-" in node["name"]:
            print "Deleting node:  {}".format(node["name"])
            jenkins_server.delete_node(node["name"])

Hello

import jenkins

SERVER = "https://jenkins.oeey.com:8443/"
USERNMAE = "username"
PASSWORD = "password"

server = jenkins.Jenkins(SERVER, username=USERNMAE, password=PASSWORD)
user = server.get_whoami()
version = server.get_version()
print('Hello %s from Jenkins %s' % (user['fullName'], version))

ref: [2]

Node Info

import jenkins

SERVER = "https://jenkins.oeey.com:8443/"
USERNMAE = "username"
PASSWORD = "password"

server = jenkins.Jenkins(SERVER, username=USERNMAE, password=PASSWORD)

node_info = server.get_node_info('SOME-NODE-NAME')
print node_info

Node Config

import jenkins

SERVER = "https://jenkins.oeey.com:8443/"
USERNMAE = "username"
PASSWORD = "password"

server = jenkins.Jenkins(SERVER, username=USERNMAE, password=PASSWORD)

node_config = server.get_node_config('SOME-NODE-NAME')
print node_config

Disable Node

import jenkins

SERVER = "https://jenkins.oeey.com:8443/"
USERNMAE = "username"
PASSWORD = "password"

server = jenkins.Jenkins(SERVER, username=USERNMAE, password=PASSWORD)

server.disable_node('SOME-NODE-NAME')
#server.enable_node('SOME-NODE-NAME')

Delete Node

import jenkins

SERVER = "https://jenkins.oeey.com:8443/"
USERNMAE = "username"
PASSWORD = "password"
NODE = 'SOME-NODE-NAME'

server = jenkins.Jenkins(SERVER, username=USERNMAE, password=PASSWORD)
server.delete_node(NODE)

Create Node

import jenkins

SERVER = "https://jenkins.oeey.com:8443/"
USERNMAE = "username"
PASSWORD = "password"
NODE = "SOME-NEW-NODE"
NODE_CREDENTIALS = 'd03e3772-783c-4749-ff93-da268e0fa9ff'
# Note: get node credentials from another existing node
#       that uses same credentials, with get_node_config()

server = jenkins.Jenkins(SERVER, username=USERNMAE, password=PASSWORD)

print "Creating Node: {}".format(NODE)

params = {
    'host': NODE,
    'port': '22',
    'credentialsId': NODE_CREDENTIALS,
    'sshHostKeyVerificationStrategy': {
        'stapler-class': 'hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy',
        'requireInitialManualTrust': False
    }
}
server.create_node(
    name=NODE,
    numExecutors=1,
    nodeDescription='Some description',
    remoteFS='/j',
    labels='some labels',
    exclusive=False,
    launcher=jenkins.LAUNCHER_SSH,
    launcher_params=params)

# Disabled new node
server.disable_node(NODE)

Update Node

import jenkins
import sys
import re

jserver = jenkins.Jenkins('https://jenkins.oeey.com', username='jenkins', password='PASSWORD')

node_name = 'MYNODE'

node_config = jserver.get_node_config(node_name)

match = re.findall('<numExecutors>.*</numExecutors>', node_config)[0]

new_node_config = node_config.replace(match, '<numExecutors>6</numExecutors>')

jserver.reconfig_node(node_name, new_node_config)

<?xml version="1.0" encoding="UTF-8"?>
<slave>
  <name>MYNODE</name>
  <description>NODE_DESCRIPTION</description>
  <remoteFS>/j</remoteFS>
  <numExecutors>3</numExecutors>
  <mode>EXCLUSIVE</mode>
  <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
  <launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.21">
    <host>MYNODE.OEEY.COM</host>
    <port>22</port>
    <credentialsId>XX8416c5-6aac-4834-9c43-f478869f7af2</credentialsId>
    <maxNumRetries>0</maxNumRetries>
    <retryWaitTime>0</retryWaitTime>
    <sshHostKeyVerificationStrategy class="hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy"/>
  </launcher>
  <label>MY_LABELS</label>
  <nodeProperties/>
</slave>

Groovy Script

Run from /script 

https://jenkins.server.com/script

Kill long running jobs

int MAX_ALLOWED_DURATION_IN_SECONDS = 60 * 60 * 6 // 24 hours

url = "http://jenkins.oeey.com"
 
def busyExecutors = Jenkins.instance.computers
                   .collect {
                     c -> c.executors.findAll { it.isBusy() }
                   }
                   .flatten() // reminder: transforms list(list(executor)) into list(executor)

def ok = true
 
println "Busy Executors list"
busyExecutors.each { e ->
  int durationInSeconds = (System.currentTimeMillis() - e.executable.getTimeInMillis())/1000.0
   
  if(durationInSeconds > MAX_ALLOWED_DURATION_IN_SECONDS )
  {
    println e ;
    nodeName = e.getOwner().nodeName
    println "\t $url/computer/$nodeName"
    durationInHours = durationInSeconds / 60 / 60
    println "\t duration (hours) = $durationInHours"
    println ""
    //e.doStop()
    ok = false;
  }
}
 
println "Done"
 
return ok

Doc:

Ref: Find builds currently running that has been executing for more than N seconds - Jenkins - Jenkins Wiki - https://wiki.jenkins.io/display/JENKINS/Find+builds+currently+running+that+has+been+executing+for+more+than+N+seconds

Shorten workspace name

A job typically runs at: 

C:\jenkins\workspace\[JOB]\[LABEL_TITLE]\[LABEL]

Windows has a 260 character path limit. To help with this, make the following changes:

  1. Remote root directory to "c:\j" instead of "c:\jenkins" (node settings)
  2. Change "workspace" to "w" (server setting that affects slaves -Dhudson.model.Slave.workspaceRoot=ws)
  3. Use shorter job names like "t1" (job's "project name")
  4. Use shorter label expression title name, like "l" (job's label expression name defaults to "label_exp")
  5. Use shorter label expression name, like "win"

The Jenkins jobs will now be running at these shortened paths: 

C:\j\w\[JOB]\l\[LABEL]

Example: 

C:\j\w\T1\l\WIN

Instead of something like: 

C:\jenkins\workspace\MYTEST\labels\WIN-NODES


-

Change "workspace" to "ws" 

-Dhudson.model.Slave.workspaceRoot=ws
or
-Dhudson.model.Slave.workspaceRoot=w

Added to /usr/share/tomcat7/bin/catalina.sh under CATALINA_OPTS. This applies to all slaves.

-

Alternative:

Windows 10 has already added the feature for paths > 260 chars to the insider preview.

You can go to Advanced Options -> Windows Update and enable Insider builds. You then have to wait for Windows to download the latest insider build. I believe the path length feature was added to the official Windows Anniversary update. Maybe the lab team can add the anniversary update to provisioning?

Windows Web Start

Run IE with Elevated Permissions

Launch agent from browser

Java slave agent -> Fil -> Install as a service


Stop service and set "Log on as" -> "This account" -> administrator

  • Running some jobs as Local System account causes issuse

Delete Jenkins Service

sc stop jenkinsslave-c__j
sc delete jenkinsslave-c__j

Don't Exit Shell on Failure

set +e
git commit -m "Bla."
set -e

- 

# Disable exit on non 0
set +e

#Do something. If something fails with exit!=0 the script continues anyway

# Enable exit on non 0
set -e

Ref:

keywords: bash shell execute failure

Issues

UTF-8 decode URLs

Warning:

"Your container doesn’t use UTF-8 to decode URLs. If you use non-ASCII characters as a job name etc, this will cause problems. See Containers and Tomcat i18n for more details."

Solution:

Some versions of Tomcat (such as 5.0.28) uses iso-8859-1 to decode URLs, which is in a clear violation of the relevant RFCs. To fix this problem, add the following URIEncoding attribute to the connector definition in $TOMCAT_HOME/conf/server.xml. 

<Connector port="8080" URIEncoding="UTF-8"/>

Unsecured Jenkins

Warning:

"Unsecured Jenkins allows anyone on the network to launch processes on your behalf. Consider at least enabling authentication to discourage misuse. "

Reverse Proxy is Broken

See #Reverse Proxy

Jenkins says my reverse proxy setup is broken - Jenkins - Jenkins Wiki - https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+says+my+reverse+proxy+setup+is+broken

java.nio.charset.UnsupportedCharsetException: UTF-8-server

SEVERE: Servlet.service() for servlet Stapler threw exception
java.nio.charset.UnsupportedCharsetException: UTF-8-server

Had to remove this line from the Tomcat6 setenv.sh... 

-Dfile.encoding=UTF-8

keywords

Retrieved from "https://aznot.com/index.php?title=Jenkins&oldid=6392"