Omnia - User contributions [en]

Go

2026-04-30T08:48:54Z

Kenneth: /* Ungoogable */

Stub for Go programming language

== Ungoogable ==

https://www.google.com/search?q=golang

Issues:
* Name "Go" is ungoogable / hastagable (use golang) *** MOST ANNOYING ***

== Go Version ==

go version

== Hello World ==

<pre>
// Hello World
package main

import "fmt"

func main() {
fmt.Println("Hello, world")
}
</pre>

== Online Go Editor and Runner ==

https://go.dev/

== Install ==

=== Ubuntu ===

Standard version:
sudo apt update
sudo apt install golang-go

Latest via PPA:
sudo add-apt-repository ppa:longsleep/golang-backports
sudo apt update
sudo apt install golang-go

Via SNAP:
sudo snap install go --classic

== keywords ==

[[Category:Go]]

Go

2026-04-30T08:48:45Z

Kenneth: /* Online Go Editor and Runner */

Stub for Go programming language

== Ungoogable ==

https://www.google.com/search?q=golang

Issues:
* Name "Go" is ungoogable / hastagable (use golang) *** MOST ANNOYING ***

== Hello World ==

<pre>
// Hello World
package main

import "fmt"

func main() {
fmt.Println("Hello, world")
}
</pre>

== Online Go Editor and Runner ==

https://go.dev/

== Install ==

=== Ubuntu ===

Standard version:
sudo apt update
sudo apt install golang-go

Latest via PPA:
sudo add-apt-repository ppa:longsleep/golang-backports
sudo apt update
sudo apt install golang-go

Via SNAP:
sudo snap install go --classic

== keywords ==

[[Category:Go]]

RLM

2026-04-28T18:03:48Z

Kenneth: /* OpenSSL 1.1.0 required */

Reprise License Manager (RLM) is considered a, or a spiritual successor to, FLEXlm (now FlexNet Publisher). It was founded in 2006 by the original developers of GLOBEtrotter, the company that created FLEXlm

* https://www.openlm.com/blog/what-is-reprise-rlm/
* https://www.openlm.com/supported-software/reprise-license-management-rlm-and-optimization-with-openlm/

{FlexLM} (now officially FlexNet Publisher) is the industry-standard software license management system developed by Flexera. It controls, monitors, and optimizes application usage across networks, typically utilizing a floating (concurrent) license model where a central server manages a pool of licenses for multiple users. It supports node-locked licenses, manages license files (often found at C:\flexlm\license on Windows), and allows for vendor-specific configurations.

See [[FlexLM]]

== OpenSSL 1.1.0 required ==

Ubuntu 24 fails with:
<pre>
$ ./getiolid
./getiolid: /lib/x86_64-linux-gnu/libcrypto.so.1.1: version `OPENSSL_1_1_0' not found (required by ./getiolid)
</pre>

Cause: Ubuntu 24.04 (Noble Numbat) does not include libcrypto.so.1.1 in its official repositories because it has migrated to OpenSSL 3.0. You must manually install the legacy libssl1.1 package to obtain this file. <ref>https://askubuntu.com/questions/1521032/libssl-so-1-1-missing-on-ubuntu-24-04#:~:text=2,this%20package%20and%20updating%20it.</ref>

One solution: (no longer exists)
curl -O http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.22_amd64.deb
dpkg -i libssl1.1_1.1.1f-1ubuntu2.22_amd64.deb

# also works: libssl1.1_1.1.1w-0+deb11u1_amd64.deb.md5sum

Docker Ubuntu 24 needed:
apt install sudo curl libterm-readline-gnu-perl libterm-ui-perl dmidecode iproute2

== Systemd Service ==

<pre>
# cat /etc/systemd/system/rlm.service
[Unit]
Description=RLM License Server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=license
Group=license
WorkingDirectory=/opt/rlm

ExecStart=/bin/bash -x /opt/rlm/rlm-wrapper.sh
ExecStop=/bin/kill -TERM $MAINPID

Restart=on-failure
RestartSec=5

# Hardening (safe defaults for RHEL 8)
#NoNewPrivileges=true
#PrivateTmp=true
#ProtectSystem=full
#ProtectHome=true
#LimitNOFILE=262144

[Install]
WantedBy=multi-user.target
</pre>

<pre>
# cat /opt/rlm/rlm-wrapper.sh
#!/usr/bin/env bash
#set -euo pipefail
set -e

RLM_HOME="/opt/rlm"
RLM_BIN="${RLM_HOME}/rlm"
LICENSE_FILE="${RLM_HOME}/license.lic"
LOG_DIR="/var/log/rlm"
LOG_FILE="${LOG_DIR}/rlm.log"

mkdir -p "${LOG_DIR}"

# -nows disables Web UI
# use -ws 5054 if you want Web UI
# -dlog with + adds timestamped debug
#exec "${RLM_BIN}" \
# -c "${LICENSE_FILE}" \
# -ws 5054 \
# -dlog +"${LOG_FILE}"

exec "${RLM_BIN}" \
-c "${LICENSE_FILE}" \
-ws 5054 \
-dlog +"${LOG_FILE}"
</pre>

== keywords ==

Docker

2026-04-28T17:29:16Z

Kenneth: /* Ubuntu Docker Install Basic Tools */

{{subpages}}

== Docker ==

https://www.docker.com/

== Installing Docker Community Edition ==

=== Ubuntu ===

=== RHEL ===

RHEL | Docker Docs
https://docs.docker.com/engine/install/rhel/

If missing dnf see [[RHEL#DNF]]

# Remove Non Docker-CE:
sudo dnf remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine \
podman \
runc

sudo dnf -y install dnf-plugins-core
sudo dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo

sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo systemctl enable --now docker

sudo docker run hello-world

== Docker Hub ==

Docker Hub
https://hub.docker.com/

=== Search Hub ===

docker search [image]

Example:
docker search ubuntu

<pre>
Usage: docker search [OPTIONS] TERM

Search Docker Hub for images

Options:
-f, --filter filter Filter output based on conditions provided
--format string Pretty-print search using a Go template
--limit int Max number of search results
--no-trunc Don't truncate output
</pre>

=== Search Tags and Images ===

==== docker-browse ====

Show all images in registry
docker-browse --registry [registry] images

Show all tags of image:
docker-browse --registry [registry] tags [image]

Registry being that from ~/.docker/config.json

Can change default registry with "save" to ~/.docker-browse.json
<pre>
$ docker-browse save --registry myregistry.example.com:5000 --protocol https

$ cat ~/.docker-browse.json
{
"defRegistry": "myregistry.example.com:5000",
"defProtocol": "https"
}
</pre>

Install with:
# global
sudo npm install -g docker-browse

# local:
cd ~
mkdir -p ~/.npm
cd ~/.npm
npm install --prefix `pwd` docker-browse
cd ~/.bin
ln -sfn ~/.npm/node_modules/docker-browse/bin/docker-browse.js ~/.local/bin/docker-browse

Or local with:
mkdir ~/.src/docker-browse
cd ~/.src/docker-browse
# npm install docker-browse # umm did this ever work?
npm install --prefix `pwd` docker-browse
# cd ~/.bin
cd ~/.local/bin
# ln -s ../.src/docker-browse/node_modules/.bin/docker-browse # umm did this change names?
ln -s ../.src/docker-browse/node_modules/bin/docker-browse.js docker-browse

Assuming ~/.local/bin or .bin is in your path

ref: https://www.npmjs.com/package/docker-browse

==== skopeo ====

Install skopeo with apt:
sudo apt install skopeo

Login:
skopeo login docker://[repo:port]

Logout:
skopeo login docker://[repo:port]

List tags in registry:
skopeo list-tags docker://[repo:port]/[image]/[tag]

Inspect image:
skopeo delete docker://[repo:port]/[image]/[tag]

Copy image in registry:
skopeo copy docker://[repo:port]/[image]/[tag] docker://[repo:port]/[new_image]/[new_tag]

Delete image in registry: (( WARNING - THIS DELETES THE IMAGE, NOT JUST THE TAG ))
skopeo delete docker://[repo:port]/[image]/[tag]

== What Is Docker ==

Package your application into a standardized unit for software development

:"Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment."

https://www.docker.com/what-docker

--

"Docker is an open-source project that automates the deployment of applications inside software containers."

Docker provides an additional layer of abstraction and automation of operating-system-level virtualization on Linux. Docker uses the resource isolation features of the Linux kernel such as cgroups and kernel namespaces, and a union-capable file system such as OverlayFS and others to allow independent "containers" to run within a single Linux instance, avoiding the overhead of starting and maintaining virtual machines.

The Linux kernel's support for namespaces mostly isolates an application's view of the operating environment, including process trees, network, user IDs and mounted file systems, while the kernel's cgroups provide resource limiting, including the CPU, memory, block I/O and network. Since version 0.9, Docker includes the libcontainer library as its own way to directly use virtualization facilities provided by the Linux kernel, in addition to using abstracted virtualization interfaces via libvirt, LXC (Linux Containers) and systemd-nspawn.

As actions are done to a Docker base image, union file system layers are created and documented, such that each layer fully describes how to recreate an action. This strategy enables Docker's lightweight images, as only layer updates need to be propagated (compared to full VMs, for example).

https://en.wikipedia.org/wiki/Docker_(software)

== Removing all containers and images ==

# Stop the container(s) using the following command:
docker-compose down
# Delete all containers using the following command:
docker rm -f $(docker ps -a -q)
# Delete all volumes using the following command:
docker volume rm $(docker volume ls -q)
# Prune system images, included stopped:
docker system prune -a

Ref [https://docs.tibco.com/pub/mash-local/4.1.1/doc/html/docker/GUID-BD850566-5B79-4915-987E-430FC38DAAE4.html] [https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes]

== A Docker Tutorial for Beginners ==

A Docker Tutorial for Beginners 
https://docker-curriculum.com/

Once you are done installing Docker, test your Docker installation by running the following:
# docker run hello-world

hello-world container: https://hub.docker.com/_/hello-world

Example:
<pre>
root@server:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:9f6ad537c5132bcce57f7a0a20e317228d382c3cd61edae14650eec68b2b345c
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/

For more examples and ideas, visit:
https://docs.docker.com/get-started/
</pre>

Example of Ubuntu container:
<pre>
root@mserver:~# docker run -it ubuntu bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
c549ccf8d472: Pull complete
Digest: sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe
Status: Downloaded newer image for ubuntu:latest
root@4d827b59d198:/#
</pre>

Try Busybox container:
<pre>
## Fetch container
docker pull busybox

## List containers
docker images

## Run container (did nothing then exited)
docker run busybox

## Run container with instructions to do something
docker run busybox echo "hello from busybox"
hello from busybox

## List running docker containers:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

## List all dockers containers that we have run:
docker ps -a

## Run in interactive mode:
docker run -it busybox sh

## Run and auto delete on exit
docker run --rm busybox echo "hello"

## Remove docker containers (clean up after yourself
docker rm [CONTAINER_ID CONTAINER_ID ...]

## Remove all exited docker containers:
docker rm $(docker ps -a -q -f status=exited)
# or in later editions
docker container prune

## Remove docker images:
docker rmi [IMAGE_ID IMAGE_ID ...]

## Run a detached instance, and name it
docker run -d --name [NAME] [IMAGE]
# example: docker run -d --rm --name u3 -it ubuntu bash

## Attach to running container: [https://docs.docker.com/engine/reference/commandline/attach/]
docker attach [NAME]

## Detach from a running container: [https://docs.docker.com/engine/reference/commandline/attach/]
CTRL-p CTRL-q

## Stop detached instance
docker stop [NAME]

## Pull a specific version:
docker pull ubuntu:latest
docker pull ubuntu:18.04
docker pull ubuntu:16.04

## Run specific version
docker run ubuntu:16.04
</pre>

<pre>
root@server:~# docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
b71f96345d44: Pull complete
Digest: sha256:930490f97e5b921535c153e0e7110d251134cc4b72bbb8133c6a5065cc68580d
Status: Downloaded newer image for busybox:latest
docker.io/library/busybox:latest

root@server:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ubuntu latest 9873176a8ff5 13 days ago 72.7MB
busybox latest 69593048aa3a 3 weeks ago 1.24MB
hello-world latest d1165f221234 3 months ago 13.3kB

root@server:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
495cf502dde2 busybox "echo 'hello from bu…" 2 minutes ago Exited (0) 2 minutes ago
</pre>

=== Create Image ===

All user images are based on a base image.

Login to Docker Hub:
docker login

Sample Dockerfile:
<pre>
FROM python:3

# set a directory for the app
WORKDIR /usr/src/app

# copy all the files to the container
COPY . .

# install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# define the port number the container should expose
EXPOSE 5000

# run the command
CMD ["python", "./app.py"]
</pre>

Simple Dockerfile:
<pre>
FROM ubuntu

RUN touch test.txt
</pre>

Build image:
docker build -t [username]/[project] .

Test imagE:
docker run [username]/[project]<:version>

Push image to Docker Hub:
docker push [username]/[project]

See parent hash of image:
<pre>
docker history --format '{{ .ID }}' --no-trunc [USERNAME]/[PROJECT]
</pre>

== Pull Image ==

docker pull

Hash specific: [https://docs.docker.com/engine/reference/commandline/pull/]
docker pull ubuntu@sha256:26c68657ccce2cb0a31b330cb0be2b5e108d467f641c62e13ab40cbec258c68d

Hash specific in Dockerfile: [https://docs.docker.com/engine/reference/commandline/pull/]
FROM ubuntu@sha256:26c68657ccce2cb0a31b330cb0be2b5e108d467f641c62e13ab40cbec258c68d

== Run Image ==

docker run -it --rm [image:tag] /bin/bash

If you want to force a specific entry point:
docker run -it --rm --entrypoint /bin/sh [image:tag]

== Attach to Running Docker ==

docker attach [NAME]

If it isn't running bash, you can do something like this:
docker exec -it [NAME] /bin/bash

== Run Docker Ubuntu Mounted to Current Directory ==

docker run --rm -it -v .:/mnt ubuntu bash

== Running ARM on x86 ==

'''Problem:'''
uname -m # Display the host architecture
#x86_64

docker run --rm -t arm64v8/ubuntu uname -m # Run an executable made for aarch64 on x86_64
#standard_init_linux.go:211: exec user process caused "exec format error"

'''Solution:'''

As expected the instructions are not recognized since the packages are not installed yet. Installing the following packages should allow you to enable support for aarch64 containers on your x86 workstation:

sudo apt-get install qemu binfmt-support qemu-user-static # Install the qemu packages
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes # This step will execute the registering scripts

docker run --rm -t arm64v8/ubuntu uname -m # Testing the emulation environment
#aarch64

ref: [https://www.stereolabs.com/docs/docker/building-arm-container-on-x86/]

== Static MAC ==

docker run ... --mac-address 12:34:56:78:9a:bc

ref: [https://docs.docker.com/engine/reference/run/#:~:text=By%20default%2C%20the%20MAC%20address,specified%20MAC%20addresses%20are%20unique.]

== Exporting ==

=== Save and Load Image ===

docker save -o [image.tar] [IMAGE]
# or
docker save [IMAGE] > [image.tar]

docker load -i [image.tar]
# or
cat [image.tar] | docker load

=== Export and Import Container ===

docker export [CONTAINER_ID_OR_NAME] > [container.tar]

docker import [container.tar] [IMAGE_NAME:VER]
# or
cat [container.tar] | docker import - [IMAGE_NAME:VER]

== Extract Build Steps ==

Don't truncate the lines - gives you the FULL command line:
docker history [IMAGE_ID] --no-trunc | tac

Show only the docker step - full command line, no other details (.CreatedBy is the command that created the layer, not the author)
<pre>
docker history [IMAGE_ID] --format '{{.CreatedBy}}' --no-trunc | tac
</pre>

Show only the docker step - Don't truncate the lines and wrap lines at && - really nice view
<pre>
docker history --no-trunc [IMAGE_ID] | tac | tr -s ' ' | cut -d " " -f 5- | sed 's,^/bin/sh -c #(nop) ,,g' | sed 's,^/bin/sh -c,RUN,g' | sed 's, && ,\n & ,g' | sed 's,\s*[0-9]*[\.]*[0-9]*\s*[kMG]*B\s*$,,g' | head -n -1
</pre>

ref: <ref>https://stackoverflow.com/questions/19104847/how-to-generate-a-dockerfile-from-an-image</ref>

See parent hash of image:
<pre>
docker history --format '{{ .ID }}' --no-trunc [IMAGE_ID]
</pre>

== Docker Compose ==

apt install docker-compose docker-compose-plugin
# docker-compose provides "docker-compose" command
# docker-compose-plugin provides "docker compose" command

docker compose up -d

Sample docker-compose.yml:
<pre>
version: "3.0" # compose API version
services:
license-service:
container_name: license
# Docker image name
image: flexlm:v1
build:
context: .
user: root
restart: always
# mac address tied to license file
mac_address: xx:xx:xx:xx:xx:xx
ports:
# 27000 for lmgrd
- 27000:27000
# 27001 for armlmd
- 27001:27001
</pre>

== Docker BuildX ==

buildx:
https://github.com/docker/buildx

apt package:
docker-buildx/jammy-updates,jammy-security 0.12.1-0ubuntu1~22.04.2 amd64
Docker CLI plugin for extended build capabilities with BuildKit

Install buildx with apt:
sudo apt install docker-buildx

Set buildx as default builder:
docker buildx install

To remove buildx alias:
docker buildx uninstall

You can build multi-platform images using three different strategies that are supported by Buildx and Dockerfiles:
# Using the QEMU emulation support in the kernel
# Building on multiple native nodes using the same builder instance
# Using a stage in Dockerfile to cross-compile to different architectures

If not using Docker Desktop:
docker run --privileged --rm tonistiigi/binfmt --install all

Add additional platforms:
docker buildx create --use --name mybuild node-amd64
docker buildx create --append --name mybuild node-arm64
docker buildx build --platform linux/amd64,linux/arm64 .

== Proxy ==

Client:
~/.docker/config.json

<pre>
{
"proxies": {
"default": {
"httpProxy": "http://proxy.example.com:3128",
"httpsProxy": "https://proxy.example.com:3129",
"noProxy": "*.test.example.com,.example.org,127.0.0.0/8"
}
}
}
</pre>

ref: Configure Docker to use a proxy server | Docker Documentation - https://docs.docker.com/network/proxy/

Configure Docker Service

systemctl edit docker.service

<pre>
[Service]
Environment="HTTP_PROXY=http://proxy.example.com:3128",
Environment="HTTPS_PROXY=httpS://proxy.example.com:3129",
Environment="NO_PROXY=localhost,127.0.0.1",
</pre>
systemctl restart docker.service

== Registry ==

=== Registry API ===

HTTP API V2 | Docker Documentation
https://docs.docker.com/registry/spec/api/

List available images:
GET /v2/_catalog
GET https://registry.example.com/v2/_catalog

Pretty print output:
curl -s https://registry.example.com/v2/_catalog | jq
curl -s https://registry.example.com/v2/_catalog | python3 -m json.tool

Pagenation
GET /v2/_catalog?n=<count>
# using a name of repo in a previous result:
GET /v2/_catalog?n=<count>&last=<name_repo_start_with>

List tags: (pagination also works with tags)
GET /v2/<image_name>/tags/list

Delete image:
# reference must be a digest or the delete will fail.
DELETE /v2/<name>/manifests/<reference>

API:
Method Path Entity Description
GET /v2/ Base Check that the endpoint implements Docker Registry API V2.
GET /v2/<name>/tags/list Tags Fetch the tags under the repository identified by name.
GET /v2/<name>/manifests/<reference> Manifest Fetch the manifest identified by name and reference where reference can be a tag or digest. A HEAD request can also be issued to this endpoint to obtain resource information without receiving all data.
PUT /v2/<name>/manifests/<reference> Manifest Put the manifest identified by name and reference where reference can be a tag or digest.
DELETE /v2/<name>/manifests/<reference> Manifest Delete the manifest or tag identified by name and reference where reference can be a tag or digest. Note that a manifest can only be deleted by digest.
GET /v2/<name>/blobs/<digest> Blob Retrieve the blob from the registry identified by digest. A HEAD request can also be issued to this endpoint to obtain resource information without receiving all data.
DELETE /v2/<name>/blobs/<digest> Blob Delete the blob identified by name and digest
POST /v2/<name>/blobs/uploads/ Initiate Blob Upload Initiate a resumable blob upload. If successful, an upload location will be provided to complete the upload. Optionally, if the digest parameter is present, the request body will be used to complete the upload in a single request.
GET /v2/<name>/blobs/uploads/<uuid> Blob Upload Retrieve status of upload identified by uuid. The primary purpose of this endpoint is to resolve the current status of a resumable upload.
PATCH /v2/<name>/blobs/uploads/<uuid> Blob Upload Upload a chunk of data for the specified upload.
PUT /v2/<name>/blobs/uploads/<uuid> Blob Upload Complete the upload specified by uuid, optionally appending the body as the final chunk.
DELETE /v2/<name>/blobs/uploads/<uuid> Blob Upload Cancel outstanding upload processes, releasing associated resources. If this is not called, the unfinished uploads will eventually timeout.
GET /v2/_catalog Catalog Retrieve a sorted, json list of repositories available in the registry.

== Reducing Image Size ==

1. Bundle Layers

RUN apt-get update -y \
&& apt install -y package_one

2. Avoid extra packages

apt install -y --no-install-recommends

3. Clean up after install

apt ... \
&& rm -rf /var/lib/apt/lists/*

4. Pick smaller image

alpine vs ubuntu

5. Use .dockerignore

ref: [https://towardsdatascience.com/5-easy-to-implement-tricks-to-trim-down-your-docker-image-size-263978a6ed29]

== Missing Docker Images Packages ==

scp:
apt update ; apt install openssh-clients

== Issues ==

=== apt getting stuck at configuring tzdata ===

<pre>
Configuring tzdata
------------------

Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.

1. Africa 4. Australia 7. Atlantic 10. Pacific 13. Etc
2. America 5. Arctic 8. Europe 11. SystemV
3. Antarctica 6. Asia 9. Indian 12. US
Geographic area:
</pre>

Solution:
<pre>
FROM ubuntu:20.04

ENV TZ=Asia/Dubai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

RUN apt update
RUN apt install -y tzdata
</pre>

ref: Fix: tzdata hangs during Docker image build - DEV Community - https://dev.to/grigorkh/fix-tzdata-hangs-during-docker-image-build-4o9m

== Ubuntu Docker Install Basic Tools ==

apt update

# basic tools
apt install sudo curl wget screen vim less

# more tools
apt install dmidecode iproute2 net-tools

# even more tools
apt install nmap

== keywords ==

Docker

2026-04-28T17:28:21Z

Kenneth: /* keywords */

{{subpages}}

== Docker ==

https://www.docker.com/

== Installing Docker Community Edition ==

=== Ubuntu ===

=== RHEL ===

RHEL | Docker Docs
https://docs.docker.com/engine/install/rhel/

If missing dnf see [[RHEL#DNF]]

# Remove Non Docker-CE:
sudo dnf remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine \
podman \
runc

sudo dnf -y install dnf-plugins-core
sudo dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo

sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo systemctl enable --now docker

sudo docker run hello-world

== Docker Hub ==

Docker Hub
https://hub.docker.com/

=== Search Hub ===

docker search [image]

Example:
docker search ubuntu

<pre>
Usage: docker search [OPTIONS] TERM

Search Docker Hub for images

Options:
-f, --filter filter Filter output based on conditions provided
--format string Pretty-print search using a Go template
--limit int Max number of search results
--no-trunc Don't truncate output
</pre>

=== Search Tags and Images ===

==== docker-browse ====

Show all images in registry
docker-browse --registry [registry] images

Show all tags of image:
docker-browse --registry [registry] tags [image]

Registry being that from ~/.docker/config.json

Can change default registry with "save" to ~/.docker-browse.json
<pre>
$ docker-browse save --registry myregistry.example.com:5000 --protocol https

$ cat ~/.docker-browse.json
{
"defRegistry": "myregistry.example.com:5000",
"defProtocol": "https"
}
</pre>

Install with:
# global
sudo npm install -g docker-browse

# local:
cd ~
mkdir -p ~/.npm
cd ~/.npm
npm install --prefix `pwd` docker-browse
cd ~/.bin
ln -sfn ~/.npm/node_modules/docker-browse/bin/docker-browse.js ~/.local/bin/docker-browse

Or local with:
mkdir ~/.src/docker-browse
cd ~/.src/docker-browse
# npm install docker-browse # umm did this ever work?
npm install --prefix `pwd` docker-browse
# cd ~/.bin
cd ~/.local/bin
# ln -s ../.src/docker-browse/node_modules/.bin/docker-browse # umm did this change names?
ln -s ../.src/docker-browse/node_modules/bin/docker-browse.js docker-browse

Assuming ~/.local/bin or .bin is in your path

ref: https://www.npmjs.com/package/docker-browse

==== skopeo ====

Install skopeo with apt:
sudo apt install skopeo

Login:
skopeo login docker://[repo:port]

Logout:
skopeo login docker://[repo:port]

List tags in registry:
skopeo list-tags docker://[repo:port]/[image]/[tag]

Inspect image:
skopeo delete docker://[repo:port]/[image]/[tag]

Copy image in registry:
skopeo copy docker://[repo:port]/[image]/[tag] docker://[repo:port]/[new_image]/[new_tag]

Delete image in registry: (( WARNING - THIS DELETES THE IMAGE, NOT JUST THE TAG ))
skopeo delete docker://[repo:port]/[image]/[tag]

== What Is Docker ==

Package your application into a standardized unit for software development

:"Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment."

https://www.docker.com/what-docker

--

"Docker is an open-source project that automates the deployment of applications inside software containers."

Docker provides an additional layer of abstraction and automation of operating-system-level virtualization on Linux. Docker uses the resource isolation features of the Linux kernel such as cgroups and kernel namespaces, and a union-capable file system such as OverlayFS and others to allow independent "containers" to run within a single Linux instance, avoiding the overhead of starting and maintaining virtual machines.

The Linux kernel's support for namespaces mostly isolates an application's view of the operating environment, including process trees, network, user IDs and mounted file systems, while the kernel's cgroups provide resource limiting, including the CPU, memory, block I/O and network. Since version 0.9, Docker includes the libcontainer library as its own way to directly use virtualization facilities provided by the Linux kernel, in addition to using abstracted virtualization interfaces via libvirt, LXC (Linux Containers) and systemd-nspawn.

As actions are done to a Docker base image, union file system layers are created and documented, such that each layer fully describes how to recreate an action. This strategy enables Docker's lightweight images, as only layer updates need to be propagated (compared to full VMs, for example).

https://en.wikipedia.org/wiki/Docker_(software)

== Removing all containers and images ==

# Stop the container(s) using the following command:
docker-compose down
# Delete all containers using the following command:
docker rm -f $(docker ps -a -q)
# Delete all volumes using the following command:
docker volume rm $(docker volume ls -q)
# Prune system images, included stopped:
docker system prune -a

Ref [https://docs.tibco.com/pub/mash-local/4.1.1/doc/html/docker/GUID-BD850566-5B79-4915-987E-430FC38DAAE4.html] [https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes]

== A Docker Tutorial for Beginners ==

A Docker Tutorial for Beginners 
https://docker-curriculum.com/

Once you are done installing Docker, test your Docker installation by running the following:
# docker run hello-world

hello-world container: https://hub.docker.com/_/hello-world

Example:
<pre>
root@server:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:9f6ad537c5132bcce57f7a0a20e317228d382c3cd61edae14650eec68b2b345c
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/

For more examples and ideas, visit:
https://docs.docker.com/get-started/
</pre>

Example of Ubuntu container:
<pre>
root@mserver:~# docker run -it ubuntu bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
c549ccf8d472: Pull complete
Digest: sha256:aba80b77e27148d99c034a987e7da3a287ed455390352663418c0f2ed40417fe
Status: Downloaded newer image for ubuntu:latest
root@4d827b59d198:/#
</pre>

Try Busybox container:
<pre>
## Fetch container
docker pull busybox

## List containers
docker images

## Run container (did nothing then exited)
docker run busybox

## Run container with instructions to do something
docker run busybox echo "hello from busybox"
hello from busybox

## List running docker containers:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

## List all dockers containers that we have run:
docker ps -a

## Run in interactive mode:
docker run -it busybox sh

## Run and auto delete on exit
docker run --rm busybox echo "hello"

## Remove docker containers (clean up after yourself
docker rm [CONTAINER_ID CONTAINER_ID ...]

## Remove all exited docker containers:
docker rm $(docker ps -a -q -f status=exited)
# or in later editions
docker container prune

## Remove docker images:
docker rmi [IMAGE_ID IMAGE_ID ...]

## Run a detached instance, and name it
docker run -d --name [NAME] [IMAGE]
# example: docker run -d --rm --name u3 -it ubuntu bash

## Attach to running container: [https://docs.docker.com/engine/reference/commandline/attach/]
docker attach [NAME]

## Detach from a running container: [https://docs.docker.com/engine/reference/commandline/attach/]
CTRL-p CTRL-q

## Stop detached instance
docker stop [NAME]

## Pull a specific version:
docker pull ubuntu:latest
docker pull ubuntu:18.04
docker pull ubuntu:16.04

## Run specific version
docker run ubuntu:16.04
</pre>

<pre>
root@server:~# docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
b71f96345d44: Pull complete
Digest: sha256:930490f97e5b921535c153e0e7110d251134cc4b72bbb8133c6a5065cc68580d
Status: Downloaded newer image for busybox:latest
docker.io/library/busybox:latest

root@server:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ubuntu latest 9873176a8ff5 13 days ago 72.7MB
busybox latest 69593048aa3a 3 weeks ago 1.24MB
hello-world latest d1165f221234 3 months ago 13.3kB

root@server:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
495cf502dde2 busybox "echo 'hello from bu…" 2 minutes ago Exited (0) 2 minutes ago
</pre>

=== Create Image ===

All user images are based on a base image.

Login to Docker Hub:
docker login

Sample Dockerfile:
<pre>
FROM python:3

# set a directory for the app
WORKDIR /usr/src/app

# copy all the files to the container
COPY . .

# install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# define the port number the container should expose
EXPOSE 5000

# run the command
CMD ["python", "./app.py"]
</pre>

Simple Dockerfile:
<pre>
FROM ubuntu

RUN touch test.txt
</pre>

Build image:
docker build -t [username]/[project] .

Test imagE:
docker run [username]/[project]<:version>

Push image to Docker Hub:
docker push [username]/[project]

See parent hash of image:
<pre>
docker history --format '{{ .ID }}' --no-trunc [USERNAME]/[PROJECT]
</pre>

== Pull Image ==

docker pull

Hash specific: [https://docs.docker.com/engine/reference/commandline/pull/]
docker pull ubuntu@sha256:26c68657ccce2cb0a31b330cb0be2b5e108d467f641c62e13ab40cbec258c68d

Hash specific in Dockerfile: [https://docs.docker.com/engine/reference/commandline/pull/]
FROM ubuntu@sha256:26c68657ccce2cb0a31b330cb0be2b5e108d467f641c62e13ab40cbec258c68d

== Run Image ==

docker run -it --rm [image:tag] /bin/bash

If you want to force a specific entry point:
docker run -it --rm --entrypoint /bin/sh [image:tag]

== Attach to Running Docker ==

docker attach [NAME]

If it isn't running bash, you can do something like this:
docker exec -it [NAME] /bin/bash

== Run Docker Ubuntu Mounted to Current Directory ==

docker run --rm -it -v .:/mnt ubuntu bash

== Running ARM on x86 ==

'''Problem:'''
uname -m # Display the host architecture
#x86_64

docker run --rm -t arm64v8/ubuntu uname -m # Run an executable made for aarch64 on x86_64
#standard_init_linux.go:211: exec user process caused "exec format error"

'''Solution:'''

As expected the instructions are not recognized since the packages are not installed yet. Installing the following packages should allow you to enable support for aarch64 containers on your x86 workstation:

sudo apt-get install qemu binfmt-support qemu-user-static # Install the qemu packages
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes # This step will execute the registering scripts

docker run --rm -t arm64v8/ubuntu uname -m # Testing the emulation environment
#aarch64

ref: [https://www.stereolabs.com/docs/docker/building-arm-container-on-x86/]

== Static MAC ==

docker run ... --mac-address 12:34:56:78:9a:bc

ref: [https://docs.docker.com/engine/reference/run/#:~:text=By%20default%2C%20the%20MAC%20address,specified%20MAC%20addresses%20are%20unique.]

== Exporting ==

=== Save and Load Image ===

docker save -o [image.tar] [IMAGE]
# or
docker save [IMAGE] > [image.tar]

docker load -i [image.tar]
# or
cat [image.tar] | docker load

=== Export and Import Container ===

docker export [CONTAINER_ID_OR_NAME] > [container.tar]

docker import [container.tar] [IMAGE_NAME:VER]
# or
cat [container.tar] | docker import - [IMAGE_NAME:VER]

== Extract Build Steps ==

Don't truncate the lines - gives you the FULL command line:
docker history [IMAGE_ID] --no-trunc | tac

Show only the docker step - full command line, no other details (.CreatedBy is the command that created the layer, not the author)
<pre>
docker history [IMAGE_ID] --format '{{.CreatedBy}}' --no-trunc | tac
</pre>

Show only the docker step - Don't truncate the lines and wrap lines at && - really nice view
<pre>
docker history --no-trunc [IMAGE_ID] | tac | tr -s ' ' | cut -d " " -f 5- | sed 's,^/bin/sh -c #(nop) ,,g' | sed 's,^/bin/sh -c,RUN,g' | sed 's, && ,\n & ,g' | sed 's,\s*[0-9]*[\.]*[0-9]*\s*[kMG]*B\s*$,,g' | head -n -1
</pre>

ref: <ref>https://stackoverflow.com/questions/19104847/how-to-generate-a-dockerfile-from-an-image</ref>

See parent hash of image:
<pre>
docker history --format '{{ .ID }}' --no-trunc [IMAGE_ID]
</pre>

== Docker Compose ==

apt install docker-compose docker-compose-plugin
# docker-compose provides "docker-compose" command
# docker-compose-plugin provides "docker compose" command

docker compose up -d

Sample docker-compose.yml:
<pre>
version: "3.0" # compose API version
services:
license-service:
container_name: license
# Docker image name
image: flexlm:v1
build:
context: .
user: root
restart: always
# mac address tied to license file
mac_address: xx:xx:xx:xx:xx:xx
ports:
# 27000 for lmgrd
- 27000:27000
# 27001 for armlmd
- 27001:27001
</pre>

== Docker BuildX ==

buildx:
https://github.com/docker/buildx

apt package:
docker-buildx/jammy-updates,jammy-security 0.12.1-0ubuntu1~22.04.2 amd64
Docker CLI plugin for extended build capabilities with BuildKit

Install buildx with apt:
sudo apt install docker-buildx

Set buildx as default builder:
docker buildx install

To remove buildx alias:
docker buildx uninstall

You can build multi-platform images using three different strategies that are supported by Buildx and Dockerfiles:
# Using the QEMU emulation support in the kernel
# Building on multiple native nodes using the same builder instance
# Using a stage in Dockerfile to cross-compile to different architectures

If not using Docker Desktop:
docker run --privileged --rm tonistiigi/binfmt --install all

Add additional platforms:
docker buildx create --use --name mybuild node-amd64
docker buildx create --append --name mybuild node-arm64
docker buildx build --platform linux/amd64,linux/arm64 .

== Proxy ==

Client:
~/.docker/config.json

<pre>
{
"proxies": {
"default": {
"httpProxy": "http://proxy.example.com:3128",
"httpsProxy": "https://proxy.example.com:3129",
"noProxy": "*.test.example.com,.example.org,127.0.0.0/8"
}
}
}
</pre>

ref: Configure Docker to use a proxy server | Docker Documentation - https://docs.docker.com/network/proxy/

Configure Docker Service

systemctl edit docker.service

<pre>
[Service]
Environment="HTTP_PROXY=http://proxy.example.com:3128",
Environment="HTTPS_PROXY=httpS://proxy.example.com:3129",
Environment="NO_PROXY=localhost,127.0.0.1",
</pre>
systemctl restart docker.service

== Registry ==

=== Registry API ===

HTTP API V2 | Docker Documentation
https://docs.docker.com/registry/spec/api/

List available images:
GET /v2/_catalog
GET https://registry.example.com/v2/_catalog

Pretty print output:
curl -s https://registry.example.com/v2/_catalog | jq
curl -s https://registry.example.com/v2/_catalog | python3 -m json.tool

Pagenation
GET /v2/_catalog?n=<count>
# using a name of repo in a previous result:
GET /v2/_catalog?n=<count>&last=<name_repo_start_with>

List tags: (pagination also works with tags)
GET /v2/<image_name>/tags/list

Delete image:
# reference must be a digest or the delete will fail.
DELETE /v2/<name>/manifests/<reference>

API:
Method Path Entity Description
GET /v2/ Base Check that the endpoint implements Docker Registry API V2.
GET /v2/<name>/tags/list Tags Fetch the tags under the repository identified by name.
GET /v2/<name>/manifests/<reference> Manifest Fetch the manifest identified by name and reference where reference can be a tag or digest. A HEAD request can also be issued to this endpoint to obtain resource information without receiving all data.
PUT /v2/<name>/manifests/<reference> Manifest Put the manifest identified by name and reference where reference can be a tag or digest.
DELETE /v2/<name>/manifests/<reference> Manifest Delete the manifest or tag identified by name and reference where reference can be a tag or digest. Note that a manifest can only be deleted by digest.
GET /v2/<name>/blobs/<digest> Blob Retrieve the blob from the registry identified by digest. A HEAD request can also be issued to this endpoint to obtain resource information without receiving all data.
DELETE /v2/<name>/blobs/<digest> Blob Delete the blob identified by name and digest
POST /v2/<name>/blobs/uploads/ Initiate Blob Upload Initiate a resumable blob upload. If successful, an upload location will be provided to complete the upload. Optionally, if the digest parameter is present, the request body will be used to complete the upload in a single request.
GET /v2/<name>/blobs/uploads/<uuid> Blob Upload Retrieve status of upload identified by uuid. The primary purpose of this endpoint is to resolve the current status of a resumable upload.
PATCH /v2/<name>/blobs/uploads/<uuid> Blob Upload Upload a chunk of data for the specified upload.
PUT /v2/<name>/blobs/uploads/<uuid> Blob Upload Complete the upload specified by uuid, optionally appending the body as the final chunk.
DELETE /v2/<name>/blobs/uploads/<uuid> Blob Upload Cancel outstanding upload processes, releasing associated resources. If this is not called, the unfinished uploads will eventually timeout.
GET /v2/_catalog Catalog Retrieve a sorted, json list of repositories available in the registry.

== Reducing Image Size ==

1. Bundle Layers

RUN apt-get update -y \
&& apt install -y package_one

2. Avoid extra packages

apt install -y --no-install-recommends

3. Clean up after install

apt ... \
&& rm -rf /var/lib/apt/lists/*

4. Pick smaller image

alpine vs ubuntu

5. Use .dockerignore

ref: [https://towardsdatascience.com/5-easy-to-implement-tricks-to-trim-down-your-docker-image-size-263978a6ed29]

== Missing Docker Images Packages ==

scp:
apt update ; apt install openssh-clients

== Issues ==

=== apt getting stuck at configuring tzdata ===

<pre>
Configuring tzdata
------------------

Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.

1. Africa 4. Australia 7. Atlantic 10. Pacific 13. Etc
2. America 5. Arctic 8. Europe 11. SystemV
3. Antarctica 6. Asia 9. Indian 12. US
Geographic area:
</pre>

Solution:
<pre>
FROM ubuntu:20.04

ENV TZ=Asia/Dubai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

RUN apt update
RUN apt install -y tzdata
</pre>

ref: Fix: tzdata hangs during Docker image build - DEV Community - https://dev.to/grigorkh/fix-tzdata-hangs-during-docker-image-build-4o9m

== Ubuntu Docker Install Basic Tools ==

apt update
apt install sudo curl wget screen vim less
apt install dmidecode iproute2 net-tools

== keywords ==

Python

2026-04-27T20:28:38Z

Kenneth: /* Variable Directory = */

{{subtop}}

== Python ==

Python (programming language) - Wikipedia - http://en.wikipedia.org/wiki/Python_(programming_language)
: Python is an interpreted, general-purpose high-level programming language whose design philosophy emphasizes code readability. Python aims to combine "remarkable power with very clear syntax", and its standard library is large and comprehensive. Its use of indentation for block delimiters is unique among popular programming languages.
: Python supports multiple programming paradigms, primarily but not limited to object-oriented, imperative and, to a lesser extent, functional programming styles. It features a fully dynamic type system and automatic memory management, similar to that of Scheme, Ruby, Perl, and Tcl. Like other dynamic languages, Python is often used as a scripting language, but is also used in a wide range of non-scripting contexts.
: The reference implementation of Python ([http://en.wikipedia.org/wiki/CPython CPython]) is free and open source software and has a community-based development model, as do all or nearly all of its alternative implementations. CPython is managed by the non-profit Python Software Foundation.
: Python interpreters are available for many operating systems, and Python programs can be packaged into stand-alone executable code for many systems using various tools.

Author:
:"Python was conceived in the late 1980s and its implementation was started in December 1989 by '''[http://www.python.org/~guido/ Guido van Rossum]''' at CWI in the Netherlands as a successor to the ABC programming language (itself inspired by SETL) capable of exception handling and interfacing with the Amoeba operating system. Van Rossum is Python's principal author, and his continuing central role in deciding the direction of Python is reflected in the title given to him by the Python community, Benevolent Dictator for Life (BDFL)." [http://en.wikipedia.org/wiki/Python_(programming_language)]

Name:
:"The language is named after the BBC show “Monty Python’s Flying Circus” and has nothing to do with reptiles. Making references to Monty Python skits in documentation is not only allowed, it is encouraged!" [http://docs.python.org/py3k/tutorial/appetite.html]

We as people are "Pythonites" [http://raspi.tv/2013/how-to-use-interrupts-with-python-on-the-raspberry-pi-and-rpi-gpio-part-2]

== [http://www.python.org/dev/peps/pep-0020/ The Zen of Python] ==

"The core philosophy of the language is summarized by the document 'PEP 20 (The Zen of Python)'" [http://en.wikipedia.org/wiki/Python_%28programming_language%29]
* PEP 20 -- The Zen of Python - http://www.python.org/dev/peps/pep-0020/

<pre>
Abstract

Long time Pythoneer Tim Peters succinctly channels the BDFL's
guiding principles for Python's design into 20 aphorisms, only 19
of which have been written down.

The Zen of Python

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Easter Egg

>>> import this
</pre>

Other Sources:
* http://www.python.org/doc/humor/
* The Zen of Python – as a Poster « CODE POETRY - http://codepoetry.wordpress.com/2010/03/23/the-zen-of-python-as-a-poster/
** Zen of Python (PDF) http://codepoetry.files.wordpress.com/2010/03/zen-of-python.pdf

== Documentation ==

* Overview — Python v2.7.1 documentation - http://docs.python.org/index.html
** Download — Python v2.7.1 documentation - http://docs.python.org/download.html

* Overview — Python v3.2 documentation - http://docs.python.org/py3k/
** Download — Python v3.2 documentation - http://docs.python.org/py3k/download.html

* PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/

== Tutorials ==

* The Python Tutorial — Python v2.7.1 documentation - http://docs.python.org/tutorial/

* Python 101 -- Introduction to Python - http://www.rexx.com/~dkuhlman/python_101/python_101.html

* 4. More Control Flow Tools — Python v2.7.1 documentation - http://docs.python.org/tutorial/controlflow.html

== Linux Installation ==

RHEL Distribution installation:
yum -y install python python-setuptools

Ubuntu Distribution installation: [https://help.ubuntu.com/community/Mercurial]
# enable 'universe' in the sources.list
# "Mercurial is included in the Universe repository - activate this repo first"
sudo apt-get install mercurial

CentOS 5 Option Recommended Dependencies:
yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel

# this leaves: 2.6.6: _bsddb, bsddb185, dl, imageop, sunaudiodev
# this leaves: 2.7.1: _bsddb, bsddb185, dl, imageop, sunaudiodev
# this leaves: 3.2: none

CentOS 9 Option Recommended Dependencies:
# gdbm-devel not found in centos9?
yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel sqlite-devel bzip2-devel tk-devel tcl-devel

Ubuntu 12.04 LTS 64bit dependencies:
apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev

# maybe: lib64readline-dev libsqlite3-dev tk-dev

# this leaves: 2.7: _bsddb, bsddb185, dl, imageop, sunaudiodev

=== 2.x ===

CentOS dependencies:
sudo yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel

Ubuntu 12.04 LTS 64bit dependencies:
sudo apt-get update
sudo apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev

Ubuntu 24.04 LTS 64bit dependencies:
sudo apt-get update
sudo apt-get install gcc make sudo libbz2-dev tcl-dev libncurses-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev libffi-dev
# optional install
sudo apt-get install checkinstall

2.7.15:
<pre>
# PYVER=2.6.6
# PYVER=2.7.5
# PYVER=2.7.6
# PYVER=2.7.9
# PYVER=2.7.10
# PYVER=2.7.15
PYVER=2.7.18
mkdir -p ~/.src ; cd ~/.src
wget --no-check-certificate http://www.python.org/ftp/python/$PYVER/Python-$PYVER.tgz
tar -zvxf Python-$PYVER.tgz
cd Python-$PYVER
# ./configure --prefix=/opt/python-$PYVER
# make clean && make
./configure --prefix=/opt/python-$PYVER && make clean && make
sudo make install

sudo /opt/python-$PYVER/bin/python -m ensurepip --upgrade
sudo /opt/python-$PYVER/bin/pip install --upgrade pip
sudo /opt/python-$PYVER/bin/pip install requests

export PATH=/opt/python-$PYVER/bin:$PATH
echo -e "\n\nexport PATH=/opt/python-$PYVER/bin:\$PATH\n" >> ~/.bash_profile
# echo -e "\n\nexport PATH=/opt/python27/bin:\$PATH\n" >> ~/.bashrc

# NOTE: remove any old versions from your .bash_profile
</pre>

Bootstrapping the pip installer - https://docs.python.org/2/library/ensurepip.html

=== 3.x ===

3.8.16:
<pre>
#VER=3.7.0
#VER=3.8.15
#VER=3.10.9
VER=3.8.16
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/${VER}/Python-${VER}.tgz
tar -zvxf Python-${VER}.tgz
cd Python-${VER}
./configure --prefix=/opt/python${VER}
make

sudo make install

## OPTIONAL alternative install path:
sudo checkinstall
## note: 'checkinstall' makes a .deb, which you can uninstall with easier, but if you are using --prefix, you do need to
## premake the folders
sudo mkdir -p /opt/python${VER}/bin
sudo mkdir -p /opt/python${VER}/lib
shortVER=`echo $VER | cut -d '.' -f 1-2` # get short version eg. 3.10 from 3.10.9
sudo mkdir -p /opt/python${VER}/lib/python${shortVER}
sudo mkdir -p /opt/python${VER}/lib/python${shortVER}/test
# sudo checkinstall # hit Y, and enter twice
## note: older versions have problems with the virtualization of the file system, so do FS Translation = no
checkinstall --fstrans=no

export PATH=/opt/python${VER}/bin:$PATH
echo -e "\n\nexport PATH=/opt/python${VER}/bin:\$PATH\n" >> ~/.bash_profile

# have to manually update version
rm /opt/python${VER}/bin/python
ln -sfn python3.8 /opt/python${VER}/bin/python
#cp /opt/python${VER}/bin/python3.8 /opt/python${VER}/bin/python

# DEPENDENCIES
sudo apt install ...
yum install libffi-devel openssl-devel
</pre>

Note, if you want the libraries to be shared (like python-dev):
./configure --enable-shared

---

VER=3.6.9

VER=3.6.8

---

3.2.0:
<pre>
mkdir -p ~/src ; cd ~/src
wget http://www.python.org/ftp/python/3.2/Python-3.2.tgz
tar -zvxf Python-3.2.tgz
cd Python-3.2
./configure --prefix=/opt/python32
make
sudo make install

export PATH=/opt/python32/bin:$PATH
echo -e "\n\nexport PATH=/opt/python32/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.3.0:
<pre>
mkdir -p ~/src ; cd ~/src
wget http://www.python.org/ftp/python/3.3.0/Python-3.3.0.tgz
tar -zvxf Python-3.3.0.tgz
cd Python-3.3.0
./configure --prefix=/opt/python3.3.0
make
sudo make install

export PATH=/opt/python3.3.0/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.3.0/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.3.4:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/3.3.4/Python-3.3.4.tgz
tar -zvxf Python-3.3.4.tgz
cd Python-3.3.4
./configure --prefix=/opt/python3.3.4
make clean
make
sudo make install

export PATH=/opt/python3.3.4/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.3.4/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.4.0:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz
tar -zvxf Python-3.4.0.tgz
cd Python-3.4.0
./configure --prefix=/opt/python3.4.0
make
sudo make install

export PATH=/opt/python3.4.0/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.4.0/bin:\$PATH\n" >> ~/.bash_profile
</pre>

=== setuptools ===

Purpose: to install pip

Note: required for several 'python setup.py install' installations.

Python Package Index : setuptools 0.6c11 - http://pypi.python.org/pypi/setuptools

By Package Manager:
yum install python-setuptools

By Source:
#VER=2.0.1
#VER=2.2
VER=3.4.4
mkdir -p ~/.src ; cd ~/.src
wget --no-check-certificate http://pypi.python.org/packages/source/s/setuptools/setuptools-$VER.tar.gz
tar -zvxf setuptools-$VER.tar.gz
cd setuptools-$VER
# sudo python setup.py install
sudo /opt/python-2.7.9/bin/python setup.py install

Fix header:
# Only needed on older versions
sed -i 's#!/usr/bin/python#!/usr/bin/env python#g' /opt/python279/bin/easy_install

Install pip:
sudo /opt/python-2.7.9/bin/easy_install pip

Sample pip usage:
sudo /opt/python-2.7.9/bin/pip install paramiko

== IDLE and Interactive Mode ==

Interactive Mode - "When commands are read from a tty, the interpreter is said to be in interactive mode. In this mode it prompts for the next command with the primary prompt, usually three greater-than signs (>>>); for continuation lines it prompts with the secondary prompt, by default three dots (...)." [http://docs.python.org/2/tutorial/interpreter.html#invoking-the-interpreter]

Can use '_' to reference last output:
>>> 5 + 6
11
>>> print _
11

References:
* 2. Using the Python Interpreter — Python v2.7.3 documentation - http://docs.python.org/2/tutorial/interpreter.html

== Python 2.7 EOL ==

<blockquote>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
</blockquote>

== Show All Python in Path ==

compgen -c | grep -i "python"

== Code ==

=== Hello World ===

[http://en.wikipedia.org/wiki/Shebang_%28Unix%29 shebang] (also called a hashbang) of Python:
#!/usr/bin/env python
print "Hello World"

NOTE: can also just use "#!/usr/bin/python"

=== Standard Script ===

Standard Script:
<pre>
def main(args):
# do logic

if __name__ == "__main__":
sys.exit(main(sys.argv))
</pre>

Needed for things like pylint

=== White Space ===

Python uses white space to indicate code blocks. Instead of ''{'' use a '':'' and indent consistently (4 spaces).

if a == b:
print 'equal'

Convert tabs to spaces:
sed -i 's/\t/ /g' *.py

Remove trailing whitespace:
sed -i 's/\s*$//' *.py
sed --in-place 's/[[:space:]]\+$//' fileName

Clean project:
# find . -name "*py" -exec dos2unix {} \;
# find . -name "*py" -exec sed -i 's/\t/ /g' {} \;
# find . -name "*py" -exec sed -i 's/\s*$//' {} \;
find . -name "*py" -exec dos2unix {} \; -exec sed -i 's/\t/ /g' {} \; -exec sed -i 's/\s*$//' {} \;

==== Trim Tailing White Space ====

<pre>
sed -e 's/[[:space:]]*$//'
</pre>

<pre>
find . -name "*.py" -exec sed -i -e 's/[[:space:]]*$//' {} \;
</pre>

ref: [https://stackoverflow.com/questions/369758/how-to-trim-whitespace-from-a-bash-variable]

=== Print ===

print - http://docs.python.org/tutorial/inputoutput.html

Print: # include new line
print("Hello World")
print "Hello World" # python 2 only

Suppress new line: [http://stackoverflow.com/questions/493386/how-to-print-in-python-without-newline-or-space]
<pre>
print('#', end='') # python 3
print "Hello World", # python 2 only (does produce a single separating white space though)

import sys
sys.stdout.write('.') # no separating white space
</pre>

# flush output
# optional, if new line won't come for some time
import sys
sys.stdout.flush()

Multiple with separator:
print("hello", "world") # separated by spaces
print("hello", "world", sep = '.') # separated by period

Formatting:
<pre>
mystr = "abc"
myint = 5
myflt = 3.141592
print( "%s %d %.3f" % (mystr, myint, myflt) )
print( "{0} {1} {2}".format(mystr, myint, myflt) ) # string's format function (python > 2.7)
print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name
print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding
</pre>

Padded floating point decimal number: (rounds)
print "%06.2f" % 2.319 # "002.32"
print "%6.2f" % 2.311 # " 2.31"
print "{:06.2f}".format(2.319) # "002.32"
print "{:6.2f}".format(2.319) # " 2.32"

# Python 3 version where var=2.321234
print(f"{var:.2f}") # "2.32"
print(f"{var:6.2f}") # " 2.32"
print(f"{var:06.2f}") # "002.32"
print(f"{var:08.2f}") # "00002.32"
print(f"{var:08.4f}") # "002.3212"
# Python 3 version where var=2.329 - rounding
print(f"{var:06.2f}") # "002.33"

=== Comments ===

# this is a comment
# this is a comment
x = y # this is a comment

Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended)
"""
multi line comment
"""

'''
multi line comment can comment out """
'''

This can be used like a C "#if 0" concept to comment out whole sections of code.

Using false block to comment out a block code: (does not ignore bad syntax though)
if False:
# code in this block is ignored

=== doc strings ===

Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended)
"""
multi line comment
"""

=== Variable Type ===

print( type(myvar).__name__ )

Used to be able to just print the type(myvar), but this doesn't seem to work anymore :-(

=== Variable Directory ===

All of the properties and functions of a variable or object

print( dir(myvar) )

=== Help ===

dir - list functions and properties of objects:
import math
dir(math) # module
dir(dir) # function
dir("") # object - "", [], {}, etc...
dir(__builtins__) # built in functions
dir() # list imported objects, and dunder methods

Note: dunder (double under) methods determine special functionality (eg. __add__ for +)

help - manual printout: (class doc string, function doc string, data, etc)
import math
help(math) # full module info
help(len) # full function info
help(["foo"]) # full object info

Pydoc: (command line, same as 'import math, help(math)'
$ pydoc math

Documentation String: (aka docstring, doc string)
print(math.pow.__doc__) # module doc string
print(len.__doc__) # function doc string

Object or function type:
type(x)

mylist = []
isinstance(mylist, list) # True
type(s) is list # True

Note: isinstance() and type() not equivalent when objects and subclass get involved

IDLE interactive help:
help()

=== Variables ===

Basic Data Types
str = "string" # string
num = 5 # integer
flt = 5.0 # float
nothing = None # None (not the same as undefined)

Undefined. Variables start off in an 'undefined' state. It is always better to initially set the variable to None to avoid this state. To check if undefined: [http://code.activestate.com/recipes/59892-testing-if-a-variable-is-defined/] [http://code.activestate.com/recipes/59892/]
<pre>
try:
x
except NameError:
x = None
</pre>

Rules: [http://www.python.org/dev/peps/pep-0008/ PEP8 - Method Names and Instance Variables]
* any length
* letters, numbers, underscore
* first character can't be number
* case sensitive
* cannot use keywords

Multiple assignment:
x = y = z = 0 # Zero x, y and z
x, y, z = 1, "two", 3.0 # x = 1, y = "two", z = 3.0

Swap variables: (done in parallel)
a, b = b, a

Unpack variables: (requires equal items on each side)
a, b, c = [1, 2, 3]

Other Data Types:
* tuple
* list
* dictionary
* set
* objects

Everything in Python is an object that has:
* identity: id(obj)
* type: type(obj)
* value: obj
** mutable - id() stays the same on modification (dictionary, lists, objects)
** immutable - id() changes on modification as new object is created (string, integer, tuple)

==== Check if Variable Exists ====

<pre>
try:
a # does a exist in the current namespace
except NameError:
a = 10 # nope
</pre>

'a' in vars() or 'a' in globals()
'a' in vars(__builtins__)

<pre>
if hasattr(a, 'property'):
doStuff(a.property)
else:
otherStuff()
</pre>

Ref: [https://stackoverflow.com/questions/1592565/determine-if-variable-is-defined-in-python] [https://stackoverflow.com/questions/610883/how-to-know-if-an-object-has-an-attribute-in-python]

=== Numbers ===

num = 5
print "num: %d" % num

>>> print "Today's stock price: %f" % 50.4625 1
50.462500
>>> print "Today's stock price: %.2f" % 50.4625 2
50.46
>>> print "Change since yesterday: %+.2f" % 1.5 3
+1.50

Arithmetic:
+ - * /
// # integer division
% # modulus
** # exponent

Compact Arithmetic:
i+=1 # Increment - Note, there is no 'i++'
i*=2 # Multiply
i%=2 # modulus
...

Math Functions:
import math
math.ceil(x)
math.floor(x)
math.trunc(x)
math.pow(x,y)

abs(x) # built in

String interpolation:
%% % character
%d integer
%x hex
%X hex (upper)
%f float
%s string

x = 1/81
print('value: %.2f' % x) # 0.01
print('value: %.5f' % x) # 0.012345
print('%d %d %d' % (1, 2, 3))

Type check:
if not isinstance(myvar, (int, float)): ...

=== Strings ===

STRINGS ARE IMMUTABLE! (to modify convert to a list)

Strings: (String Constants)
str = "Hello"
str = 'Hello'
str = """
Hello
World
""" # can also use '''...'''

String literals can span multiple lines: (no wrapping when \ used)
hello = "hello\
world"

Length:
len("str")

Raw string: (back slashes are not escaped)
myrawstr = r"hello\nworld\
how are you"
myrawstr = r"C:\" # ERROR - limitation of what won't be escaped

Concatenation:
str = 'hot' + 'dog'
print "Hello" "World"

Can't concat numbers
str = 5 + 'test' # error
num = 5
str = num + 'test' # error
str = str(5) + str(num) + 'test' # ok

Repeated Concatenation:
str = 10 * 'ha'
str = 'ha' * 10 # same
print '-'*60 # --------------------

String Indexing:
s = 'hello'
print( s[0] ) # is 'h'
print( s[-1] ) # is 'o'
print( s[len(s) - 1 ] ) # is also 'o'

Modify Strings: [http://stackoverflow.com/questions/1228299/change-one-character-in-a-string-in-python]
s[1] = 'x' # ERROR!
ERROR: can't modify string by index

# solution:
s = list("Hello World")
s[0] = 'C'
print( "".join(s) )

String Slicing:
s = 'hello'
print( s[0:3] ) # 'hel'
i = 1
print( s[i:i+1] ) # 'e'
print( s[:3] ) # 0:3 'hel'
print( s[4:] ) # 4:end 'o'
print( s[:] ) # start:end 'hello'
print( s[-2:] ) # -2:end 'lo'

For loop:
s = 'hello'
for c in s:
print c # c = 'h' first time

Escape characters:
\\ \' \" \n \r \t

Alternate loop:
s = 'hello'
for i in range( len(s) ):
print s[i]

Ord and Char (ASCII):
ord('a') # 97
chr(97) # 'a'

0-9 : 48-57
A-Z : 65-90
a-z : 97-122

List Functions: (dir on any string)
dir('')
dir(str)

Help:
help('')
help(str)

Padding:
"a".rjust(2, '0') # '0a'
"1".ljust(4, '0') # '1000'

String Test Functions:
"HELLO".isupper() # True
"hello".islower() # True
"hello".isalpha() # True
"12.22".isdigit() # False (because of '.')
"12.22".isnumeric() # False (because of '.')
"hel2".isalnum() # True
"e" in "hello" # True
"hello".startswith('he') # True
"hello".endswith('lo') # True

Note: To check for floats: [http://stackoverflow.com/questions/354038/how-do-i-check-if-a-string-is-a-number-in-python]
<pre>
def is_number(s):
try:
float(s)
return True
except ValueError:
return False
</pre>

String Modify Functions: (sort of... Strings are immutable)
"HELLO".lower() # 'hello'
"hello".upper() # 'HELLO'
" hello ".strip() # 'hello' remove leading/tailing whitespace
" hello ".lstrip() # 'hello ' remove left whitespace
" hello ".rstrip() # ' hello' remove right whitespace
"www.com".split('.') # ['www', 'com'] <- list
"www.com".partition('.') # ('www', '.', 'com') <- tuple
"".join( ('a','b','c') ) # "abc" join strings, list or tuple
"-".join( ('a','b','c') ) # "a-b-c" join with separator
"23".zfill(4) # "0023"

String Search Functions:
"hello".find('lo') # 3 (-1 on fail)
"hello".index('lo') # 3 (exception on fail)
"hello".rfind('lo') # 3 right to left (-1 on fail)
"hello".rindex('lo') # 3 right to left (exception on fail)

Formatting: (form of concatenation)
<pre>
mystr = "abc"
myint = 5
myflt = 3.141592
print( "%s %d %.3f" % (mystr, myint, myflt) )
print( "{0} {1} {2}".format(mystr, myint, myflt) ) # pep 3101 style - string's format function (python > 2.7)
print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name
print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding
</pre>

Split multi line string:
for line in str.split('\n'): print(line)
lines = str.splitlines() # returns list
lines = str.splitlines(True) # returns list (each includes new line characters)

Input a string:
name = input('What is your name: ').strip() # python 3
name = raw_input('What is your name: ').strip() # python 2

Leet Speak:
leet_table = ''.maketrans('EIOBT', '31087')
'BE COOL. SPEAK LEET!'.translate(leet_table)

String Functions:
newstr = mystr.strip() # remove white space on both sides
newstr = mystr.rstrip() # remove white space on right side
newstr = mystr.lstrip() # remove white space on left side
newstr = mystr.replace('\n', ' ') # replace string with another
newstr = mystr.replace('jello', 'hello') # replace string with another

==== String Formatting ====

PyFormat: Using % and .format() for great good! - https://pyformat.info/

===== Zero Padding =====

3 zero padding:
"{:03}".format(1) # 003

3 zero padding, specify position:
"{1:03}".format(1, 2) # 002

Old method:
"%03d" % 2 # 002

=== chr and ord ===

Ord and Char (ASCII):
ord('a') # 97
chr(97) # 'a'

=== Range ===

Creates a list of numbers. Remember: range always ends one less than you want!

Count:
ten = 10
one = 1
range(ten) # [0,1,...,9] starts with zero by default
range(one, ten + 1) # [1,...,10] actually get 1 to 10
range(ten, one - 1, -1) # [10,9,...,1] count down
range(2, 10 + 1, 2) # [2,4,...,10] even numbers

=== drange ===

Solution for decimal range() step value:
<pre>
def drange(start, stop, step):
r = start
while r < stop:
yield r
r += step

sub_ten = drange(0.0, 1.0, 0.1)
</pre>

References:
* floating point - Python decimal range() step value - Stack Overflow - http://stackoverflow.com/questions/477486/python-decimal-range-step-value

=== Casting ===

Convert:
float(3)
str(85)
int(8.6)
round(5.5)

Check Type:
type(5) # <class 'int'>
type(5.0) # <class 'float'>
type('5') # <class 'str'>
type(None) # <class 'NoneType'>
type(print) # <class 'builtin_function_or_method'>
type(()) # <class 'tuple'>
type([]) # <class 'list'>

mytype = int
if type(1) is mytype: ...

=== Boolean Logic ===

True and False:
bol = True
bol = False

Boolean Operators: (in order precedence)
a == b
a != b
not a
a and b
a or b

Note: Short-circuit evaluated - 'and' and 'or'

Return Boolean:
return myint >= 16

Get Boolean from string: [http://stackoverflow.com/questions/715417/converting-from-a-string-to-boolean-in-python]
# Note: no built in way to do this

myString = "false"
val = (myString == "true")

True if myString=="True" else False

s in ['true', '1', 't', 'y', 'yes', 'yeah', 'yup', 'certainly', 'uh-huh']

def str2bool(v):
return v.lower() in ("yes", "true", "t", "1")

=== None ===

None is Python's version of NULL or NIL.

None is a singleton (Python only has one copy of None in the interpreter).

if myvar is None:
...

=== If Conditionals ===

Operators:
==, !=, not, and, or, <, >, <>, <=, >=, in, not in, is, is not

If statement:
<pre>
if a == b:
...
elif c < age < d:
...
else:
...
</pre>

Shorthand conditional: (ternary operator)
val = True if a == b else False

is - checks for identical objects - same id()
# None is a singleton, so this works great for None
if myvar is None:
...

Other shorthand:
<pre>
def mycmp(x, y):
if x > y: return True
else: return False
</pre>

Switch/Case statement:
* there is no built in switch/case statement, use "if ... elif ... elif ..." sequence instead

=== For Loop ===

''for'' is used for iterating over an iterable sequence (eg. lists, dictionaries, tuples). (for each)

# range(start, end + 1, skip)

for i in range(10):
print(i) # 0 .. 9

for i in range(5, 10):
print(i) # 5 .. 9

for i in range(10, 0, -1):
print(i) # 10 .. 1

for i in [1, 2, 3]: # iterate over list
print(i) # 1 2 3

Discard variable: [http://bytes.com/topic/python/answers/757444-loop-without-variable]
for _ in range(10): # repeat and discard variable (good for pep8 check)
print "hello world"

for key in my_dict.keys(): ...
for value in my_dict.values(): ...
for key, value in my_dict.items(): ...

Compact version:
for i in range(10): print(i) # 0 .. 9

Access index: [http://stackoverflow.com/questions/522563/accessing-the-index-in-python-for-loops]
# using enumeration
for idx, val in enumerate(mylist):
print idx, val

# using range
for ix in range(len(mylist)):
print mylist[ix]

# list comprehension
[ (ix, mylist[ix]) for ix in range(len(mylist))]

=== While Loop ===

i = 0
while i < 10:
i = i + 1

while True:
... # infinite loop

=== Breaking Loops ===

break statement:
while True:
if s == 'done':
break # jump out of the loop
...

continue statement: (works with while too)
for i in range(10):
if i == 2:
continue # skip 2
...

=== Functions ===

No return value: (None)
x = print('')
# x is 'None'

Note: specifying no return value is the same as this:
return None

No Operation - 'pass':
def noop():
pass

Function:
<pre>
def area(radius):
""" This is a doc string """ # can be " ... " also...
import math
return math.pi * radius ** 2
</pre>

Function Doc string:
print(area.__doc__)
foo.func_doc

Function name:
foo.func_name

Local variable:
name = "old"
def cname():
name = "new"
print(name) # is still 'old'

Global variable:
name = "old"
def cname():
global name
name = "new"
cname()
print(name) # is now 'new'

By convention starting point is: (not required)
def main():

Parameters: (with default)
def foo(x, y, z = "default"):

Note: don't use mutable for defaults, as the object is remembered across runs! (unless that is what you want)
<pre>
# each call to t() will increase list
def t(foo=[]):
foo.append('1')
print(foo)
</pre>

<pre>
# fix mutable issue
def named_param(a, foo=None):
foo = foo or []
if not foo:
foo.append(a)
</pre>

Parameters by keyword:
def foo(x = '1', y = '2', z = '3'):
...
foo(y = 'a', x = 'b') # order is not important with keywords

Trick for optional parameter:
def do(x = None):
if x == None:

Function tests - functions are callable
callable(myfunc)

=== Type Hints ===

: "Note The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers, IDEs, linters, etc."

def surface_area_of_cube(edge_length: float) -> str:
return f"The surface area of the cube is {6 * edge_length ** 2}."

Without the hints:
def surface_area_of_cube(edge_length):
...

ref: https://docs.python.org/3/library/typing.html

=== Pass ===

Pass is a null operation (noop):

Do nothing, noop, take no action:
pass

Great for creating minimal classes:
class MyEmptyClass:
pass

Greate for making stub functions/methods:
def initlog(*args):
pass # remember to implement this!

=== Slice ===

Take take a slice of strings, lists, tuples, etc...
a = [0,1,2,3,4,5]
a[0] # 0 - not a slice
a[-1] # 5 - not a slice
a[0:1] # [0]
a[1:3] # [1, 2]
a[:2] # [0, 1]
a[:-1] # [0, 1, 2, 3, 4] - all but last
a[2:] # [2, 3, 4, 5]
a[::2] # [0, 2, 4] - stride
a[::-1] # [5, 4, 3, 2, 1, 0] - reverse
range(0,10)[::2] # [0, 2, 4, 6, 8] - stride

a = '012345'
a[1:3] # '12'
a[::2] # '024'

=== [http://docs.python.org/reference/compound_stmts.html Compound Statements] ===

Compound statements: (generally discouraged)
fun1(); fun2(); fun3()

Compound with statements: [http://docs.python.org/reference/compound_stmts.html#the-with-statement]
with A() as a, B() as b: ...

#is equivalent to
with A() as a:
with B() as b:
...

[http://www.python.org/dev/peps/pep-0343 PEP 0343] - The “with” statement

f = open('file')
with f: ...

Try: (quick ignore all exceptions, very dangerous)
try: my_broken_function()
except: pass

=== Passing Command Line Arguments ===

Parameters are stored in the 'sys.argv' variable:
import sys
print( sys.argv[0] ) # name of script
print( sys.argv[1] ) # first argument
for arg in sys.argv: print( arg ) # list all arguments

Print script name:
os.path.basename(sys.argv[0])

Print path to script:
print sys.path[0]

Paths: [http://code.activestate.com/recipes/474083-get-the-path-of-the-currently-executing-python-scr/]
<pre>
import os,sys
print "CWD: ",os.getcwd()
print "Script: ",sys.argv[0]
print ".EXE: ",os.path.dirname(sys.executable)
print "Script dir: ", os.path.realpath(os.path.dirname(sys.argv[0]))
pathname, scriptname = os.path.split(sys.argv[0])
print "Relative script dir: ",pathname
print "Script dir: ", os.path.abspath(pathname)
</pre>

os.path.realpath(__file__)

print 'sys.argv[0] =', sys.argv[0]
pathname = os.path.dirname(sys.argv[0])
print 'path =', pathname
print 'full path =', os.path.abspath(pathname)

=== Environment Variables ===

import os
a = os.environ.get('MYVAR') # get environment var
a = os.environ.get('MYVAR','default') # get environment var
a = os.environ['MYVAR'] # get environment var
os.environ['MYVAR'] = 'something' # set environment var
os.environ.update({'MYVAR':'something'}) # set environment var

for env in os.environ:
print "%s = %s" % (env, os.environ[env])

with open("environment.txt", "w") as f:
for env in os.environ:
f.write("%s = %s\n" % (env, os.environ[env]))

=== PYTHONPATH ===

Environment Search path:
PYTHONPATH=...

From Code:
sys.path.append('...')

Get site-package directory from shell: [http://stackoverflow.com/questions/122327/how-do-i-find-the-location-of-my-python-site-packages-directory]
python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

=== Import and Modules ===

Import and Modules - http://docs.python.org/tutorial/modules.html

Import module:
import package # standard import
import package.code # nested name space
from package import code # import specifics from module
import module # standard import
from math import sin # import specifics from module
from module import myfunc as fun1 # alias function/object
import math as other_math # alias module
import longname as ln # alias module
from module import * # BAD BAD BAD!

import math
math.pow(x,y)

Group and organize by:
* stdlib libraries
* local libraries
* 3rd party libraries

Remove import:
del math

from math import *
pow(x,y)

from math import pow
pow(x,y)

import math as math_lib # rename
math_lib.pow(x,y)

List functions in module:
import math
dir(math)

List all built in functions:
dir(__builtins__)

Note: Modules have the extension .py

Simply module Example:

# mymod.py:
def hello():
""" this is a doc string """
print "Hello World"

# Use the module:
import mymod
mymod.hello():

# Use everything in the the module, except for names beginning with '_': (bad practice!)
from mymod import *
hello()

# Use specific functions in the module:
from mymod import hello, hi
hello()

# Function assignment:
h = mymod.hello
h()

# module name
mymod.__name__ # mymod
__name__ # used inside of module will print current module or "__main__" if directly called

Note: careful when building own modules, as the import literally imports and executes the code, so any code not in functions is executed!

Check if being called directly or indirectly:
if __name__ == "__main__":
# execute tests...

Standard Script Setup:
<pre>
def main(args):
# do logic
if __name__ == "__main__":
sys.exit(main(sys.argv))
</pre>

Easter egg:
# The Zen of Python
import this

Search path:
* environment variable 'PYTHONPATH' (on Unix, this is usually .:/usr/local/lib/python)
* sys.path (list of paths, initialized to PYTHONPATH)
** sys.path.append('path')

Module folder: [http://stackoverflow.com/questions/391879/organising-my-python-project] [http://stackoverflow.com/questions/279237/python-import-a-module-from-a-folder] [http://stackoverflow.com/questions/72852/how-to-do-relative-imports-in-python]
/kenlib/kenmod.py
# so you can do "import kenlib.kenmod"
# create file '__init__.py' in kenlib/

Check if package exists:
<pre>
try:
import argparse
except ImportError:
print >> sys.stderr, "Please install argparse from http://code.google.com/p/
argparse."
exit(1)
</pre>

=== ifmain ===

if __name__ == '__main__':
sys.exit(main(sys.argv))

if __name__ == '__main__':
sys.exit(main(sys.argv[1:]) or 0)

=== Lists ===

* Lists can be modified (mutable)
* Tuples are constant (immutable)

List: (mutable or modifiable)
x = [] # empty list
x = [5] # singleton list

s = list("Hello World")
print(s) # s = ['H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd']
s[0] = 'X'

s.append(x) # append x
s.insert(i, x) # insert x at i
del(s[i]) # delete item at i
s.sort() # sort (modify in place!)
s.reverse() # reverse
s.index(x) # index of x in s
new = old[:] # create copy of list
new = list(old) # create copy of list

Check if item in list:
if x in s: ...

Filter list of strings based on contents: [https://stackoverflow.com/questions/2152898/filtering-a-list-of-strings-based-on-contents]
res = [k for k in lst if 'ab' in k]

List comprehension
[n * n for n in range(1, 11)] # squares of numbers 1 to 10
[c for c in 'pizza'] # list of string 'pizza'
[c.upper() for c in 'pizza'] # list of string 'pizza' uppercase
[n for n in nums if n > 0] # filters out only positive numbers in nums list
[fname for fname in os.listdir(path) # on multiple lines
if os.path.isfile(fname)
if fname.endswith('.py')]

Enumeration
for index, value in enumerate(mylist): print index, value

Length:
len(x)

Concatenation:
x + s
x * 2

Convert list to string:
"".join(mylist) # no separator
" ".join(mylist) # space separator
",".join(mylist) # comma separator

Test Membership
x in s

Multi dimensional list:
mylist = [ [0,"hi"], [1,"bye"] ]
print( mylist[0][1] )

Sorting: (see lambda)
data.sort() # alpha numerica sort
data.reverse() # alpha numerica sort

<pre>
# custom sort
data = [1, 5, 3, 9]
def mylistcmp(x, y):
if x == y: return 0
if x < y:
return -1
else: # swap 1 and -1 here to reverse
return 1

data.sort(cmp=mylistcmp)
</pre>

Slices:
lst = ['1a', '2b', '3c', '4d', '5e', '6f']
lst[0] # '1a'
lst[-1] # '6f' last item
lst[0:2] # ['1a', '2b']
lst[0:2] = ['11', '22'] # lst = ['11', '22', '3c', '4d', '5e', '6f'] - assignment to slice
lst[0:2] = [] # remove items
lst[0:0] = ['1a', '2b'] # insert items (at beginning)
lst[-1:-1] = ['7g', '8h'] # insert items (at item BEFORE last), not really what you want, use append()
lst[-1:] = ['7g', '8h'] # replace last item
lst[:] = [] # clear entire list
del(lst[0:2]) # delete slice items

Note: it is not safe to modify a sequence while being iterated over. Create a copy first:
for x in mylst[:]: # make a slice copy of the entire list
if len(x) > 6: a.insert(0,x)

Make list from tuple:
list( range(10) )

Merge multiple lists: [http://www.testingreflections.com/node/view/4930]
<pre>
def merge(seq):
merged = []
for s in seq:
for x in s:
merged.append(x)
return merged

# sample usage:
foo = [['a', 'b'],['c'],['d', 'e', 'f']]
print merge(foo)
>>>['a', 'b', 'c', 'd', 'e', 'f']
</pre>

=== Tuple ===

Tuple: (immutable or constant)
x = () # empty tuple
x = (5,) # singleton tuple
x = (5) # NOTE: integer with parenthesis, not tuple

s = tuple("Hello World")
print(s) # s = ('H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd')
# ERROR: s[0] = 'X'

Make list from tuple:
list( range(10) )

Use in conditionals:
if answer in ('y', 'ye', 'yes'): ...

Convert tuple string to tuple: [http://stackoverflow.com/questions/3945856/converting-string-to-tuple-and-adding-to-tuple]
x = "(1,2,3)"
t = tuple(int(v) for v in re.findall("[0-9]+", x))

Another Convert tuple string to tuple: [http://stackoverflow.com/questions/3945856/converting-string-to-tuple-and-adding-to-tuple]
import ast
ast.literal_eval("(1,2,3,4)") # (1,2,3,4)

=== Named Tuple ===

What are "named tuples" in Python? - Stack Overflow - http://stackoverflow.com/questions/2970608/what-are-named-tuples-in-python

<pre>
from collections import namedtuple
Point = namedtuple('Point', 'x y')
pt1 = Point(1.0, 5.0)
pt2 = Point(2.5, 1.5)
print p1 # point(x=1, y=5)
print p1.x # 1.0
print p1[0] # 1.0
</pre>

=== Dictionary ===

NOTE: Dictionaries are also known as associative arrays, maps or hash tables.

Dictionary: (mutable or modifiable)

color = {} # empty dictionary
color = {'red' : 1, 'blue', 2}
print( color['red'] ) # 1
print( color.get('red') ) # 1
print( color.get('red', 'default') ) # red if found, or return 'default'
color['green'] = 3 # able to add new item on the fly!
'red' in color # True - check if index exists
del(color["red"]) # delete item
len(color) # 2 - count of items

List in dictionary
<pre>
d.setdefault(key,[]).append(member) # tricky!

# more tricky: set default type as list
from from collections import defaultdict
d = defaultdict(list)
d[key].append(member)
</pre>

Enumeration:
for index, key in enumerate(mylist): print index, key

functions:
d.clear()
d.copy()
d.get(key)
d.get(key, default_val)
d.update(e) # update with (key, value) pairs in e
d.setdefault(key, value) # only update if not exist, and return set value

Views: (adjust on the fly as dictionary changes)
d.items() (key, values)
for k, v in d.items(): print(k, v)

d.keys()
for k in d.keys(): print(k)

d.values()
for v in d.values(): print(v)

Sorting: (see also lambda)
<pre>
# custom sort of dictionaries in list
data = [dict(number=x) for x in '036149'] # create list of dictionaries
def mydictcmp(x, y):
if x['number'] == y['number']: return 0
if x['number'] < y['number']:
return -1
else: # swap 1 and -1 here to reverse
return 1

# data.sort(cmp=mydictcmp) # modifies data

new_data = sorted(data, cmp=mydictcmp) # better solution
</pre>

Sorting:
<pre>
my_dict = {'a':1, 'b':4, 'c':3, 'd':9}
my_dict_keys = my_dict.keys()

def my_dict_sort(x, y):
if my_dict[x] == my_dict[y]: return 0
if my_dict[x] < my_dict[y]: return -1
if my_dict[x] > my_dict[y]: return 1

#sorted_my_dict = sorted(my_dict_keys, cmp=my_dict_sort, reverse=True)
sorted_tuple = sorted(my_dict_keys, cmp=my_dict_sort)
print "Sorted Dictionary:"
for key in sorted_tuple:
print "%s: %d" % (key, my_dict[key])

# a: 1, c: 3, b: 4, d: 9,
# reverse: d: 9, b: 4, c: 3, a: 1,
</pre>

Sort and keys:
keys = sorted(mydict.keys())

Sort by value: [http://stackoverflow.com/questions/613183/python-sort-a-dictionary-by-value]
import operator
x = {1: 2, 3: 4, 4:3, 2:1, 0:0}
sorted_x = sorted(x.iteritems(), key=operator.itemgetter(1))

Sort options:
# LIST.sort(cmp=None, key=None, reverse=False) -- stable sort *IN PLACE*;
# sorted(iterable, cmp=None, key=None, reverse=False)

Another: (works well!)
sorted(my_dict.items(), key=lambda x: x[1])

See: http://wiki.python.org/moin/HowTo/Sorting/

=== Sets ===

Sets are special lists with unique values (good for removing duplicates)
lst = [1, 1, 6, 8, 1, 5]
s = set(lst)
print(s) # set([8, 1, 5, 6])
s.add(item)
item in s # true
s.remove(item)

Union (return combination of all):
s.union(t)

Intersection (return matching):
s.intersection(t)

Symmetric Difference (return only items not matching either, inverse of intersection):
s.symmetric_difference(t)
list( set(mylist).symmetric_difference(set(mylist2)) )

<pre>
Operation Equivalent Result
len(s) cardinality of set s
x in s test x for membership in s
x not in s test x for non-membership in s
s.issubset(t) s <= t test whether every element in s is in t
s.issuperset(t) s >= t test whether every element in t is in s
s.union(t) s | t new set with elements from both s and t
s.intersection(t) s & t new set with elements common to s and t
s.difference(t) s - t new set with elements in s but not in t
s.symmetric_difference(t) s ^ t new set with elements in either s or t but not both
s.copy() new set with a shallow copy of s
</pre>

References:
* sets — Unordered collections of unique elements - http://docs.python.org/library/sets.html

=== Files ===

Open file, get file descriptor:
fd = open('/dev/null')

File Modes:
'r' read (default)
'w' write (truncate and create if needed)
'a' append (create if needed)

'b' binary
't' text (default)

'+' read and write

Common Functions: (fd)

f.read() # read whole file to string
f.read(n) # read n bytes from file to string
f.readline() # read single line as string (while loop)
f.readlines() # read all lines to list

f.write(str) # write string to file (does not auto include new line)
f.writelines(list) # write list to file (does not auto include new lines)

f.seek(0) # seek to start of file
f.seek(pos) # seek to position
f.truncate() # truncate file

f.close() # close file (fd)
f.closed # True/False flag that reports if file is closed

Help:
help(open('/dev/null'))
pydoc file

Read Text File Line by Line
f = open(fname, 'r') # ('r' optional)
for line in f: # or f.readlines()
print(line)
f.close() # optional

Write list to file (with new lines):
f.writelines('\n'.join(mylist))

One liner read whole file:
print( open( fname, 'r' ).read() )

Write text file:
# f = open(fname, 'a') # append text file
f = open(fname, 'w')
f.write("Hello\n")
f.write( str(10) ) # convert numbers to strings first
f.writelines(["line one\n", "line two\n"])
f.close

Read binary file:
def is_gif(fname):
f = open(fname, 'br') # not sure 'b' is correct
first4 = tuple(f.read(4))
return first4 == (0x47, 0x49, 0x46, 0x38)

Read all characters of file:
c = f.read(1)
while c:
# if c == ... logic
c = f.read(1)

Reading file in while loop: [http://ubuntuforums.org/showthread.php?t=1092300]
<pre>
f=open("file")
while True:
line=f.readline()
if not line: break
print line
f.close()
</pre>

with - implicit close - [http://www.python.org/dev/peps/pep-0343/ PEP 343 -- The "with" Statement]: [http://docs.python.org/reference/compound_stmts.html#the-with-statement] (python >= 2.5)
with open("/etc/passwd") as f:
# ..., f.read(), etc

With statement: (promptly close open handle at end of block)
with open(fname, 'r') as f:
for ...

<pre>
# eqivalent to:
f = open("/etc/passwd")
f.__enter__()
try:
...
finally:
f.__exit__()
</pre>

cat: (for grabbing a configuration line)
<pre>
def cat(filename):
with open(filename) as f:
return f.readline().strip()
</pre>

=== With ===

PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/

'with' structure:
<pre>
with EXPR as VAR:
BLOCK
</pre>

translates to:
<pre>
VAR = EXPR
VAR.__enter__()
try:
BLOCK
finally:
VAR.__exit__()
</pre>

Note: The "as VAR" part is optional. Good for locking:
<pre>
with locking(myLock):
BLOCK
</pre>

with - implicit close - [http://www.python.org/dev/peps/pep-0343/ PEP 343 -- The "with" Statement]: [http://docs.python.org/reference/compound_stmts.html#the-with-statement] (python >= 2.5)
with open("/etc/passwd") as f:
...

With statement: (promptly close open handle at end of block)
with open(fname, 'r') as f:
for ...

See also [[#Files]]

References:
* PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/
* PEP 340 -- Anonymous Block Statements - http://www.python.org/dev/peps/pep-0340/

=== Try Exception Handling ===

Exception Handling - http://docs.python.org/release/2.6.6/tutorial/errors.html

throw execption:
raise [EXCEPTION]
raise IOError("This is a test!")

Catch exceptions:
try:
except ( [EXCEPTION], [EXCEPTION] ):
except [EXCEPTION]:
except [EXCEPTION] as err: # use "[EXCEPTION], err" in python < 2.6
print("Error: {}".format(err))
except:
raise # reraise caught error that is not handled
else: # if no exception caught
finally: # always executed

Note: Usually a good idea to re-raise exception if you don't handle it (just 'raise')

General Exception: (all exceptions inherit from Exception)
try:
raise Exception('spam', 'eggs')
except Exception as e:
pass

Custom Exception Class:
<pre>
class MyError(Exception):
"""My Error Exception"""

class MyException(Exception):
pass

rasie MyError("My Error")
</pre>

<pre>
class ValidationError(Exception):
def __init__(self, message, Errors):

# Call the base class constructor with the parameters it needs
Exception.__init__(self, message)

# Now for your custom code...
self.Errors = Errors
def __str__(self):
return "MyErrors - %s" % self.Errors
</pre>

Rename standard exception:
<pre>
class Failure(StandardError):
"""Exception for unexpected failures."""
</pre>

Note: a good place to hide the exception classes in the __init__.py file.

==== exception line number ====

Python When I catch an exception, how do I get the type, file, and line number? - Stack Overflow [http://stackoverflow.com/questions/1278705/python-when-i-catch-an-exception-how-do-i-get-the-type-file-and-line-number]
<pre>
import sys, os

try:
raise NotImplementedError("No error")
except Exception as e:
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print(exc_type, fname, exc_tb.tb_lineno)
</pre>

==== traceback ====

<pre>
import traceback
try:
...
except Exception as E:
...
print 'Traceback: %s', traceback.format_exc()
</pre>

Catching multiple exceptions and getting properties from the exception:
<pre>
import traceback
import sys

class MyError(Exception):
"""My Error Exception"""

try:
#raise ValueError("a value error") # <type 'exceptions.ValueError'>
raise MyError("a my error") # <class '__main__.MyError'>
raise Exception("won't be caught") # <type 'exceptions.Exception'>
except (ValueError, MyError) as e:
print type(e) # <class '__main__.MyError'>
print e # __str__() = a my error
msg = str(e) # msg = 'a my error'
print e.__doc__ # My Error Exception
print e.__str__() # a my error
print e.__repr__() # MyError('a my error',)
print e.args # ('a my error',)
x = e # __getitem__() allows args to be unpacked directly
# x, y = e # if two args were passed, use len(e.args) to see count

# manually print stack trace (import traceback, sys)
print "Exception in user code:"
print '-'*60
traceback.print_exc(file=sys.stdout)
# stack_trace = traceback.format_exc()
print '-'*60
</pre>

=== Object Oriented Programming ===

Terms:
* encapsulates
* inheritance
* polymorphism
* method overriding

==== class ====

class animal.py:
<pre>
class Animal(object): # default "(object)" not required
" Animal class "
def __init__(self, name):
self.name = name # name is private variable
self.__iam = "animal" # __iam is private variable
def __str__(self): # string representation
return "name = '%s'" % (self.name)
def __repr__(self): # object representation (just typing instance in IDLE, or repr())
return 'Animal(%s)' % str(self) # returns __str__
def speak(self, msg='Nothing to say'):
print(msg)
</pre>

Note: if no __str__ defined, but __repr__ is, then: __str__ = __repr__

Usage:
<pre>
from animal import Animal
a = Animal('good')
print(a.__doc__) # Animal class
print(a) # name = 'good'
a # Animal(name = 'good')
print(a.name) # good
a.speak('hi') # hi
print(a._Animal__iam) # animal - access to private variable (shouldn't do)
</pre>

subclass dog.py:
<pre>
from animal import Animal
class Dog(Animal):
" Dog class "
def __init__(self, name):
self.name = name # name is private variable
self.__iam = "dog" # __iam is private variable
# NOTICE: no __str__, inherited from super class
def __repr__(self): # override to reflect "dog"
return 'Dog(%s)' % str(self) # returns __str__
def speak(self, msg='Nothing to say'): # override method
super(Dog, self).speak('Dog says: ' + msg) # call super class method
</pre>

Subclass usage:
<pre>
from dog import Dog
d = Dog('bad')
print(d.__doc__) # Dog class
print(d) # name = 'bad'
d # Dog(name = 'bad')
print(d.name) # bad
d.speak('hi') # Dog says: hi
print(d._Dog__iam) # dog - access to private variable (shouldn't do)
</pre>

Super: [http://stackoverflow.com/questions/576169/understanding-python-super-with-init-methods]
# python 3.0
super().__init__()

# python 2.7
super(self.__class__, self).__init__()
super([CLASSNAME], self).__init__()

<pre>
class Rectangle(Polygon):
def __init__(self, id, width, height):
# super(Rectangle, self).__init__(id)
super(self.__class__, self).__init__(id)
self.shape = (width, height)
</pre>

Get class name:
self.__class__.__name__
obj.__class__.__name__

Full name:
obj.__module__ + "." + obj.__class__.__name__

Private variables: (uses name mangling)
self.__age
# direct access: _classname__varname (eg. p._Person__age = 44)

Tests:
isinstance(myobj, myclass) # check if myobj is a class or subclass of myclass
issubclass(A, B) # check if A is subclass of B
issubclass(Exception, BaseException) # True

Multiple Inheritance: (methods searched in order)
class DerviedClass(Base1, Base2, Base3):
pass

Standard startup script:
class MyClass():
...
@classmethod
def main(cls, argv=None):
...
# return cls() # creator
if __name__ == "__main__";
sys.exit(MyClass.main(sys.argv))

Raise an error if a method is not overridden (not implemented)
def _get_dist(self):
"""Find linux disto. Implemented in subclasses."""
raise NotImplementedError

==== Multiple Constructors ====

<pre>
class Animal(object):

def __init__(self, name):
self.name = name

@classmethod
def create_fido(cls):
return cls("fido")
</pre>

==== Descriptor Classes ====

"In general, a descriptor is an object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol: __get__(), __set__(), and __delete__(). If any of those methods are defined for an object, it is said to be a descriptor." [http://docs.python.org/2/reference/datamodel.html#implementing-descriptors]

<pre>
object.__get__(self, instance, owner)

Called to get the attribute of the owner class (class attribute access) or of an instance of that
class (instance attribute access). owner is always the owner class, while instance is the instance
that the attribute was accessed through, or None when the attribute is accessed through the owner.
This method should return the (computed) attribute value or raise an AttributeError exception.

object.__set__(self, instance, value)

Called to set the attribute on an instance instance of the owner class to a new value, value.
</pre>

Example: [http://stackoverflow.com/questions/3798835/understanding-get-and-set-and-python-descriptors]
<pre>
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)

class Temperature(object):
celsius = Celsius()

t = Temperature()
t.celsius = 5 # calls Celsius.__set__
print t.celsius # calls Celsius.__get__
</pre>

Note: Also appears in 'pydoc' under "Data descriptors defined here" with object's docstring. Trick: the __doc__ can be overridden in the __init__ method, which makes for dynamic docstrings!

References:
* Descriptor HowTo Guide — Python v2.7.5 documentation - http://docs.python.org/2/howto/descriptor.html
* 3.4.2.2. Implementing Descriptors¶ - 3. Data model — Python v2.7.5 documentation - http://docs.python.org/2/reference/datamodel.html#implementing-descriptors

=== [http://docs.python.org/reference/simple_stmts.html#the-exec-statement exec code] ===

DANGEROUS!

Execute code in string:
exec( 'print("hi")' )

Example of executing a startup script for IDLE:
if os.path.isfile('.pythonrc.py'): exec(open('.pythonrc.py').read())

import os
filename = os.environ.get('PYTHONSTARTUP')
if filename and os.path.isfile(filename):
exec(open(filename).read())

=== eval ===

VERY DANGEROUS

Will execute a string as though it were pure python code!

eval("print 'hello'")

=== Common Header Format ===

<pre>
__author__ = "software team"
__date__ ="$Feb 15, 2011 1:42:17 PM$"
__version__ = "1.1.0.441 bandelier@66d9fe1edf3e"
</pre>

<pre>
Next should be authorship information. This information should follow this format:

__author__ = "Rob Knight, Gavin Huttley, and Peter Maxwell"
__copyright__ = "Copyright 2007, The Cogent Project"
__credits__ = ["Rob Knight", "Peter Maxwell", "Gavin Huttley",
"Matthew Wakefield"]
__license__ = "GPL"
__version__ = "1.0.1"
__maintainer__ = "Rob Knight"
__email__ = "rob@spot.colorado.edu"
__status__ = "Production"
</pre>
Source: Python Coding Guidelines - http://bayes.colorado.edu/PythonGuidelines.html

These will show up in the help() like such:
<pre>
Help on module test:

NAME
test

FILE
/home/kenneth/test.py

FUNCTIONS
test()

DATA
__author__ = 'test author'
__version__ = '1.0'

VERSION
1.0

AUTHOR
test author
</pre>

References:
* Python: What is the common header format? - Stack Overflow - http://stackoverflow.com/questions/1523427/python-what-is-the-common-header-format

=== Common Environment Variables ===

Command line and environment — Python v2.7.2 documentation - http://docs.python.org/using/cmdline.htm

PYTHONUNBUFFERED
: If this is set to a non-empty string it is equivalent to specifying the -u option. [http://docs.python.org/using/cmdline.html#envvar-PYTHONUNBUFFERED]
: -u - Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
* See also stdout - Python output buffering - Stack Overflow - http://stackoverflow.com/questions/107705/python-output-buffering
export PYTHONUNBUFFERED=1
set PYTHONUNBUFFERED=1

PYTHONVERBOSE
:If this is set to a non-empty string it is equivalent to specifying the -v option. If set to an integer, it is equivalent to specifying -v multiple times.
: -v - Print a message each time a module is initialized, showing the place (filename or built-in module) from which it is loaded.
export PYTHONVERBOSE=1
set PYTHONVERBOSE=1

PYTHONPATH
: Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored. [http://docs.python.org/using/cmdline.html#envvar-PYTHONPATH]
export PYTHONPATH=/path1/:path2/
set PYTHONPATH=C:\path1\;C:\path2\

=== Queue ===

Good for threading.

"The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics." [http://docs.python.org/2/library/queue.html]

<pre>
import Queue

bucket = Queue.Queue()
bucket.put("something")

while True:
try:
item = bucket.get(block=False)
except Queue.Empty:
pass
else:
# do stuff with queue
</pre>

Important methods:
Queue.qsize()
Queue.empty()
Queue.put(item[, block[, timeout]])
Queue.get([block[, timeout]])

-

Example: http://docs.python.org/2/library/queue.html
<pre>
def worker():
while True:
item = q.get()
do_work(item)
q.task_done()

q = Queue()
for i in range(num_worker_threads):
t = Thread(target=worker)
t.daemon = True
t.start()

for item in source():
q.put(item)

q.join() # block until all tasks are done
</pre>

-

"The module implements three types of queue, which differ only in the order in which the entries are retrieved. In a FIFO queue, the first tasks added are the first retrieved. In a LIFO queue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the heapq module) and the lowest valued entry is retrieved first."

Queue Types:
class Queue.Queue(maxsize=0) # FIFO Queue
class Queue.LifoQueue(maxsize=0) # LIFO Queue
class Queue.PriorityQueue(maxsize=0) # Priority Queue

References:
* 8.10. Queue — A synchronized queue class — Python v2.7.5 documentation - http://docs.python.org/2/library/queue.html

== Recipes ==

=== Remove Numbers from String ===

Using list comprehension: [http://stackoverflow.com/questions/12851791/removing-numbers-from-string]
<pre>
result = ''.join([i for i in s if not i.isdigit()])
</pre>

Using pattern matching:
<pre>
>>> re.compile(r'\D+').findall("test 1234 test 1234")
['test ', ' test ']

>>> re.compile(r"\d").sub("", "test 1234 test 1234")
'test test '
</pre>

=== Retry Wrapper ===

<pre>
import functool
class SomethingRetry:

def retry_with_backoff(retries: int, base_delay: float, exceptions: tuple[type[Exception], ...] = (Exception,)) -> callable:
"""
Retries a function with exponential backoff, doubling on each attempt.
Arguments:
retries: Number of retry attempts.
base_delay: Initial delay between retries in seconds.
exceptions: Tuple of exceptions to catch and retry.
"""

def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(1, retries + 1):
try:
return func(*args, **kwargs)
except exceptions as e:
print(f"Warning: Attempt {attempt} failed with error: {e}")
if attempt == retries:
print("Error: All retry attempts failed.")
raise
sleep_time = base_delay * (2 ** (attempt - 1))
print(f"Notice: Retrying in {sleep_time} seconds...")
time.sleep(sleep_time)
return wrapper
return decorator

@retry_with_backoff(retries=6, base_delay=10, exceptions=(Exception,))
def do_something(self, sample_param: str):
</pre>

== Standard Libraries ==

=== user input ===

<pre>
line = raw_input() # Python 2
# line = input() # Python 2
line = input('What is your name: ')
print(": " + line.strip() + " :")
</pre>

=== sys - stdout stderr stdin ===

STDOUT:
import sys
sys.stdout.write("Hello ") # does not write new line
sys.stdout.write("World\n")

STDERR:
import sys
sys.stderr.write("ERROR!\n")

STDIN:
import sys
line = sys.stdin.readline()

line = input() # use raw_input() in Python 2
line = input('What is your name: ')
print(": " + line.strip() + " :")

Read piped stdin line by line: [http://stackoverflow.com/questions/1450393/how-do-you-read-from-stdin-in-python]
# cat /etc/passwd | ./readlines.py # linux
# type file.txt | python readlines.py # windows

# fileinput.input() - returns fileinput object which is iterative
import fileinput
for line in fileinput.input():
print(": " + line + " :")

# sys.stdin.readlines() - returns list of strings
import sys
for line in sys.stdin.readlines():
print(": " + line + " :")

# sys.stdin - returns object which is iterative
import sys
for line in sys.stdin:
print(": " + line + " :")

# ugly while loop: [http://ubuntuforums.org/showthread.php?t=1092300]
while True:
try:
s = raw_input("Say something ")
except EOFError:
break
print ": %s" % s

Check if stdin has any waiting data: [http://stackoverflow.com/questions/3762881/how-do-i-check-if-stdin-has-some-data]
<pre>
import sys
import select

# if select.select([sys.stdin,],[],[],0.0)[0]: # alternative old method

if not sys.stdin.isatty():
print "Have data!"
print sys.stdin.readlines()
else:
print "No data"
</pre>

<pre>
if len(sys.argv) > 1:
message = ' '.join(sys.argv[1:])
elif not sys.stdin.isatty():
# read from pipe in
message = sys.stdin.readline()
else:
# read from user
sys.stdout.write("Message: ")
message = raw_input().strip()
</pre>

=== [http://docs.python.org/library/re.html Regular Expressions] ===

Import:
import re # user regular expressions

Match vs Search:
re.match("cde", "abcdef") # no match (match searches beginning of string, first match)
re.match("abc", "abcdef") # match (match searches beginning of string, first match)
re.search("cd", "abcdef") # match (matches any location, first match)

Both re.match() and re.search() return objects that are not intuitive to use. You can check the object's group() method for what matched, but I prefer re.findall().

Match:
s = 'done'
if re.match('done|quit', s) != None # True

Search:
s = 'done'
if re.search('do', s) != None # True

Replace:
msg = 'jello world'
msg = re.sub('jello', 'hello', msg)

For case intensive searches append 3rd parameter "re.I".

Find all:
re.findall("[a-z]+", "aaa111bbb222") # ['aaa', 'bbb']
re.findall("111([a-z]+)", "aaa111bbb222") # ['bbb']
re.findall("hello", "HELLO", re.I) # ['HELLO'] - Case Insensitive

Split:
>>> re.split('[a-z]+','111aaa222bbb')
['111', '222', '']

>>> re.split('([a-z]+)','111aaa222bbb') # parenthesis cause return of all
['111', 'aaa', '222', 'bbb', '']

Replacement Substitution:
# re.sub(pattern, repl, string[, count, flags])
re.sub(r'\\', r'/', r'\\data\files')

Compiled regular express for repeated use: (performance)
valid = re.compile(r"[0-9]+")
valid.match("aaa111bbb") # not valid
match = valid.search("aaa111bbb222") # valid
print(match.group()) # what matched: '111'

=== Date and Time ===

Sleep in seconds:
import time
time.sleep(seconds)

Use the following functions to convert between time representations:
<table border="1">
<tr><th>From</th><th>To</th><th>Use</th></tr>
<tr><td>seconds since the epoch</td><td>struct_time in UTC</td><td>gmtime()</td></tr>
<tr><td>seconds since the epoch</td><td>struct_time in local time</td><td>localtime()</td></tr>
<tr><td>struct_time in UTC</td><td>seconds since the epoch</td><td>calendar.timegm()</td></tr>
<tr><td>struct_time in local time</td><td>seconds since the epoch</td><td>mktime()</td></tr>
</table>

Date formats:
<pre>
>>> time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
'2014-04-04 12:22:43'

>>> time.time()
1396635763.0581281

>>> datetime.datetime.now()
datetime.datetime(2014, 4, 4, 12, 22, 43, 58456)
# datetime(year, month, day, hour, minute, second, microsecond)

>>> time.localtime()
time.struct_time(tm_year=2014, tm_mon=4, tm_mday=4, tm_hour=12, tm_min=22, tm_sec=43, tm_wday=4, tm_yday=94, tm_isdst=1)
</pre>

Current epoch time:
int(time.time())

Time zone offset:
standard_tz_offset = - time.timezone / 3600 # 7
daylight_tz_offset = - time.altzone / 3600 # 6

Is daylight saving time: [http://stackoverflow.com/questions/2881025/python-daylight-savings-time]
import time
time.localtime()
_.tm_isdst # 0 or 1

mst_epoch_time = time.time() # server in MST/MDT
if time.localtime().tm_isdst: # are we in daylight saving time?
utc_epoch_time = mst_epoch_time + time.altzone
else:
utc_epoch_time = mst_epoch_time + time.timezone

Format current time: [http://docs.python.org/2/library/time.html#time.strftime]
time.strftime('%d%b%y') # '12May11'
time.strftime('%Y.%m.%d %H:%M:%S') # '2013.08.09 14:01:28'

Time conversions:
<pre>
# epoch to str
epoch_time = time.time() # 1375992088.7867229
dt = datetime.datetime.fromtimestamp( epoch_time ) # datetime.datetime(2013, 8, 8, 14, 1, 28, 786723)
date_str = dt.strftime("%Y-%m-%d %H:%M:%S") # '2013-08-08 14:01:28'
tz_offset = - time.altzone / 3600
if tz_offset < 0:
tz_offset_str = "-%02d00" % abs(tz_offset)
else:
tz_offset_str = "+%02d00" % abs(tz_offset)
print date_str + " " + tz_offset_str # '2013-08-08 14:01:28 -0600'
</pre>

<pre>
# UTC epoch to str
gm_epoch_time = calendar.timegm(time.gmtime()) # 1375992114
dt = datetime.utcfromtimestamp( gm_epoch_time ) # datetime.datetime(2013, 8, 8, 20, 1, 54)
dt.strftime("%Y-%m-%d %H:%M:%S +0000") # '2013-08-08 20:01:54 +0000'
dt.strftime("%Y-%m-%dT%H:%M:%SZ") # '2013-08-08 20:01:54 +0000'
</pre>

<pre>
# Local time str to epoch
iso_string='2013-08-08T14:01:28'
epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S' ) ) # 1375992088.0

iso_string='2013-08-08T14:01:28 MDT'
epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S %Z' ) ) # 1375992088.0

# not sure how to specify numerical time zone and pull it in!

# UTC (Zulu) str to epoch, and back again
iso_string = '2013-08-08T20:01:54Z'
epoch_time = calendar.timegm( time.strptime( iso_string.replace('Z', 'GMT'), '%Y-%m-%dT%H:%M:%S%Z' ) )
print epoch_time # 1375992114
print "%sZ" % datetime.utcfromtimestamp(timestamp).isoformat() # 2013-08-08T20:01:54Z
</pre>

<pre>
# Convert epoch from Local to UTC
epoch_time = 1375992114
tz_offset = - time.altzone / 3600
gm_epoch_time = epoch_time - tz_offset * 3600 # 1376013714
</pre>

<pre>
# Convert epoch from UTC to Local
gm_epoch_time = 1376013714
tz_offset = - time.altzone / 3600
epoch_time = gm_epoch_time + tz_offset * 3600 # 1375992114
</pre>

struct_time:
time.localtime() # local zime zone
time.gmtime() # UTC time

today = time.localtime()
today_year = today.tm_year
today_month = today.tm_mon
today_day = today.tm_mday

time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime())

Zulu time to epoch time: [http://stackoverflow.com/questions/127803/how-to-parse-iso-formatted-date-in-python]
s = '2008-09-03T20:56:35.450686Z'
event_time_struct = time.strptime(s.split('.')[0]+ "UTC", "%Y-%m-%dT%H:%M:%S%Z")
epoch_time = calendar.timegm(event_time_struct)

Parse string to time.struct_time (like gmtime()): [http://docs.python.org/2/library/time.html#time.strptime]
time.strptime(time.ctime()) # current time - default format is ctime format: "%a %b %d %H:%M:%S %Y"
time.strptime(time.ctime(time.time())) # default format is ctime format: "%a %b %d %H:%M:%S %Y"
time.strptime("30 Nov 00", "%d %b %y")

Build from ticks: [http://www.tutorialspoint.com/python/python_date_time.htm]
localtime = time.localtime( time.time() ) # build time.struct_time from ticks
print time.asctime( localtime ) # Tue Jan 13 10:17:09 2009

dt = datetime.datetime.fromtimestamp( time.time() ) # build datetime.datetime struct from ticks
dt.strftime("%Y-%m-%d %H:%M:%S") # '2013-07-31 23:07:48'
time.strptime( time.time(), ")

<pre>
from datetime import date
now = date.today() # now = datetime.date(2003, 12,2)
now.strftime("%m-%d-%y") # '12-02-03'
# dates support calendar arthimetic
birthday = date(1964, 7, 31)
age = now - birthday
age.days # 14368
</pre>

ts = (datetime.datetime.now().strftime("%Y%m%d%H%M")) # 201105191118
tstxt = (datetime.datetime.now().strftime("%m-%d-%Y %H:%M")) # 05-19-2011 11:18

Seconds since epoch:
int(time.time()) # 1305842304

Zulu and back again: ([http://aboutsimon.com/2013/06/05/datetime-hell-time-zone-aware-to-unix-timestamp/ Datetime hell])
<pre>
import time
from calendar import timegm
from datetime import datetime

if __name__ == '__main__':
iso_string = '2013-06-05T15:19:10Z'

timestamp = timegm(
time.strptime(
iso_string.replace('Z', 'GMT'),
'%Y-%m-%dT%H:%M:%S%Z'
)
)

print iso_string
print timestamp
print '-> %sZ' % datetime.utcfromtimestamp(timestamp).isoformat()
</pre>

http://wiki.python.org/moin/TimeTransitionsImage?action=AttachFile&do=get&target=v1.png [http://wiki.python.org/moin/WorkingWithTime]

References:
* WorkingWithTime - PythonInfo Wiki - http://wiki.python.org/moin/WorkingWithTime
* Datetime hell. Time zone aware to UNIX timestamp. | About me and the things I do - http://aboutsimon.com/2013/06/05/datetime-hell-time-zone-aware-to-unix-timestamp/
* WorkingWithTime - Python Wiki - http://wiki.python.org/moin/WorkingWithTime

---

Something I cooked up...

<pre>
def utc_time(self, years_offset=60):
mst_epoch_time = time.time() # server is in MST
if time.localtime().tm_isdst: # are we daylight saving time?
utc_epoch_time = mst_epoch_time + time.altzone
else:
utc_epoch_time = mst_epoch_time + time.timezone
return utc_epoch_time

def tz_time(self, tz_offset, daylight_adjust=True):
""" return the future time based on timezone offset
@tz_offset is timezone offset, eg -7 for MST
@daylight_adjust is flag to auto adjust for daylight time"""
if daylight_adjust:
if time.localtime().tm_isdst: # are we daylight saving time?
_tz_offset = tz_offset - 1
else:
_tz_offset = tz_offset
_utc_epoch_time = self.utc_time()
_adj_time = _utc_epoch_time + _tz_offset * 60 * 60
return _adj_time

def future_time(self, tz_offset=-7, daylight_adjust=True, years_offset=60):
""" Calculate some future year """
_tz_time = self.tz_time(tz_offset)
future_utc_epoch_time = _tz_time + years_offset * 365 * 24 * 60 * 60
# account for leap years
cur_year = datetime.datetime.now().year
cur_year_leap = cur_year / 4
future_year = cur_year + years_offset
future_year_leap = future_year / 4
leap_days = future_year_leap - cur_year_leap
future_utc_epoch_time += leap_days * 24 * 60 * 60
return future_utc_epoch_time
</pre>

=== Math ===

Math Functions:
import math
math.ceil(x)
math.floor(x)
math.trunc(x)
math.pow(x,y)

abs(x) # built in

=== Random ===

import random
random.choice(['a', 'b', 'c'])
random.sample(range(100), 10)
random.random() # random float 0 < x < 1
random.randrange(3) # integer in [0, 1, 2]
random.randrange(1, 3) # integer in [1, 2]
random.randint(1, 3) # integer in [1, 2, 3]

=== Operating System Operations ===

Import OS Library:
import os

Common Functions:
# directory functions
os.getcwd() # get current directory
os.listdir(p) # get list of directory contents
os.chdir(p) # change directory
os.mkdir(p) # make directory
os.makedirs(p) # make directory and all intermediate
os.rmdir(p) # remove empty directory, see shutil.rmtree()
os.removedirs(p) # remove empty directories, see shutil.rmtree()
os.path.exists(p) # does file/directory exist?
os.path.isdir(p) # does directory exist
os.path.join(folder1, folder2, file, etc) # portable path join

# file functions
os.remove(p) # remove file
os.unlink(p) # same as os.remove()
os.path.exists(p) # does file/directory exist?
os.path.isfile(p) # does file exist
os.stat(fname) # get file stats
os.stat(fname).st_size # file size
os.link(src, dst) # create hard link
os.readlink(p) # read symbolic link
os.rename(old, new) # rename/move file or directory
os.tempnam(p) # temporary unique name for target path
os.tmpfile() # create temporary file
os.tmpnam() # temporary file name
os.path.getsize(filename) # get file size

# system functions
os.fork() # fork process
os.getpid() # get process id
os.getppid() # get parent's process id
os.kill(pid, sig) # kill a process
os.getlogin() # get user's login name

# execution
os.system(cmd) # output to stdout, return exit_status
os.popen(cmd) # returns a pipe to command

Note: to get path to script:
sys.path[0]

Get directory and file names:
os.path.dirname(path) # '/root/test.txt' -> '/root'
os.path.basename(path) # '/root/test.txt' -> 'test.txt'
os.path.splitext(path) # '/root/test.test.txt' -> ('/root/test.test', '.txt') - extract extention
filename.split(".")[-1] # '/root/test.test.txt' -> 'txt' - extract extention

Normalize path, eliminating double slashes, etc.
os.path.normpath(path) # '/root/..' -> '/' or 'test/..' -> '.'
os.path.abspath(path) # '/root/..' -> '/' or 'test/..' -> '/root/test'

List dir: (returns list)
os.listdir( os.getcwd() )

List of files:
[p for p in os.listdir(os.getcwd())
if os.path.isfile(p)]

List of folders:
[p for p in os.listdir(os.getcwd())
if os.path.isdir(p)]

Build path for local OS: (using appropriate slash)
pathstr = os.path.join("folder", "folder", "file_or_folder")

File and directory management with shutil
import shutil
shutil.copyfile('file1', 'file2')
shutil.move('file1', 'folder2/file1')

File Wildcards:
import glob
glob.glob('*.py') # return list of files

Walk directory tree: [http://stackoverflow.com/questions/120656/directory-listing-in-python]
<pre>
import os
for dirname, dirnames, filenames in os.walk('.'):
for subdirname in dirnames:
print os.path.join(dirname, subdirname)
for filename in filenames:
print os.path.join(dirname, filename)
</pre>

ugly 'deltree' (by kenneth)
<pre>
import os
def deltree(path):
files = os.listdir(path)
for file in files:
file_path = os.path.join(path, file)
if os.path.isdir(file_path):
deltree(file_path)
os.rmdir(file_path)
else:
os.remove(file_path)
</pre>

Cleaner 'deltree' (by Kep)
<pre>
import shutil
# Blow away (if needed), and reclone to proper version
if os.path.isdir("xxx"):
shutil.rmtree("xxx")
</pre>

== System Platform ==

<pre>
import platform

if platform.system() == "Linux":
# ...
if platform.system() == "Windows":
# ...
else:
# ...
</pre>

=== Temp Files ===

import tempfile
filename = tempfile.mktemp()
f = open(filename, 'w')
...
f.close()

import os
os.remove(filename) # delete temp file

=== Pickling ===

Pickling (serializing, making objects persistent):
import pickle
grades = {'bob' : [4, 5, 6],
'sue' : [3, None, 7, 7]}
outfile = open(fname, 'wb')
pickle.dump(grades, outfile)

infile = open(fname, 'rb')
grades = pickle.load(infile)

=== [http://docs.python.org/library/logging.html Logging] ===

* logging — Logging facility for Python — Python 3 documentation - https://docs.python.org/3/library/logging.html
* Logging HOWTO — Python Dev Version documentation - http://docs.python.org/dev/howto/logging.html
* Logging Cookbook — Python Dev Version documentation - http://docs.python.org/dev/howto/logging-cookbook.html

Setup logger:
import logging
logging.basicConfig(level=logging.DEBUG,format='[%(asctime)s.%(msecs)03d] %(levelname)s %(filename)s(line:%(lineno)d): %(message)s')

Log Levels:
import logging
msg = "i broke"
logging.debug(msg)
logging.info(msg)
logging.warning(msg) # WARNING:root:i broke
logging.error(msg) # ERROR:root:i broke
logging.critical(msg) # CRITICAL:root:i broke

log.setLevel(level) # debug, info, warn/warning, error, critical
log.setLevel(logging.DEBUG)

A good convention to use when naming loggers is to use a module-level logger, in each module which uses logging, named as follows: [http://docs.python.org/dev/howto/logging.html]
logger = logging.getLogger(__name__)

Simple example: [http://stackoverflow.com/questions/345991/python-no-handlers-could-be-found-for-logger-opengl-error]
<pre>
log_format='%(asctime)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
logging.basicConfig(format=log_format, datefmt=log_date_format)
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)

log.info('hi')
</pre>

Formatter function:
<pre>
def makeFormatter(format_str="", date_format="%Y-%m-%d %H:%M:%S"):
if not format_str:
format_str = "%(asctime)s | %(levelname)s | %(filename)s:%(lineno)d:%(funcName)s | %(message)s"
return logging.Formatter(fmt=format_str, datefmt=date_format)
</pre>

Medium example:
<pre>
# Create logger
log_format='%(asctime)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
log_formatter = logging.Formatter(fmt=log_format, datefmt=log_date_format)
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)

# Log to console
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(log_formatter)
log.addHandler(ch)

# Log to file
fh = logging.FileHandler('my_logger.log')
fh.setLevel(logging.DEBUG)
fh.setFormatter(log_formatter)
log.addHandler(fh)
</pre>

Log all loggers to file - attach to root logger
<pre>
import logging

logger = logging.getLogger(__name__)

# Configure Logger
log_format='%(asctime)s | %(name)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
log_formatter = logging.Formatter(fmt=log_format, datefmt=log_date_format)
logging.basicConfig(level=logging.DEBUG, format=log_format, datefmt=log_date_format)

# Setup Log to File handler, and attach to root logger
root_logger = logging.getLogger() # Calling getLogger() without a name returns the root logger
fh = logging.FileHandler('logger.txt')
fh.setLevel(logging.DEBUG)
fh.setFormatter(log_formatter)
root_logger.addHandler(fh)
</pre>

Application Setup (not basic log configuration carries across imported modules):
<pre>
import logging
def main():
logging.basicConfig(filename='myapp.log', level=logging.INFO)
logging.info('Started')
...
logging.info('Finished')

if __name__ == '__main__':
main()
</pre>

Log multiple lines:
<pre>
for line in output.splitlines():
log.debug(line)
</pre>

Set log level: (warning is default)
logging.basicConfig(level=logging.DEBUG)

Set log level from string value: [http://docs.python.org/dev/howto/logging.html]
<pre>
# assuming loglevel is bound to the string value obtained from the
# command line argument. Convert to upper case to allow the user to
# specify --log=DEBUG or --log=debug
numeric_level = getattr(logging, loglevel.upper(), None)
if not isinstance(numeric_level, int):
raise ValueError('Invalid log level: %s' % loglevel)
logging.basicConfig(level=numeric_level, ...)
</pre>

Log to file:
logging.basicConfig(filename='example.log',level=logging.DEBUG)
logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG) # do not append

Format output: (once set, can't be changed?)
# WARNING:MESSAGE
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)

# 05/23/2011 12:50:02 PM MESSAGE
logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')

# 05/23/2011 12:55:28 PM : WARNING : MESSAGE
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')

# 2011-05-23 13:05:30 : WARNING : MESSAGE
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

# 2011-05-23 13:05:30 | WARNING | FILE:FUNC | MESSAGE
logging.basicConfig(format='%(asctime)s | %(levelname)s | %(filename)s:%(funcName)s | %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

Log record attributes - http://docs.python.org/library/logging.html#logrecord-attributes

Format Example:
<pre>
FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' # doesn't work!
logging.basicConfig(format=FORMAT)
d = { 'clientip' : '192.168.0.1', 'user' : 'fbloggs' }
logger = logging.getLogger('tcpserver')
logger.warning('Protocol problem: %s', 'connection reset', extra=d)
</pre>

<pre>
import logging

# create logger with 'spam_application'
logger = logging.getLogger('spam_application')
logger.setLevel(logging.DEBUG)

# create file handler which logs even debug messages
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)

# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)

# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(fh)
logger.addHandler(ch)

logger.info('Message to loggers and handlers')
</pre>

=== Templating ===

from string import Template
t = Template('${name}'s fav dish is $food')
t.substitute(name = 'ken', food = 'beans') # throws exception if missing one, but can have more
t.safe_substitute(name = 'ken') # will fill in what it can

Change delimiter:
class MYT(Template):
delimiter = '%'
minput = input("File name format (%d - date, %s - size): ")
myt = MYT(minput)
myt.substitute(%d=date, %s=size)

=== [http://docs.python.org/library/threading.html Threading] ===

Basic thread: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import thread

def someFunc():
print "someFunc was called"

thread.start_new_thread(someFunc, ())
</pre>

More complete basic thread:
<pre>
import thread
import threading
import time

def run_thread (threadname, sleeptime):
print "Thread start: " + threadname
time.sleep(sleeptime)
print "Thread end: " + threadname
thread.interrupt_main() # throw keyboard interrupt to kill main program

if __name__ == "__main__":
thread.start_new_thread(run_thread, ("Thread1", 3))
try:
while 1:
pass
except:
print "Thread1 exited...."
</pre>

Can use a global counter instead of exception to track number of threads:
<pre>
import thread
import threading
import time

threadcount = 3

def run_thread (threadname, sleeptime):
global threadcount
print "Thread start: " + threadname
time.sleep(sleeptime)
print "Thread end: " + threadname
threadcount -= 1

if __name__ == "__main__":
thread.start_new_thread(run_thread, ("Thread1", 3))
thread.start_new_thread(run_thread, ("Thread2", 4))
thread.start_new_thread(run_thread, ("Thread3", 2))
while 1:
if threadcount == 0:
break
else:
pass
print "completed"
</pre>

---

Thread object: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import threading

t1 = threading.Thread(target=someFunc)
# with parameters: Thread(target=myfunction, args=('MyStringHere',1)).start()
t1.start()
t1.join()
</pre>

Basic threading with parameters:
<pre>
import threading
t = threading.Thread(None, someFunc, None, (param1, param2))
t.start()
t.join()
</pre>

<pre>
t = threading.Thread(target=client_recv, args=(s, name))
while True:
t.join(timeout=2)
if t.isAlive():
print "Thread is dead..."
break
</pre>

Threading as a class: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import threading

class FuncThread(threading.Thread):
def __init__(self, target, *args):
self._target = target
self._args = args
threading.Thread.__init__(self)

def run(self):
self._target(*self._args)

# Example usage
def someOtherFunc(data, key):
print "someOtherFunc was called : data=%s; key=%s" % (str(data), str(key))

t1 = FuncThread(someOtherFunc, [1,2], 6)
t1.start()
t1.join()
</pre>

Threading as a class #2:
<pre>
import threading
import time
import sys

class MyT(threading.Thread):
def __init__(self, name, time):
threading.Thread.__init__(self)
self.name = name
self.time = time
def run(self):
print("Thread Start: " + self.name)
for i in range(self.time):
print("[" + self.name + "]", end='')
sys.stdout.flush()
time.sleep(1)
print("Thread Stop: " + self.name)

myt1 = MyT('1', 10)
myt1.start()

myt2 = MyT('2', 15)
myt2.start()

myt3 = MyT('3', 5)
myt3.start()

myt4 = MyT('4', 10)
myt4.start()

myt1.join()
myt2.join()
myt3.join()
myt4.join()
</pre>

Locking:
<pre>
lock=thread.allocate_lock()

# pass lock to thread, and use in thread as such:
lock.acquire()
#some operation
lock.release()
</pre>

Returning data from a thread (using a [http://docs.python.org/library/queue.html queue]): [http://stackoverflow.com/questions/1886090/return-value-from-thread] [http://stackoverflow.com/questions/2829329/catch-a-threads-exception-in-the-caller-thread-in-python]
<pre>
# Pass exception back to parent
import sys
import threading
import Queue

class ExcThread(threading.Thread):
def __init__(self, bucket):
threading.Thread.__init__(self)
self.bucket = bucket
def run(self):
try:
raise Exception('An error occured here.')
except Exception:
self.bucket.put(sys.exc_info())

def main():
bucket = Queue.Queue()
thread_obj = ExcThread(bucket)
thread_obj.start()
while True:
try:
exc = bucket.get(block=False)
except Queue.Empty:
pass
else:
exc_type, exc_obj, exc_trace = exc
# deal with the exception
print "Exception Caught:", exc_type, exc_obj
print "Exception Trace:", exc_trace
thread_obj.join(0.1)
if thread_obj.isAlive():
continue
else:
break

if __name__ == '__main__':
main()
</pre>

<pre>
Exception Caught: <type 'exceptions.Exception'> An error occured here.
Exception Trace: <traceback object at 0x2b2230a285f0>
</pre>

Another Exception catching queue example: (kenneth's example)
<pre>
# thread function
def run_thread(queue, item):
...
queue.put( Exception("Failure on item:" + item) )

threads = []
queue = Queue.Queue()

# start threads
for item in items:
t = Thread(None, run_thread, None, (queue, item))
t.start()
threads.append(t)

# wait for threads
for t in threads:
t.join()

# catch and reraise exceptions
while not queue.empty():
raise(queue.get())
</pre>

Returning data from a thread (using a callback function): [http://stackoverflow.com/questions/1886090/return-value-from-thread]
:"Another approach is to pass a callback function to the thread. This gives a simple, safe and flexible way to return a value to the parent, anytime from the new thread."
<pre>
import threading
import time
import sys

class MyThread(threading.Thread):
def __init__(self, cb):
threading.Thread.__init__(self)
self.callback = cb
def run(self):
for i in range(10):
self.callback(i)
time.sleep(1)

def count(x):
print x
sys.stdout.flush()

t = MyThread(count)
t.start()
</pre>

References:
* 16.2. threading — Higher-level threading interface — Python v2.7.1 documentation - http://docs.python.org/library/threading.html
* An Introduction to Thread Programming - http://www.prasannatech.net/2008/08/introduction-to-thread-programming.html
* Python - Multithreaded Programming - http://www.tutorialspoint.com/python/python_multithreading.htm
* Running Functions as Threads in Python « Software Ramblings - http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html
* Understanding Threading in Python LG #107 - http://linuxgazette.net/107/pai.html
* multithreading - Simple threading in Python 2.6 using thread.start_new_thread() - Stack Overflow - http://stackoverflow.com/questions/849674/simple-threading-in-python-2-6-using-thread-start-new-thread
* python - Return value from thread - Stack Overflow - http://stackoverflow.com/questions/1886090/return-value-from-thread

---

Interrupt main thread:
<pre>
# source: http://stackoverflow.com/questions/4591917/python-how-to-quit-cli-when-stuck-in-blocking-raw-input

import thread
import time

def main():
try:
m = thread.start_new_thread(killable_input, tuple())
while 1:
time.sleep(0.1)
except KeyboardInterrupt:
print "exception"

def killable_input():
w = thread.start_new_thread(normal_input, tuple())
i = thread.start_new_thread(wait_sometime, tuple())

def normal_input():
s = raw_input("input:")

def wait_sometime():
time.sleep(4) # or any other condition to kill the thread
print "too slow, killing imput"
thread.interrupt_main()

if __name__ == '__main__':
main()
</pre>

=== urllib ===

See [[Python/urllib]]

=== Command line arguments ===

<pre>
import sys
for x in sys.argv:
print x
</pre>

=== Execute other python script ===

import sys
sys.argv = ['', 'my', 'args', 'here']
execfile('myscript.py') # does not support arguments directly

=== Exit Code ===

sys.exit(n) http://docs.python.org/library/sys.html:
import sys
sys.exit(1) # exit code of 1 (default is 0)
sys.exit("some error") # exit code of 1, and message displayed

=== [http://docs.python.org/library/syslog.html syslog] ===

<pre>
import syslog
syslog.openlog('[WATER]')
syslog.syslog('Processing started')
if error:
syslog.syslog(syslog.LOG_ERR, 'Processing started')
</pre>

<pre>
import syslog
syslog.openlog('[WATER]')
def log(msg):
syslog.syslog(str(msg))
log("Starting...")
</pre>

=== Subprocess ===

Subprocess - http://docs.python.org/library/subprocess.html

Methods of subprocess (or external command execution):
* subprocess (only method to rule/replace all others)
* os.system
* os.spawn*
* os.popen*
* popen2.*
* commands.*

My New Favorite:
<pre>
import subprocess
class Run(object):
class RC(int):
pass
def __new__(cls, cmd, *args, **kwargs):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
rc = cls.RC(p.wait())
rc.output = "\n".join(p.stdout.readlines())
rc.error = "\n".join(p.stderr.readlines())
return rc

if Run("some command"): print "do something" # on rc of not 0 (negative or positive)
</pre>

My Favorite:
<pre>
def exe(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
#output = " | ".join(p.stdout.readlines())
output = "\n".join(p.stdout.readlines())
rc = p.wait()
return (rc, output)

def exe_checked(cmd, checked=True):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
#output = " | ".join(p.stdout.readlines())
output = "\n".join(p.stdout.readlines())
rc = p.wait()
if rc and checked:
raise Exception("Command returned non zero [{0}]: {1}".format(rc, cmd)
return (rc, output)

def exe_with_stderr(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subproc
ess.PIPE)
output = " | ".join(p.stdout.readlines())
outputerr = " | ".join(p.stderr.readlines())
rc = p.wait()
return (rc, output, outputerr)
</pre>

<pre>
def exec_program(cmd):
s = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = s.communicate()
result = s.wait()
return stdout, stderr, result
</pre>

Pass along output and return true error code:
import subprocess
retcode = subprocess.call(["ls", "-l"]) # true return code, output sent to STDOUT

retcode = subprocess.call(["foo.sh","args"],shell=True) # Execute in shell, with shell processing
retcode = subprocess.call(['C:\\Temp\\a b c\\Notepad.exe'])
retcode = subprocess.call(["D:/Python24/Python.exe", "Skripta.py", "arg1", "arg2"])

Note: shell=True is equivalent to: subprocess.Popen(['/bin/sh', '-c', args[0], args[1], ...])

Process STDOUT and STDERR and get return code:
<pre>
#!/usr/bin/env python
import subprocess
import string

cmd = "cat /etc/passwd | grep ken"
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) # stderr=subprocess.STDOUT to combine

out = string.join(p.stdout.readlines())
outerr = string.join(p.stderr.readlines())
retcode = str(p.wait())

print("STDOUT: " + out)
print("STDERR: " + outerr)
print("RETCODE: " + retcode)
</pre>

Above with stderr redirected to stdout (combined):
<pre>
import subprocess
p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
print line,
retval = p.wait()
</pre>

Replacing shell backtick: [http://docs.python.org/library/subprocess.html]
# shell code:
# OUTPUT=`mycmd myarg`
from subprocess import Popen
output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]

Replacing shell pipeline [http://docs.python.org/library/subprocess.html]
# shell code:
# OUTPUT=`dmesg | grep hda`
from subprocess import Popen
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

Replacing os.system(): [http://docs.python.org/library/subprocess.html]
# old:
sts = os.system("mycmd" + " myarg")

# new:
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)[1]

Control an executed process:
<pre>
cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

# cycle while process continues alive
while p.poll() == None:

# check if input
if select.select([p.stdout.fileno()], [], [], 1)[0]:
while select.select([p.stdout.fileno()], [], [], .1)[0]:
line = p.stdout.readline()
# ... process

# write something to process
p.stdin.write("tp %s %s\n" % (user1, user2))

...

# time to force a quit: (choose which you need)
p.kill() # SIGKILL
p.terminate() # SIGTERM
os.kill(p.pid, signal.SIGTERM) # same as first, but not as clean
</pre>

==== Other Command Execution ====

Old method with os.system(): [http://docs.python.org/library/os.html]
import os
encoded_wait_code = os.system("some command") # output sent to STDOUT
# Note: returns code of 0 if command returns 0, but wait() encoded value for all other values

Quick and dirty old method (prints stderr, captures stdout)
<pre>
import os
out = string.join( os.popen(cmd).readlines() )
</pre>

Quick and dirty old method (prints stderr, captures stdout, captures ugly wait() encoded return code)
<pre>
import os
p = os.popen(cmd)
stdout = "".join( p.readlines() )
encoded_wait_code = p.close() # None if 0
rcode = os.WEXITSTATUS(ugly_rcode) # fix ugly wait() encoded rcode
print(stdout)
print(rcode)

if os.popen(cmd).close():
print "error with cmd: %s" % cmd
sys.exit(1)
</pre>

???:
<pre>
os.spawnvp(os.P_WAIT,"/usr/bin/mplayer",["","-ao","pcm",file])
# or
os.spawnvp(os.P_NOWAIT,"/usr/bin/mplayer",["","-ao","pcm",file])
</pre>

Replacing os.system(): [http://docs.python.org/library/subprocess.html]
# old:
sts = os.system("mycmd" + " myarg")

# new:
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)[1]

More realistic example replacing os.system(): [http://docs.python.org/library/subprocess.html]
<pre>
try:
retcode = call("mycmd" + " myarg", shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e
</pre>

==== select ====

Select:

select.select(rlist, wlist, xlist[, timeout])
: This is a straightforward interface to the Unix select() system call. The first three arguments are sequences of ‘waitable objects’: either integers representing file descriptors or objects with a parameterless method named fileno() returning such an integer
: The optional timeout argument specifies a time-out as a floating point number in seconds. When the timeout argument is omitted the function blocks until at least one file descriptor is ready. A time-out value of zero specifies a poll and never blocks.

if select.select([sys.stdin.fileno()], [], [], .1)[0]:
line = p.stdout.readline()

==== full wrapped execution ====

Process timeout (by Kenneth):
<pre>
import subprocess
import time
import os
import signal

def cmd(cmd, timeout=300, retry=0):
# Execute command, time progress, and kill on timeout

# (status, output) = commands.getstatusoutput(cmd)
start_time = None
status = None
output = None
for retry_count in range(retry+1):
if retry_count > 0:
print("RETRY: " + str(retry_count) + " " + cmd)
start_time = time.time()
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
timeout_fail = False
while p.poll() == None:
if time.time() - start_time > timeout:
# kill process
print("TIMEOUT: " + str(timeout) + "sec - killing (" + str(p.pid) + ") - " + cmd)
try:
os.kill(p.pid, signal.SIGTERM)
except OSError as e:
print("Kill exception: " + str(e))
timeout_fail = True
break
#print(".")
time.sleep(1)
status = p.poll()
output = p.stdout.read()
if timeout_fail == False:
break
seconds = int(time.time() - start_time)
return (status, output, seconds)
</pre>

---

Process input and output wrapper (from minecraft):
<pre>
import subprocess
import select
import re
import sys

cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)

# cycle while process continues alive
while p.poll() == None:

if select.select([p.stdout.fileno()], [], [], 1)[0]:
while select.select([p.stdout.fileno()], [], [], .1)[0]:
line = p.stdout.readline()
# <kiloforce> -tp joe
match = re.findall("<(.*)> -tp (.*)", line)
if match:
user1 = match[0][0]
user2 = match[0][1]
print ">> ACTION: Teleporting %s to %s" % (user1, user2)
p.stdin.write("tp %s %s\n" % (user1, user2))
if line:
print ">> SERVER: %s" % line.strip()
else:
print "The server died!"
sys.exit(1)

if select.select([sys.stdin.fileno()], [], [], .1)[0]:
line = sys.stdin.readline()
print ">> CONSOLE: %s" % line.strip()
p.stdin.write("%s" % line)
</pre>

==== Command With Timeout ====

python - Using module 'subprocess' with timeout - Stack Overflow - http://stackoverflow.com/questions/1191374/using-module-subprocess-with-timeout

<pre>
import subprocess, threading

class Command(object):
def __init__(self, cmd):
self.cmd = cmd
self.process = None

def run(self, timeout):
def target():
print 'Thread started'
self.process = subprocess.Popen(self.cmd, shell=True)
self.process.communicate()
print 'Thread finished'

thread = threading.Thread(target=target)
thread.start()

thread.join(timeout)
if thread.is_alive():
print 'Terminating process'
self.process.terminate()
thread.join()
print self.process.returncode

command = Command("echo 'Process started'; sleep 2; echo 'Process finished'")
command.run(timeout=3)
command.run(timeout=1)
</pre>

My version:
<pre>
import subprocess
import threading

class Command(object):
def __init__(self, cmd, capture=False, shell=True):
self.cmd = cmd
self.process = None
self.thread = None
self.shell = shell
if type(cmd) == list:
self.shell = False
self.rc = None
self.pid = None
self.stdout = None
self.stderr = None
self.capture = capture # capture stdout?

def run(self, timeout=None, wait=True):
def target():
print '>> Thread started (%s)' % self.cmd
if self.capture:
self.process = subprocess.Popen(self.cmd, shell=self.shell, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
else:
self.process = subprocess.Popen(self.cmd, shell=self.shell)
print '>> Thread PID: %d' % self.process.pid
self.pid = self.process.pid
self.stdout, self.stderr = self.process.communicate()
self.rc = self.process.returncode
print '>> Thread finished: %d' % self.rc

self.thread = threading.Thread(target=target)
self.thread.start()

if wait:
self.wait(timeout)
return self # so you can do: c = Command().run()

def wait(self, timeout=None):
print '>> Waiting for thread...'
if timeout:
self.thread.join(timeout)
else:
self.thread.join()
if self.thread.is_alive():
print '>> Terminating process'
self.process.terminate()
self.rc = self.process.returncode
self.thread.join()
if self.thread.is_alive():
print '>> Failed to terminate'
else:
print '>> Thread completed'
#print self.process.returncode

def is_alive(self ):
return self.thread.is_alive()

command = Command('do something').run()

command2 = Command(['do', 'something', 'long']).run(wait=False)
command2.wait(timeout=10)

command3 = Command('want output', capture=True).run()
print command3.stdout
</pre>

=== System call ===

os.system("some_command with args")
os.system("some_command < input_file | another_command > output_file")
stream = os.popen("some_command with args")

Here's a summary of the ways to call external programs and the advantages and disadvantages of each:

1.
: os.system("some_command with args") passes the command and arguments to your system's shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example,
: os.system("some_command < input_file | another_command > output_file")
: However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs.
: http://docs.python.org/lib/os-process.html

2.
: stream = os.popen("some_command with args") will do the same thing as os.system except that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything.
: http://docs.python.org/lib/os-newstreams.html

3.
: The Popen class of the subprocess module. This is intended as a replacement for os.popen but has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you'd say
: print Popen("echo Hello World", stdout=PIPE, shell=True).stdout.read()
: instead of
: print os.popen("echo Hello World").read()
: but it is nice to have all of the options there in one unified class instead of 4 different popen functions.
: http://docs.python.org/lib/node528.html

4.
: The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply wait until the command completes and gives you the return code. For example:
: return_code = call("echo Hello World", shell=True)
: http://docs.python.org/lib/node529.html

5.
: The os module also has all of the fork/exec/spawn functions that you'd have in a C program, but I don't recommend using them directly.

: The subprocess module should probably be what you use.

Source: [http://stackoverflow.com/questions/89228/how-to-call-external-command-in-python]

== Advanced Code ==

=== Executing Code from Command Line ===

python -c [cmd]
# -c cmd : program passed in as string (terminates option list)

<pre>
python -c "import sys;exec(sys.stdin.read())" << "EOF"
def test1():
print "test1"
test1()
print "test"
EOF
</pre>

=== Integer Return Code Class ===

Python3 version:
<pre>
class RC(int):
def __new__(cls, rc, stdout='', stderr=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, stdout='', stderr=''):
super().__init__()
self.rc = rc # redundant
self.stdout = stdout
self.stderr = stderr
def __repr__(self):
"""Optional string output"""
return "(%s, '%s', '%s')" % (self.rc, self.stdout, self.stderr)
</pre>

Python2 version:
<pre>
class RC(int):
def __new__(cls, rc, stdout='', stderr=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, stdout='', stderr=''):
super(self.__class__, self).__init__(rc)
self.rc = rc # redundant
self.stdout = stdout
self.stderr = stderr
def __repr__(self):
"""Optional string output"""
return "(%s, '%s', '%s')" % (self, self.stdout, self.stderr)
</pre>

sample:
<pre>
>>> rc1 = RC(5)
>>> rc2 = RC(0, 'out', 'err')
>>> rc1
(5, '', '')
>>> rc2
(5, 'out', 'err')
>>> rc2.stdout
'out'
>>> int(rc1)
5
>>> 0 + rc1
5
>>> if rc1 <= 5: print "yes"
...
yes
>>> if rc2: print "non-zero"
...
</pre>

=== Smart Execute ===

<pre>
class RC(int):
def __new__(cls, rc, output=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, output=''):
super().__init__()
self.rc = rc # redundant
self.output = output
def __repr__(self):
"""Optional string output"""
return "(%s, '%s')" % (self.rc, self.output)

def smart_execute(cmd, display=True, checked=False):
output = ""
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, shell=True, encoding='utf-8')
for c in iter(lambda: process.stdout.read(1), ""):
if display:
sys.stdout.buffer.write(c.encode('utf-8'))
output += c
process.stdout.close()
rc = process.wait()
if checked and rc:
raise subprocess.CalledProcessError(ret, cmd)
return RC(rc, output.strip())
</pre>

Not smart version:
<pre>
def execute(cmd, return_output=False):
out = ""
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, shell=True, encoding='utf-8')
for c in iter(lambda: process.stdout.read(1), ""):
sys.stdout.buffer.write(c.encode('utf-8'))
if return_output:
out += c
process.stdout.close()
ret = process.wait()
if ret:
raise subprocess.CalledProcessError(ret, cmd)
if return_output:
return out.strip()
</pre>
</pre>

=== select ===

usage: select.select(rlist, wlist, xlist[, timeout]) [http://docs.python.org/2/library/select.html]
* rlist: wait until ready for reading
* wlist: wait until ready for writing
* xlist: wait for an “exceptional condition” (see the manual page for what your system considers such a condition)
* The optional timeout argument specifies a time-out as a floating point number in seconds.

if select.select([p.stdout.fileno()], [], [], 1)[0]:

=== ANSI Color ===

* ANSI escape code - http://en.wikipedia.org/wiki/ANSI_escape_code

---

unicode - Print in terminal with colors using Python? - Stack Overflow - https://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python

<pre>
class bcolors:
HEADER = '\033[95m'
OKBLUE = '\033[94m'
OKGREEN = '\033[92m'
WARNING = '\033[93m'
FAIL = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'

print bcolors.WARNING + "Warning: No active frommets remain. Continue?" + bcolors.ENDC
</pre>

<pre>
class colors:
black = 0
red = 1
green = 2
yellow = 3
blue = 4
magenta = 5
cyan = 6
white = 7
fg = 30 # foreground add
bg = 40 # background add
bfg = 90 # bold foreground add
bbg = 100 # bold background add

print '\033[%dm red foreground \033[0m' % (colors.red + colors.fg)
print '\033[%dm bold red foreground \033[0m' % (colors.red + colors.bfg)
print '\033[%dm blue background \033[0m' % (colors.blue + colors.bg)
print '\033[%dm bold blue background \033[0m' % (colors.blue + colors.bbg)
</pre>

---

Before you start spouting escape sequences, you should check that stdout is a tty. You can do this with sys.stdout.isatty()

<pre>
sys.stdout.isatty()
</pre>

<pre>
def hilite(string, status, bold):
attr = []
if status:
# green
attr.append('32')
else:
# red
attr.append('31')
if bold:
attr.append('1')
return '\x1b[%sm%s\x1b[0m' % (';'.join(attr), string)
</pre>

Python | change text color in shell - Stack Overflow - https://stackoverflow.com/questions/2330245/python-change-text-color-in-shell

---

For reference, a small bash script which prints a table of escape sequences: colortable16.sh: ([http://www.frexx.de/xterm-256-notes/ The 256 color mode of xterm], [http://plug.org/pipermail/plug/2008-March/016821.html The 90s called; they want their 256 colors back])
<pre>
#!/bin/bash

# prints a color table of 8bg * 8fg * 2 states (regular/bold)
echo
echo Table for 16-color terminal escape sequences.
echo Replace ESC with \\033 in bash.
echo
echo "Background | Foreground colors"
echo "---------------------------------------------------------------------"
for((bg=40;bg<=47;bg++)); do
for((bold=0;bold<=1;bold++)) do
echo -en "\033[0m"" ESC[${bg}m | "
for((fg=30;fg<=37;fg++)); do
if [ $bold == "0" ]; then
echo -en "\033[${bg}m\033[${fg}m [${fg}m "
else
echo -en "\033[${bg}m\033[1;${fg}m [1;${fg}m"
fi
done
echo -e "\033[0m"
done
echo "--------------------------------------------------------------------- "
done

echo
echo
</pre>

Python version: (modified by me)
<pre>
#!/usr/bin/env python

import sys

# prints a color table of 8bg * 8fg * 2 states (regular/bold)
print
print "Table for 16-color terminal escape sequences."
print "Replace ESC with \\033 in bash."
print
print "Background | Foreground colors"
print "---------------------------------------------------------------------"
for bg in range(40, 47+1):
for bold in range(0, 1+1):
sys.stdout.write("\033[0m"" ESC[{bg}m | ".format(bg=bg))
for fg in range(30, 37+1):
if bold == 0:
sys.stdout.write("\033[{bg}m\033[{fg}m [{fg}m ".format(bg=bg, fg=fg))
else:
sys.stdout.write("\033[{bg}m\033[1;{fg}m [1;{fg}m".format(bg=bg, fg=fg))
sys.stdout.write("\033[0m\n")
print "--------------------------------------------------------------------- "

print
print
</pre>

=== Execute External Application ===

Execute program: (and send output to STDOUT)
encoded_wait_code = os.system('mkdir mydir') # output sent to STDOUT

Get output and ugly return code: (STDERR goes to STDOUT)
p = os.popen(cmd)
print p.readlines()
encoded_wait_code = p.close()

Execute other python script
import sys
sys.argv = [, 'my', 'args', 'here']
execfile('myscript.py') # does not support arguments directly

Subprocess:
* See [[#Subprocess]]

=== Program Layout ===

<pre>
PackageName/
README
setup.py
bin/
script
docs/
test/ # some include in package_name
pacakgename/
__init__.py
code1.py
subpackage/
__init__.py
</pre>

=== __init__.py ===

Purpose: Makes the folder into a package library

NOTE: This file is most commonly blank

Good place to put your package functions.
<pre>
# ken/__init__.py
def load():
...
</pre>

<pre>
import ken
ken.load()
</pre>

Can import other modules to then use as package modules:
<pre>
# ken/__init__.py
from ken.client import Server
</pre>

<pre>
import ken
ken.Server()
</pre>

Good place to put your Exceptions.
<pre>
class MyException(Exception):
""" My Error Exception """
</pre>

Use the __all__ to specify what gets imported on 'from package import *'
__all__ = ['func1', 'func2']

=== Metadata ===

VERSION = (0, 2, 1)
or
__version__ = '0.2.1'

Also See [[#pydoc]]

---

__version__ = '1.10'
__author__ = 'Joshua Roesslein'
__license__ = 'MIT'

=== setup.py ===

See [[#Distributing Python Modules with Distutils]]

=== Distributing Python Modules with Distutils ===

Distributing Python Modules — http://docs.python.org/distutils/index.html

Uses setup.py for package installation.
python setup.py install

==== Simple example ====

To install foo.py:
<pre>
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
</pre>

Source: http://docs.python.org/distutils/introduction.html

==== klib example ====

klib setup.py:
<pre>
from distutils.core import setup

setup(name = "klib",
version = "101",
description = "klib library",
url = "http://www.oeey.com/klib",
author = "Kenneth",
author_email = "kenneth@oeey.com",
license = "Kenneth License",
long_description = """Really long text here.""",
platforms = "Any Platform",

packages = ['klib','klib.messenger'],
)
</pre>

Installed to:
/opt/python26/lib/python2.6/site-packages/klib/
/opt/python26/lib/python2.6/site-packages/klib-100-py2.6.egg-info

Metadata: (/opt/python26/lib/python2.6/site-packages/klib-101-py2.6.egg-info)
<pre>
Metadata-Version: 1.0
Name: klib
Version: 101
Summary: klib library
Home-page: http://www.oeey.com/klib
Author: Kenneth
Author-email: kenneth@oeey.com
License: Kenneth License
Description: Really long text here.
Platform: Any platform
</pre>

Any fields not included will be marked:
Platform: UNKNOWN

References:
* 2. Writing the Setup Script — Python v2.7.2 documentation - http://docs.python.org/distutils/setupscript.html
* http://wiki.python.org/moin/Distutils/Tutorial

==== Uninstall Package ====

Collect files during install:
python setup.py install --record install-record.txt
sudo rm $(cat install-record.txt)

References:
* installation - python setup.py uninstall - Stack Overflow - http://stackoverflow.com/questions/1550226/python-setup-py-uninstall

=== Distributing Python Modules with Setuptools ===

<pre>
#!/usr/bin/env python
import setuptools

setuptools.setup(name="tweepy",
version="1.10",
description="Twitter library for python",
license="MIT",
author="Joshua Roesslein",
author_email="tweepy@googlegroups.com",
url="http://github.com/tweepy/tweepy",
packages = setuptools.find_packages(),
keywords= "twitter library",
zip_safe = True)

</pre>

=== Decorators ===

Decorators wrap functions

Function Counter: [http://stackoverflow.com/questions/739654/understanding-python-decorators]
<pre>
def fcount(func):
def wrapper(*args, **kw):
wrapper.count = wrapper.count + 1
return func(*args, **kw)
wrapper.count = 0
return wrapper

@fcount
def f(n):
return n*n

print "f.count = ", f.count
print "f(10) = ", f(10)
print "f.count = ", f.count
print "f(3) = ", f(3)
print "f.count = ", f.count
print "f(12) = ", f(12)
print "f.count = ", f.count
print "f(15) = ", f(15)
print "f.count = ", f.count

# Output:
# f.count = 0
# f(10) = 100
# f.count = 1
# f(3) = 9
# f.count = 2
# f(12) = 144
# f.count = 3
# f(15) = 225
# f.count = 4
</pre>

Great reference: Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

Example 1:
@dec1
def func(arg1, arg2, ...): pass

# equivalent to:
def func(arg1, arg2, ...): pass
func = dec1(func)

Example 2:
<pre>
@dec2
@dec1
def func(arg1, arg2, ...):
pass
</pre>

<pre>
# equivalent to:
def func(arg1, arg2, ...):
pass
func = dec2(dec1(func))
</pre>

Example 3:
@decomaker(argA, argB, ...)
def func(arg1, arg2, ...):
pass

# equivalent to:
func = decomaker(argA, argB, ...)(func)

# example:
@accepts(int,int)
@returns(float)
def bar(low,high):
pass

Example 4: (overrides and ignores passed in function - bad, unless desired)
<pre>
def spamrun(fn):
def sayspam(*args):
print "spam, spam, spam"
return sayspam

@spamrun
def useful(a, b):
print a**2 + b**2
</pre>

Example 5: (Decorator that modifies behavior of undecorated func)
<pre>
def addspam(fn):
def new(*args):
print "spam, spam, spam"
return fn(*args)
return new

@addspam
def useful(a, b):
print a**2 + b**2
</pre>

My Example 1: (print args and continue to function)
<pre>
def showargs(fn):
def doshowargs(*args, **kw):
print "Args for:", fn.__name__, args, kw
return fn(*args, **kw)
return doshowargs

@showargs
def useful(a, b):
print a**2 + b**2

</pre>

Another Example: [http://stackoverflow.com/questions/739654/understanding-python-decorators]
<pre>
def makebold(fn):
def wrapped(*args, **kw):
return "" + fn(*args, **kw) + ""
return wrapped

def makeitalic(fn):
def wrapped(*args, **kw):
return "" + fn(*args, **kw) + ""
return wrapped

@makebold
@makeitalic
def hello(msg):
return msg

print hello("hello world")
# output: hello world
</pre>

References:
* PEP 318 -- Decorators for Functions and Methods - http://www.python.org/dev/peps/pep-0318/
* Charming Python: Decorators make magic easy - http://www.ibm.com/developerworks/linux/library/l-cpdecor/index.html
* What are some common uses for Python decorators? - Stack Overflow - http://stackoverflow.com/questions/489720/what-are-some-common-uses-for-python-decorators
* Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

---

Utah Python July 2013 Meeting - Decorator Examples: [https://www.youtube.com/watch?v=uh9uhzitdvw]

Example of uses:
* Count function calls
* Add additional output (debugging)
* Timing code
* Thread synchronization (locks)
* Type checking

<pre>
## Simple wrapping call
def shout(f):
def wrapper(*args):
print('This is before {name}'.format(name=f.__name__))
ret = f(*args)
print('This is after {name}'.format(name=f.__name__))
return ret
wrapper.__name__ = f.__name__
return wrapper

@shout
def echo(text):
print(text)

echo('this is a test')
</pre>

<pre>
## Counting function calls
def count(f):
def wrapper(*args):
wrapper.counter += 1
print("Function called {count} times".format(count=wrapper.counter))
return f(*args)
wrapper.counter = 0
# change wrapper name, or it will just be 'wrapper' (for inspection)
wrapper.__name__ = f.__name__
return wrapper

@count
def donothing():
pass

donothing()
donothing()
print("Function called a total of {count} times".format(count=donothing.counter))
</pre>

<pre>
## Timing your code
import time

def timer(f):
def wrapper(*arg):
t = time.time()
ret = f(*arg)
print("Took {elapsedtime} seconds".format(elapsedtime=time.time()-t))
return ret
return wrapper

@timer
def fact(n):
prod = 1
while n > 0:
prod *= n
n -= 1
return prod

total = fact(10000)
</pre>

<pre>
## Synchronization
import threading

def synchronized(lock):
def wrap(f):
def wrapper(*args, **kwargs):
lock.acquire()
try:
return f(*args, **kwargs)
finally:
lock.release()
return wrapper
return wrap

mylock = threading.Lock()

@synchronized(mylock)
def myfunc():
pass
</pre>

<pre>
## Type checking
def myMethod(ID, name):
''' Old way '''
if not (isinstance(ID, 'uint') and isinstance(name, 'utf8string')):
raise Exception()

def accepts(*typeargs):
''' Decorator way '''
def wrap(f):
def wrapper(*args):
for a, t in zip(args, typeargs):
if not isinstance(a, t):
raise Exception('{a} not of type {t}'.format(a=a, t=t))
return f(*args)
return wrapper
return wrap

@accepts(int)
def add1(n):
return n + 1
</pre>

<pre>
## Memoize
def memoize(f):
def wrapper(*args):
if not wrapper.ret:
wrapper.ret = f(*args)
return wrapper.ret
wrapper.ret = False
wrapper.__name__ = f.__name__
return wrapper

import time

@memoize
def sumto10():
count = 0
sum = 0
while count < 10:
sum += count
time.sleep(1)
count += 1
return sum

# first time will count slow
print sumto10()
# second time will return cached value instantly
print sumto10()
</pre>

Memoize: "Caches a function's return value each time it is called. If called later with the same arguments, the cached value is returned (not reevaluated)" [http://wiki.python.org/moin/PythonDecoratorLibrary#Memoize]

<pre>
## Nested decorators
import time
@shout
@memoize
def test1():
print "calculating"
time.sleep(3)
return 100
</pre>

Order is important! For tested decorators the execution order is from the top to the bottom. If you reverse @shout and @memoize, the @shout will not run because of the way @memoize was written.

<pre>
## registry
registry = []
# can also decorate decorators:
@shout
def registry(f):
registry.append(f)
return f

@register
def test2():
pass
</pre>

Decorator is just a function that returns a function.

Can use functools to automatically update decorator wrappers with __name__, etc [http://docs.python.org/2/library/functools.html]
<pre>
## Auto syntax sygar the wrapper for you
import functools
def my_decorator(f):
@functools.wraps(f)
def wrapper(*args, **kwds):
print 'Calling decorated function'
return f(*args, **kwds)
return wrapper
</pre>

=== Class Method Decorators ===

Static methods: [http://docs.python.org/library/functions.html#staticmethod]
: Similar to C++ and Java static methods
<pre>
class C:
@staticmethod
def say(msg):
print(msg)

C.say('hi')
</pre>

Class method: [http://docs.python.org/library/functions.html#classmethod]
: A class method receives the class as implicit first argument, just like an instance method receives the instance.
: Class methods are different than C++ or Java static methods. If you want those, see staticmethod() in this section.
<pre>
@classmethod
def setup_parser(cls, parser):
</pre>

See also [[#Class Properties]]

=== Class Properties ===

Properties: (not using decorators)
<pre>
class C(object):
def __init__(self):
self._x = None
def getx(self):
print("get x")
return self._x
def setx(self, value):
print("set x")
self._x = value
def delx(self):
print("del x")
del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")
</pre>

<pre>
#usage
c = C()
c.x = 5
print(c.x)
del(c.x)
</pre>

Getter and Setter Property Decorators: [http://docs.python.org/library/functions.html#property]
<pre>
class C(object): # make sure to inherit from object for these properties to work!
def __init__(self):
self._x = None

@property
def x(self):
"""I'm the 'x' property."""
print("get x")
return self._x

@x.setter # python >= 2.6
def x(self, value):
print("set x")
self._x = value

@x.deleter # python >= 2.6
def x(self):
print("del x")
del self._x
</pre>

<pre>
#usage
c = C()
c.x = 5
print(c.x)
del(c.x)
</pre>

Note: If you don't add the @x.setter, the property will be read only, and will throw a "AttributeError: can't set attribute" if attempted to be set.

Using x.getter example: [http://wiki.python.org/moin/AlternativeDescriptionOfProperty]
<pre>
class Cls(object):
def __init__(self):
self._x = 100

x = property()

@x.getter
def x(self):
"""I'm the 'x' property."""
return self._x

@x.setter
def x(self, value):
self._x = value

@x.deleter
def x(self):
del self._x
</pre>

=== Custom Decorators ===

decorator
@dec1
def func(arg1, arg2, ...): pass

equivalent to:
def func(arg1, arg2, ...): pass
func = dec1(func)

Basic layout (names are arbitrary):
<pre>
def my_decorator(function_name):
# update wrapper.__doc__ and .func_name
# or @functools.wraps(my_wrapper)
def mywrapper(*args, **kw):
# do something before
result = function_name(*args, **kw)
# do something after
return result
return my_wrapper
</pre>

Example:
<pre>
def limit(length):
def decorator(function):
def wrapper(*args, **kw):
result = function(*args, **kw)
result = result[:length]
return result
return wrapper
return decorator
</pre>

<pre>
@limit(3)
def echo(foo): return foo

echo('123456') # '123'
</pre>

=== assert ===

assert True
assert False # exception
assert 1 == 1
assert 1 == 2, "some message" # AssertionError: some message

=== Snake Case ===

Convert Camel Case to Snake Case: [http://stackoverflow.com/questions/1175208/elegant-python-function-to-convert-camelcase-to-snake-case]

<pre>
def snake_case(name):
s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name)
return re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1).lower()
</pre>

<pre>
>>> convert('CamelCase')
'camel_case'
</pre>

=== Camel Case ===

Camel Case: [http://stackoverflow.com/questions/8347048/camelcase-every-string-any-standard-library]

>>> ''.join(x for x in 'make IT camel CaSe'.title() if not x.isspace())
'MakeItCamelCase'

To titleize a sentence:
>>> "this is a test".title()
"This Is A Test"

=== Lambda ===

Simple functions
mul = lambda a, b: a * b

# equivalent to:
def mul(a, b):
return a * b

Sort usage:
data = [dict(number=x) for x in '036149']
data.sort(key=lambda x: float(x['number']))

Compare usage:
data = [dict(number=x) for x in '036149']
data.sort(cmp=lambda x,y: cmp(x['number'], y['number']))

Boolean:
lambda x: x % 3 == 0

Multiple parameter:
lambda x, y: x + y

Call another function:
lambda word: len(word)

Strings:
lambda line: line.split()[2]

Embedded:
def transform(n):
return lambda x: x + n

No parameters:
lambda: 1

Default parameters:
lambda x=1: x+1

References:
* Python: Lambda Functions - http://www.secnetix.de/olli/Python/lambda_functions.hawk

=== Map ===

Apply a function to items of a sequence:
map(str, range(3)) # ['0', '1', '2']

Note: list comprehensions can replace map and filter

=== Reduce ===

Apply a function to pairs of the sequence:
import operator
reduce(operator.mul, [1,2,3,4]) # 24 - ((1*2)*3)*4)

Note: sum or for loop can replace reduce

=== Filter ===

Return a sequence of items for which function(item) is True
filter(lambda x: x >= 0, [0, -1, 3]) # [0, 3]

Note: list comprehensions can replace map and filter

Find matching items in two lists:
a = ['a','b','c']
b = ['c','d']
filter(lambda x: x in a, b) # returns only 'c' as it is in both lists

=== List comprehension ===

Nothing more than Syntactic Sugar

5. Data Structures — Python v2.7.5 documentation - http://docs.python.org/2/tutorial/datastructures.html#list-comprehensions
: "List comprehensions provide a concise way to create lists."
: "A list comprehension consists of '''brackets''' containing an '''expression''' followed by a '''for clause''', then '''zero or more for or if clauses'''. "

Remove newlines from each item in a list:
mylist = [x.strip() for x in mylist]

Simple:
doubled = [x*2 for x in mylist]
squares = [x**2 for x in range(10)]

If:
[ 2*x for x in range(1,10) if x >= 5 ]

Complex:
[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]

Expanding PI:
[str(round(pi, i)) for i in range(1, 6)]

Return list of words
my_list = [current_word.lower() for current_word in words]

Utah Python July 2013 Meeting: (http://basepi.net)
<pre>
# Iterative
squares = []
for x in range(10):
squares.append(x**2)
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

## Functional
squares = map(lambda x: x**2, range(10))

## List comprehension
squares = [x**2 for x in range(10)]
even_squares = [x**2 for x in range(10) if x%2 == 0]

# get hash of strings in list:
import sha
strings = ['Python is cool!', 'I love python']
hashes = [sha.sha(x).digest() for x in strings]

# flatten matrix:
matrix = [[1, 2, 3], [4, 5, 6]]
[x for row in matrix for x in row]
# [1, 2, 3, 4, 5, 6]

# build matrix:
[(x,y) for x in range(10) for y in range(10)]
# matrix of not both odd and not both even
[(x,y) for x in range(10) for y in range(10) if x%2 != y%2]

# suck lines of files into list, ignoring commented lines
[line.strip().split("\t") for line in open("my_file.tab")
if not line.startswith('#')]
</pre>

References:
* 5.1.4. List Comprehensions - Data Structures — Python v2.7.2 documentation - http://docs.python.org/tutorial/datastructures.html#list-comprehensions
* Utah Python July 2013 Meeting - YouTube - https://www.youtube.com/watch?v=uh9uhzitdvw [http://basepi.net]

=== [http://docs.python.org/reference/expressions#calls Mass Parameter Passing] ===

Using: *args and **kw
* *args a tuple of remaining unnamed parameters
* **kw a dictionary of remaining named parameters

def param_func(a, b='b', *args, **kw):
print( [a, b, args, kw] )

WARNING: If not careful, this can result in this error: "TypeError: param_func() got multiple values for keyword argument 'myvar'". Might be better to just use:

def param_func(*args, **kw):
print( [args, kw] )

param_func(1, 2, 3, 4, 5) # [1, 2, (3, 4, 5), {}]
param_func(1, 2, 3, 4, 5, joe='test') # [1, 2, (3, 4, 5), {'joe': 'test'}]
mylist = ("a", "c", "g", "h", 5)
param_func(mylist) # [('a', 'c', 'g', 'h', 5), 'b', (), {}]
param_func(*mylist) # ['a', 'c', ('g', 'h', 5), {}] *tricky*
Note: *mydict will strip out just keys and treat like list
Note: **mydict will treat like named parameters

def mywrapper(*args, **kw):
# do something before
result = function_name(*args, **kw)

Passing arbitrary parameters:
<pre>
def fun2(**kw):
print kw # {'a': '1', 'b': '2', 'c': '3'}

def fun1(c='3', **kw):
print kw # {'a': '1', 'b': '2'}
fun2(c='3', **kw)

fun1(a='1', b='2')
</pre>

=== [http://www.python.org/dev/peps/pep-0234/ Iteration Protocol] ===

[http://www.python.org/dev/peps/pep-0234/ PEP 234 -- Iterators]

Making a class iterable:
<pre>
class mycount(object):
def __init__(self, max):
self.x = 0
self.max = max
def __iter__(self): # return object that implements __next__() or next()
return self
def next(self):
if self.x < self.max:
self.x += 1
return self.x
else:
self.x = 0
raise StopIteration()
</pre>

for i in mycount(10): print i # 1 ... 10
print [x for x in mycount(10) ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10)
for i in c: print i # 1 ... 10
[ x for x in c ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

=== Generators ===

Generators are functions that return to their previous state (using 'yield') when iterated over, which can be used in for loops.

==== Yield ====

Improve on Iterators:
* Infinite data (infinite iterators)
* Memory-intensive iteration
* Cooperative Multi-threading

Generators are functions that return to their previous state when iterated over, which can be used in for loops.

Note: Generators throw StopIteration when completed

Note: the 'yield' keyword will stash away the state of the function and return the value. (Continuations)

<pre>
def gen_range(end):
cur = 0
while cur < end:
yield cur
# return here next time
cur += 1

# for loop iteration
for i in gen_range(10): print i

# manual iteration
my_gen = gen_range(10)
try:
while True:
print my_gen.next()
except StopIteration:
pass

# list comprehension
numbers = [x for x in gen_range(10)]
</pre>

Note: one can place an empty 'return' in a generator if needed. A 'return' with argument is invalid.

Making a class generate:
<pre>
class mycount(object):
def __init__(self, max):
self.max = max
def __iter__(self):
return self.next() # change here
def next(self): # change here
x = 0
while x < self.max:
x += 1
yield x

for i in mycount(10): print i # 1 ... 10
print [x for x in mycount(10) ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10)
for i in c: print i # 1 ... 10
[ x for x in c ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

itr = mycount(10).next()
itr.next() # 1
itr.next() # 2 - eventually throws StopIteration exception
</pre>

Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
<pre>
* Iterators are used to traverse a list
* Generators are more powerful Iterators

# simple 'n' number in functional form
def firstn(n):
num = 0
nums = []
while num < n:
nums.append(num)
num += 1
return nums

# simple 'n' number in generator form
def firstn(n):
num = 0
while num < n:
yield num
num += 1

-

# Infinite list generator
def counter():
num = 0
while True:
yield num
num += 1

# Fibbonacci generator
def fib():
num1 = 0
num2 = 1
yield num1
while True:
num1, num2 = num2, num1 + num2
yield num1

# Fibbonacci generator 2
def fib(n):
num1 = 0
num2 = 1
count = 0
yield num1
while count < n:
count += 1
num1, num2 = num2, num1 + num2
yield num1
</pre>

Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
<pre>
## Cooperative Multi-Threading:
# Cooperative multithreading relies on the threads themselves to relinquish control once they are at a stopping point.

def mythread(name):
print('Starting Thread: {name}...'.format(name=name))
counter = 10
while counter > 0:
print('Thread: {name} doing some work!'.format(name=name))
counter -= 1
yield

thread1 = mythread('Alice')
thread2 = mythread('Bob')
thread3 = mythread('Trudy')

try:
while True:
thread1.next()
thread2.next()
thread3.next()
except StopIteration:
pass
</pre>

References:
* Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
** http://files.basepi.net/talks/pyarmory.zip

=== dunder methods ===

Some dunder (double under) methods determine what will happen when + (__add__) or other similar numerical symbols used.

<pre>
class str_add(object):
def __init__(self, name):
self.name = name
def __add__(self, other):
self.name += other.name
return self
# note: only need to return self if you want to go beyond 'a + b'

a = str_add('a_')
b = str_add('b_')
c = str_add('c_')
d = a + b + c
print(a.name) # a_b_c_
print(b.name) # b_
print(c.name) # c_
print(d.name) # a_b_c_
</pre>

Other common methods to override:
* __add__ for +
* __sub__ for -
* __mul__ for *
* __div__ for /
* __mod__ for %

References:
* Emulating numeric types - http://docs.python.org/2/reference/datamodel.html#emulating-numeric-types

=== [http://docs.python.org/library/doctest.html doctest] ===

test.py:
<pre>
#!/usr/bin/env python
"""
This is the "example" module.

>>> double(5)
10
"""

def double(n):
"""
This is double function

>>> double(3)
6
"""
return n * 2

if __name__ == "__main__":
import doctest
doctest.testmod()
</pre>

=== unittest ===

unittest.TestCase

Here is a short script to test three functions from the random module:
<pre>
import random
import unittest

class TestSequenceFunctions(unittest.TestCase):

def setUp(self):
self.seq = range(10)

def test_shuffle(self):
# make sure the shuffled sequence does not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, range(10))

# should raise an exception for an immutable sequence
self.assertRaises(TypeError, random.shuffle, (1,2,3))

def test_choice(self):
element = random.choice(self.seq)
self.assertTrue(element in self.seq)

def test_sample(self):
with self.assertRaises(ValueError):
random.sample(self.seq, 20)
for element in random.sample(self.seq, 5):
self.assertTrue(element in self.seq)

if __name__ == '__main__':
unittest.main()
</pre>

The unittest module can be used from the command line to run tests from modules, classes or even individual test methods:
python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method
python -m unittest -v test_module # verbose
python -m unittest -h # help

References:
* 25.3. unittest — Unit testing framework — Python v2.7.4 documentation - http://docs.python.org/2/library/unittest.html
* Python Debugging Tips - Google Drive - https://docs.google.com/presentation/d/130Wl--IzbDh9ymF6KHMu0WrsIwz6k32pav1gFbsw7NI/edit?usp=sharing

=== Internet Access ===

<pre>
# Python 3
from urllib.request import urlopen
for line in urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'):
line = line.decode('utf-8')
if 'EST' in line or 'EDT' in line: # look for eastern time
print(line)
</pre>

=== Simple SMTP ===

Requires local STMP server.

<pre>
import smtplib
server = smtplib.SMTP("localhost")
server.sendmail('mrken@demo.oeey.com', 'kenneth@demo.oeey.com',
"""To: kenneth@demo.oeey.com
From: mrken@demo.oeey.com
Subject: This is fun

Beware of the dog.
""")
server.quit()
</pre>

<pre>
s = smtplib.SMTP(smtp_server)
</pre>

with TLS:
<pre>
s = smtplib.SMTP("smtp.gmail.com",587)
s.ehlo()
s.starttls()
s.ehlo()
</pre>

SSL:
<pre>
s = smtplib.SMTP_SSL('smtp.mail.yahoo.com', 465)
</pre>

=== Data Compress ===

zlib, gzip, bz2, zipfile, tarfile

<pre>
import zlib
s = b'hello world' # only in python >3
len(s)
t = zlib.compress(s)
zlib.decompress(t)
zlib.crc32(s)
</pre>

=== gzip ===

<pre>
import gzip
content = "Lots of content here"
f = gzip.open('file.txt.gz', 'wb')
f.write(content)
f.close()
</pre>

<pre>
import gzip
f_in = open('file.txt', 'rb')
f_out = gzip.open('file.txt.gz', 'wb')
f_out.writelines(f_in)
f_out.close()
f_in.close()
</pre>

=== Performance Timer ===

Time a task repeated 10 times:
<pre>
from timeit import Timer
Timer('t=a; a=b; b=t', 'a=1; b=2').timeit()
Timer('print("hi")').timeit(10) # repeat 10 times and time
</pre>

Simple method:
<pre>
start_time = int(time.time())
... task ...
compute_time = int(time.time()) - start_time
</pre>

=== Python Version Check ===

Solution:

__init__.py:
<pre>
# Check Python Version:
import sys
assert sys.version_info >= (2, 6, 0), 'requires Python 2.6 or higher'
del sys
</pre>

resulting error if not > 2.6:
<pre>
>>> import nucleon
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "nucleon/__init__.py", line 9, in ?
assert sys.version_info >= (2, 6, 0), 'nucleon requires Python 2.6 or higher'
AssertionError: nucleon requires Python 2.6 or higher
</pre>

Other solutions:

Note: this will throw syntax error on 2.4 due to ternary
<pre>
import sys
if sys.version_info < (2, 4):
raise "must use python 2.5 or greater"
else:
# syntax error in 2.4, ok in 2.5
x = 1 if True else 2
print x
</pre>

maybe solution:
<pre>
try:
eval("1 if True else 2")
except SyntaxError:
# doesn't have ternary
</pre>

<pre>
>>> try:
... from my_module import twoPointSixCode
... except Exception:
... print "can't import, probably because your python is too old!"
>>>
</pre>

<pre>
#!/usr/bin/env/python
import sys
if sys.version_info<(2,6,0):
sys.stderr.write("You need python 2.6 or later to run this script\n")
exit(1)
</pre>

<pre>
>>> import sys
>>> sys.hexversion # hex printed as integer
33817584
>>> '%x' % sys.hexversion # output as hex, not zero fill 2.4.3
'20403f0'
>>> sys.hexversion < 0x02060000 # test if version less than 2.6
True
>>> print "0x%08x" % sys.hexversion # print as hex and zero fill and prepend '0x'
0x020403f0
</pre>

<pre>
import sys
if sys.hexversion < 0x02060500:
print >> sys.stderr, "This script requires python 2.6.5 or higher."
exit(1)
</pre>

<pre>
>>> import sys
>>> sys.version_info
(2, 6, 4, 'final', 0)
>>> if not sys.version_info[:2] == (2, 6):
... print "Error, I need python 2.6"
... else:
... from my_module import twoPointSixCode
>>>
</pre>

References:
* http://stackoverflow.com/questions/446052/python-best-way-to-check-for-python-version-in-program-that-uses-new-language-fe
* http://stackoverflow.com/questions/1093322/how-do-i-check-what-version-of-python-is-running-my-script

=== [http://docs.python.org/library/struct.html Struct - Binary Data Record Layout] ===

Format Characters:
<pre>
Format C Type Python type Standard size
x pad byte no value
c char str of len 1 1
b signed char integer 1
B unsigned char integer 1
? _Bool bool 1
h short integer 2
H unsigned short integer 2
i int integer 4
I unsigned int integer 4
l long integer 4
L unsigned long integer 4
q long long integer 8
Q unsigned long long integer 8
f float float 4
d double float 8
s char[] string
p char[] string
P void * integer
</pre>

H # two byte unassigned number
I # four byte unassigned number
< # standard size and in little-endian byte order

import struct
data = struct.pack('<IIIHH', 1, 2, 3, 4, 5)
# data = '\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x05\x00'

fields = struct.unpack('<IIIHH', data)
# fields = (1L, 2L, 3L, 4, 5)
# using splice: fields = struct.unpack('<IIIHH', data[start:start+16])

f1, f2, f3, f4, f5 = fields

hex(f1) # hex a field:
# '0x1L'

>>> struct.pack("<5sh", "helloo", 11)
'hello\x0b\x00'
>>> struct.unpack("<5sh", 'hello\x0b\x00')
('hello', 11)

=== pdb interactive debugger ===

# uncomment the following to use the interactive debugger:
# import pdb; pdb.set_trace()

=== [http://docs.python.org/library/telnetlib.html Telnet] ===

create:
class telnetlib.Telnet([host[, port[, timeout]]])

Note: The instance is initially not connected by default; the open() method must be used to establish a connection. Alternatively, the host name and optional port number can be passed to the constructor, to, in which case the connection to the server will be established before the constructor returns.

open if not initially done so:
Telnet.open(host[, port[, timeout]])

Functions:
Telnet.read_until(expected[, timeout]) # Read until a given string, expected, is encountered or until timeout seconds have passed.
# on timeout return whatever is in queue, possibly empty string
Telnet.close() # close connection
Telnet.read_all() # Read all data until EOF; block until connection closed.
Telnet.write(buffer) # Write a string to the socket, doubling any IAC characters. This can block if the connection is blocked.
Telnet.expect(list[, timeout]) # Read until one from a list of a regular expressions matches.
# Return a tuple of three items: the index in the list of matches; the match object; and the text read to match.

Read until end of line:
line = tn.read_until("\n")

<pre>
import getpass
import sys
import telnetlib

HOST = "localhost"
user = raw_input("Enter your remote account: ")
password = getpass.getpass()

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ")
tn.write(user + "\n")
if password:
tn.read_until("Password: ")
tn.write(password + "\n")

tn.write("ls\n")
tn.write("exit\n")

print tn.read_all()
</pre>

<pre>
#!/usr/bin/env python

import sys
import telnetlib

HOST = "216.119.193.252"
USER = "admin"
PASS = "@lt12345"

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ")
tn.write(USER + "\n")
tn.read_until("Password: ")
tn.write(PASS + "\n")
tn.read_until("# ")

tn.write("terminal length 0\n")
tn.read_until("# ")

tn.write("show running-config\n")
configlines = tn.read_until("# ")

tn.write("exit\n")
tn.read_all()

print(configlines)
</pre>

=== SSH ===

Cheap method:
os.system('ssh user@server command')

SecureShell - PythonInfo Wiki - http://wiki.python.org/moin/SecureShell
: There are several ways to use Secure Shell (SSH) in Python
* pyssh - http://pyssh.sourceforge.net/
* paramiko - Paramiko is a native Python implementation of SSH. - http://www.lag.net/paramiko/
* conch - conch is another native implementation of SSH and part of the Twisted Matrix project - http://twistedmatrix.com/projects/conch
* SSH wrapper - Some tools just wrap around existing ssh/sftp implementations
** keyphrene - http://www.keyphrene.com/products/org.keyphrene/
** keyphrene SSH wrapper - http://membres.lycos.fr/fredp/python/pyscp.html
* Fabric - http://docs.fabfile.org/en/1.4.0/index.html
* pexpect - Noah.org - http://www.noah.org/wiki/Pexpect

SFTP:
* SFTP in Python: Really Simple SSH | The best of Zeth - http://zeth.net/post/332/

pexpect:
wget http://pexpect.sourceforge.net/pexpect-2.3.tar.gz
tar xzf pexpect-2.3.tar.gz
cd pexpect-2.3
sudo python ./setup.py install

=== SFTP ===

SFTP in Python? (platform independent) - Stack Overflow - http://stackoverflow.com/questions/432385/sftp-in-python-platform-independent

paramiko: ssh2 protocol for python - http://www.lag.net/paramiko/

pip install paramiko

mkdir -p ~/.src ; cd ~/.src
wget http://www.lag.net/paramiko/download/paramiko-1.7.7.1.zip
unzip paramiko-1.7.7.1.zip
cd paramiko-1.7.7.1
python setup.py install

SFTP Methods:
* chdir
* chmod
* chown
* close
* file
* from_transport
* get
* get_channel
* getcwd
* listdir
* listdir_attr
* logger
* lstat
* mkdir
* normalize
* open
* put
* readlink
* remove
* rename
* request_number
* rmdir
* sock
* stat
* symlink
* truncate
* ultra_debug
* unlink
* utime

<pre>
import paramiko

host = "THEHOST.com" #hard-coded
port = 22
transport = paramiko.Transport((host, port))

password = "THEPASSWORD" #hard-coded
username = "THEUSERNAME" #hard-coded
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

import sys
path = './THETARGETDIRECTORY/' + sys.argv[1] #hard-coded
localpath = sys.argv[1]
sftp.put(localpath, path)

sftp.close()
transport.close()
print 'Upload done.'
</pre>

VMware Example:
<pre>
#!/usr/bin/env python

import paramiko
import sys

host = "sftp2.engx.vmware.com"
port = 443
transport = paramiko.Transport((host, port))

username = "xxxx"
password = "xxx"
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

print dir(sftp)

print sftp.chdir("build-uploader")
print sftp.listdir()

sftp.close()
transport.close()
</pre>

=== paramiko ===

See [[Python/paramiko]]

=== [http://docs.python.org/library/getpass.html Get Password] ===

Prompt for password:
getpass.getpass(prompt='Password: ')

import getpass
user = raw_input("Enter your remote account: ")
password = getpass.getpass()

=== [http://docs.python.org/library/configparser.html ini files] ===

<pre>
ini_filename = 'my.ini'

import ConfigParser
parser = ConfigParser.SafeConfigParser()
parser.read(ini_filename)

sections = parser.sections() # get sections
for section in sections:
print "[" + section + "]"
for name, value in parser.items(section):
print name + " = " + value
</pre>

<pre>
if parser.has_option(section, option):
print( parser.get(section, option) )
</pre>

To have the option names not convert to lower case:
parser.optionxform = str

Ignore section: [http://stackoverflow.com/questions/2885190/using-pythons-configparser-to-read-a-file-without-section-name]
ini_str = '[root]\n' + open(ini_path, 'r').read()
ini_fp = StringIO.StringIO(ini_str)
config = ConfigParser.RawConfigParser()
config.readfp(ini_fp)

Get option:
config.has_option('root', 'some_key')
config.get('root', 'some_key')

config.items('root') # list of Key Value tuples

=== optparse - Option Parser ===

[http://docs.python.org/library/optparse.html - Option Parser

Note: Deprecated since version 2.7: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

<pre>
import optparse
parser = optparse.OptionParser()

parser.add_option('--name', default=None)

parser.add_option(
'-l', '--loglevel',
help='Log Level (default: info)',
default="info", action='store', type='string')

(options, args) = parser.parse_args()

# if no parameters passed show help
if len(sys.argv) == 1:
parser.print_help()
sys.exit(1)

# if a stand alone parameter passed, override option
if args:
options.target = args[0]

# show option:
print(options.loglevel)
</pre>

Note: Order is important when listing add_options!

Pretty Description:
parser = optparse.OptionParser(description="Purpose: to do something")
# or
parser.description = "Purpose: to do something"

Parse arguments:
(options, args) = parser.parse_args()
Note: If you like, you can pass a custom argument list to parse_args(), but that’s rarely necessary: by default it uses sys.argv[1:].

Change Usage Message:
<pre>
usage = "%prog [options]" # default message
# Usage: <prog> [options]" # where <prog> is replaced with the script name

parser = OptionParser(usage=usage)
# or
parser.usage = usage # default message
# or
parser.set_usage(usage)
</pre>

Remove option: (does not remove default value from 'options' variable!!! - stupid!)
parser.remove_option('--loglevel')

Modify an option: (changes are saved to parser automatically, unless in an option group)
option = parser.get_option('--loglevel')
option.help = 'something else'
option.default = 'debug'

# or
parser.remove_option("--loglevel")
parser.add_option("--loglevel", help="something else", default="info")

Modify an option in an option group:
opt_group = parser.get_option_group("--loglevel")
opt_group.remove_option("--loglevel")
opt_group.add_option("--loglevel", help="something else", default="info")

Types: "string", "int", "long", "choice", "float" and "complex" [http://docs.python.org/library/optparse.html#optparse-standard-option-types]
Defaults:
* Default actions is 'store'.
* Default type is 'string'.
* Default dest is the long name of the parameter (or short name if no long name provided)
* Default value is None

<pre>
from optparse import OptionParser
[...]
parser = OptionParser()

# string
parser.add_option("-f", "--file", dest="filename",
help="write report to FILE", metavar="FILE")

# boolean - action is what to do when flag found
parser.add_option("-q", "--quiet",
action="store_true", dest="verbose", default=False,
help="don't print status messages to stdout")

# integer
parser.add_option(
"-m", "--minutes",
help="Minutes (default: 60)",
default=60, type='int')

(options, args) = parser.parse_args()

print "Filename: " + options.filename
</pre>

fake handling of required option:
<pre>
if not option.filename:
parser.print_help()
parser.error("filename is required")
</pre>

change default usage:
parser = OptionParser(usage="Usage: %prog [args] arg1 arg2")

Really ugly way to print help usage :-(
<pre>
def Usage():
pname = sys.argv[0]
cmd='%s -h' % (pname)
os.system(cmd)

if (len(sys.argv) < 2):
Usage()
sys.exit(1)
</pre>

Groups:
<pre>
# OptionGroup(parser, title, description=None)
group = OptionGroup(parser, "Dangerous Options",
"Caution: use these options at your own risk. "
"It is believed that some of them bite.")
group.add_option("-g", action="store_true", help="Group option.")
parser.add_option_group(group)
</pre>

Hide an option:
* Option.help
: "Help text to print for this option when listing all available options after the user supplies a help option (such as --help). If no help text is supplied, the option will be listed without help text. To hide this option, use the special value optparse.SUPPRESS_HELP."
<pre>
parser.add_option("--secret", help=SUPPRESS_HELP)
</pre>

Issues with Option Parser:
* No handling for positional arguments
* No adding help for positional arguments
* No handling for required field

=== argparse ===

argparse - http://docs.python.org/library/argparse.html

argparse — Parser for command-line options, arguments and sub-commands

NOTE: MUCH MORE POWERFUL THAN OPTION PARSER!

Note: New in version 2.7.

Installation on <= 2.6
<pre>
# Install argparse: - http://code.google.com/p/argparse/
# REQUIRES SETUPTOOLS
mkdir -p ~/.src ; cd ~/.src
wget http://argparse.googlecode.com/files/argparse-1.2.1.tar.gz
tar -zvxf argparse-1.2.1.tar.gz
cd argparse-1.2.1
sudo python setup.py install
</pre>

The argparse module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse will figure out how to parse those out of sys.argv. The argparse module also automatically generates help and usage messages and issues errors when users give the program invalid arguments.

<pre>
import argparse

parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
const=sum, default=max,
help='sum the integers (default: find the max)')

args = parser.parse_args()
print args.accumulate(args.integers)
</pre>

<pre>
import argparse

argparser = demoproject.util.argparse.ArgumentParser(
usage='Kick off demoproject benchmarks.')
argparser.add_argument(
'--card',
help='What cards to run on. Depends on Jenkins NODE_LABEL environment'
'variable for correct mappings.')
argparser.add_argument(
'--suite', metavar='SUITE',
help='Specify the appropriate benchmark suite as defined in '
'the benchmark ini file. Defaults to standard_perf',
default='standard_perf')
args = argparser.parse_args()

print args.card
</pre>

Remove an argument: [http://stackoverflow.com/questions/32807319/disable-remove-argument-in-argparse]
<pre>
def remove_options(parser, options):
""" remove_options(parser, ['--some-option', '--some-option2']) """
for option in options:
for action in parser._actions:
if vars(action)['option_strings'][0] == option:
parser._handle_conflict_resolve(None,[(option,action)])
break
</pre>

=== [http://docs.python.org/library/commands.html commands] ===

35.16. commands — Utilities for running commands — Python v2.7.1 documentation - http://docs.python.org/library/commands.html

Note: Deprecated since version 2.6: The commands module has been removed in Python 3.0. Use the subprocess module instead.

The commands module contains wrapper functions for os.popen() which take a system command as a string and return any output generated by the command and, optionally, the exit status.

Get status and output:
status, output = commands.getstatusoutput(cmd)

Get output ignoring status:
output = commands.getoutput(cmd)

NOTE: DO NOT USE getstatus() AS IT DOES NOT DO WHAT YOU THINK IT DOES!
status = commands.getstatus(file)

<pre>
>>> import commands

>>> commands.getstatusoutput('ls /bin/ls')
(0, '/bin/ls')

>>> commands.getstatusoutput('cat /bin/junk')
(256, 'cat: /bin/junk: No such file or directory')

>>> commands.getstatusoutput('/bin/junk')
(256, 'sh: /bin/junk: not found')

>>> commands.getoutput('ls /bin/ls')
'/bin/ls'

>>> commands.getstatus('/bin/ls')
'-rwxr-xr-x 1 root 13352 Oct 14 1994 /bin/ls'
</pre>

=== [http://docs.python.org/library/_winreg.html Windows Registry] ===

Note: The _winreg module has been renamed to winreg in Python 3.0.

* 32.3. winreg – Windows registry access — Python v3.2 documentation - http://docs.python.org/py3k/library/winreg.html
* 34.3. _winreg – Windows registry access — Python v2.7.1 documentation - http://docs.python.org/library/_winreg.html

Read from registry:
<pre>
from _winreg import *
key = OpenKey(HKEY_LOCAL_MACHINE, r'Software\Microsoft\Outlook Express', 0, KEY_ALL_ACCESS)
QueryValueEx(key, "InstallRoot")
</pre>

Write to registry:
<pre>
keyVal = r'Software\Microsoft\Internet Explorer\Main'
try:
key = OpenKey(HKEY_CURRENT_USER, keyVal, 0, KEY_ALL_ACCESS)
except:
key = CreateKey(HKEY_CURRENT_USER, keyVal)
SetValueEx(key, "Start Page", 0, REG_SZ, "http://www.blog.pythonlibrary.org/")
CloseKey(key)
</pre>

Enumerate registry keys:
<pre>
from _winreg import EnumKey, HKEY_USERS

try:
i = 0
while True:
subkey = EnumKey(HKEY_USERS, i)
print subkey
i += 1
except WindowsError:
# WindowsError: [Errno 259] No more data is available
pass
</pre>

Another enumeration example:
<pre>
# File: winreg-example-1.py

import _winreg

explorer = _winreg.OpenKey(
_winreg.HKEY_CURRENT_USER,
"Software\\Microsoft\\Windows\\CurrentVersion\\Explorer"
)

# list values owned by this registry key
try:
i = 0
while 1:
name, value, type = _winreg.EnumValue(explorer, i)
print repr(name),
i += 1
except WindowsError:
print

value, type = _winreg.QueryValueEx(explorer, "Logon User Name")

print
print "user is", repr(value)
</pre>

Enable Remote Desktop Example: (by Kenneth)
<pre>
from _winreg import *
key = OpenKey(HKEY_LOCAL_MACHINE, r'SYSTEM\CurrentControlSet\Control\Terminal Server', 0, KEY_ALL_ACCESS)
SetValueEx(key, "fDenyTSConnections", 0, REG_DWORD, 0)
ValueTypeTuple = QueryValueEx(key, "fDenyTSConnections") # (0, 4)
print("Please reboot the system to complete the changes")
</pre>

References:
* Python’s _winreg: Editing the Windows Registry « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/03/20/pythons-_winreg-editing-the-windows-registry/

=== JSON ===

JSON - http://docs.python.org/library/json.html

ENCODING TO JSON STRING (type str)

import json
json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
# '["foo", {"bar": ["baz", null, 1.0, 2]}]'

compact encoding:
import json
json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',',':'))
# '[1,2,3,{"4":5,"6":7}]'

Pretty printing:
import json
print json.dumps({'4': 5, '6': 7}, sort_keys=True, indent=4)
# {
# "4": 5,
# "6": 7
# }

DECODING JSON STRING TO PYTHON LIST OR DICTIONARY OBJECT

import json
json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]')
# [u'foo', {u'bar': [u'baz', None, 1.0, 2]}]

Load from file:
f = open(filename)
j = json.load(f)
print j["kind"] # "dual-controller-adapter"
#{
# "kind": "dual-controller-adapter",
#}

From Command Line:
echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool

# get value
echo '{"test":1,"test2":2}' | python -c 'import sys,json;data=json.loads(sys.stdin.read()); print data["test"]'

# read large file
python -c "import json;f=open('file.json');data=json.load(f);print json.dumps(data,indent=2)"
python -c "import json;print json.dumps(json.load(open('file.json')),indent=2)"

Load from web:
<pre>
import httplib
import json

def get_http_json(server, path):
""" This performs an HTTP GET against a server and path and returns the
resulting page data.

server The server we're hitting, e.g. demo.com
path The path we're hitting, e.g. api/part/MA12345/
"""
connection = httplib.HTTPConnection(server)
try:
url = 'http://%s/%s' % (server, path)
print 'Querying url %s' % url
connection.request('GET', url)
response = connection.getresponse()
if response.status == httplib.OK:
return json.loads(response.read())
else:
print 'Unable to get %s. %s %s' % (url, response.reason,
response.status)
return None
finally:
connection.close()

j = get_http_json(server, path)
</pre>

Pretty printing from the command line:
echo '{"json":"obj"}' | python -mjson.tool
#{
# "json": "obj"
#}

References:
* 18.2. json — JSON encoder and decoder — Python v2.7.1 documentation - http://docs.python.org/library/json.html

=== Sockets ===

See [[Python/Sockets]]

=== [http://docs.python.org/library/base64.html base64] ===

import base64

Encode:
encoded = base64.b64encode('data to be encoded')

Decode:
base64.b64decode(encoded)

=== daemonize ===

Creating a daemon the Python way « Python recipes « ActiveState Code - http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/

<pre>
import os
import sys

try:
pid = os.fork()
except OSError, e:
raise Exception, "%s [%d]" % (e.strerror, e.errno)

if (pid != 0): # parent
os._exit(0)

# create new process session
os.setsid()

# redirect stdin, stdout, stderr
os.open('/dev/null', os.O_RDWR)
os.dup2(0, 1)
os.dup2(0, 2)

#... tasks ...
</pre>

My Solution:
<pre>
import os
import sys
import signal
import syslog

HUP = False

def log(msg):
syslog.syslog(str(msg))
syslog.openlog('[WATER]')

def kill_signal_handler(signal, frame):
log("Being killed off: {0} {1}".format(signal, frame))
log("Shutting Down!")
sys.exit(0)
signal.signal(signal.SIGTERM, kill_signal_handler)
signal.signal(signal.SIGALRM, kill_signal_handler)

def hup_signal_handler(signal, frame):
global HUP
log("Signal reload received: {0} {1}".format(signal, frame))
HUP=True
signal.signal(signal.SIGHUP, hup_signal_handler)

def main_loop():
global HUP

while True:
# ... handle HUP=True ...

# ... main code ...

# wait 60 seconds before we start the next loop
time.sleep(60) # signals will interrupt this

if __name__ == "__main__":
log("Starting Daemon...")
child_id = os.fork()
if child_id:
log("Daemonized!")
sys.exit(0)
os.setsid()
# redirect stdin, stdout, stderr
os.open('/dev/null', os.O_RDWR)
os.dup2(0, 1)
os.dup2(0, 2)

# write pid
with open("water.pid", "w") as f:
f.write(str(os.getpid()))

while True:
try:
main_loop()
except Exception as e:
log("Critical Failure Caught: %s" % str(e))
log("Critical Failure - Restarting!")
time.sleep(30)
#raise e
</pre>

ln -s /opt/pyton27/bin/python pyton-water
./python-water water.py
killall python-water

==== fork ====

<pre>
import os
import sys
import time

child_id = os.fork()
if child_id:
print "parent exiting"
sys.exit(0)

while True:
time.sleep(3)
print "child"
</pre>

For windows, to hide the console, see [[#Hide Console]]

=== csv ===

13.1. csv — CSV File Reading and Writing — Python v2.7.2 documentation - http://docs.python.org/library/csv.html

Read:
<pre>
import csv
spamReader = csv.reader(open('eggs.csv', 'rb'), delimiter=' ', quotechar='|')
for row in spamReader:
print ', '.join(row)
</pre>

Write:
<pre>
import csv
spamWriter = csv.writer(open('eggs.csv', 'wb'), delimiter=' ',
quotechar='|', quoting=csv.QUOTE_MINIMAL)
spamWriter.writerow(['Spam'] * 5 + ['Baked Beans'])
spamWriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam'])
</pre>

=== SQLite ===

11.13. sqlite3 — DB-API 2.0 interface for SQLite databases — Python v2.7.2 documentation - http://docs.python.org/library/sqlite3.html

sqlite3 - New in version 2.5.

<pre>
import sqlite3

conn = sqlite3.connect('/tmp/example')

c = conn.cursor()

# Create table
c.execute('''create table stocks
(date text, trans text, symbol text,
qty real, price real)''')

# Insert a row of data
c.execute("""insert into stocks
values ('2006-01-05','BUY','RHAT',100,35.14)""")

# Show affected row count
print c.rowcount

# Save (commit) the changes
conn.commit()

# We can also close the cursor if we are done with it
c.close()
</pre>

parameter substitution:
<pre>
# Never do this -- insecure!
symbol = 'IBM'
c.execute("... where symbol = '%s'" % symbol)

# Do this instead
t = (symbol,)
c.execute('select * from stocks where symbol=?', t)
c.execute('select * from stocks where symbol=?', (symbol,))

# Larger example
for t in [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00),
('2006-04-06', 'SELL', 'IBM', 500, 53.00),
]:
c.execute('insert into stocks values (?,?,?,?,?)', t)
</pre>

select results:
<pre>
c = conn.cursor()
c.execute('select * from stocks order by price')

for row in c:
print row # row is a tuple

# OR (but not both)

rows = c.fetchall() # rows is list of tuples
print len(rows)
for row in rows:
print row
</pre>

Show which version of sqlite3: [http://zetcode.com/db/sqlitepythontutorial/]
<pre>
#!/usr/bin/python
# -*- coding: utf-8 -*-

import sqlite3 as lite
import sys

con = None

try:
con = lite.connect('test.db')

cur = con.cursor()
cur.execute('SELECT SQLITE_VERSION()')

data = cur.fetchone()

print "SQLite version: %s" % data

except lite.Error, e:

print "Error %s:" % e.args[0]
sys.exit(1)

finally:

if con:
con.close()
</pre>

Or simply: [http://stackoverflow.com/questions/1545479/force-python-to-forego-native-sqlite3-and-use-the-installed-latest-sqlite3-ver]
# echo -e "import sqlite3\nprint sqlite3.sqlite_version_info" | python
(3, 3, 6)

=== MySQL Database ===

Ubuntu packages:
* python-mysqldb - Python interface to MySQL
* python-mysql.connector - pure Python implementation of MySQL Client/Server protocol

apt-get install python-mysqldb
>>> import _mysql

apt-get install python-mysql.connector

PIP:
* https://pypi.python.org/pypi/MySQL-python
* https://pypi.python.org/pypi/mysqlclient

pip install MySQL-python

pip install mysqlclient

---

_mysql - http://mysql-python.sourceforge.net/MySQLdb.html

apt-get install python-mysqldb

<pre>
import _mysql

db=_mysql.connect(host="localhost",user="user",
passwd="pass",db="db")

db.query("""SELECT spam, eggs, sausage FROM breakfast
WHERE price < 5""")

r=db.store_result()
# ...or...
r=db.use_result()

row = r.fetch_row()
</pre>

=== couchdb ===

Getting started with couchdb-python — couchdb-python v0.8 documentation - http://packages.python.org/CouchDB/getting-started.html

<pre>
couch = couchdb.Server('http://127.0.0.1:5984/')

# database
db = couch.create('test') # create new
db = couch['mydb'] # or get existing

# print databases:
for db in couch:
print db

doc = {'foo': 'bar'}
doc_id, doc_rev = db.save(doc)
# (u'9278f7691c3421227a9f8436540008f9', u'1-4c6114c65e295552ab1019e2b046b10e')
# note: this also updates doc with _rev and _id
doc = db['9278f7691c3421227a9f8436540008f9']

# better to create a document using an id you want, instead of the randomly generated one
db[doc.id] = doc # this will write immediately to the database

# print all docs:
for id in db:
print id

# delete doc
db.delete(doc)

# delete database
couch.delete('test')
</pre>

Search database:
<pre>
# show all records
map_fun = '''function(doc) { emit(doc.name, null); }'''

# row = <Row id=u'12341234123412341234', key=None, value=None>
# emit(key, value) - returns the values for key and value

map_fun = '''
function(doc) {
if (doc.username == 'virt-01')
emit(doc.username, null);
}
'''

# row = <Row id=u'12341234123412341234', key=None, value=None>
for row in db.query(map_fun):
print row.id

for row in db.query(map_fun, descending=True):
print row.id
</pre>

couchdb-python - CouchDB Python Library - Google Project Hosting - http://code.google.com/p/couchdb-python/

Installation:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/C/CouchDB/CouchDB-0.8.tar.gz#md5=e32b39e459e1fa2586ea252712a11a59
tar -zvxf CouchDB-0.8.tar.gz
cd CouchDB-0.8
python setup.py install
</pre>

Test:
echo "import couchdb" | python

=== nose ===

nose: nose extends unittest to make testing easier

nose : Python Package Index - http://pypi.python.org/pypi/nose

Install:
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/n/nose/nose-1.1.2.tar.gz#md5=144f237b615e23f21f6a50b2183aa817
tar -zvxf nose-1.1.2.tar.gz
cd nose-1.1.2
sudo python setup.py install

skeleton:
mkdir skeleton
cd skeleton
mkdir bin NAME tests docs
touch NAME/__init__.py
touch tests/__init__.py
touch tests/NAME_tests.py

nose test skeleton:
<pre>
# tests/NAME_tests.py
from nose.tools import *
import klib

def setup():
print "SETUP!"

def teardown():
print "TEAR DOWN!"

def test_basic():
print "I RAN!"

# def test_something_else():
</pre>

Usage:
nosetests

See Also:
* [[#doctest]]
* [[#unittest]]

=== pydoc ===

25.1. pydoc — Documentation generator and online help system — Python v2.7.2 documentation - http://docs.python.org/library/pydoc.html

pydoc - The pydoc module automatically generates documentation from Python modules.

pydoc - same as 'import [library], help([library])'
$ pydoc [library]

Note: lists some special 'module metadata variables' separately (e.g. version, author).

special pydoc recognized metadata sections:
__version__
__author__
__credits__
__date__

==== pydoc function module ====

<pre>
Help on package klib:

NAME
klib - This is the name

FILE
/root/klib/klib/__init__.py

DESCRIPTION
These are module doc strings
that give helpful details

PACKAGE CONTENTS
markone

FUNCTIONS
func1()
This is a doc string
for func1

DATA
PI = 3.141
__all__ = ['func1', 'PI']
__author__ = 'Kenneth <kenneth@oeey.com>'
__copyright__ = 'Copyright 2007, The Project'
__credits__ = 'Credits go to me'
__date__ = '26 February 2001'
__email__ = 'ken@oeey.com'
__license__ = 'GPL'
__maintainer__ = 'Ken'
__status__ = 'Production'
__version__ = '$Revision: 83492 $'

VERSION
83492

DATE
26 February 2001

AUTHOR
Kenneth <kenneth@oeey.com>

CREDITS
Credits go to me
</pre>

Source:
<pre>
"""
This is the name

These are module doc strings
that give helpful details
"""

#__version__ = "1.0"
__version__ = "$Revision: 83492 $"
__author__ = "Kenneth <kenneth@oeey.com>"

__date__ = "26 February 2001"
__credits__ = "Credits go to me"

__all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license'
__license__ = "GPL"
__copyright__ = "Copyright 2007, The Project"
__maintainer__ = "Ken"
__email__ = "ken@oeey.com"
__status__ = "Production"

def func1():
"""
This is a doc string
for func1
"""
</pre>

==== pydoc class module ====

Sample:
<pre>
Help on module class_pydoc:

NAME
class_pydoc - This is the name

FILE
/root/class_pydoc.py

DESCRIPTION
These are module doc strings
that give helpful details

CLASSES
Toad

class Toad
| doc string for object
|
| Methods defined here:
|
| __init___(self, name)
| Constructor
|
| instance_function(self)
| Instance Function
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| class_function(cls) from __builtin__.classobj
| Class Function
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| my_class_attribute = 'sample class attribute'

DATA
PI = 3.141
__author__ = 'Kenneth <kenneth@oeey.com>'
__copyright__ = 'Copyright 2007, The Project'
__credits__ = 'Credits go to me'
__date__ = '26 February 2001'
__email__ = 'ken@oeey.com'
__license__ = 'GPL'
__maintainer__ = 'Ken'
__status__ = 'Production'
__version__ = '1.0'

VERSION
1.0

DATE
26 February 2001

AUTHOR
Kenneth <kenneth@oeey.com>

CREDITS
Credits go to me
</pre>

Code:
<pre>
"""
This is the name

These are module doc strings
that give helpful details
"""

__version__ = "1.0"
#__version__ = "$Revision: 83492 $"
__author__ = "Kenneth <kenneth@oeey.com>"

__date__ = "26 February 2001"
__credits__ = "Credits go to me"

#__all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license'
__license__ = "GPL"
__copyright__ = "Copyright 2007, The Project"
__maintainer__ = "Ken"
__email__ = "ken@oeey.com"
__status__ = "Production"

class Toad:
""" doc string for object """

my_class_attribute = 'sample class attribute'

def __init___(self, name):
""" Constructor """

def instance_function(self):
""" Instance Function """

@classmethod
def class_function(cls):
""" Class Function """
</pre>

=== username ===

User ID:
os.getuid()

User Name:
# fails it stdin piped to program
os.getlogin()

User Name:
import sys
import pwd
def get_username():
return pwd.getpwuid( os.getuid() )[ 0 ]

Environment:
os.environ['USER']

=== hostname ===

import socket
print(socket.gethostname())

os.uname()[1]

hostname = os.system('hostname')

=== Networking ===

==== Get Hostname ====

import socket
print(socket.gethostname())

==== Convert hostname to ip address ====

>>> socket.gethostbyaddr('demo')
('demo.oeey.com', [], ['10.10.10.10'])

/usr/local/bin/whatip: (or nametoip)
<pre>
#!/usr/bin/env python

#
# whatip - convert hostname to ip address
# author: Kenneth Burgener <kenneth@oeey.com> (FEB 2012)
#

import socket
import sys

if len(sys.argv) != 2:
print "Convert Hostname to IP Address"
print "Usage: %s HOSTNAME" % sys.argv[0].split('/')[-1]
sys.exit(1)

hostname = sys.argv[1]
try:
result = socket.gethostbyaddr(hostname)
print result[2][0]
except (socket.gaierror, socket.herror) as e:
print "Error: Unable to resolve '%s'" % hostname
sys.exit(1)
</pre>

==== Get Public IP Address ====

<pre>
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("gmail.com",80))
print(s.getsockname()[0])
s.close()
</pre>

networking - Finding local IP addresses using Python's stdlib - Stack Overflow - http://stackoverflow.com/questions/166506/finding-local-ip-addresses-using-pythons-stdlib

==== Python Network Server ====

<pre>
#!/usr/bin/python # This is server.py file

import socket # Import socket module

s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.
s.bind((host, port)) # Bind to the port

s.listen(5) # Now wait for client connection.
while True:
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
c.send('Thank you for connecting')
c.close() # Close the connection
</pre>

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
* 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

==== Python Network Client ====

<pre>
#!/usr/bin/python # This is client.py file

import socket # Import socket module

s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.

s.connect((host, port))
print s.recv(1024)
s.close # Close the socket when done
</pre>

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
* 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

=== Run Library Module as Script ===

Run python with -m option:
-m mod : run library module as a script (terminates option list)

Example:
cat [json_file] | python -m json.tool
# same as:
cat [json_file] | python /opt/python26/lib/python2.6/json/tool.py

Good for when library is buried deep in the library directories

=== Hex ===

<pre>
>>> hex(10) # print integer as hex string
'0xa'
>>> sys.hexversion # python version as integer
33817584
>>> hex(sys.hexversion) # python version as hex
'0x20403f0'
>>> print "0x%08x" % sys.hexversion # python version as hex
'0x20403f0'
>>> int('0xa', 16) # hex string to integer
10
>>> int('0a', 16) # hex string to integer
10
>>> int('0xa', 0) # hex string to integer auto detected du to '0x'
10
</pre>

=== Binary ===

Integer to Binary String with Negative Number Support: (2's complement)
<pre>
# source: http://www.gossamer-threads.com/lists/python/python/645216
def bin(x):
if x < -32768 or x > 65535:
raise Exception("Out of Range Error: %d" % x)
if x < 0:
x = 65535 + x + 1
bin_str = ''
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(15, 11, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(11, 7, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(7, 3, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(3, -1, -1))
return bin_str
</pre>

Integer to Binary String:
<pre>
# source: http://www.gossamer-threads.com/lists/python/python/645216
def bin(x):
if x < 0 or x > 65535:
raise Exception("Out of Range Error: %d" % x)
bin_str = ''
bin_str = ''.join(x & (1 << i) and '1' or '0' for i in range(15, 7, -1))
bin_str = bin_str + ' '
bin_str = bin_str + ''.join(x & (1 << i) and '1' or '0' for i in range(7, -1, -1))
return bin_str

print bin(0) # '00000000 00000000'
print bin(1) # '00000000 00000001'
print bin(255) # '00000000 11111111'
print bin(256) # '00000001 00000000'
print bin(43690) # '10101010 10101010'
print bin(65535) # '11111111 11111111'
print bin(65536) # '00000000 00000000' - rolled over to next set!
</pre>

Integer to Binary:
bin(170) # '0b10101010'

Binary to Integer:
int('10101010', 2) # 170

=== bitwise ===

See [[#Binary]]

BitwiseOperators - Python Wiki - http://wiki.python.org/moin/BitwiseOperators

<pre>
The Operators:

x << y
Returns x with the bits shifted to the left by y places (and new bits on the right-hand-side are zeros). This is the same as multiplying x by 2**y.
x >> y
Returns x with the bits shifted to the right by y places. This is the same as //'ing x by 2**y.
x & y
AND - Does a "bitwise and". Each bit of the output is 1 if the corresponding bit of x AND of y is 1, otherwise it's 0.
x | y
OR - Does a "bitwise or". Each bit of the output is 0 if the corresponding bit of x AND of y is 0, otherwise it's 1.
~ x
Returns the complement of x - the number you get by switching each 1 for a 0 and each 0 for a 1. This is the same as -x - 1.
x ^ y
XOR - Does a "bitwise exclusive or". Each bit of the output is the same as the corresponding bit in x if that bit in y is 0, and it's the complement of the bit in x if that bit in y is 1.
</pre>

=== Script Path ===

<pre>
if __name__ == "__main__":
# Change directory to where this script lives so we can find relative
# paths from here regardless of where the working directory was when we
# ran the script.
# APP_DIR = os.path.dirname(sys.argv[0])
APP_DIR = os.path.dirname(os.path.abspath(sys.argv[0))
if APP_DIR:
os.chdir(APP_DIR)
</pre>

normpath(path) # Normalize path, eliminating double slashes, etc.
abspath(path) # Return an absolute path

Path to a module: [http://stackoverflow.com/questions/247770/retrieving-python-module-path] (couldn't fine a_module)
import a_module
import os
path = os.path.dirname(a_module.__file__)

Path to module: [http://stackoverflow.com/questions/50499/in-python-how-do-i-get-the-path-and-name-of-the-file-that-is-currently-executin]
import inspect, os
print inspect.getfile(inspect.currentframe()) # script filename (usually with path)
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))) # script directory

Even better:
print os.path.realpath(__file__)
print os.path.dirname(__file__)
print os.path.basename(__file__)

references:
* http://stackoverflow.com/questions/595305/python-path-of-script

keywords: scriptpath script path program path pathname

=== Singleton ===

GLOBAL OBJECT!

None is a singleton (Python only has one copy of None in the interpreter).

Make your own singleton:

override the new method like this:
<pre>
class Singleton(object):
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Singleton, cls).__new__(
cls, *args, **kwargs)
return cls._instance

if __name__ == '__main__':
s1=Singleton()
s2=Singleton()
if(id(s1)==id(s2)):
print "Same"
else:
print "Different"
</pre>

Using decorator -implementation from PEP318? Implementing the singleton pattern with a decorator:
<pre>
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance

@singleton
class MyClass:
...
</pre>

* Python and the Singleton Pattern - Stack Overflow - http://stackoverflow.com/questions/42558/python-and-the-singleton-pattern

=== XML ===

<pre>
from xml.dom.minidom import parse, parseString

dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name

datasource = open('c:\\temp\\mydata.xml')
dom2 = parse(datasource) # parse an open file

dom3 = parseString('<myxml>Some data<empty/> some more data</myxml>')
</pre>

Pretty Print: [http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python]
<pre>
import xml.dom.minidom

xml = xml.dom.minidom.parse(xml_fname) # or xml.dom.minidom.parseString(xml_string)
pretty_xml_as_string = xml.toprettyxml()
</pre>

Pretty Print with etree [http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python]
<pre>
import lxml.etree as etree

x = etree.parse("filename")
print etree.tostring(x, pretty_print = True)
</pre>

References:
* 19.7. xml.dom.minidom — Lightweight DOM implementation — Python v2.7.3 documentation - http://docs.python.org/library/xml.dom.minidom.html
* Pretty printing XML in python - Stack Overflow - http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python

---

Simply don't include self.output.startElement and self.output.endElement for elements you don't want:

simple_filter_parser.py: (my version of [http://freegeographytools.com/2010/removin-time-data-out-of-a-gpx-or-kml-file-to-make-it-work-right-in-google-earth])
<pre>
# this parses out the time element of a gpx for Google Earth
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class my_filter_xml(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self.indent = 0
self.ignore = False
self.output.startDocument()

def startElement(self, name, attrs):
print " " * self.indent + "start: " + name
if attrs.keys():
keys = " " * (self.indent + 2) + "attrs:"
for key in attrs.keys():
keys = keys + " " + key + "=" + attrs[key]
print keys
self.indent = self.indent + 2
#self.output.startElement(name, {})
if name == "time":
self.ignore = True
#if not self.ignore:
self.output.startElement(name, attrs)

def characters(self, ch):
if ch.strip():
print " " * self.indent + "ch: " + ch.strip()
if not self.ignore:
self.output.characters(ch)
else:
if not self.ignore:
self.output.characters("\n")

def endElement(self, name):
self.indent = self.indent - 2
print " " * self.indent + "/end: " + name
#if not self.ignore:
self.output.endElement(name)
if name == "time":
self.ignore = False

if __name__ == '__main__':
source = open("tracks.gpx")
dest = open("tracks_out.gpx", 'w+')
xml.sax.parse(source, my_filter_xml(dest, True))
</pre>

---

fixdescriptor.py:
<pre>
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self._outfile = outfile
self._verbose = verbose
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self._filter = [ 'payload',
'payloads',
'file-list',
'file',
'checksum' ]
self._keep = [ 'vib',
'maintenance-mode' ]
self._inside = None

def startElement(self, name, attrs):
# Certain elements are kept unchanged
if name in self._keep:
if self._verbose:
print '%s element start found' % (name)
self.output.startElement(name, {})
if name == 'maintenance-mode':
if self._verbose:
print 'inside maintenance-mode'
self._inside = name
return
# Other elements are filtered (removed)
elif name in self._filter:
if self._verbose:
print '%s element start found' % (name)
self._inside = name
return
self.output.startElement(name, attrs)

def characters(self, ch):
if self._inside == 'maintenance-mode':
if self._verbose:
print 'new content for maintenance-mode'
self.output.characters('false')
elif self._inside not in self._filter:
self.output.characters(ch)

def endElement(self, name):
# If it's not a checksum element, only print it
if name in self._filter + self._keep:
self._inside = None
if self._verbose:
print '%s element end found' % (name)
if name in self._filter:
return
self.output.endElement(name)

if __name__ == '__main__':
#
parser = optparse.OptionParser(usage='%prog ...')
parser.add_option('-v', '--verbose',
action='store_true', default=False,
help='Print more information.')
parser.add_option('--input-xml',
type='str', default='descriptor.xml',
help='Name of input XML [default: %default]')
parser.add_option('--output-xml',
type='str', default='fixed_descriptor.xml',
help='Name of output XML [default: %default]')
opts, args = parser.parse_args(argv[1:])
source = open(opts.input_xml)
dest = open(opts.output_xml, 'w+')
xml.sax.parse(source, fix_descriptor(dest, opts.verbose))
</pre>

--

Simple read and output parsing:
<pre>
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self._outfile = outfile
self._verbose = verbose
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self._inside = None

def startElement(self, name, attrs):
# attrs.items() is a list of tuples (name, value)
print "start:", name, "/", attrs.items()
self.output.startElement(name, attrs)

def characters(self, ch):
print "char:", self._inside, "/", ch
self.output.characters(ch)

def endElement(self, name):
print "end:", name
self.output.endElement(name)

if __name__ == '__main__':
parser = optparse.OptionParser(usage='%prog ...')
parser.add_option('-v', '--verbose',
action='store_true', default=False,
help='Print more information.')
parser.add_option('--input-xml',
type='str', default='descriptor.xml',
help='Name of input XML [default: %default]')
parser.add_option('--output-xml',
type='str', default='fixed_descriptor.xml',
help='Name of output XML [default: %default]')
opts, args = parser.parse_args(argv[1:])
source = open(opts.input_xml)
dest = open(opts.output_xml, 'w+')
xml.sax.parse(source, fix_descriptor(dest, opts.verbose))
</pre>

=== Unicode ===

Unicode string:
u'Hello World !'

Raw Unicode string:
>>> ur'Hello\u0020World !'
u'Hello World !'

Convert:
u = u'hello\xe4'
str(u) # UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 5: ordinal not in range(128)
s = u.encode('utf8') # 's' is now type str

References:
* Introduction to Unicode Strings - http://docs.python.org/tutorial/introduction.html#unicode-strings

=== Zip ===

See [[Python/Zip]]

=== signals ===

signal.SIGTERM : kill [pid]
signal.SIGALRM : kill -ALRM [pid]
signal.SIGHUP : kill -HUP [pid]

<pre>
import signal, os

def handler(signum, frame):
print 'Signal handler called with signal', signum
raise IOError("Couldn't open device!")

# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)

# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)

signal.alarm(0) # Disable the alarm
</pre>

Note: use signal.SIGTERM to handle "kill" command

References:
* 17.4. signal — Set handlers for asynchronous events — Python v2.7.5 documentation - http://docs.python.org/2/library/signal.html

==== Ctrl+C ====

<pre>
#!/usr/bin/env python
import signal
import sys
def signal_handler(signal, frame):
print 'You pressed Ctrl+C!'
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
print 'Press Ctrl+C'

# linux only:
signal.pause()

# windows alternative:
#import time
#time.sleep(60)
</pre>

References:
* control - How do I capture SIGINT in Python? - Stack Overflow - http://stackoverflow.com/questions/1112343/how-do-i-capture-sigint-in-python

==== Non-blocking raw_input for Python ====

Non-blocking raw_input for Python:
<pre>
# source: http://www.garyrobinson.net/2009/10/non-blocking-raw_input-for-python.html

import signal

class AlarmException(Exception):
pass

def alarmHandler(signum, frame):
raise AlarmException

def nonBlockingRawInput(prompt='', timeout=20):
signal.signal(signal.SIGALRM, alarmHandler)
signal.alarm(timeout)
try:
text = raw_input(prompt)
signal.alarm(0)
return text
except AlarmException:
print '\nPrompt timeout. Continuing...'
signal.signal(signal.SIGALRM, signal.SIG_IGN)
return ''
</pre>

=== email ===

<pre>
import smtplib
import email.mime.text

msg_text = "Hello World"
msg = email.mime.text.MIMEText(msg_text)

# me == the sender's email address
# you == the recipient's email address
msg['Subject'] = 'The contents of %s' % textfile
msg['From'] = me
msg['To'] = you

# Send the message via our own SMTP server, but don't include the
# envelope header.
s = smtplib.SMTP('localhost')
s.sendmail(me, [you], msg.as_string())
s.quit()
</pre>

my nagios fake /bin/mail:
<pre>
#!/usr/bin/env python

import smtplib
import email.mime.text
import optparse
import os
import sys
import pwd

os.system('logger mail called')

parser = optparse.OptionParser()
parser.add_option(
'-s', '--subject',
help='Subject',
default="")
parser.add_option(
'-t', '--to',
help='To',
dest="send_to",
default=None)
parser.add_option(
'-f', '--from',
help='From',
dest="send_from",
default=None)
parser.add_option(
'-S', '--server',
help='server',
default="prime")
(options, args) = parser.parse_args()

def get_username():
return pwd.getpwuid( os.getuid() )[ 0 ]

if options.send_to:
send_to = options.send_to
else:
if len(args) == 1:
send_to = args[0]
else:
parser.print_help()
sys.exit(1)

if options.send_from:
send_from = options.send_from
else:
username = get_username()
hostname = os.uname()[1]
send_from = username + "@" + hostname

msg_text = sys.stdin.read()
msg = email.mime.text.MIMEText(msg_text)

msg['Subject'] = options.subject
msg['From'] = send_from
msg['To'] = send_to

s = smtplib.SMTP(options.server)
s.sendmail(send_from, [send_to], msg.as_string())
s.quit()
</pre>

Authenticated (with gmail):
<pre>
import smtplib
import email.mime.text

# from_email = "kenneth@demo.oeey.com"
# to_email = "kenneth@demo.oeey.com"
from_email = "Kenneth Burgener <kenneth@demo.oeey.com>"
to_emails = ["Kenneth Burgener <kenneth@demo.oeey.com>", "Kenneth Cell <xxx@xxx.com>"]

msg_text = "hello world"
msg = email.mime.text.MIMEText(msg_text)
msg['Subject'] = 'Test text'
msg['From'] = from_email
# msg['To'] = to_email
msg['To'] = ", ".join(to_emails)
# s = smtplib.SMTP('localhost')
print "sending..."
s = smtplib.SMTP_SSL('smtp.gmail.com', '465')
s.login('xxx', 'xxx')
# s.sendmail(from_email, [to_email], msg.as_string())
s.sendmail(from_email, to_emails, msg.as_string())
s.quit()
print "sent!"
</pre>

Source:
* 20.12. smtplib — SMTP protocol client — Python v2.7.5 documentation - http://docs.python.org/2/library/smtplib.html
* 18.1.11. email: Examples — Python v2.7.5 documentation - http://docs.python.org/2/library/email-examples.html

=== pprint ===

8.18. pprint — Data pretty printer — Python v2.7.5 documentation - http://docs.python.org/2/library/pprint.html

import pprint
pprint.pprint(myVar)

import pprint
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(myVar)

Note: Useless on objects!

Dump object:
<pre>
def dumpobject(obj):
print "<<<---"
print type(obj), obj.__class__.__name__
for attr in dir(obj):
if attr[0] == "_": continue # ignore under/dunder properties
if str(type(getattr(obj, attr))) == "<type 'instancemethod'>":
print attr, ": METHOD"
else:
print attr, ":", type(getattr(obj, attr)), ":", getattr(obj, attr)
print "END --->>>"
</pre>

=== Curses ===

Curses Programming with Python — Python v3.3.2 documentation - http://docs.python.org/3/howto/curses.html

"The curses library supplies a terminal-independent screen-painting and keyboard-handling facility for text-based terminals; such terminals include VT100s, the Linux console, and the simulated terminal provided by various programs. Display terminals support various control codes to perform common operations such as moving the cursor, scrolling the screen, and erasing areas. Different terminals use widely differing codes, and often have their own minor quirks.

In a world of graphical displays, one might ask “why bother”? It’s true that character-cell display terminals are an obsolete technology, but there are niches in which being able to do fancy things with them are still valuable. One niche is on small-footprint or embedded Unixes that don’t run an X server. Another is tools such as OS installers and kernel configurators that may have to run before any graphical support is available."

---

Thread from Utah Python User's Group mailing list:

-

Hello, I am trying to learn more about curses in python for a desktop application. Does anybody have any recommendations on tutorials or documentation?

-Corban

-

We've use urwid at work. Seems to be a good approach if you aren't already tied to something. Learning curve can be a bit steep (well, compared to what you'd think it should be). Good luck!

http://excess.org/urwid/

Gabriel Gunderson

-

This video series looks pretty good:

http://www.youtube.com/user/pythoncursestutorial?feature=watch

Official HOWTOs from the python.org site:

http://docs.python.org/2/howto/curses.html
http://docs.python.org/3/howto/curses.html

As Gabe mentioned, there are other libraries that provide similar or
greater functionality, so if you aren't doing this specifically to
learn curses, it's worth at least looking at a few of them to compare.

There's a brief comparison of three possibilities here:

http://stackoverflow.com/questions/8349085/python-ncurses-cdk-urwid-difference

Daniel Fackrell

-

Here's another option that looks promising:

https://pypi.python.org/pypi/blessings/

Daniel Fackrell

=== dialog ===

pythondialog - http://pythondialog.sourceforge.net/
: "pythondialog is a Python wrapper for the dialog utility originally written by Savio Lam, and later rewritten by Thomas E. Dickey. Its purpose is to provide an easy to use, pythonic and comprehensive Python interface to dialog. This allows one to make simple text-mode user interfaces on Unix-like systems (including Linux). "

Note: The latest versions (2.12 and later) only support Python 3; users who really want to stick to Python 2 should use version 2.11 for now.

Install:
# latest:
# wget http://sourceforge.net/projects/pythondialog/files/latest/download?source=dlp

wget http://sourceforge.net/projects/pythondialog/files/pythondialog/2.11/python2-pythondialog-2.11.tar.gz/download
tar -zvxf python2-pythondialog-2.11.tar.gz
cd pythondialog-2.11

import dialog
d = dialog.Dialog()
if d.yesno("Continue?") == d.DIALOG_OK:
print "Continuing..."

---

My solution

kiloforce / dialog / source / — Bitbucket - https://bitbucket.org/kiloforce/dialog

<pre>
#!/usr/bin/env python

""" Wrapper for Linux 'dialog' command """

__author__ = "Kenneth Burgener <kenneth@k.ttak.org>"
__copyright__ = "(c) 2013"
__date__ = "$Sep 27, 2013$"
__version__ = "1.0"
__credits__ = r"""
XXXXXXXXXXXXXXXXXX
X Dialog Wrapper #
##################
"""

import subprocess
import time
import sys

class Dialog():
""" Python wrapper for Linux 'dialog' command

dialog - display dialog boxes from shell scripts

>>> import dialog
>>> dlg = dialog.Dialog()
>>> rc = dlg.msgbox('hello world')

Return codes:
YES = 0
OK = 0
EXIT = 0
NO = 1
CANCEL = 1
ESC = 255
"""

YES = 0
OK = 0
EXIT = 0
NO = 1
CANCEL = 1
ESC = 255

def __init__(self):
rc = self._exe('which dialog > /dev/null')[0]
if rc != 0:
print "Error: failed to find Linux dialog command"
sys.exit(1)

def _exe(self, cmd, strip=True):
p = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE)
output = "\n".join(p.stderr.readlines())
if strip:
output = output.strip()
rc = p.wait()
return (rc, output)

def rc_to_str(self, rc):
if rc == 0:
return "YES"
elif rc == 1:
return "NO"
elif rc == 255:
return "ESX"
else:
raise Exception("Unknown return code: %s" % rc)

def clear(self):
"""
Clear screen

Example:
>>> dialog.clear()"""
rc, output = self._exe('clear')
assert rc in (0,), "clear rc"
return rc

def msgbox(self, msg):
"""
Simple message box

@param msg (str) = display message
@return: rc (int)

Example:
>>> rc = dialog.msgbox('hello world')"""
rc, output = self._exe('dialog --msgbox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.ESC), "msgbox rc"
return rc

def yesno(self, msg):
"""
Yes/No dialog

@param msg (str) = display message
@return: rc (int)

Example:
>>> rc = dialog.yesno('continue')"""
rc, output = self._exe('dialog --yesno "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.NO, self.ESC), "yesno rc"
return rc

def infobox(self, msg):
"""
Dialog that does not wait for user's response
(do your own pause)

@param msg (str) = display message
@return: rc (int) always zero

Example:
>>> for i in range(10):
>>> dialog.infobox('counting: ' + str(i))
>>> time.sleep(3)"""
rc, output = self._exe('dialog --infobox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES,), "infobox rc"
return rc

def inputbox(self, msg):
"""
Get user input

@param msg (str) = display message
@return: rc (int), output (str/None)

Example:
>>> rc, output = dialog.inputbox('name:')"""
rc, output = self._exe('dialog --inputbox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.NO, self.ESC), "inputbox rc"
if not output:
output = None
return rc, output

def textbox(self, filename):
"""
Display a file

@param filename (str) = file to display
@return: rc (int)

Example:
>>> dialog.textbox('/etc/passwd')"""
rc, output = self._exe('dialog --textbox {filename} 0 0'.format(filename=filename))
assert rc in (self.YES, self.ESC), "textbox rc"
return rc

def menu(self, msg, item_list):
"""
Single selection checkbox menu

@param msg (str) = display message
@param item_list (str list) = list of items to display
@return: rc (int), selection (int/None)

Example:
>>> rc, selection = dialog.menu('Select multiple options:',
... ('option 1', 'option 2'))"""
item_str = ""
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
rc, output = self._exe('dialog --menu "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
if output:
selection = int(output)
else:
selection = None
assert rc in (self.YES, self.NO, self.ESC), "menu rc"
return rc, selection

def checklist(self, msg, item_list, checked_list=None):
"""
Multiple selection checkbox menu

@param msg (str) = display message
@param item_list (str list) = list of items
@param checked_list (int list) = default items to check
@return: rc (int), selections (int list/None)

Example:
>>> rc, selections = dialog.checklist('Select multiple options:',
... ('option 1', 'option 2')
"""
item_str = ""
assert type(item_list) in (type(None), type(()), type([])), "checklist: item_list must be a list"
assert type(checked_list) in (type(None), type(()), type([])), "checklist: checked_list must be a list"
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
if checked_list and i+1 in checked_list:
item_str += " on"
else:
item_str += " off"
rc, output = self._exe('dialog --checklist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
print item_str
selections = []
for choice in output.split('"'):
if choice.strip():
selections.append(int(choice))
if selections:
selections = tuple(selections)
else:
selections = None
assert rc in (self.YES, self.NO, self.ESC), "checklist rc"
return rc, selections

def radiolist(self, msg, item_list, checked_item=None):
"""
Single selection radio menu, with default

@param msg (str) = display message
@param item_list (str list) = list of items
@param checked_item (int) = default item to select
@return: rc (int), selection (int)

Example:
>>> rc, selection = dialog.radiolist('Select one option:',
>>> ('option 1', 'option 2'), 2)"""
item_str = ""
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
if checked_item and i+1 == int(checked_item):
item_str += " on"
else:
item_str += " off"
rc, selection = self._exe('dialog --radiolist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
if not selection:
selection = None
assert rc in (self.YES, self.NO, self.ESC), "radiolist rc"
return rc, selection

def gauge(self, msg, percent):
"""
Progress bar (does not wait for user's input)

@param msg (str) = display message
@param percent (int) = progress
@return: rc (int) always zero

Example:
>>> for i in range(0, 100+1, 2):
>>> dialog.gauge('Progress:', i)
>>> time.sleep(.1)"""
rc, output = self._exe('echo {percent} | dialog --gauge "{msg}" 0 0'.format(percent=percent, msg=msg))
assert rc in (self.YES,), "guage rc"
return rc

def progress(self, msg, percent):
"""
Alias for self.guage()
"""
return self.gauge(msg=msg, percent=percent)

def form(self, msg, item_list):
"""
Multiple item entry form

@param msg (str) = display message
@param item_list (str list) = list of items
@return: rc (int), answers (str list)

Example:
>>> rc, answers = dialog.form('Enter details',
... ('first name', 'last name'))"""
item_str = ""
name_len = 1
for i in range(len(item_list)):
if len(item_list[i])+3 > name_len:
name_len = len(item_list[i])+3
for i in range(len(item_list)):
item_str += " \"" + item_list[i] + ":\" " + str(i+1) + " 1 \"\" " + str(i+1) +\
" " + str(name_len) + " 40 0"
rc, answers = self._exe('dialog --form "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str),
strip=False)
answers = answers.split('\n\n')
if not answers:
answers = None
assert rc in (self.YES, self.NO, self.ESC), "form rc"
return rc, answers

def test(self):
"""
Test features

>>> import dialog
>>> dlg = dialog.Dialog()
>>> dlg.test()
"""

def wait_user(msg):
self.msgbox("%s" % msg)

# rc_to_str
wait_user("Testing return code...")
print self.rc_to_str(self.YES) # YES
print self.rc_to_str(self.NO) # NO
print self.rc_to_str(self.ESC) # ESX
try:
print self.rc_to_str(111) # exception
raise Exception("Did not throw exception")
except Exception as rc:
pass

# msgbox
wait_user("Testing msgbox...")
rc = self.msgbox("msgbox")
if rc not in (self.YES, self.ESC):
raise Exception("Unknown return code for msgbox: %s" % rc)

# yesno
wait_user("Testing yesno...")
rc = self.yesno("yesno")
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for yesno: %s" % rc)

# infobox
wait_user("Testing infobox...")
rc = self.infobox("infobox sleep 1 sec")
if rc not in (self.YES,):
raise Exception("Unknown return code for infobox %s" % rc)
time.sleep(1)

# inputbox
wait_user("Testing inputbox...")
rc, output = self.inputbox("inputbox")
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for inputbox: %s" % rc)
wait_user("Result: %s" % output)

# textbox
wait_user("Testing textbox...")
rc = self.textbox("/etc/passwd")
if rc not in (self.YES, self.ESC):
raise Exception("Unknown return code for textbox: %s" % rc)

# menu
wait_user("Testing menu...")
rc, selection = self.menu("menu", ("option 1", "option 2"))
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for menu: %s" % rc)
wait_user("Result: %s" % selection)

# checklist
wait_user("Testing checklist...")
rc, selection = self.checklist("menu", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for checklist: %s" % rc)
wait_user("Result: %s" % str(selection))
wait_user("Testing checklist with default selection...")
rc, selection = self.checklist("menu", ["option 1", "option 2"], [1, 2])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for checklist: %s" % rc)
wait_user("Result: %s" % str(selection))

# radiolist
wait_user("Testing radiolist...")
rc, selection = self.radiolist("menu", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for radiolist: %s" % rc)
wait_user("Result: %s" % str(selection))
wait_user("Testing radiolist with default selection...")
rc, selection = self.radiolist("menu", ["option 1", "option 2"], 2)
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for radiolist: %s" % rc)
wait_user("Result: %s" % str(selection))

# guage
wait_user("Testing guage...")
for i in range(4+1):
rc = self.gauge("test guage", i*25)
# rc = self.progress("test progress", i*25)
if rc not in (self.YES,):
raise Exception("Unknown return code for guage: %s" % rc)
time.sleep(1)

# form
wait_user("Testing form...")
rc, answers = self.form("form", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for form: %s" % rc)
wait_user("Result: %s" % str(answers))

# done with test, cleanup
wait_user("Done with tests...")
self.clear()

if __name__ == "__main__":
# dialog = Dialog()
# dialog.test()
pass
</pre>

=== Busy Loop and Thread.yield() ===

"I want to tell my Python threads to yield, and so avoid hogging the CPU unnecessarily. In Java, you could do that using the Thread.yield() function"

Solution:
time.sleep(0)
time.sleep(0.00001)
time.sleep(epsilon)

References:
* multithreading - In there something similar to Java's Thread.yield() in Python? Does that even make sense? - Stack Overflow - http://stackoverflow.com/questions/1908206/in-there-something-similar-to-javas-thread-yield-in-python-does-that-even-ma
* multithreading - How does a threading.Thread yield the rest of its quantum in Python? - Stack Overflow - http://stackoverflow.com/questions/787803/how-does-a-threading-thread-yield-the-rest-of-its-quantum-in-python

=== USB ===

Programming with PyUSB 1.0 - http://pyusb.sourceforge.net/docs/1.0/tutorial.html

PyUSB - http://pyusb.sourceforge.net/

git clone https://github.com/walac/pyusb

import usb.core
dev = usb.core.find()

# find our device
dev = usb.core.find(idVendor=0xfffe, idProduct=0x0001)

"'''Dealing with multiple identical devices''' - Sometimes you may have two identical devices connected to the computer. How can you differentiate them? Device objects come with two additional attributes which are not part of the USB Spec, but are very useful: bus and address attributes. First of all, it is worth to say that these attributes come from the backend and a backend is free to not support them, in which case they are set to None. That said, these attributes represent the bus number and bus address of the device and, as you might already have imagined, can be used to differentiate two devices with the same idVendor and idProduct attributes."

=== Tkinter ===

Python GUI Bitcoin: [http://bitcoin.stackexchange.com/questions/10594/simplest-way-of-obtaining-mtgox-exchange-rate-via-an-api]
<pre>
import json
import urllib2
from Tkinter import *

url='http://api.bitcoincharts.com/v1/markets.json'
req=urllib2.Request(url)
response=urllib2.urlopen(req).read()
output=json.loads(response)

mtusd_price = output[-10]['avg']
mteur_price = output[31]['avg']
mtgbp_price = output[44]['avg']

MTUSD= 'USD' + ' ' + str(mtusd_price)
MTEUR= 'EUR' + ' ' + str(mteur_price)
MTGBP = 'GBP' + ' ' + str(mtgbp_price)

root= Tk()
root.title("Bitcoin Price")
root.geometry("250x100")

USD = Label(root, text=MTUSD)
EUR = Label(root, text=MTEUR)
GBP = Label(root, text=MTGBP)
USD.pack()
EUR.pack()
GBP.pack()
mainloop()
</pre>

=== ctypes ===

"ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python."
:-- 15.17. ctypes — A foreign function library for Python — Python v2.7.7 documentation - https://docs.python.org/2/library/ctypes.html

Use "c scope" to build a database of symbols to see what options are available.

---

testlib.c
<pre>
#include <stdio.h>

void myprint(void);

void myprint()
{
printf("hello world\n");
}
</pre>

Compile:
gcc -shared -Wl,-soname,testlib -o testlib.so -fPIC testlib.c

testlibwrapper.py

Python:
<pre>
import ctypes

testlib = ctypes.CDLL('/full/path/to/testlib.so')
testlib.myprint()
</pre>

Source: python - ctypes - Beginner - Stack Overflow - http://stackoverflow.com/questions/5081875/ctypes-beginner

---

nzjrs's Gists - https://gist.github.com/nzjrs

---

<pre>
#include <stdio.h>

int sum(double *x,int n)
{
int i;
double counter;
counter = 0;
for(i=0;i<n;i++)
{
counter=counter+x[i];

}
return counter;
}
</pre>

<pre>
gcc -c sum.c
gcc -shared -o sum.so sum.o
</pre>

<pre>
from ctypes import *
my_sum=CDLL('sum.so')
a=numpy.array(range(10),dtype=float)
my_sum.sum(a.ctypes.data_as(c_void_p),int(10))
</pre>

<pre>
a=(c_double*10)()
for i in range(10):
a[i]=i
my_sum.sum(a,int(10))
</pre>

Source: Ctypes — Numerical Computing with Sage v6.2 - http://www.sagemath.org/doc/numerical_sage/ctypes.html

---

=== Warnings ===

if sys.version_info >= (1, 0, 0):
warnings.warn("Danger Will Robinson: trying Phi on untested Python 3.0!")

=== enum ===

Python 3: https://docs.python.org/3/library/enum.html

<pre>
from enum import Enum

# class syntax
class Color(Enum):
RED = 1
GREEN = 2
BLUE = 3

# functional syntax
Color = Enum('Color', ['RED', 'GREEN', 'BLUE'])
</pre>

---

Python 2: [http://stackoverflow.com/questions/36932/how-can-i-represent-an-enum-in-python]
<pre>
class Enum(set):
""" Basic enumerated type

Example
>>> Animals = phi.Enum(["DOG", "CAT", "HORSE"])
>>> Animals.DOG
'DOG'
>>> 'HORSE' in Animals
True
"""
def __getattr__(self, name):
if name in self:
return name
raise AttributeError("Missing Enum property: %s" % name)
def __setattr__(self, name, value): # this makes it read-only
raise AttributeError("Unable to modify Enum property: %s (%s)" % (name, value))

Animals = Enum(["DOG", "CAT", "HORSE"])

print(Animals.DOG) # 'DOG'
</pre>

----------

Alternative:
<pre>
class Stationery:
Pen, Pencil, Eraser = range(0, 3)
</pre>

One liner:
class Enum(tuple): __getattr__ = tuple.index

<pre>
>>> State = Enum(['Unclaimed', 'Claimed'])
>>> State.Claimed
1
>>> State[1]
'Claimed'
>>> State
('Unclaimed', 'Claimed')
>>> range(len(State))
[0, 1]
>>> [(k, State[k]) for k in range(len(State))]
[(0, 'Unclaimed'), (1, 'Claimed')]
>>> [(k, getattr(State, k)) for k in State]
[('Unclaimed', 0), ('Claimed', 1)]
</pre>

Python 3 supports enum natively with the enum.Enum library:

=== pySerial ===

https://pythonhosted.org/pyserial/

"This module encapsulates the access for the serial port. It provides backends for Python running on Windows, OSX, Linux, BSD (possibly any POSIX compliant system) and IronPython. The module named “serial” automatically selects the appropriate backend."

https://github.com/pyserial/

Intro - https://pythonhosted.org/pyserial/shortintro.html

---

<pre>
import serial
import time

ser = serial.Serial(
port='COM4',\
baudrate=9600,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)

print("connected to: " + ser.portstr)

#this will store the line
seq = []
count = 1

while True:
for c in ser.read():
seq.append(chr(c)) #convert from ANSII
joined_seq = ''.join(str(v) for v in seq) #Make a string from array

if chr(c) == '\n':
print("Line " + str(count) + ': ' + joined_seq)
seq = []
count += 1
break

ser.close()
</pre>

Ref: http://stackoverflow.com/questions/16077912/python-serial-how-to-use-the-read-or-readline-function-to-read-more-than-1-char

---

I modified the code to monitor a COM port: (Windows)

Monitor Serial:

<pre>
import serial, time, sys

logger = open("serial.txt", "w")
timer = time.time()

def log(msg, head=False):
msg = msg.strip()
if head:
msg = ">>> " + msg
logger.write(msg + "\n")
sys.stdout.write(msg + "\n")
newtime = time.time()
if newtime > timer + 1:
logger.flush()
sys.stdout.flush()

while True:

ser = None
try:
ser = serial.Serial(
port='COM1',\
baudrate=115200,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)
except serial.serialutil.SerialException as e:
log(str(e), True)
time.sleep(1)
continue

log("Connected to: " + ser.portstr, True)
count=1

try:
line = []
while True:
for c in ser.readline():
line.append(c)
if c == "\n":
log("".join(line))
line = []
except serial.serialutil.SerialException as e:
log(str(e), True)
time.sleep(1)
ser.close()
</pre>

=== MyBool ===

<pre>
>>> class NewBool(int):
... def __new__(cls, value):
... return int.__new__(cls, bool(value))

>>> y = NewBool(2)
>>> y == 1
True
</pre>

https://jfine-python-classes.readthedocs.io/en/latest/subclass-int.html

Also check out the __cmp__

== md5 ==

import md5
md5.new("Nobody inspects the spammish repetition").hexdigest()
# 'bb649c83dd1ea5c9d9dec9a18df0ffe9'

<pre>
$ echo -n "Nobody inspects the spammish repetition" | md5sum
bb649c83dd1ea5c9d9dec9a18df0ffe9 -
<pre>

== External Libraries ==

=== execnet ===

Python Package Index - execnet - https://pypi.python.org/pypi/execnet

Examples - http://codespeak.net/execnet/examples.html
* basic local and remote communication - http://codespeak.net/execnet/example/test_info.html

Execute source code in subprocess, communicate through a channel:
<pre>
>>> import execnet
>>> gw = execnet.makegateway()
>>> channel = gw.remote_exec("channel.send(channel.receive()+1)")
>>> channel.send(1)
>>> channel.receive()
2
</pre>

Compare current working directories
<pre>
>>> import execnet, os
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec("import os; channel.send(os.getcwd())")
>>> res = ch.receive()
>>> assert res == os.getcwd()
</pre>

Get information from remote ssh account
<pre>
>>> import execnet, os
>>> gw = execnet.makegateway("ssh=codespeak.net")
>>> channel = gw.remote_exec("""
... import sys, os
... channel.send((sys.platform, sys.version_info, os.getpid()))
... """)
>>> platform, version_info, remote_pid = channel.receive()
>>> platform
'linux2'
>>> version_info
(2, 4, 2, 'final', 0)
</pre>

remote-exec a function (avoiding inlined source part I)
<pre>
import execnet

def multiplier(channel, factor):
while not channel.isclosed():
param = channel.receive()
channel.send(param * factor)

gw = execnet.makegateway()
channel = gw.remote_exec(multiplier, factor=10)

for i in range(5):
channel.send(i)
result = channel.receive()
assert result == i * 10

gw.exit()
</pre>

remote-exec a module (avoiding inlined source part II)
:"You can pass a module object to remote_exec in which case its source code will be sent. No dependencies will be transferred so the module must be self-contained or only use modules that are installed on the “other” side. Module code can detect if it is running in a remote_exec situation by checking for the special __name__ attribute."

<pre>
# content of a module remote1.py

if __name__ == '__channelexec__':
channel.send('initialization complete')
</pre>

<pre>
>>> import execnet, remote1
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec(remote1)
>>> print (ch.receive())
initialization complete
</pre>

Use a callback instead of receive() and wait for completion
<pre>
>>> import execnet
>>> gw = execnet.makegateway()
>>> channel = gw.remote_exec("for i in range(10): channel.send(i)")
>>> l = []
>>> channel.setcallback(l.append, endmarker=None)
>>> channel.waitclose() # waits for closing, i.e. remote exec finish
>>> l
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, None]
</pre>

a simple command loop pattern
<pre>
# contents of: remotecmd.py
import os

def simple(arg):
return arg + 1

def listdir(path):
return os.listdir(path)

if __name__ == '__channelexec__':
for item in channel:
channel.send(eval(item))
</pre>

<pre>
>>> import execnet, remotecmd
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec(remotecmd)
>>> ch.send('simple(10)') # execute func-call remotely
>>> ch.receive()
11
</pre>

=== evdev ===

Install:
apt-get install python-setuptools python-dev
easy_install pip
pip install evdev

Simple read usage:
<pre>
from evdev import InputDevice
from evdev import ecodes

# look for a /dev/input/by-id/usb...kbd or similar
DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd"

dev = InputDevice(DEVICE)

while True:
r, w, x = select([dev], [], [], 5)
for event in dev.read():
if event.type == 1 and event.value == 1:
print event.code
</pre>

List LED states:
dev.leds()
dev.leds(verbose=True)

Inject usage: (does not appear to really work)
<pre>
from evdev import InputDevice
from evdev import ecodes
from evdev import UInput

DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd"
dev = InputDevice(DEVICE)

ui = UInput()

ui.write(ecodes.EV_KEY, ecodes.KEY_4, 1)
ui.write(ecodes.EV_KEY, ecodes.KEY_4, 0)
ui.syn()

ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 1)
ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 0)
ui.syn()

ui.close()
</pre>

Tutorial:
* evdev documentation — python-evdev 0.4.3 documentation - http://pythonhosted.org/evdev/index.html
References:
* python-evdev/doc/tutorial.rst at master · gvalkov/python-evdev · GitHub - https://github.com/gvalkov/python-evdev/blob/master/doc/tutorial.rst

=== ldap-python ===

python-ldap: LDAP client API for Python - http://www.python-ldap.org/

Index of Packages : Python Package Index - https://pypi.python.org/pypi/python-ldap/

yum install python-devel openldap-devel
pip install python-ldap

Articles:
* Python LDAP Applications: Part 1 - Installing and Configuring the Python-LDAP Library and Binding to an LDAP Directory | Packt Publishing - http://www.packtpub.com/article/installing-and-configuring-the-python-ldap-library-and-binding-to-an-ldap-directory

=== xmmp ===

PDF:
* XMPP: The Definitive Guide - Oreilly - http://oriolrius.cat/blog/wp-content/uploads/2009/10/Oreilly.XMPP.The.Definitive.Guide.May.2009.pdf

xmmp is the Jabber protocol

Best options:

MIT libraries.
* sleekxmpp (was: sleekxmpp) is pretty popular and is used for examples in Peter Saint-Andre's XMPP book from O'Reilly. It reportedly works well, and finally got an email list in April 2010, and has a chat room at sleek@conference.jabber.org.

GPL libraries.
* xmpppy is used by gajim.

LPGL libraries.
* pyxmpp is pretty good and uses libxml2 internally for xml parsing.
* pyxmpp2 is the next version of pyxmpp, runs on python 2.7 and 3.2, and removes the libxml2 requirement. Like many, it requires dnspython.
* jabberpy is the original and still works for a lot of tasks, but is thoroughly unmaintained.

Other libraries.
* Wokkel, mentioned in another post. That's a new one for me, based on twisted.

chat - Best python XMPP / Jabber client library? - Stack Overflow - http://stackoverflow.com/questions/1901828/best-python-xmpp-jabber-client-library

=== xmpppy ===

xmpppy: the jabber python project - http://xmpppy.sourceforge.net/

"xmpppy is a Python library that is targeted to provide easy scripting with Jabber. Similar projects are Twisted Words and jabber.py.

This library was not designed from scratch. It inherits some code from jabberpy and have very similar API in many places. Though it is separate project since it have almost completely different architecture and primarily aims to work with jabberd2 - the new Open Source Jabber Server.

xmpppy is distributed under the terms of GNU General Public License and can be freely redistributed without any charge."

wget http://sourceforge.net/projects/xmpppy/files/xmpppy/0.4.1/xmpppy-0.4.1.tar.gz/download -O xmpppy-0.4.1.tar.gz
tar -zvxf xmpppy-0.4.1.tar.gz
pip install pydns

--- basic connection ---

<pre>
#!/usr/bin/env python

import xmpp

user="username@gmail.com"
password="password"
server="gmail.com"

jid = xmpp.JID(user)
connection = xmpp.Client(server,debug=[])
connection.connect()
result = connection.auth(jid.getNode(), password,"LFY-client")

connection.sendInitPresence()

while connection.Process(1):
pass
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- basic receive ---

Print messages when received:
<pre>
#!/usr/bin/env python

import xmpp

user="username@gmail.com"
password="password"
server="gmail.com"

def message_handler(connect_object, message_node):
message = "Welcome to my first Gtalk Bot :)"
connect_object.send( xmpp.Message( message_node.getFrom() ,message))

jid = xmpp.JID(user)
connection = xmpp.Client(server)
connection.connect()
result = connection.auth(jid.getNode(), password, "LFY-client")
connection.RegisterHandler('message', message_handler)

connection.sendInitPresence()

while connection.Process(1):
pass
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- remote-control shell bot ---

Replace the simple bot’s message_handler() function with this new one:

<pre>
def message_handler(connect_object,message_node):
command = str(message_node.getBody())

process = subprocess.Popen(command,shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
message = process.stdout.read()
if message=="":
message=process.stderr.read()

connect_object.send( xmpp.Message( message_node.getFrom() ,message))
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- roster ---

<pre>
import xmpp
import sys

userID = 'myname@gmail.com'
password = 'mypassword'
ressource = 'Script'

jid = xmpp.protocol.JID(userID)
jabber = xmpp.Client(jid.getDomain(), debug=[])

connection = jabber.connect(('talk.google.com',5222))
auth = jabber.auth(jid.getNode(), password, ressource)

jabber.sendInitPresence(requestRoster=1)
myroster = jabber.getRoster()
</pre>

To get status, you have to wait for the other clients to send you their status:
<pre>
jabber.RegisterHandler('presence', myPresenceHandler)

def myPresenceHandler(con, event):
fromjid = event.getFrom().getStripped()
status = myroster.getStatus(fromjid)
</pre>

References:
python - XMPP chat: accessing contacts' status messages with xmppPy's Roster - Stack Overflow - http://stackoverflow.com/questions/2381597/xmpp-chat-accessing-contacts-status-messages-with-xmpppys-roster

--- full roster script ---

kparal/jabber-roster · GitHub - https://github.com/kparal/jabber-roster

A simple Python tool for listing your Jabber roster contacts. You can use it to easily backup list of your buddies.

This program is maintained, but not further developed. Bugs will be fixed, but no new features will be added. If you want to work on this program, don't hesitate to contact me, I will gladly assign you to the development team.

git clone https://github.com/kparal/jabber-roster
python jabber_roster.py

https://pypi.python.org/pypi/jabber-roster

--- vcard ---

This information is not in the roster. You will need to query the clients individually and get their vCard by sending this IQ :

<pre>
<iq from='stpeter@jabber.org/roundabout'
id='v1'
type='get'>
<vCard xmlns='vcard-temp'/>
</iq>
</pre>

My solution:
<pre>
def vcard(disp, jid):
msg = xmpp.protocol.Iq()
msg.setType('get')
msg.setTo(jid)
qc = msg.addChild('vCard')
qc.setAttr('xmlns', 'vcard-temp')
rep = disp.SendAndWaitForResponse(msg)
# to see what other fields are available in the XML output:
# print rep
userid=fname=lname=title=department=region=None
for i in rep.getChildren():
for j in i.getChildren():
if j.getName() == "TITLE":
title = j.getData().encode('utf-8')
for k in j.getChildren():
if k.getName() == "GIVEN":
fname = k.getData().encode('utf-8')
if k.getName() == "FAMILY":
lname = k.getData().encode('utf-8')
if k.getName() == "ORGUNIT":
department = k.getData().encode('utf-8')
if k.getName() == "REGION":
region = k.getData().encode('utf-8')
return fname, lname, title, department, region
</pre>

References:
* xmpppy - Retrieve gtalk nickname in python xmpp - Stack Overflow - http://stackoverflow.com/questions/4539477/retrieve-gtalk-nickname-in-python-xmpp
* XEP-0054: vcard-temp - http://xmpp.org/extensions/xep-0054.html

=== google-api-python-client ===

Google APIs Client Library for Python, which includes:
* google-api-python-client: The core Python library for accessing Google APIs.
* oauth2client: A Python client library for OAuth 2.0.
* Sample applications using google-api-python-client and oauth2client

-

https://developers.google.com/api-client-library/python/start/installation
* Calendar API - Python/Command Line

-
Download:
wget https://code.google.com/p/google-api-python-client/downloads/detail?name=google-api-python-client-1.2.zip
wget https://code.google.com/p/google-api-python-client/downloads/detail?name=oauth2client-1.2.zip

Git:
git clone https://github.com/google/google-api-python-client
git clone https://github.com/google/oauth2client

References:
* google-api-python-client - Google APIs Client Library for Python - Google Project Hosting - https://code.google.com/p/google-api-python-client/

== Documentation Tools ==

=== pydoc ===

pydoc - http://docs.python.org/library/pydoc.html
: Documentation generator and online help system

=== Doxygen and Doxypy ===

See [[Python/Doxygen]]

=== Sphinx ===

Sphinx - http://sphinx.pocoo.org/

"Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license.

It was originally created for the new Python documentation, and it has excellent facilities for the documentation of Python projects, but C/C++ is already supported as well, and it is planned to add special support for other languages as well. Of course, this site is also created from reStructuredText sources using Sphinx!"

=== epydoc ===

epydoc - http://epydoc.sourceforge.net/

"Epydoc is a tool for generating API documentation for Python modules, based on their docstrings. For an example of epydoc's output, see the API documentation for epydoc itself (html, pdf). A lightweight markup language called epytext can be used to format docstrings, and to add information about specific fields, such as parameters and instance variables. Epydoc also understands docstrings written in reStructuredText, Javadoc, and plaintext. For a more extensive example of epydoc's output, see the API documentation for Python 2.5. "

== Python 3 ==

=== Python 3 Wall of Superpowers ===

Python 3 Wall of Superpowers - https://python3wos.appspot.com/

"As listed on PyPI - packages in red don't support python 3, packages in green do. Hopefully one day everything will be greener. "

=== New in Python 3 ===

==== qualnames ====

New in 3.3
<pre>
>>> def f():
... def g():
... class H: pass
... return H
... return g
...

>>> H=f()()

>>> # 3.3:
>>> H
<class '__main__.f.<locals>.g.<locals>.H'>
>>> H.__qualname__
'f.<locals>.g.<locals>.H'

>>> # 2.6:
>>> H
<class __main__.H at 0x2b885044b890>
>>> H.__qualname__
AttributeError: class H has no attribute '__qualname__'
</pre>

== Python 2 or Python 3 ==

Python2orPython3 - PythonInfo Wiki - http://wiki.python.org/moin/Python2orPython3
:Short version: Python 2.x is the status quo, Python 3.x is the shiny new thing.
:At the time of writing (July 4, 2010), the final 2.7 release is out, with a statement of extended support for this end-of-life release. The 2.x branch will see no new major releases after that. 3.x is under active and continued development, with 3.1 already available and 3.2 due for release around the turn of the year.

Python 2.7.1 Release - http://www.python.org/download/releases/2.7.1/

What’s New In Python 3.0 — Python v3.0.1 documentation - http://docs.python.org/release/3.0.1/whatsnew/3.0.html
* print() requires parenthesis
* raw_input() is now input()

== Python for Windows ==

Python for Windows - http://www.python.org/download/windows/

=== Windows IDEs ===

==== PyCharm ====

http://www.pycharm.com/

=== Hide Console ===

fork() does not work on windows. So what else can we do?

How to hide console window in python? - http://stackoverflow.com/questions/764631/how-to-hide-console-window-in-python
* "Simply save it with a .pyw extension. This will prevent the console window from opening. Explanation at the bottom of section [http://docs.python.org/2/tutorial/interpreter.html#executable-python-scripts]"

=== 32bit vs 64bit ===

IDLE:

32bit:
<pre>
C:\Python27>python
Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
</pre>

64bit: (note: default path is C:\Python27)
<pre>
C:\Python27-64bit>python.exe
Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
</pre>

Task Manager:
python.exe *32

python.exe

Platform:
>>> import platform
>>> platform.python_compiler()
'MSC v.1500 32 bit (Intel)'

>>> import platform
>>> platform.python_compiler()
'MSC v.1500 64 bit (AMD64)'

Code: [http://stackoverflow.com/questions/2208828/detect-64bit-os-windows-in-python]
import struct
def is_python_64bit():
return (struct.calcsize("P") == 8)

=== Stand-Alone Executable ===

==== Python ZIP Application ====

Python can be run from a single bundled zip file - similar to a Java WAR! Python has been able to (Allow interpreter to execute a zip file http://bugs.python.org/issue1739468)

"Python quietly added a new feature in 2.6 that makes it possible to bundle up a directory full of Python code into a single executable file." [http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html]

When a python zip application is executed, it will execute '__main__.py' [http://docs.python.org/using/cmdline.html#interface-options]

mkdir app
touch app/__main__.py
cd app
zip -r ../app.zip *
cd ..
python app.zip

Load a resource - The easiest way to find and load a program bundle like this is to use the pkg_resources module. - http://packages.python.org/distribute/pkg_resources.html

<pre>
# file: app/__main__.py

import pkg_resources

def main():
print('The rain in Spain falls mainly in the plain.')
print(pkg_resources.resource_string('resources', 'inFrance.txt'))

if __name__ == '__main__':
main()
</pre>

Now use a bit of UNIX magic to turn app2.zip into an executable:
echo '#!/usr/bin/env python' | cat - app2.zip > app2
chmod +x app2

"The zip file format is designed to allow a small executable program to be inserted at the front (that's how self-extracting zip files are created), so this is kosher and doesn't corrupt the zip file." [http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html]

Now you can simply run app2 like any executable.
$ ./app2

References:
* Able Pear Software: Bundling Python files into a stand-alone executable - http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html
* PEP 441 -- Improving Python ZIP Application Support - http://www.python.org/dev/peps/pep-0441/
* Issue 1739468: Allow interpreter to execute a zip file - Python tracker - http://bugs.python.org/issue1739468
* Package Discovery and Resource Access using pkg_resources — Distribute documentation - http://packages.python.org/distribute/pkg_resources.html

==== Shell Python Loader ====

Came before the Python ZIP Application support.

Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe

"This takes your Python application and packages it into a single executable shell script. You application can be made of up multiple python scripts. The executable shell script is also compressed, so the resulting script is smaller than the original. This should work on any system that has Python and a Bourne shell (works on Cygwin too). Note that this does not package the Python interpreter into the executable. This packages only you python code into a zip file, but this still allows you to distribute your application as a single shell script that will run on any system with Python installed. In theory, you could zip the .pyc files, but the byte code is guaranteed to run only on exactly the same version Python interpreter (including minor version). It is safer to just package the .py files."

First you need to zip all your python files:
zip main.zip main.py spam.py eggs.py

zipheader.sh:
<pre>
#!/bin/sh
# This is a self-extracting executable.
# Execute this like any normal executable.
# You may need to "chmod a+x" this file.
# This is a binary ZIP file with a Python loader header.
#
# Bourne shell loader:
PYTHON=$(which python 2>/dev/null)
if [ ! -x "$PYTHON" ] ; then
echo "Python not found!"
exit 1
fi
exec $PYTHON -c "
# Python loader:
import sys, os
if int(sys.version[0])<2:
print 'Python version 2.3 final or greater is required.'
print 'Your version is', sys.version
os._exit(1)
major = sys.version_info[0]
minor = sys.version_info[1]
releaselevel = sys.version_info[3]
if (major==2 and minor<3) or (major==2 and minor==3 and releaselevel!='final'):
print 'Python version 2.3 final or greater is required.'
print 'Your version is', sys.version
os._exit(1)
sys.path.insert(0, sys.argv[1])
del sys.argv[0:1]
import main
main.main()
" $0 $@
# Zip file:
</pre>

Concatinate zipheader.sh with the zip file.
cat zipheader.sh main.zip > main

Set main as executable.
chmod +x main

"This is an improvement on Joerg Raedler's Python recipe, [http://code.activestate.com/recipes/215301-binding-main-skript-and-modules-to-one-executable-/ 215301]. The original closes stdin. The herefile in the shell script redirects stdin before python gets a chance to start. This disables raw_input() and anything else that reads sys.stdin. Unfortunately, Raedler's boot script closes stdin, which is a fairly big limitation. The herefile in the shell script (END_OF_PYTHON_CODE) redirects stdin before python starts; stdin is closed at the end of the herefile. This disables raw_input() and anything else that reads sys.stdin. "

References:
* Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe
* Build a compressed self-extracting executable script on UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/497000-build-a-compressed-self-extracting-executable-scri/
* binding main skript and modules to one executable with python-2.3 under UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/215301-binding-main-skript-and-modules-to-one-executable-/

==== py2exe ====

py2exe - http://www.py2exe.org/
: "py2exe is a Python Distutils extension which converts Python scripts into executable Windows programs, able to run without requiring a Python installation. "

Tutorial - http://www.py2exe.org/index.cgi/Tutorial

Note: Install the version that matches your version of python

Note: run installer "as administrator"

setup.py
<pre>
from distutils.core import setup
import py2exe

setup(console=['hello.py'])
</pre>

No console window version: [http://ubuntuforums.org/showthread.php?t=728170] [http://stackoverflow.com/questions/3275293/hiding-console-window-of-python-gui-app-with-py2exe]
<pre>
from distutils.core import setup
import py2exe

setup(windows=['hello.py'])
</pre>

Build:
python setup.py py2exe

pygame is more complex: http://www.pygame.org/wiki/Pygame2exe

==== py2app ====

==== pyinstaller ====

==== cx_Freeze ====

=== Windows Libraries ===

==== Python for Windows Extensions - PyWin32 ====

Python for Windows Extensions - http://starship.python.net/~skippy/win32/
* There is a build for each python version and architecture (32bit vs 64bit)

Follow event viewer
<pre>
import win32evtlog # install Python for Windows Extensions - http://starship.python.net/~skippy/win32/

# open event viewer
server = "localhost"
source = "System"
handle = win32evtlog.OpenEventLog(server, source)

# seek to end of log
flags = win32evtlog.EVENTLOG_BACKWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
events = win32evtlog.ReadEventLog(handle, flags, 0)

# return to normal reading direction
flags = win32evtlog.EVENTLOG_FORWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
win32evtlog.ReadEventLog(handle, flags, 0) # ignore initial logs

# loop through events
while True:
events = win32evtlog.ReadEventLog(handle, flags, 0)
if events:
for event in events:
data = event.StringInserts
if data:
# Example: 07/03/14 12:34:35 - MYEVENTSOURCE - 0 - 1 - 4 "HELLO"
print event.TimeGenerated, '-', event.SourceName, '-', event.EventCategory, '-', event.EventID, '-', event.EventType, '-', '"%s"' % " ".join(data)
time.sleep(.5) # yield cpu
</pre>

References:
* PyWin32: Getting Windows Event Logs « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/07/27/pywin32-getting-windows-event-logs/

==== PyCrypto for Windows ====

Download Install: http://www.voidspace.org.uk/python/modules.shtml#pycrypto

Installation: [http://stackoverflow.com/questions/11405549/how-do-i-install-pycrypto-on-windows]
easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win32-py2.7.exe
easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win-amd64-py2.7.exe

==== Paramiko for Windows ====

# Install PyCrypto (above)
# pip install paramiko

==== PyGame for Windows ====

Download - http://www.pygame.org/download.shtml
* http://pygame.org/ftp/pygame-1.9.1.win32-py2.7.msi

== Indentation ==

Python: Myths about Indentation - http://www.secnetix.de/olli/Python/block_indentation.hawk
:There are quite some prejudices and myths about Python's indentation rules among people who don't really know Python. I'll try to address a few of these concerns on this page.
*"Whitespace is significant in Python source code."
*"Python forces me to use a certain indentation style."
*"You cannot safely mix tabs and spaces in Python."
*"I just don't like it."
*"How does the compiler parse the indentation?"

== PEP - Python Enhancement Proposals ==

Python Enhancement Proposals (PEPs)

* Index of Python Enhancement Proposals (PEPs) - http://www.python.org/dev/peps/

== Convert Tabs to Spaces ==

Convert tabs to 4 spaces:
sed -i 's/\t/ /g' script.py

== Linting ==

* pep8 - python style guide checker
* pychecker - more verbose, imports code, slower
* pylint - most verbose, configurable, "rates" code
* pyflakes - least verbose (dead/redundant code)

rpmforge:
yum install pyflakes pychecker
# pylint - not sure which pacakge includes

=== pep8 ===

See [[Python/pep8]]

=== pychecker ===

See [[Python/pychecker]]

=== pylint ===

See [[Python/pylint]]

== PEP8 Style Guidlines ==

PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/

* Use spaces - Use 4 spaces per indentation level.
* Never mix tabs and spaces.
* Limit all lines to a maximum of 79 characters.
* Line wrapping - The preferred way of wrapping long lines is by using Python's implied line continuation inside parentheses, brackets and braces. Long lines can be broken over multiple lines by wrapping expressions in parentheses.
* Encoding - For Python 3.0 and beyond, UTF-8 is preferred over Latin-1, see PEP 3120.
* Imports should usually be on separate lines
* Avoid extraneous whitespace in most situations
* Always use 'self' for the first argument to instance methods.

=== PEP8 Style Checker ===

See [[#pep8]]

== Benefits of Python ==

What is the return on investment for this group by learning Python? The list of benefits below sums up what the language has to offer:
* Python is designed from the ground up to encourage maintainable, readable and reusable code. It has great support for particular paradigms such as object-oriented programming.
* Python code is usually around a quarter of the length of code generated by C++ or Java - less typing, less debugging and less maintenance.
* Python does not need to be compiled or linked, which allows for a more iterative programming style and a quicker development cycle.
* Python is extremely portable. Code can be easily exchanged between Windows, Mac OS and Linux with little to no modification.
* Python comes with a staggering array of standard libraries and mature third-party projects. Web development, scientific computing, operator system access and network scripting are all supported.
* Python can talk with many other languages easily. It has support for invoking C and C++ code and can itself be called from libraries written in these languages.

Source: http://quantstart.com/articles/Quant-Reading-List-Python-Programming/ - Quant Reading List Python Programming

Benefits of Python:
* Python is available for most operating systems, including Windows, UNIX, Linux, and Mac OS.
* Python's clean object-oriented design and extensive support libraries offer two to ten fold the programmer productivity seen with languages like C, C++, C#, Java, VB, and Perl.

Source: Python Development - Benefits of Python - Digital Mesh - http://www.digitalmesh.com/offshore-development-center/python-development/benefits-of-python.html

== Python Package Management ==

=== Install Module ===

Installing Python Modules — Python v2.7.1 documentation - http://docs.python.org/install/index.html

The new standard: Distutils
python setup.py install

Download and extract target package, then:
sudo python setup.py install

Note: Running setup.py install builds and installs all modules in one run

Help:
python setup.py --help

Incremental method:
python setup.py build
python setup.py install
python setup.py build --help # help

Force rebuild and install:
python setup.py build --force # if have to repeat
python setup.py install --force # if have to repeat

Default installation paths:
<pre>
Platform Standard installation location Default value
Unix (pure) prefix/lib/pythonX.Y/site-packages /usr/local/lib/pythonX.Y/site-packages
Unix (non-pure) exec-prefix/lib/pythonX.Y/site-packages /usr/local/lib/pythonX.Y/site-packages
Windows prefix C:\Python
</pre>

Alternate install path:
python setup.py install --home=<dir>

Alternate base folders:
<pre>
Type of file Installation Directory Override option
pure module distribution home/lib/python --install-purelib
non-pure module distribution home/lib/python --install-platlib
scripts home/bin --install-scripts
data home/share --install-data
</pre>

Modifying Python’s Search Path:
import sys
sys.path.append('/www/python/')

=== Python Package Index ===

user-submitted Python packages:
http://pypi.python.org/pypi

sudo python setup.py install

Package Site:
http://pypi.python.org/pypi/[PACKAGE]
http://pypi.python.org/pypi/test

If they are on http://pypi.python.org/:
pip install foo

[http://stackoverflow.com/questions/410163/for-python-programmers-is-there-anything-equivalent-to-perls-cpan For Python programmers, is there anything equivalent to Perl's CPAN? - Stack Overflow]

=== Easy Installation ===

EasyInstall - The PEAK Developers' Center - http://peak.telecommunity.com/DevCenter/EasyInstall
: Easy Install is a python module (easy_install) bundled with setuptools that lets you automatically download, build, install, and manage Python packages.

CheeseShopTutorial - PythonInfo Wiki - http://wiki.python.org/moin/CheeseShopTutorial
: EasyInstall (easy_install) gives you a quick and painless way to install packages remotely by connecting to the Package Index or even other websites via HTTP. It is somewhat analogous to the CPAN and PEAR tools for Perl and PHP, respectively.

=== pip ===

See [[Python/pip]]

==== Create PIP Package ====

See [[Python/pip#Create PIP Package]]

=== Distribute ===

distribute - Easily download, build, install, upgrade, and uninstall Python packages

distribute 0.6.24 : Python Package Index - http://pypi.python.org/pypi/distribute

Installation:
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/d/distribute/distribute-0.6.24.tar.gz#md5=17722b22141aba8235787f79800cc452
tar -zvxf distribute-0.6.24.tar.gz
cd distribute-0.6.24
sudo python setup.py install

Test:
echo "import distribute" | grep python

== Named Parameters ==

http://diveintopython.org/power_of_introspection/optional_arguments.html

Python allows function arguments to have default values; if the function is called without the argument, the argument gets its default value. Futhermore, arguments can be specified in any order by using named arguments. Stored procedures in SQL Server Transact/SQL can do this, so if you're a SQL Server scripting guru, you can skim this part.

Here is an example of info, a function with two optional arguments:

def info(object, spacing=10, collapse=1):

spacing and collapse are optional, because they have default values defined. object is required, because it has no default value. If info is called with only one argument, spacing defaults to 10 and collapse defaults to 1. If info is called with two arguments, collapse still defaults to 1.

Say you want to specify a value for collapse but want to accept the default value for spacing. In most languages, you would be out of luck, because you would need to call the function with three arguments. But in Python, arguments can be specified by name, in any order.

http://www.wellho.net/mouth/1871_Optional-and-named-parameters-in-Python.html

If you add a parameter with two asterixes (**) on the end of a function definition, that parameter is taken as being the name of a dictionary into which all otherwise unidentified parameters are stored.

<pre>
def getnet(gross, taxrate=17.5, **others):
net = gross / (1.0 + taxrate*0.01)
for more in others.keys():
print "GETNET:",more," - ",others[more]
return neto
</pre>

== Python vs other languages ==

[http://www.strombergers.com/python/ Python is Cool (and Perl is not), Especially for C/C++ Programmers]

==Experimenting with Memory Remanence==
[http://citp.princeton.edu/memory/exp/ Experimenting with Memory Remanence]

<pre>
#!/usr/bin/env python

# a pirate's favorite chemical element
a = ""
while 1: a += "ARGON"
</pre>

sudo strings /dev/mem | less

==UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)==
See [[UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)]]

== Program Layout Best Practices Presentation ==

Program layout best practices - Seth House <seth@eseth.com> - Utah Django User Group - 2011-06-23
* Slides - https://github.com/whiteinge/presentations/tree/master/upyug_2011-06-09_program-layout
* Video - http://blip.tv/utah-open-source/python-layout-best-practices-for-modules-django-cli-utah-django-user-group-5315808

Module skeleton:
<pre>
#!/usr/bin/env python
# coding: utf-8

"""Module docstring"""

__author__ = ’Mr. Me <me@example.net>’
__version__ = ’1.2.3’

if __name__==’__main__’:
main()
</pre>

Executing modules as scripts
# PEP 338:
python -m SimpleHTTPServer

== CLI package ==

cli — command line tools — pyCLI v1.1.1 documentation - http://packages.python.org/pyCLI/

== Python IDE ==

IntegratedDevelopmentEnvironments - PythonInfo Wiki - http://wiki.python.org/moin/IntegratedDevelopmentEnvironments
* IDEs with introspection-based code completion and integrated debugger
* IDEs with introspection-based code completion /or/ integrated debugger
* IDEs with integrated gui builder
* Editors
* Reviews

Is there a good, free Python IDE for Windows? - Stack Overflow - http://stackoverflow.com/questions/126753/is-there-a-good-free-python-ide-for-windows
* Eclipse with PyDev
* Aptana Studio 3
* Active State also offers Komodo Edit
* WingWare - http://www.wingware.com/
* Eric IDE - http://www.die-offenbachs.de/eric/index.html
* MMM-Experts - Products - http://mmm-experts.com/Products.aspx?ProductId=4
** PyScripter is a Python IDE built in Python for Delphi(P4D) components and created with the ambition to become competitive in functionality with commercial Windows-based IDEs available for other languages. Being built in a compiled language is rather snappier than some of the other Python IDEs and provides an extensive blend of features that make it a productive Python development environment.
* ActiveState Python comes with PythonWin which is reasonably good
* PyCharm from JetBrains is not free in general, but has free early access program.
* SPE - Stani's Python Editor, is pretty awesome. See http://pythonide.blogspot.com/.
* You may want to take a look at the pdb module - http://docs.python.org/lib/module-pdb.html
* Geany (http://www.geany.org/) is a fairly light text editor that does a lot of IDE-like stuff.
* SciTE: http://www.scintilla.org/SciTE.html
* Editra: http://editra.org/
* Netbeans has support for python development in their Python Early Access version. - http://www.netbeans.org/features/python/
* I use Notepad++ with the appropriate settings.
* The Eric Python IDE - http://www.die-offenbachs.de/eric/index.html
* BOa Constructor www.boa-constructor.sourceforge.net really nice! drag and drop stuff!
* DrPtyhon is also very nice, and has many plug-ins. http://drpython.sourceforge.net/
* Vim
* I will make a plug for my preference of Python development environments: SPE. SPE is an integrated development environment that provides code completion, integrated Python shell, calltips, and various helper tools like notes and todo lists. It also includes wxGlade so you can make wxPython applications easily. Plus, it includes the python debugger WinPdb.
* http://groups.google.com.tr/group/python_opengl this software is my sample visual python ide.its demo version.
* I use PyScripter, and find it pretty awesome.
* There is one Python IDE written in Java: jHepWork: http://jwork.org/jhepwork/ it can detect Python's indentation rules rather nicely (+ detects structure of a code and syntax)
* gvim built with your python version support and with this plugins:
** NERDTree (don't read he's blog ;)
** matchit (it's builtin in new versions just put a symlink or junction)
** pyflakes
** pysmell
* Try Python Tools for Visual Studio. Supports CPython and IronPython. - http://pytools.codeplex.com/

PythonEditors - PythonInfo Wiki - http://wiki.python.org/moin/PythonEditors
* List of editors

Python Ide - http://c2.com/cgi/wiki?PythonIde
: There are a lot of Ides for PythonLanguage springing up. Many of them written in Python

Is there a better Python IDE? « Lennart Regebro: Python, Plone, Web - http://regebro.wordpress.com/2010/10/09/is-there-a-better-python-ide/
: I’m currently a user of WingIDE, but sometimes I feel the need for something better, mainly something that is a bit more stable. WingIDE often hangs when switching projects, sometimes the file listing decides that you must double-click on the expand arrow instead of just click on it, etc. It still doesn’t have macros despite this being a much requested feature (how hard can it be to add, really?) and I never really get the hang of the SVN integration, so I just ignore it. And they keep recommending me to use WingIDE’s separate instance of GTK instead of the System GTK, which is daft. Their designs are ugly, and it should work with the system GTK so it looks the same. Anything else is just silly.

=== Open Komodo ===

Open Komodo - http://www.openkomodo.com/
: Open Komodo is the source code repository for Komodo Edit - a free multi-language editor for dynamic languages, based on the award-winning Komodo IDE. ActiveState created the Open Komodo code repository in August 2007 and it has been used since then for building all Komodo Edit releases.

=== Eclipse ===

Build an Eclipse development environment for Perl, Python, and PHP - http://www.eclipse.org/resources/resource.php?id=485
: This tutorial shows how Eclipse's DLTK makes it possible to build development tools for scripting languages. In particular, it explains how to implement syntax coloring, user preferences, and interpreter integration in a plug-in-based project.

See [[#PyDev]]

=== EasyEclipse ===

EasyEclipse - http://www.easyeclipse.org/
:EasyEclipse packages together Eclipse, the open-source software
development platform, and selected open source plugins.
:We select, assemble, test, patch, build installers and document
a full IDE, offered as reliable distributions and plugins.
:EasyEclipse is:
* Free and open-source,
* Easy to download and install, and
* Simple to maintain, without version and dependency issues.

EasyEclipse for Python - http://www.easyeclipse.org/site/distributions/python.html

EclipsePythonIntegration - PythonInfo Wiki - http://wiki.python.org/moin/EclipsePythonIntegration
: Using Eclipse as a Python editor

=== PyDev ===

PyDev - http://pydev.org/
: PyDev is a Python IDE for Eclipse, which may be used in Python, Jython and IronPython development.

---

Pydev - http://pydev.sourceforge.net/ (old link)

How to Configure Eclipse for Python - http://www.rose-hulman.edu/class/csse/resources/Eclipse/eclipse-python-configuration.htm

Python Development with PyDev and Eclipse - Tutorial - http://www.vogella.de/articles/Python/article.html
: This article describes how to write and debug Python programs with Eclipse

PyDev Getting Started - http://pydev.org/manual_101_root.html

Running a program - http://pydev.org/manual_101_run.html

---

Supported Task Tags:
* TODO:
* TASK:
* FIXME:

---

Installation:
# Platform Runtime Binary (download around 45-50 MB) - http://download.eclipse.org/eclipse/downloads/ or http://www.eclipse.org/downloads/
#* Example: Eclipse Classic 4.2.2
# Eclipse Software Add - PyDev: http://pydev.org/updates
# Eclipse Software Add - MercurialEclipse: http://cbes.javaforge.com/update [http://javaforge.com/project/HGE] [http://mercurial.selenic.com/wiki/MercurialEclipse]
# Switch perspectives to PyDev

=== The Eric Python IDE ===

The Eric Python IDE - http://eric-ide.python-projects.org/
:Eric is a full featured Python and Ruby editor and IDE, written in python. It is based on the cross platform Qt gui toolkit, integrating the highly flexible Scintilla editor control. It is designed to be usable as everdays' quick and dirty editor as well as being usable as a professional project management tool integrating many advanced features Python offers the professional coder. eric4 includes a plugin system, which allows easy extension of the IDE functionality with plugins downloadable from the net.

Eric Python IDE - Wikipedia - http://en.wikipedia.org/wiki/Eric_Python_IDE

=== Wingware Python IDE ===

The Intelligent Development Environment for Python Programmers - Wingware Python IDE - http://www.wingware.com/
:Wingware's Python IDE is an Integrated Development Environment designed specifically for the Python programming language. Since 1999, Wingware has focused on Python.

=== VIM ===

Python .vimrc:
<pre>
" Kenneth's Preferences

set nobackup " don't create backup files
"set nohlsearch " don't highlight search results
set hlsearch " do highlight search results
set incsearch " incremental search
set ignorecase " ignore case during search
set smartcase " do case sensitive search when upper case is typed
set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
set showmatch " briefly highlight matching bracket
set matchtime=1 " highlight match for 0.1 seconds
set background=dark " black background
set shiftwidth=4 " shift blocks by 4 spaces
set shortmess=at " use short messages so we don't have to hit enter
set expandtab " expand tabs into spaces
set scrolloff=5 " always show 5 lines before & after cursor
set shell=bash " use bash for the command shell
set shellcmdflag=-lc " make bash startup as a login shell for ~/.bashrc
"set nowrap " do not wrap long lines
set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set wildmenu " enable filename tab completion
set ruler " display the ruler: shows cursor position
set cpoptions+=$ " display '$' at end of cw sequence
set title " display filename and vim info in window title bar
set modelines=5 " search the first and last 5 lines for "vim:" modeline
set backspace=2 " backspace deletes

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

"if v:version > 700
" set cursorline " underline the current line
"endif

if &t_Co > 2 || has( "gui_running" )
syntax on
endif

if has( 'mouse' )
set mouse=a " enable mouse
endif

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

"folding settings
set foldmethod=indent "fold based on indent
set foldnestmax=10 "deepest fold is 10 levels
set nofoldenable "dont fold by default
set foldlevel=10 "this is just what i use

" Auto clear trailing white space in Python files
func! DeleteTrailingWS()
exec "normal mz"
%s/\s\+$//ge
exe "normal `z"
endfunc
autocmd BufWrite *.py :call DeleteTrailingWS()
</pre>

----

Older versions:
<pre>
" KENNETH SETTINGS
set nocp " turn off compatibility mode
set nomodeline " do not allow files to dictate vim behavior

set incsearch " incremental search
set ignorecase " ignore case during search

set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set background=dark " black background
syntax on " turn on syntax highlighting
set hlsearch " highligh last search pattern

set nowrap " do not wrap long lines
set scrolloff=5 " always show 5 lines before & after cursor
set ruler " show the cursor position all the time
set backspace=2 " allow backspace deletion
set bs=indent,eol,start " allow backspacing over everything in insert mode

set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set expandtab " expand tab into spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
filetype indent on " turn on auto indent for specific file types
</pre>

Other options:
<pre>
set number " show line numbers
set numberwidth=1 " remove space before largest line numbers

set smartindent " ?? - maybe included in 'filetype indent on'

set ai " always auto indent

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

</pre>

Kenneth's fio .vimrc:
<pre>

"source $VIM/mswin.vim

set nobackup " don't create backup files
"set nohlsearch " don't highlight search results
set incsearch " incremental search
set ignorecase " ignore case during search
set smartcase " do case sensitive search when upper case is typed
set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
set showmatch " briefly highlight matching bracket
set matchtime=1 " highlight match for 0.1 seconds
set background=dark " black background
set shiftwidth=4 " shift blocks by 4 spaces
set shortmess=at " use short messages so we don't have to hit enter
set expandtab " expand tabs into spaces
set scrolloff=5 " always show 5 lines before & after cursor
set shell=bash " use bash for the command shell
set shellcmdflag=-lc " make bash startup as a login shell for ~/.bashrc
"set nowrap " do not wrap long lines
set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set wildmenu " enable filename tab completion
set ruler " display the ruler: shows cursor position
set cpoptions+=$ " display '$' at end of cw sequence
set title " display filename and vim info in window title bar
set modelines=5 " search the first and last 5 lines for "vim:" modeline

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

if v:version > 700
set cursorline " underline the current line
endif

if &t_Co > 2 || has( "gui_running" )
syntax on
endif

if has( 'mouse' )
set mouse=a " enable mouse
endif

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

"folding settings
set foldmethod=indent "fold based on indent
set foldnestmax=10 "deepest fold is 10 levels
set nofoldenable "dont fold by default
set foldlevel=10 "this is just what i use
</pre>

== Interpreter vs Script ==

Interpreter:
>>> import os
>>> os.uname()
('Linux', 'demo.oeey.com', '2.6.18-194.8.1.el5', '#1 SMP Thu Jul 1 19:07:06 EDT 2010', 'i686')

Script:
#!/usr/bin/env python
import os
print os.uname()

== Web Development ==

WebProgramming - PythonInfo Wiki - http://wiki.python.org/moin/WebProgramming
: This topic guide attempts to cover every aspect of programming Web applications (both clients and servers) using Python.

HOWTO Use Python in the web — Python v2.7.2 documentation - http://docs.python.org/howto/webservers.html
: This document shows how Python fits into the web. It presents some ways to integrate Python with a web server, and general practices useful for developing web sites.

=== Plug Simple Python Web App ===

<pre>
-------- Original Message --------
Subject: Re: Writing a simple python web app
Date: Mon, 25 Oct 2010 19:03:31 -0400
From: Roberto Mello <roberto.mello@gmail.com>
To: Provo Linux Users Group <plug@plug.org>

On Sat, Sep 11, 2010 at 5:40 PM, Dave Smith <dave@thesmithfam.org> wrote:
> I'm creating a very simple web application, and I want to use python. My first guess was that mod_python
would provide the easiest entry point. Boy, was I wrong. All the mod_python tutorials spend 80% of their
time extolling the virtues of mod_python over CGI, but they are quite lean on specific examples. It's quite
disappointing. A natural progression then leads me to mod_wsgi, and since that has equally poor
documentation, to Django. However, Django is overkill for my app (I don't even have a database).

For something as simple as you have described, and to run on an
embedded platform, I would probably stay away from frameworks designed
to run on full hardware, and go with something more minimalistic.

Take a look at CherryPy [1] or web.py [2].

[1] http://www.cherrypy.org/
[2] http://webpy.org/

Roberto Mello
</pre>

---

See also [[web.py]]

See also [[CherryPy]]

See also [[Django]]

=== Apache CGI ===

See [[CGI]]

Great tutorial: https://www.tutorialspoint.com/python/python_cgi_programming.htm

My favorite option, make all .cgi files executable: (or .py)
<pre>
DirectoryIndex index.html index.php index.py
<Location />
# AddHandler cgi-script .cgi
AddHandler cgi-script .py
Options +ExecCGI
</Location>
</pre>

If you want to allow .htaccess:
* see [[Apache#.htaccess]]

To enable on Ubuntu:
sudo a2enmod cgi
service apache2 restart

Directory index:
DirectoryIndex index.py

Or if you really want to go crazy, this will try to run *every* file: [http://blog.kapsobor.de/archives/2006/09/05/apache_addhandler_multiple_extensions/]
<pre>
# this has problems with / directory being run as a script, instead of index too!
#<Location />
# SetHandler cgi-script
# Options +ExecCGI
#</Location>

<Files ~ ".">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>

or:
<pre>
<Files ~ "^.[a-zA-Z]*$">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>

Or just have index file:
<Files "index">

Note: Place within a <VirtualHost> section if needed.

Note: Make sure to 'chmod +x' the files.

Start all HTML output with:
print "Content-Type: text/html\n"

If you want just text output:
print "Content-Type: text/plain\n"

GET (from Environment):
<pre>
print "<pre>"
for env in os.environ:
print "%s = %s" % (env, os.environ[env])
print "</pre>"
</pre>

<pre>
# sample for: http://oeey.com/py/test.py/test?var1=ken&var2=barbie

REQUEST_URI = /py/index.py/test?var1=ken&var2=barbie
QUERY_STRING = var1=ken&var2=barbie
PATH_INFO = /test
REQUEST_METHOD = GET

SCRIPT_NAME = /py/test.py
SERVER_NAME = oeey.com
REMOTE_ADDR = 216.51.42.66
SERVER_ADDR = 192.168.108.4
DOCUMENT_ROOT = /www/py
SCRIPT_FILENAME = /www/py/test.py
HTTP_HOST = cgi.oeey.com
</pre>

POST (from stdin):
<pre>
post = sys.stdin.readline()
if post:
print post
</pre>

<pre>
CONTENT_TYPE = application/x-www-form-urlencoded 

yin=one+two+three&yang=four+five+six
</pre>

cgi helper class:
<pre>
import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage()

# field keys
print "fields keys: ", form.keys() # list of field keys

# fild values
if "name" in form: # if form.has_key('name'):
print "field 'name' is: ", form["name"].value
print "field 'name' is: ", form.getvalue('name')

# headers
print "dictionary of headers: ", form.headers
print form.type
print form.headers['content-type']
</pre>

To prevent XSS escape the characters:
print cgi.escape(form.getvalue('name'))

<pre>
import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage()
if (form.has_key("action") and form.has_key("name") \
and form.has_key("age")):
if (form["action"].value == "display"):
print "%s = %s" % (form["name"].value form["age"].value)
</pre>

Note: cgi helper class also does sessions and cookies.

References:
* CGI Tutorial | Web Python - http://webpython.codepoint.net/cgi_tutorial
* Python CGI: An Interactive Instruction - http://www.cs.virginia.edu/~lab2q/

See also:
* Web Python - http://webpython.codepoint.net/

---

REST Example:

<pre>
import os, re
def get_rest():
if not "PATH_INFO" in os.environ:
return ()
# filter for garbage
#print os.environ["PATH_INFO"]
path_info = re.findall('[/A-Za-z0-9_-]*', os.environ["PATH_INFO"])
if not path_info:
return ()
path_info = ''.join(path_info)
# collapse empty paths
paths = path_info.split('/')
final_paths = []
for path in paths:
if path:
final_paths.append(path)
return tuple(final_paths)
rest = get_rest();
</pre>

=== mod_python ===

AVOID! AVOID! AVOID! - if you want this simple, just use CGI!

See [[Apache mod_python]]

mod_python - http://docs.python.org/howto/webservers.html#mod-python

Install:
yum install mod_python

Config:
/etc/httpd/conf.d/python.conf

Simple:
<pre>

</pre>

:"People coming from PHP often find it hard to grasp how to use Python in the web. Their first thought is mostly mod_python, because they think that this is the equivalent to mod_php. Actually, there are many differences. What mod_python does is embed the interpreter into the Apache process, thus speeding up requests by not having to start a Python interpreter for each request. On the other hand, it is not “Python intermixed with HTML” in the way that PHP is often intermixed with HTML. The Python equivalent of that is a template engine. mod_python itself is much more powerful and provides more access to Apache internals. It can emulate CGI, work in a “Python Server Pages” mode (similar to JSP) which is “HTML intermingled with Python”, and it has a “Publisher” which designates one file to accept all requests and decide what to do with them.

:mod_python does have some problems. Unlike the PHP interpreter, the Python interpreter uses caching when executing files, so changes to a file will require the web server to be restarted. Another problem is the basic concept – Apache starts child processes to handle the requests, and unfortunately every child process needs to load the whole Python interpreter even if it does not use it. This makes the whole web server slower. Another problem is that, because mod_python is linked against a specific version of libpython, it is not possible to switch from an older version to a newer (e.g. 2.4 to 2.5) without recompiling mod_python. mod_python is also bound to the Apache web server, so programs written for mod_python cannot easily run on other web servers.

:These are the reasons why mod_python should be avoided when writing new programs. In some circumstances it still might be a good idea to use mod_python for deployment, but WSGI makes it possible to run WSGI programs under mod_python as well."

=== WSGI ===

WSGI - http://docs.python.org/howto/webservers.html#mod-wsgi

: "The Web Server Gateway Interface, or WSGI for short, is defined in PEP 333 and is currently the best way to do Python web programming. While it is great for programmers writing frameworks, a normal web developer does not need to get in direct contact with it. When choosing a framework for web development it is a good idea to choose one which supports WSGI."

See [[mod_wsgi]]

=== Django ===

See [[Django]]

=== CherryPy ===

See [[CherryPy]]

=== web.py ===

web.py: makes web apps

Think about the ideal way to write a web app. Write the code to make it happen.

web.py : Python Package Index - http://pypi.python.org/pypi/web.py

Home Page: http://webpy.org/

"Django lets you write web apps in Django. TurboGears lets you write web apps in TurboGears. Web.py lets you write web apps in Python."
:— Adam Atlas

NOTE: Can modify source code without having to restart app! :-)

---

PIP Install:
pip install web.py

---

Hello World: [http://webpy.org/]
<pre>
import web

urls = (
'/(.*)', 'hello'
)
app = web.application(urls, globals())

class hello:
def GET(self, name):
if not name:
name = 'World'
return 'Hello, ' + name + '!'

if __name__ == "__main__":
app.run()
</pre>

Note: app.py can be renamed.

Running:
python app.py

Note: You can add an IP address/port after the "app.py" bit to control where web.py launches the server. You can also tell it to run a fastcgi or scgi server.
python app.py 1234

---

Templates

<pre>
# templates/hello.html
$def with (name)

$if name:
I just wanted to say hello to $name.
$else:
Hello, world!
</pre>

app = ...
render = web.template.render('templates/')

index.GET:
name = 'Bob'
return render.hello(name) # render.[TEMPLATE_NAME](vars)

get variable: (/?name=Joe)
i = web.input(name=None)
return render.hello(i.name)

variable in url:
urls = ('/(.*)', 'hello')

class index:
def GET(self, name):
return render.hello(name)

# visit: /Joe

multiple variables in url:
urls = ('/(.*)/(.*)', 'hello')

class hello:
def GET(self, fname, lname):
return render.hello(fname, lname)

# visit: /Joe/Smith

=== lpthw.web ===

lpthw.web: a fork of web.py for Learn Python The Hard Way

Locks web.py at a specific version and removes some magic. Thanks to Aaron Swartz for making it originally.

lpthw.web: a fork of web.py for Learn Python The Hard Way - http://pypi.python.org/pypi/lpthw.web

Home Page: http://learnpythonthehardway.org/

This conflicts with web.py, so only install one.

=== web ===

web - Web modules for CGI and WSGI web programming

web : Python Package Index - http://pypi.python.org/pypi/web

Warning: This is an alpha release for those who are interested in
checking progress. No documentation and not for production use.

Auth, session, mail, form and templating tools for writing CGI scripts or WSGI web applications.

This version incompatible with previous versions becuase large portions of the code have been refactored into separate components.

== Issues ==

=== Hide DeprecationWarning ===

==== Import DeprecationWarning ====

Error:
<pre>
/opt/xmpppy-0.4.1/xmpp/auth.py:24: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import sha,base64,random,dispatcher,re
/opt/xmpppy-0.4.1/xmpp/auth.py:26: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5
</pre>

Hide deprecation warnings with code: [http://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python]
<pre>
import warnings

with warnings.catch_warnings():
warnings.filterwarnings("ignore",category=DeprecationWarning)
import md5, sha
</pre>

Hide with header:
#!/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line:
python -W ignore::DeprecationWarning script.py

References:
* How to ignore deprecation warnings in Python - Stack Overflow - http://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python

==== Funciton DeprecationWarning ====

Error:
<pre>
/opt/xmpppy-0.4.1/xmpp/transports.py:307: DeprecationWarning: socket.ssl() is deprecated. Use ssl.wrap_socket() instead.
tcpsock._sslObj = socket.ssl(tcpsock._sock, None, None)
</pre>

Hide with header:
#!/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line:
python -W ignore::DeprecationWarning script.py

==== Ubuntu 24 pip packages ====

<pre>
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
</pre>

== keywords ==

[[Category:Programming]]
[[Category:Python]]

Python

2026-04-27T20:28:29Z

Kenneth: /* Variable Type = */

{{subtop}}

== Python ==

Python (programming language) - Wikipedia - http://en.wikipedia.org/wiki/Python_(programming_language)
: Python is an interpreted, general-purpose high-level programming language whose design philosophy emphasizes code readability. Python aims to combine "remarkable power with very clear syntax", and its standard library is large and comprehensive. Its use of indentation for block delimiters is unique among popular programming languages.
: Python supports multiple programming paradigms, primarily but not limited to object-oriented, imperative and, to a lesser extent, functional programming styles. It features a fully dynamic type system and automatic memory management, similar to that of Scheme, Ruby, Perl, and Tcl. Like other dynamic languages, Python is often used as a scripting language, but is also used in a wide range of non-scripting contexts.
: The reference implementation of Python ([http://en.wikipedia.org/wiki/CPython CPython]) is free and open source software and has a community-based development model, as do all or nearly all of its alternative implementations. CPython is managed by the non-profit Python Software Foundation.
: Python interpreters are available for many operating systems, and Python programs can be packaged into stand-alone executable code for many systems using various tools.

Author:
:"Python was conceived in the late 1980s and its implementation was started in December 1989 by '''[http://www.python.org/~guido/ Guido van Rossum]''' at CWI in the Netherlands as a successor to the ABC programming language (itself inspired by SETL) capable of exception handling and interfacing with the Amoeba operating system. Van Rossum is Python's principal author, and his continuing central role in deciding the direction of Python is reflected in the title given to him by the Python community, Benevolent Dictator for Life (BDFL)." [http://en.wikipedia.org/wiki/Python_(programming_language)]

Name:
:"The language is named after the BBC show “Monty Python’s Flying Circus” and has nothing to do with reptiles. Making references to Monty Python skits in documentation is not only allowed, it is encouraged!" [http://docs.python.org/py3k/tutorial/appetite.html]

We as people are "Pythonites" [http://raspi.tv/2013/how-to-use-interrupts-with-python-on-the-raspberry-pi-and-rpi-gpio-part-2]

== [http://www.python.org/dev/peps/pep-0020/ The Zen of Python] ==

"The core philosophy of the language is summarized by the document 'PEP 20 (The Zen of Python)'" [http://en.wikipedia.org/wiki/Python_%28programming_language%29]
* PEP 20 -- The Zen of Python - http://www.python.org/dev/peps/pep-0020/

<pre>
Abstract

Long time Pythoneer Tim Peters succinctly channels the BDFL's
guiding principles for Python's design into 20 aphorisms, only 19
of which have been written down.

The Zen of Python

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Easter Egg

>>> import this
</pre>

Other Sources:
* http://www.python.org/doc/humor/
* The Zen of Python – as a Poster « CODE POETRY - http://codepoetry.wordpress.com/2010/03/23/the-zen-of-python-as-a-poster/
** Zen of Python (PDF) http://codepoetry.files.wordpress.com/2010/03/zen-of-python.pdf

== Documentation ==

* Overview — Python v2.7.1 documentation - http://docs.python.org/index.html
** Download — Python v2.7.1 documentation - http://docs.python.org/download.html

* Overview — Python v3.2 documentation - http://docs.python.org/py3k/
** Download — Python v3.2 documentation - http://docs.python.org/py3k/download.html

* PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/

== Tutorials ==

* The Python Tutorial — Python v2.7.1 documentation - http://docs.python.org/tutorial/

* Python 101 -- Introduction to Python - http://www.rexx.com/~dkuhlman/python_101/python_101.html

* 4. More Control Flow Tools — Python v2.7.1 documentation - http://docs.python.org/tutorial/controlflow.html

== Linux Installation ==

RHEL Distribution installation:
yum -y install python python-setuptools

Ubuntu Distribution installation: [https://help.ubuntu.com/community/Mercurial]
# enable 'universe' in the sources.list
# "Mercurial is included in the Universe repository - activate this repo first"
sudo apt-get install mercurial

CentOS 5 Option Recommended Dependencies:
yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel

# this leaves: 2.6.6: _bsddb, bsddb185, dl, imageop, sunaudiodev
# this leaves: 2.7.1: _bsddb, bsddb185, dl, imageop, sunaudiodev
# this leaves: 3.2: none

CentOS 9 Option Recommended Dependencies:
# gdbm-devel not found in centos9?
yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel sqlite-devel bzip2-devel tk-devel tcl-devel

Ubuntu 12.04 LTS 64bit dependencies:
apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev

# maybe: lib64readline-dev libsqlite3-dev tk-dev

# this leaves: 2.7: _bsddb, bsddb185, dl, imageop, sunaudiodev

=== 2.x ===

CentOS dependencies:
sudo yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel

Ubuntu 12.04 LTS 64bit dependencies:
sudo apt-get update
sudo apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev

Ubuntu 24.04 LTS 64bit dependencies:
sudo apt-get update
sudo apt-get install gcc make sudo libbz2-dev tcl-dev libncurses-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev libffi-dev
# optional install
sudo apt-get install checkinstall

2.7.15:
<pre>
# PYVER=2.6.6
# PYVER=2.7.5
# PYVER=2.7.6
# PYVER=2.7.9
# PYVER=2.7.10
# PYVER=2.7.15
PYVER=2.7.18
mkdir -p ~/.src ; cd ~/.src
wget --no-check-certificate http://www.python.org/ftp/python/$PYVER/Python-$PYVER.tgz
tar -zvxf Python-$PYVER.tgz
cd Python-$PYVER
# ./configure --prefix=/opt/python-$PYVER
# make clean && make
./configure --prefix=/opt/python-$PYVER && make clean && make
sudo make install

sudo /opt/python-$PYVER/bin/python -m ensurepip --upgrade
sudo /opt/python-$PYVER/bin/pip install --upgrade pip
sudo /opt/python-$PYVER/bin/pip install requests

export PATH=/opt/python-$PYVER/bin:$PATH
echo -e "\n\nexport PATH=/opt/python-$PYVER/bin:\$PATH\n" >> ~/.bash_profile
# echo -e "\n\nexport PATH=/opt/python27/bin:\$PATH\n" >> ~/.bashrc

# NOTE: remove any old versions from your .bash_profile
</pre>

Bootstrapping the pip installer - https://docs.python.org/2/library/ensurepip.html

=== 3.x ===

3.8.16:
<pre>
#VER=3.7.0
#VER=3.8.15
#VER=3.10.9
VER=3.8.16
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/${VER}/Python-${VER}.tgz
tar -zvxf Python-${VER}.tgz
cd Python-${VER}
./configure --prefix=/opt/python${VER}
make

sudo make install

## OPTIONAL alternative install path:
sudo checkinstall
## note: 'checkinstall' makes a .deb, which you can uninstall with easier, but if you are using --prefix, you do need to
## premake the folders
sudo mkdir -p /opt/python${VER}/bin
sudo mkdir -p /opt/python${VER}/lib
shortVER=`echo $VER | cut -d '.' -f 1-2` # get short version eg. 3.10 from 3.10.9
sudo mkdir -p /opt/python${VER}/lib/python${shortVER}
sudo mkdir -p /opt/python${VER}/lib/python${shortVER}/test
# sudo checkinstall # hit Y, and enter twice
## note: older versions have problems with the virtualization of the file system, so do FS Translation = no
checkinstall --fstrans=no

export PATH=/opt/python${VER}/bin:$PATH
echo -e "\n\nexport PATH=/opt/python${VER}/bin:\$PATH\n" >> ~/.bash_profile

# have to manually update version
rm /opt/python${VER}/bin/python
ln -sfn python3.8 /opt/python${VER}/bin/python
#cp /opt/python${VER}/bin/python3.8 /opt/python${VER}/bin/python

# DEPENDENCIES
sudo apt install ...
yum install libffi-devel openssl-devel
</pre>

Note, if you want the libraries to be shared (like python-dev):
./configure --enable-shared

---

VER=3.6.9

VER=3.6.8

---

3.2.0:
<pre>
mkdir -p ~/src ; cd ~/src
wget http://www.python.org/ftp/python/3.2/Python-3.2.tgz
tar -zvxf Python-3.2.tgz
cd Python-3.2
./configure --prefix=/opt/python32
make
sudo make install

export PATH=/opt/python32/bin:$PATH
echo -e "\n\nexport PATH=/opt/python32/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.3.0:
<pre>
mkdir -p ~/src ; cd ~/src
wget http://www.python.org/ftp/python/3.3.0/Python-3.3.0.tgz
tar -zvxf Python-3.3.0.tgz
cd Python-3.3.0
./configure --prefix=/opt/python3.3.0
make
sudo make install

export PATH=/opt/python3.3.0/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.3.0/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.3.4:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/3.3.4/Python-3.3.4.tgz
tar -zvxf Python-3.3.4.tgz
cd Python-3.3.4
./configure --prefix=/opt/python3.3.4
make clean
make
sudo make install

export PATH=/opt/python3.3.4/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.3.4/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.4.0:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz
tar -zvxf Python-3.4.0.tgz
cd Python-3.4.0
./configure --prefix=/opt/python3.4.0
make
sudo make install

export PATH=/opt/python3.4.0/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.4.0/bin:\$PATH\n" >> ~/.bash_profile
</pre>

=== setuptools ===

Purpose: to install pip

Note: required for several 'python setup.py install' installations.

Python Package Index : setuptools 0.6c11 - http://pypi.python.org/pypi/setuptools

By Package Manager:
yum install python-setuptools

By Source:
#VER=2.0.1
#VER=2.2
VER=3.4.4
mkdir -p ~/.src ; cd ~/.src
wget --no-check-certificate http://pypi.python.org/packages/source/s/setuptools/setuptools-$VER.tar.gz
tar -zvxf setuptools-$VER.tar.gz
cd setuptools-$VER
# sudo python setup.py install
sudo /opt/python-2.7.9/bin/python setup.py install

Fix header:
# Only needed on older versions
sed -i 's#!/usr/bin/python#!/usr/bin/env python#g' /opt/python279/bin/easy_install

Install pip:
sudo /opt/python-2.7.9/bin/easy_install pip

Sample pip usage:
sudo /opt/python-2.7.9/bin/pip install paramiko

== IDLE and Interactive Mode ==

Interactive Mode - "When commands are read from a tty, the interpreter is said to be in interactive mode. In this mode it prompts for the next command with the primary prompt, usually three greater-than signs (>>>); for continuation lines it prompts with the secondary prompt, by default three dots (...)." [http://docs.python.org/2/tutorial/interpreter.html#invoking-the-interpreter]

Can use '_' to reference last output:
>>> 5 + 6
11
>>> print _
11

References:
* 2. Using the Python Interpreter — Python v2.7.3 documentation - http://docs.python.org/2/tutorial/interpreter.html

== Python 2.7 EOL ==

<blockquote>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
</blockquote>

== Show All Python in Path ==

compgen -c | grep -i "python"

== Code ==

=== Hello World ===

[http://en.wikipedia.org/wiki/Shebang_%28Unix%29 shebang] (also called a hashbang) of Python:
#!/usr/bin/env python
print "Hello World"

NOTE: can also just use "#!/usr/bin/python"

=== Standard Script ===

Standard Script:
<pre>
def main(args):
# do logic

if __name__ == "__main__":
sys.exit(main(sys.argv))
</pre>

Needed for things like pylint

=== White Space ===

Python uses white space to indicate code blocks. Instead of ''{'' use a '':'' and indent consistently (4 spaces).

if a == b:
print 'equal'

Convert tabs to spaces:
sed -i 's/\t/ /g' *.py

Remove trailing whitespace:
sed -i 's/\s*$//' *.py
sed --in-place 's/[[:space:]]\+$//' fileName

Clean project:
# find . -name "*py" -exec dos2unix {} \;
# find . -name "*py" -exec sed -i 's/\t/ /g' {} \;
# find . -name "*py" -exec sed -i 's/\s*$//' {} \;
find . -name "*py" -exec dos2unix {} \; -exec sed -i 's/\t/ /g' {} \; -exec sed -i 's/\s*$//' {} \;

==== Trim Tailing White Space ====

<pre>
sed -e 's/[[:space:]]*$//'
</pre>

<pre>
find . -name "*.py" -exec sed -i -e 's/[[:space:]]*$//' {} \;
</pre>

ref: [https://stackoverflow.com/questions/369758/how-to-trim-whitespace-from-a-bash-variable]

=== Print ===

print - http://docs.python.org/tutorial/inputoutput.html

Print: # include new line
print("Hello World")
print "Hello World" # python 2 only

Suppress new line: [http://stackoverflow.com/questions/493386/how-to-print-in-python-without-newline-or-space]
<pre>
print('#', end='') # python 3
print "Hello World", # python 2 only (does produce a single separating white space though)

import sys
sys.stdout.write('.') # no separating white space
</pre>

# flush output
# optional, if new line won't come for some time
import sys
sys.stdout.flush()

Multiple with separator:
print("hello", "world") # separated by spaces
print("hello", "world", sep = '.') # separated by period

Formatting:
<pre>
mystr = "abc"
myint = 5
myflt = 3.141592
print( "%s %d %.3f" % (mystr, myint, myflt) )
print( "{0} {1} {2}".format(mystr, myint, myflt) ) # string's format function (python > 2.7)
print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name
print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding
</pre>

Padded floating point decimal number: (rounds)
print "%06.2f" % 2.319 # "002.32"
print "%6.2f" % 2.311 # " 2.31"
print "{:06.2f}".format(2.319) # "002.32"
print "{:6.2f}".format(2.319) # " 2.32"

# Python 3 version where var=2.321234
print(f"{var:.2f}") # "2.32"
print(f"{var:6.2f}") # " 2.32"
print(f"{var:06.2f}") # "002.32"
print(f"{var:08.2f}") # "00002.32"
print(f"{var:08.4f}") # "002.3212"
# Python 3 version where var=2.329 - rounding
print(f"{var:06.2f}") # "002.33"

=== Comments ===

# this is a comment
# this is a comment
x = y # this is a comment

Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended)
"""
multi line comment
"""

'''
multi line comment can comment out """
'''

This can be used like a C "#if 0" concept to comment out whole sections of code.

Using false block to comment out a block code: (does not ignore bad syntax though)
if False:
# code in this block is ignored

=== doc strings ===

Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended)
"""
multi line comment
"""

=== Variable Type ===

print( type(myvar).__name__ )

Used to be able to just print the type(myvar), but this doesn't seem to work anymore :-(

== Variable Directory ===

All of the properties and functions of a variable or object

print( dir(myvar) )

=== Help ===

dir - list functions and properties of objects:
import math
dir(math) # module
dir(dir) # function
dir("") # object - "", [], {}, etc...
dir(__builtins__) # built in functions
dir() # list imported objects, and dunder methods

Note: dunder (double under) methods determine special functionality (eg. __add__ for +)

help - manual printout: (class doc string, function doc string, data, etc)
import math
help(math) # full module info
help(len) # full function info
help(["foo"]) # full object info

Pydoc: (command line, same as 'import math, help(math)'
$ pydoc math

Documentation String: (aka docstring, doc string)
print(math.pow.__doc__) # module doc string
print(len.__doc__) # function doc string

Object or function type:
type(x)

mylist = []
isinstance(mylist, list) # True
type(s) is list # True

Note: isinstance() and type() not equivalent when objects and subclass get involved

IDLE interactive help:
help()

=== Variables ===

Basic Data Types
str = "string" # string
num = 5 # integer
flt = 5.0 # float
nothing = None # None (not the same as undefined)

Undefined. Variables start off in an 'undefined' state. It is always better to initially set the variable to None to avoid this state. To check if undefined: [http://code.activestate.com/recipes/59892-testing-if-a-variable-is-defined/] [http://code.activestate.com/recipes/59892/]
<pre>
try:
x
except NameError:
x = None
</pre>

Rules: [http://www.python.org/dev/peps/pep-0008/ PEP8 - Method Names and Instance Variables]
* any length
* letters, numbers, underscore
* first character can't be number
* case sensitive
* cannot use keywords

Multiple assignment:
x = y = z = 0 # Zero x, y and z
x, y, z = 1, "two", 3.0 # x = 1, y = "two", z = 3.0

Swap variables: (done in parallel)
a, b = b, a

Unpack variables: (requires equal items on each side)
a, b, c = [1, 2, 3]

Other Data Types:
* tuple
* list
* dictionary
* set
* objects

Everything in Python is an object that has:
* identity: id(obj)
* type: type(obj)
* value: obj
** mutable - id() stays the same on modification (dictionary, lists, objects)
** immutable - id() changes on modification as new object is created (string, integer, tuple)

==== Check if Variable Exists ====

<pre>
try:
a # does a exist in the current namespace
except NameError:
a = 10 # nope
</pre>

'a' in vars() or 'a' in globals()
'a' in vars(__builtins__)

<pre>
if hasattr(a, 'property'):
doStuff(a.property)
else:
otherStuff()
</pre>

Ref: [https://stackoverflow.com/questions/1592565/determine-if-variable-is-defined-in-python] [https://stackoverflow.com/questions/610883/how-to-know-if-an-object-has-an-attribute-in-python]

=== Numbers ===

num = 5
print "num: %d" % num

>>> print "Today's stock price: %f" % 50.4625 1
50.462500
>>> print "Today's stock price: %.2f" % 50.4625 2
50.46
>>> print "Change since yesterday: %+.2f" % 1.5 3
+1.50

Arithmetic:
+ - * /
// # integer division
% # modulus
** # exponent

Compact Arithmetic:
i+=1 # Increment - Note, there is no 'i++'
i*=2 # Multiply
i%=2 # modulus
...

Math Functions:
import math
math.ceil(x)
math.floor(x)
math.trunc(x)
math.pow(x,y)

abs(x) # built in

String interpolation:
%% % character
%d integer
%x hex
%X hex (upper)
%f float
%s string

x = 1/81
print('value: %.2f' % x) # 0.01
print('value: %.5f' % x) # 0.012345
print('%d %d %d' % (1, 2, 3))

Type check:
if not isinstance(myvar, (int, float)): ...

=== Strings ===

STRINGS ARE IMMUTABLE! (to modify convert to a list)

Strings: (String Constants)
str = "Hello"
str = 'Hello'
str = """
Hello
World
""" # can also use '''...'''

String literals can span multiple lines: (no wrapping when \ used)
hello = "hello\
world"

Length:
len("str")

Raw string: (back slashes are not escaped)
myrawstr = r"hello\nworld\
how are you"
myrawstr = r"C:\" # ERROR - limitation of what won't be escaped

Concatenation:
str = 'hot' + 'dog'
print "Hello" "World"

Can't concat numbers
str = 5 + 'test' # error
num = 5
str = num + 'test' # error
str = str(5) + str(num) + 'test' # ok

Repeated Concatenation:
str = 10 * 'ha'
str = 'ha' * 10 # same
print '-'*60 # --------------------

String Indexing:
s = 'hello'
print( s[0] ) # is 'h'
print( s[-1] ) # is 'o'
print( s[len(s) - 1 ] ) # is also 'o'

Modify Strings: [http://stackoverflow.com/questions/1228299/change-one-character-in-a-string-in-python]
s[1] = 'x' # ERROR!
ERROR: can't modify string by index

# solution:
s = list("Hello World")
s[0] = 'C'
print( "".join(s) )

String Slicing:
s = 'hello'
print( s[0:3] ) # 'hel'
i = 1
print( s[i:i+1] ) # 'e'
print( s[:3] ) # 0:3 'hel'
print( s[4:] ) # 4:end 'o'
print( s[:] ) # start:end 'hello'
print( s[-2:] ) # -2:end 'lo'

For loop:
s = 'hello'
for c in s:
print c # c = 'h' first time

Escape characters:
\\ \' \" \n \r \t

Alternate loop:
s = 'hello'
for i in range( len(s) ):
print s[i]

Ord and Char (ASCII):
ord('a') # 97
chr(97) # 'a'

0-9 : 48-57
A-Z : 65-90
a-z : 97-122

List Functions: (dir on any string)
dir('')
dir(str)

Help:
help('')
help(str)

Padding:
"a".rjust(2, '0') # '0a'
"1".ljust(4, '0') # '1000'

String Test Functions:
"HELLO".isupper() # True
"hello".islower() # True
"hello".isalpha() # True
"12.22".isdigit() # False (because of '.')
"12.22".isnumeric() # False (because of '.')
"hel2".isalnum() # True
"e" in "hello" # True
"hello".startswith('he') # True
"hello".endswith('lo') # True

Note: To check for floats: [http://stackoverflow.com/questions/354038/how-do-i-check-if-a-string-is-a-number-in-python]
<pre>
def is_number(s):
try:
float(s)
return True
except ValueError:
return False
</pre>

String Modify Functions: (sort of... Strings are immutable)
"HELLO".lower() # 'hello'
"hello".upper() # 'HELLO'
" hello ".strip() # 'hello' remove leading/tailing whitespace
" hello ".lstrip() # 'hello ' remove left whitespace
" hello ".rstrip() # ' hello' remove right whitespace
"www.com".split('.') # ['www', 'com'] <- list
"www.com".partition('.') # ('www', '.', 'com') <- tuple
"".join( ('a','b','c') ) # "abc" join strings, list or tuple
"-".join( ('a','b','c') ) # "a-b-c" join with separator
"23".zfill(4) # "0023"

String Search Functions:
"hello".find('lo') # 3 (-1 on fail)
"hello".index('lo') # 3 (exception on fail)
"hello".rfind('lo') # 3 right to left (-1 on fail)
"hello".rindex('lo') # 3 right to left (exception on fail)

Formatting: (form of concatenation)
<pre>
mystr = "abc"
myint = 5
myflt = 3.141592
print( "%s %d %.3f" % (mystr, myint, myflt) )
print( "{0} {1} {2}".format(mystr, myint, myflt) ) # pep 3101 style - string's format function (python > 2.7)
print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name
print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding
</pre>

Split multi line string:
for line in str.split('\n'): print(line)
lines = str.splitlines() # returns list
lines = str.splitlines(True) # returns list (each includes new line characters)

Input a string:
name = input('What is your name: ').strip() # python 3
name = raw_input('What is your name: ').strip() # python 2

Leet Speak:
leet_table = ''.maketrans('EIOBT', '31087')
'BE COOL. SPEAK LEET!'.translate(leet_table)

String Functions:
newstr = mystr.strip() # remove white space on both sides
newstr = mystr.rstrip() # remove white space on right side
newstr = mystr.lstrip() # remove white space on left side
newstr = mystr.replace('\n', ' ') # replace string with another
newstr = mystr.replace('jello', 'hello') # replace string with another

==== String Formatting ====

PyFormat: Using % and .format() for great good! - https://pyformat.info/

===== Zero Padding =====

3 zero padding:
"{:03}".format(1) # 003

3 zero padding, specify position:
"{1:03}".format(1, 2) # 002

Old method:
"%03d" % 2 # 002

=== chr and ord ===

Ord and Char (ASCII):
ord('a') # 97
chr(97) # 'a'

=== Range ===

Creates a list of numbers. Remember: range always ends one less than you want!

Count:
ten = 10
one = 1
range(ten) # [0,1,...,9] starts with zero by default
range(one, ten + 1) # [1,...,10] actually get 1 to 10
range(ten, one - 1, -1) # [10,9,...,1] count down
range(2, 10 + 1, 2) # [2,4,...,10] even numbers

=== drange ===

Solution for decimal range() step value:
<pre>
def drange(start, stop, step):
r = start
while r < stop:
yield r
r += step

sub_ten = drange(0.0, 1.0, 0.1)
</pre>

References:
* floating point - Python decimal range() step value - Stack Overflow - http://stackoverflow.com/questions/477486/python-decimal-range-step-value

=== Casting ===

Convert:
float(3)
str(85)
int(8.6)
round(5.5)

Check Type:
type(5) # <class 'int'>
type(5.0) # <class 'float'>
type('5') # <class 'str'>
type(None) # <class 'NoneType'>
type(print) # <class 'builtin_function_or_method'>
type(()) # <class 'tuple'>
type([]) # <class 'list'>

mytype = int
if type(1) is mytype: ...

=== Boolean Logic ===

True and False:
bol = True
bol = False

Boolean Operators: (in order precedence)
a == b
a != b
not a
a and b
a or b

Note: Short-circuit evaluated - 'and' and 'or'

Return Boolean:
return myint >= 16

Get Boolean from string: [http://stackoverflow.com/questions/715417/converting-from-a-string-to-boolean-in-python]
# Note: no built in way to do this

myString = "false"
val = (myString == "true")

True if myString=="True" else False

s in ['true', '1', 't', 'y', 'yes', 'yeah', 'yup', 'certainly', 'uh-huh']

def str2bool(v):
return v.lower() in ("yes", "true", "t", "1")

=== None ===

None is Python's version of NULL or NIL.

None is a singleton (Python only has one copy of None in the interpreter).

if myvar is None:
...

=== If Conditionals ===

Operators:
==, !=, not, and, or, <, >, <>, <=, >=, in, not in, is, is not

If statement:
<pre>
if a == b:
...
elif c < age < d:
...
else:
...
</pre>

Shorthand conditional: (ternary operator)
val = True if a == b else False

is - checks for identical objects - same id()
# None is a singleton, so this works great for None
if myvar is None:
...

Other shorthand:
<pre>
def mycmp(x, y):
if x > y: return True
else: return False
</pre>

Switch/Case statement:
* there is no built in switch/case statement, use "if ... elif ... elif ..." sequence instead

=== For Loop ===

''for'' is used for iterating over an iterable sequence (eg. lists, dictionaries, tuples). (for each)

# range(start, end + 1, skip)

for i in range(10):
print(i) # 0 .. 9

for i in range(5, 10):
print(i) # 5 .. 9

for i in range(10, 0, -1):
print(i) # 10 .. 1

for i in [1, 2, 3]: # iterate over list
print(i) # 1 2 3

Discard variable: [http://bytes.com/topic/python/answers/757444-loop-without-variable]
for _ in range(10): # repeat and discard variable (good for pep8 check)
print "hello world"

for key in my_dict.keys(): ...
for value in my_dict.values(): ...
for key, value in my_dict.items(): ...

Compact version:
for i in range(10): print(i) # 0 .. 9

Access index: [http://stackoverflow.com/questions/522563/accessing-the-index-in-python-for-loops]
# using enumeration
for idx, val in enumerate(mylist):
print idx, val

# using range
for ix in range(len(mylist)):
print mylist[ix]

# list comprehension
[ (ix, mylist[ix]) for ix in range(len(mylist))]

=== While Loop ===

i = 0
while i < 10:
i = i + 1

while True:
... # infinite loop

=== Breaking Loops ===

break statement:
while True:
if s == 'done':
break # jump out of the loop
...

continue statement: (works with while too)
for i in range(10):
if i == 2:
continue # skip 2
...

=== Functions ===

No return value: (None)
x = print('')
# x is 'None'

Note: specifying no return value is the same as this:
return None

No Operation - 'pass':
def noop():
pass

Function:
<pre>
def area(radius):
""" This is a doc string """ # can be " ... " also...
import math
return math.pi * radius ** 2
</pre>

Function Doc string:
print(area.__doc__)
foo.func_doc

Function name:
foo.func_name

Local variable:
name = "old"
def cname():
name = "new"
print(name) # is still 'old'

Global variable:
name = "old"
def cname():
global name
name = "new"
cname()
print(name) # is now 'new'

By convention starting point is: (not required)
def main():

Parameters: (with default)
def foo(x, y, z = "default"):

Note: don't use mutable for defaults, as the object is remembered across runs! (unless that is what you want)
<pre>
# each call to t() will increase list
def t(foo=[]):
foo.append('1')
print(foo)
</pre>

<pre>
# fix mutable issue
def named_param(a, foo=None):
foo = foo or []
if not foo:
foo.append(a)
</pre>

Parameters by keyword:
def foo(x = '1', y = '2', z = '3'):
...
foo(y = 'a', x = 'b') # order is not important with keywords

Trick for optional parameter:
def do(x = None):
if x == None:

Function tests - functions are callable
callable(myfunc)

=== Type Hints ===

: "Note The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers, IDEs, linters, etc."

def surface_area_of_cube(edge_length: float) -> str:
return f"The surface area of the cube is {6 * edge_length ** 2}."

Without the hints:
def surface_area_of_cube(edge_length):
...

ref: https://docs.python.org/3/library/typing.html

=== Pass ===

Pass is a null operation (noop):

Do nothing, noop, take no action:
pass

Great for creating minimal classes:
class MyEmptyClass:
pass

Greate for making stub functions/methods:
def initlog(*args):
pass # remember to implement this!

=== Slice ===

Take take a slice of strings, lists, tuples, etc...
a = [0,1,2,3,4,5]
a[0] # 0 - not a slice
a[-1] # 5 - not a slice
a[0:1] # [0]
a[1:3] # [1, 2]
a[:2] # [0, 1]
a[:-1] # [0, 1, 2, 3, 4] - all but last
a[2:] # [2, 3, 4, 5]
a[::2] # [0, 2, 4] - stride
a[::-1] # [5, 4, 3, 2, 1, 0] - reverse
range(0,10)[::2] # [0, 2, 4, 6, 8] - stride

a = '012345'
a[1:3] # '12'
a[::2] # '024'

=== [http://docs.python.org/reference/compound_stmts.html Compound Statements] ===

Compound statements: (generally discouraged)
fun1(); fun2(); fun3()

Compound with statements: [http://docs.python.org/reference/compound_stmts.html#the-with-statement]
with A() as a, B() as b: ...

#is equivalent to
with A() as a:
with B() as b:
...

[http://www.python.org/dev/peps/pep-0343 PEP 0343] - The “with” statement

f = open('file')
with f: ...

Try: (quick ignore all exceptions, very dangerous)
try: my_broken_function()
except: pass

=== Passing Command Line Arguments ===

Parameters are stored in the 'sys.argv' variable:
import sys
print( sys.argv[0] ) # name of script
print( sys.argv[1] ) # first argument
for arg in sys.argv: print( arg ) # list all arguments

Print script name:
os.path.basename(sys.argv[0])

Print path to script:
print sys.path[0]

Paths: [http://code.activestate.com/recipes/474083-get-the-path-of-the-currently-executing-python-scr/]
<pre>
import os,sys
print "CWD: ",os.getcwd()
print "Script: ",sys.argv[0]
print ".EXE: ",os.path.dirname(sys.executable)
print "Script dir: ", os.path.realpath(os.path.dirname(sys.argv[0]))
pathname, scriptname = os.path.split(sys.argv[0])
print "Relative script dir: ",pathname
print "Script dir: ", os.path.abspath(pathname)
</pre>

os.path.realpath(__file__)

print 'sys.argv[0] =', sys.argv[0]
pathname = os.path.dirname(sys.argv[0])
print 'path =', pathname
print 'full path =', os.path.abspath(pathname)

=== Environment Variables ===

import os
a = os.environ.get('MYVAR') # get environment var
a = os.environ.get('MYVAR','default') # get environment var
a = os.environ['MYVAR'] # get environment var
os.environ['MYVAR'] = 'something' # set environment var
os.environ.update({'MYVAR':'something'}) # set environment var

for env in os.environ:
print "%s = %s" % (env, os.environ[env])

with open("environment.txt", "w") as f:
for env in os.environ:
f.write("%s = %s\n" % (env, os.environ[env]))

=== PYTHONPATH ===

Environment Search path:
PYTHONPATH=...

From Code:
sys.path.append('...')

Get site-package directory from shell: [http://stackoverflow.com/questions/122327/how-do-i-find-the-location-of-my-python-site-packages-directory]
python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

=== Import and Modules ===

Import and Modules - http://docs.python.org/tutorial/modules.html

Import module:
import package # standard import
import package.code # nested name space
from package import code # import specifics from module
import module # standard import
from math import sin # import specifics from module
from module import myfunc as fun1 # alias function/object
import math as other_math # alias module
import longname as ln # alias module
from module import * # BAD BAD BAD!

import math
math.pow(x,y)

Group and organize by:
* stdlib libraries
* local libraries
* 3rd party libraries

Remove import:
del math

from math import *
pow(x,y)

from math import pow
pow(x,y)

import math as math_lib # rename
math_lib.pow(x,y)

List functions in module:
import math
dir(math)

List all built in functions:
dir(__builtins__)

Note: Modules have the extension .py

Simply module Example:

# mymod.py:
def hello():
""" this is a doc string """
print "Hello World"

# Use the module:
import mymod
mymod.hello():

# Use everything in the the module, except for names beginning with '_': (bad practice!)
from mymod import *
hello()

# Use specific functions in the module:
from mymod import hello, hi
hello()

# Function assignment:
h = mymod.hello
h()

# module name
mymod.__name__ # mymod
__name__ # used inside of module will print current module or "__main__" if directly called

Note: careful when building own modules, as the import literally imports and executes the code, so any code not in functions is executed!

Check if being called directly or indirectly:
if __name__ == "__main__":
# execute tests...

Standard Script Setup:
<pre>
def main(args):
# do logic
if __name__ == "__main__":
sys.exit(main(sys.argv))
</pre>

Easter egg:
# The Zen of Python
import this

Search path:
* environment variable 'PYTHONPATH' (on Unix, this is usually .:/usr/local/lib/python)
* sys.path (list of paths, initialized to PYTHONPATH)
** sys.path.append('path')

Module folder: [http://stackoverflow.com/questions/391879/organising-my-python-project] [http://stackoverflow.com/questions/279237/python-import-a-module-from-a-folder] [http://stackoverflow.com/questions/72852/how-to-do-relative-imports-in-python]
/kenlib/kenmod.py
# so you can do "import kenlib.kenmod"
# create file '__init__.py' in kenlib/

Check if package exists:
<pre>
try:
import argparse
except ImportError:
print >> sys.stderr, "Please install argparse from http://code.google.com/p/
argparse."
exit(1)
</pre>

=== ifmain ===

if __name__ == '__main__':
sys.exit(main(sys.argv))

if __name__ == '__main__':
sys.exit(main(sys.argv[1:]) or 0)

=== Lists ===

* Lists can be modified (mutable)
* Tuples are constant (immutable)

List: (mutable or modifiable)
x = [] # empty list
x = [5] # singleton list

s = list("Hello World")
print(s) # s = ['H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd']
s[0] = 'X'

s.append(x) # append x
s.insert(i, x) # insert x at i
del(s[i]) # delete item at i
s.sort() # sort (modify in place!)
s.reverse() # reverse
s.index(x) # index of x in s
new = old[:] # create copy of list
new = list(old) # create copy of list

Check if item in list:
if x in s: ...

Filter list of strings based on contents: [https://stackoverflow.com/questions/2152898/filtering-a-list-of-strings-based-on-contents]
res = [k for k in lst if 'ab' in k]

List comprehension
[n * n for n in range(1, 11)] # squares of numbers 1 to 10
[c for c in 'pizza'] # list of string 'pizza'
[c.upper() for c in 'pizza'] # list of string 'pizza' uppercase
[n for n in nums if n > 0] # filters out only positive numbers in nums list
[fname for fname in os.listdir(path) # on multiple lines
if os.path.isfile(fname)
if fname.endswith('.py')]

Enumeration
for index, value in enumerate(mylist): print index, value

Length:
len(x)

Concatenation:
x + s
x * 2

Convert list to string:
"".join(mylist) # no separator
" ".join(mylist) # space separator
",".join(mylist) # comma separator

Test Membership
x in s

Multi dimensional list:
mylist = [ [0,"hi"], [1,"bye"] ]
print( mylist[0][1] )

Sorting: (see lambda)
data.sort() # alpha numerica sort
data.reverse() # alpha numerica sort

<pre>
# custom sort
data = [1, 5, 3, 9]
def mylistcmp(x, y):
if x == y: return 0
if x < y:
return -1
else: # swap 1 and -1 here to reverse
return 1

data.sort(cmp=mylistcmp)
</pre>

Slices:
lst = ['1a', '2b', '3c', '4d', '5e', '6f']
lst[0] # '1a'
lst[-1] # '6f' last item
lst[0:2] # ['1a', '2b']
lst[0:2] = ['11', '22'] # lst = ['11', '22', '3c', '4d', '5e', '6f'] - assignment to slice
lst[0:2] = [] # remove items
lst[0:0] = ['1a', '2b'] # insert items (at beginning)
lst[-1:-1] = ['7g', '8h'] # insert items (at item BEFORE last), not really what you want, use append()
lst[-1:] = ['7g', '8h'] # replace last item
lst[:] = [] # clear entire list
del(lst[0:2]) # delete slice items

Note: it is not safe to modify a sequence while being iterated over. Create a copy first:
for x in mylst[:]: # make a slice copy of the entire list
if len(x) > 6: a.insert(0,x)

Make list from tuple:
list( range(10) )

Merge multiple lists: [http://www.testingreflections.com/node/view/4930]
<pre>
def merge(seq):
merged = []
for s in seq:
for x in s:
merged.append(x)
return merged

# sample usage:
foo = [['a', 'b'],['c'],['d', 'e', 'f']]
print merge(foo)
>>>['a', 'b', 'c', 'd', 'e', 'f']
</pre>

=== Tuple ===

Tuple: (immutable or constant)
x = () # empty tuple
x = (5,) # singleton tuple
x = (5) # NOTE: integer with parenthesis, not tuple

s = tuple("Hello World")
print(s) # s = ('H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd')
# ERROR: s[0] = 'X'

Make list from tuple:
list( range(10) )

Use in conditionals:
if answer in ('y', 'ye', 'yes'): ...

Convert tuple string to tuple: [http://stackoverflow.com/questions/3945856/converting-string-to-tuple-and-adding-to-tuple]
x = "(1,2,3)"
t = tuple(int(v) for v in re.findall("[0-9]+", x))

Another Convert tuple string to tuple: [http://stackoverflow.com/questions/3945856/converting-string-to-tuple-and-adding-to-tuple]
import ast
ast.literal_eval("(1,2,3,4)") # (1,2,3,4)

=== Named Tuple ===

What are "named tuples" in Python? - Stack Overflow - http://stackoverflow.com/questions/2970608/what-are-named-tuples-in-python

<pre>
from collections import namedtuple
Point = namedtuple('Point', 'x y')
pt1 = Point(1.0, 5.0)
pt2 = Point(2.5, 1.5)
print p1 # point(x=1, y=5)
print p1.x # 1.0
print p1[0] # 1.0
</pre>

=== Dictionary ===

NOTE: Dictionaries are also known as associative arrays, maps or hash tables.

Dictionary: (mutable or modifiable)

color = {} # empty dictionary
color = {'red' : 1, 'blue', 2}
print( color['red'] ) # 1
print( color.get('red') ) # 1
print( color.get('red', 'default') ) # red if found, or return 'default'
color['green'] = 3 # able to add new item on the fly!
'red' in color # True - check if index exists
del(color["red"]) # delete item
len(color) # 2 - count of items

List in dictionary
<pre>
d.setdefault(key,[]).append(member) # tricky!

# more tricky: set default type as list
from from collections import defaultdict
d = defaultdict(list)
d[key].append(member)
</pre>

Enumeration:
for index, key in enumerate(mylist): print index, key

functions:
d.clear()
d.copy()
d.get(key)
d.get(key, default_val)
d.update(e) # update with (key, value) pairs in e
d.setdefault(key, value) # only update if not exist, and return set value

Views: (adjust on the fly as dictionary changes)
d.items() (key, values)
for k, v in d.items(): print(k, v)

d.keys()
for k in d.keys(): print(k)

d.values()
for v in d.values(): print(v)

Sorting: (see also lambda)
<pre>
# custom sort of dictionaries in list
data = [dict(number=x) for x in '036149'] # create list of dictionaries
def mydictcmp(x, y):
if x['number'] == y['number']: return 0
if x['number'] < y['number']:
return -1
else: # swap 1 and -1 here to reverse
return 1

# data.sort(cmp=mydictcmp) # modifies data

new_data = sorted(data, cmp=mydictcmp) # better solution
</pre>

Sorting:
<pre>
my_dict = {'a':1, 'b':4, 'c':3, 'd':9}
my_dict_keys = my_dict.keys()

def my_dict_sort(x, y):
if my_dict[x] == my_dict[y]: return 0
if my_dict[x] < my_dict[y]: return -1
if my_dict[x] > my_dict[y]: return 1

#sorted_my_dict = sorted(my_dict_keys, cmp=my_dict_sort, reverse=True)
sorted_tuple = sorted(my_dict_keys, cmp=my_dict_sort)
print "Sorted Dictionary:"
for key in sorted_tuple:
print "%s: %d" % (key, my_dict[key])

# a: 1, c: 3, b: 4, d: 9,
# reverse: d: 9, b: 4, c: 3, a: 1,
</pre>

Sort and keys:
keys = sorted(mydict.keys())

Sort by value: [http://stackoverflow.com/questions/613183/python-sort-a-dictionary-by-value]
import operator
x = {1: 2, 3: 4, 4:3, 2:1, 0:0}
sorted_x = sorted(x.iteritems(), key=operator.itemgetter(1))

Sort options:
# LIST.sort(cmp=None, key=None, reverse=False) -- stable sort *IN PLACE*;
# sorted(iterable, cmp=None, key=None, reverse=False)

Another: (works well!)
sorted(my_dict.items(), key=lambda x: x[1])

See: http://wiki.python.org/moin/HowTo/Sorting/

=== Sets ===

Sets are special lists with unique values (good for removing duplicates)
lst = [1, 1, 6, 8, 1, 5]
s = set(lst)
print(s) # set([8, 1, 5, 6])
s.add(item)
item in s # true
s.remove(item)

Union (return combination of all):
s.union(t)

Intersection (return matching):
s.intersection(t)

Symmetric Difference (return only items not matching either, inverse of intersection):
s.symmetric_difference(t)
list( set(mylist).symmetric_difference(set(mylist2)) )

<pre>
Operation Equivalent Result
len(s) cardinality of set s
x in s test x for membership in s
x not in s test x for non-membership in s
s.issubset(t) s <= t test whether every element in s is in t
s.issuperset(t) s >= t test whether every element in t is in s
s.union(t) s | t new set with elements from both s and t
s.intersection(t) s & t new set with elements common to s and t
s.difference(t) s - t new set with elements in s but not in t
s.symmetric_difference(t) s ^ t new set with elements in either s or t but not both
s.copy() new set with a shallow copy of s
</pre>

References:
* sets — Unordered collections of unique elements - http://docs.python.org/library/sets.html

=== Files ===

Open file, get file descriptor:
fd = open('/dev/null')

File Modes:
'r' read (default)
'w' write (truncate and create if needed)
'a' append (create if needed)

'b' binary
't' text (default)

'+' read and write

Common Functions: (fd)

f.read() # read whole file to string
f.read(n) # read n bytes from file to string
f.readline() # read single line as string (while loop)
f.readlines() # read all lines to list

f.write(str) # write string to file (does not auto include new line)
f.writelines(list) # write list to file (does not auto include new lines)

f.seek(0) # seek to start of file
f.seek(pos) # seek to position
f.truncate() # truncate file

f.close() # close file (fd)
f.closed # True/False flag that reports if file is closed

Help:
help(open('/dev/null'))
pydoc file

Read Text File Line by Line
f = open(fname, 'r') # ('r' optional)
for line in f: # or f.readlines()
print(line)
f.close() # optional

Write list to file (with new lines):
f.writelines('\n'.join(mylist))

One liner read whole file:
print( open( fname, 'r' ).read() )

Write text file:
# f = open(fname, 'a') # append text file
f = open(fname, 'w')
f.write("Hello\n")
f.write( str(10) ) # convert numbers to strings first
f.writelines(["line one\n", "line two\n"])
f.close

Read binary file:
def is_gif(fname):
f = open(fname, 'br') # not sure 'b' is correct
first4 = tuple(f.read(4))
return first4 == (0x47, 0x49, 0x46, 0x38)

Read all characters of file:
c = f.read(1)
while c:
# if c == ... logic
c = f.read(1)

Reading file in while loop: [http://ubuntuforums.org/showthread.php?t=1092300]
<pre>
f=open("file")
while True:
line=f.readline()
if not line: break
print line
f.close()
</pre>

with - implicit close - [http://www.python.org/dev/peps/pep-0343/ PEP 343 -- The "with" Statement]: [http://docs.python.org/reference/compound_stmts.html#the-with-statement] (python >= 2.5)
with open("/etc/passwd") as f:
# ..., f.read(), etc

With statement: (promptly close open handle at end of block)
with open(fname, 'r') as f:
for ...

<pre>
# eqivalent to:
f = open("/etc/passwd")
f.__enter__()
try:
...
finally:
f.__exit__()
</pre>

cat: (for grabbing a configuration line)
<pre>
def cat(filename):
with open(filename) as f:
return f.readline().strip()
</pre>

=== With ===

PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/

'with' structure:
<pre>
with EXPR as VAR:
BLOCK
</pre>

translates to:
<pre>
VAR = EXPR
VAR.__enter__()
try:
BLOCK
finally:
VAR.__exit__()
</pre>

Note: The "as VAR" part is optional. Good for locking:
<pre>
with locking(myLock):
BLOCK
</pre>

with - implicit close - [http://www.python.org/dev/peps/pep-0343/ PEP 343 -- The "with" Statement]: [http://docs.python.org/reference/compound_stmts.html#the-with-statement] (python >= 2.5)
with open("/etc/passwd") as f:
...

With statement: (promptly close open handle at end of block)
with open(fname, 'r') as f:
for ...

See also [[#Files]]

References:
* PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/
* PEP 340 -- Anonymous Block Statements - http://www.python.org/dev/peps/pep-0340/

=== Try Exception Handling ===

Exception Handling - http://docs.python.org/release/2.6.6/tutorial/errors.html

throw execption:
raise [EXCEPTION]
raise IOError("This is a test!")

Catch exceptions:
try:
except ( [EXCEPTION], [EXCEPTION] ):
except [EXCEPTION]:
except [EXCEPTION] as err: # use "[EXCEPTION], err" in python < 2.6
print("Error: {}".format(err))
except:
raise # reraise caught error that is not handled
else: # if no exception caught
finally: # always executed

Note: Usually a good idea to re-raise exception if you don't handle it (just 'raise')

General Exception: (all exceptions inherit from Exception)
try:
raise Exception('spam', 'eggs')
except Exception as e:
pass

Custom Exception Class:
<pre>
class MyError(Exception):
"""My Error Exception"""

class MyException(Exception):
pass

rasie MyError("My Error")
</pre>

<pre>
class ValidationError(Exception):
def __init__(self, message, Errors):

# Call the base class constructor with the parameters it needs
Exception.__init__(self, message)

# Now for your custom code...
self.Errors = Errors
def __str__(self):
return "MyErrors - %s" % self.Errors
</pre>

Rename standard exception:
<pre>
class Failure(StandardError):
"""Exception for unexpected failures."""
</pre>

Note: a good place to hide the exception classes in the __init__.py file.

==== exception line number ====

Python When I catch an exception, how do I get the type, file, and line number? - Stack Overflow [http://stackoverflow.com/questions/1278705/python-when-i-catch-an-exception-how-do-i-get-the-type-file-and-line-number]
<pre>
import sys, os

try:
raise NotImplementedError("No error")
except Exception as e:
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print(exc_type, fname, exc_tb.tb_lineno)
</pre>

==== traceback ====

<pre>
import traceback
try:
...
except Exception as E:
...
print 'Traceback: %s', traceback.format_exc()
</pre>

Catching multiple exceptions and getting properties from the exception:
<pre>
import traceback
import sys

class MyError(Exception):
"""My Error Exception"""

try:
#raise ValueError("a value error") # <type 'exceptions.ValueError'>
raise MyError("a my error") # <class '__main__.MyError'>
raise Exception("won't be caught") # <type 'exceptions.Exception'>
except (ValueError, MyError) as e:
print type(e) # <class '__main__.MyError'>
print e # __str__() = a my error
msg = str(e) # msg = 'a my error'
print e.__doc__ # My Error Exception
print e.__str__() # a my error
print e.__repr__() # MyError('a my error',)
print e.args # ('a my error',)
x = e # __getitem__() allows args to be unpacked directly
# x, y = e # if two args were passed, use len(e.args) to see count

# manually print stack trace (import traceback, sys)
print "Exception in user code:"
print '-'*60
traceback.print_exc(file=sys.stdout)
# stack_trace = traceback.format_exc()
print '-'*60
</pre>

=== Object Oriented Programming ===

Terms:
* encapsulates
* inheritance
* polymorphism
* method overriding

==== class ====

class animal.py:
<pre>
class Animal(object): # default "(object)" not required
" Animal class "
def __init__(self, name):
self.name = name # name is private variable
self.__iam = "animal" # __iam is private variable
def __str__(self): # string representation
return "name = '%s'" % (self.name)
def __repr__(self): # object representation (just typing instance in IDLE, or repr())
return 'Animal(%s)' % str(self) # returns __str__
def speak(self, msg='Nothing to say'):
print(msg)
</pre>

Note: if no __str__ defined, but __repr__ is, then: __str__ = __repr__

Usage:
<pre>
from animal import Animal
a = Animal('good')
print(a.__doc__) # Animal class
print(a) # name = 'good'
a # Animal(name = 'good')
print(a.name) # good
a.speak('hi') # hi
print(a._Animal__iam) # animal - access to private variable (shouldn't do)
</pre>

subclass dog.py:
<pre>
from animal import Animal
class Dog(Animal):
" Dog class "
def __init__(self, name):
self.name = name # name is private variable
self.__iam = "dog" # __iam is private variable
# NOTICE: no __str__, inherited from super class
def __repr__(self): # override to reflect "dog"
return 'Dog(%s)' % str(self) # returns __str__
def speak(self, msg='Nothing to say'): # override method
super(Dog, self).speak('Dog says: ' + msg) # call super class method
</pre>

Subclass usage:
<pre>
from dog import Dog
d = Dog('bad')
print(d.__doc__) # Dog class
print(d) # name = 'bad'
d # Dog(name = 'bad')
print(d.name) # bad
d.speak('hi') # Dog says: hi
print(d._Dog__iam) # dog - access to private variable (shouldn't do)
</pre>

Super: [http://stackoverflow.com/questions/576169/understanding-python-super-with-init-methods]
# python 3.0
super().__init__()

# python 2.7
super(self.__class__, self).__init__()
super([CLASSNAME], self).__init__()

<pre>
class Rectangle(Polygon):
def __init__(self, id, width, height):
# super(Rectangle, self).__init__(id)
super(self.__class__, self).__init__(id)
self.shape = (width, height)
</pre>

Get class name:
self.__class__.__name__
obj.__class__.__name__

Full name:
obj.__module__ + "." + obj.__class__.__name__

Private variables: (uses name mangling)
self.__age
# direct access: _classname__varname (eg. p._Person__age = 44)

Tests:
isinstance(myobj, myclass) # check if myobj is a class or subclass of myclass
issubclass(A, B) # check if A is subclass of B
issubclass(Exception, BaseException) # True

Multiple Inheritance: (methods searched in order)
class DerviedClass(Base1, Base2, Base3):
pass

Standard startup script:
class MyClass():
...
@classmethod
def main(cls, argv=None):
...
# return cls() # creator
if __name__ == "__main__";
sys.exit(MyClass.main(sys.argv))

Raise an error if a method is not overridden (not implemented)
def _get_dist(self):
"""Find linux disto. Implemented in subclasses."""
raise NotImplementedError

==== Multiple Constructors ====

<pre>
class Animal(object):

def __init__(self, name):
self.name = name

@classmethod
def create_fido(cls):
return cls("fido")
</pre>

==== Descriptor Classes ====

"In general, a descriptor is an object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol: __get__(), __set__(), and __delete__(). If any of those methods are defined for an object, it is said to be a descriptor." [http://docs.python.org/2/reference/datamodel.html#implementing-descriptors]

<pre>
object.__get__(self, instance, owner)

Called to get the attribute of the owner class (class attribute access) or of an instance of that
class (instance attribute access). owner is always the owner class, while instance is the instance
that the attribute was accessed through, or None when the attribute is accessed through the owner.
This method should return the (computed) attribute value or raise an AttributeError exception.

object.__set__(self, instance, value)

Called to set the attribute on an instance instance of the owner class to a new value, value.
</pre>

Example: [http://stackoverflow.com/questions/3798835/understanding-get-and-set-and-python-descriptors]
<pre>
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)

class Temperature(object):
celsius = Celsius()

t = Temperature()
t.celsius = 5 # calls Celsius.__set__
print t.celsius # calls Celsius.__get__
</pre>

Note: Also appears in 'pydoc' under "Data descriptors defined here" with object's docstring. Trick: the __doc__ can be overridden in the __init__ method, which makes for dynamic docstrings!

References:
* Descriptor HowTo Guide — Python v2.7.5 documentation - http://docs.python.org/2/howto/descriptor.html
* 3.4.2.2. Implementing Descriptors¶ - 3. Data model — Python v2.7.5 documentation - http://docs.python.org/2/reference/datamodel.html#implementing-descriptors

=== [http://docs.python.org/reference/simple_stmts.html#the-exec-statement exec code] ===

DANGEROUS!

Execute code in string:
exec( 'print("hi")' )

Example of executing a startup script for IDLE:
if os.path.isfile('.pythonrc.py'): exec(open('.pythonrc.py').read())

import os
filename = os.environ.get('PYTHONSTARTUP')
if filename and os.path.isfile(filename):
exec(open(filename).read())

=== eval ===

VERY DANGEROUS

Will execute a string as though it were pure python code!

eval("print 'hello'")

=== Common Header Format ===

<pre>
__author__ = "software team"
__date__ ="$Feb 15, 2011 1:42:17 PM$"
__version__ = "1.1.0.441 bandelier@66d9fe1edf3e"
</pre>

<pre>
Next should be authorship information. This information should follow this format:

__author__ = "Rob Knight, Gavin Huttley, and Peter Maxwell"
__copyright__ = "Copyright 2007, The Cogent Project"
__credits__ = ["Rob Knight", "Peter Maxwell", "Gavin Huttley",
"Matthew Wakefield"]
__license__ = "GPL"
__version__ = "1.0.1"
__maintainer__ = "Rob Knight"
__email__ = "rob@spot.colorado.edu"
__status__ = "Production"
</pre>
Source: Python Coding Guidelines - http://bayes.colorado.edu/PythonGuidelines.html

These will show up in the help() like such:
<pre>
Help on module test:

NAME
test

FILE
/home/kenneth/test.py

FUNCTIONS
test()

DATA
__author__ = 'test author'
__version__ = '1.0'

VERSION
1.0

AUTHOR
test author
</pre>

References:
* Python: What is the common header format? - Stack Overflow - http://stackoverflow.com/questions/1523427/python-what-is-the-common-header-format

=== Common Environment Variables ===

Command line and environment — Python v2.7.2 documentation - http://docs.python.org/using/cmdline.htm

PYTHONUNBUFFERED
: If this is set to a non-empty string it is equivalent to specifying the -u option. [http://docs.python.org/using/cmdline.html#envvar-PYTHONUNBUFFERED]
: -u - Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
* See also stdout - Python output buffering - Stack Overflow - http://stackoverflow.com/questions/107705/python-output-buffering
export PYTHONUNBUFFERED=1
set PYTHONUNBUFFERED=1

PYTHONVERBOSE
:If this is set to a non-empty string it is equivalent to specifying the -v option. If set to an integer, it is equivalent to specifying -v multiple times.
: -v - Print a message each time a module is initialized, showing the place (filename or built-in module) from which it is loaded.
export PYTHONVERBOSE=1
set PYTHONVERBOSE=1

PYTHONPATH
: Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored. [http://docs.python.org/using/cmdline.html#envvar-PYTHONPATH]
export PYTHONPATH=/path1/:path2/
set PYTHONPATH=C:\path1\;C:\path2\

=== Queue ===

Good for threading.

"The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics." [http://docs.python.org/2/library/queue.html]

<pre>
import Queue

bucket = Queue.Queue()
bucket.put("something")

while True:
try:
item = bucket.get(block=False)
except Queue.Empty:
pass
else:
# do stuff with queue
</pre>

Important methods:
Queue.qsize()
Queue.empty()
Queue.put(item[, block[, timeout]])
Queue.get([block[, timeout]])

-

Example: http://docs.python.org/2/library/queue.html
<pre>
def worker():
while True:
item = q.get()
do_work(item)
q.task_done()

q = Queue()
for i in range(num_worker_threads):
t = Thread(target=worker)
t.daemon = True
t.start()

for item in source():
q.put(item)

q.join() # block until all tasks are done
</pre>

-

"The module implements three types of queue, which differ only in the order in which the entries are retrieved. In a FIFO queue, the first tasks added are the first retrieved. In a LIFO queue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the heapq module) and the lowest valued entry is retrieved first."

Queue Types:
class Queue.Queue(maxsize=0) # FIFO Queue
class Queue.LifoQueue(maxsize=0) # LIFO Queue
class Queue.PriorityQueue(maxsize=0) # Priority Queue

References:
* 8.10. Queue — A synchronized queue class — Python v2.7.5 documentation - http://docs.python.org/2/library/queue.html

== Recipes ==

=== Remove Numbers from String ===

Using list comprehension: [http://stackoverflow.com/questions/12851791/removing-numbers-from-string]
<pre>
result = ''.join([i for i in s if not i.isdigit()])
</pre>

Using pattern matching:
<pre>
>>> re.compile(r'\D+').findall("test 1234 test 1234")
['test ', ' test ']

>>> re.compile(r"\d").sub("", "test 1234 test 1234")
'test test '
</pre>

=== Retry Wrapper ===

<pre>
import functool
class SomethingRetry:

def retry_with_backoff(retries: int, base_delay: float, exceptions: tuple[type[Exception], ...] = (Exception,)) -> callable:
"""
Retries a function with exponential backoff, doubling on each attempt.
Arguments:
retries: Number of retry attempts.
base_delay: Initial delay between retries in seconds.
exceptions: Tuple of exceptions to catch and retry.
"""

def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(1, retries + 1):
try:
return func(*args, **kwargs)
except exceptions as e:
print(f"Warning: Attempt {attempt} failed with error: {e}")
if attempt == retries:
print("Error: All retry attempts failed.")
raise
sleep_time = base_delay * (2 ** (attempt - 1))
print(f"Notice: Retrying in {sleep_time} seconds...")
time.sleep(sleep_time)
return wrapper
return decorator

@retry_with_backoff(retries=6, base_delay=10, exceptions=(Exception,))
def do_something(self, sample_param: str):
</pre>

== Standard Libraries ==

=== user input ===

<pre>
line = raw_input() # Python 2
# line = input() # Python 2
line = input('What is your name: ')
print(": " + line.strip() + " :")
</pre>

=== sys - stdout stderr stdin ===

STDOUT:
import sys
sys.stdout.write("Hello ") # does not write new line
sys.stdout.write("World\n")

STDERR:
import sys
sys.stderr.write("ERROR!\n")

STDIN:
import sys
line = sys.stdin.readline()

line = input() # use raw_input() in Python 2
line = input('What is your name: ')
print(": " + line.strip() + " :")

Read piped stdin line by line: [http://stackoverflow.com/questions/1450393/how-do-you-read-from-stdin-in-python]
# cat /etc/passwd | ./readlines.py # linux
# type file.txt | python readlines.py # windows

# fileinput.input() - returns fileinput object which is iterative
import fileinput
for line in fileinput.input():
print(": " + line + " :")

# sys.stdin.readlines() - returns list of strings
import sys
for line in sys.stdin.readlines():
print(": " + line + " :")

# sys.stdin - returns object which is iterative
import sys
for line in sys.stdin:
print(": " + line + " :")

# ugly while loop: [http://ubuntuforums.org/showthread.php?t=1092300]
while True:
try:
s = raw_input("Say something ")
except EOFError:
break
print ": %s" % s

Check if stdin has any waiting data: [http://stackoverflow.com/questions/3762881/how-do-i-check-if-stdin-has-some-data]
<pre>
import sys
import select

# if select.select([sys.stdin,],[],[],0.0)[0]: # alternative old method

if not sys.stdin.isatty():
print "Have data!"
print sys.stdin.readlines()
else:
print "No data"
</pre>

<pre>
if len(sys.argv) > 1:
message = ' '.join(sys.argv[1:])
elif not sys.stdin.isatty():
# read from pipe in
message = sys.stdin.readline()
else:
# read from user
sys.stdout.write("Message: ")
message = raw_input().strip()
</pre>

=== [http://docs.python.org/library/re.html Regular Expressions] ===

Import:
import re # user regular expressions

Match vs Search:
re.match("cde", "abcdef") # no match (match searches beginning of string, first match)
re.match("abc", "abcdef") # match (match searches beginning of string, first match)
re.search("cd", "abcdef") # match (matches any location, first match)

Both re.match() and re.search() return objects that are not intuitive to use. You can check the object's group() method for what matched, but I prefer re.findall().

Match:
s = 'done'
if re.match('done|quit', s) != None # True

Search:
s = 'done'
if re.search('do', s) != None # True

Replace:
msg = 'jello world'
msg = re.sub('jello', 'hello', msg)

For case intensive searches append 3rd parameter "re.I".

Find all:
re.findall("[a-z]+", "aaa111bbb222") # ['aaa', 'bbb']
re.findall("111([a-z]+)", "aaa111bbb222") # ['bbb']
re.findall("hello", "HELLO", re.I) # ['HELLO'] - Case Insensitive

Split:
>>> re.split('[a-z]+','111aaa222bbb')
['111', '222', '']

>>> re.split('([a-z]+)','111aaa222bbb') # parenthesis cause return of all
['111', 'aaa', '222', 'bbb', '']

Replacement Substitution:
# re.sub(pattern, repl, string[, count, flags])
re.sub(r'\\', r'/', r'\\data\files')

Compiled regular express for repeated use: (performance)
valid = re.compile(r"[0-9]+")
valid.match("aaa111bbb") # not valid
match = valid.search("aaa111bbb222") # valid
print(match.group()) # what matched: '111'

=== Date and Time ===

Sleep in seconds:
import time
time.sleep(seconds)

Use the following functions to convert between time representations:
<table border="1">
<tr><th>From</th><th>To</th><th>Use</th></tr>
<tr><td>seconds since the epoch</td><td>struct_time in UTC</td><td>gmtime()</td></tr>
<tr><td>seconds since the epoch</td><td>struct_time in local time</td><td>localtime()</td></tr>
<tr><td>struct_time in UTC</td><td>seconds since the epoch</td><td>calendar.timegm()</td></tr>
<tr><td>struct_time in local time</td><td>seconds since the epoch</td><td>mktime()</td></tr>
</table>

Date formats:
<pre>
>>> time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
'2014-04-04 12:22:43'

>>> time.time()
1396635763.0581281

>>> datetime.datetime.now()
datetime.datetime(2014, 4, 4, 12, 22, 43, 58456)
# datetime(year, month, day, hour, minute, second, microsecond)

>>> time.localtime()
time.struct_time(tm_year=2014, tm_mon=4, tm_mday=4, tm_hour=12, tm_min=22, tm_sec=43, tm_wday=4, tm_yday=94, tm_isdst=1)
</pre>

Current epoch time:
int(time.time())

Time zone offset:
standard_tz_offset = - time.timezone / 3600 # 7
daylight_tz_offset = - time.altzone / 3600 # 6

Is daylight saving time: [http://stackoverflow.com/questions/2881025/python-daylight-savings-time]
import time
time.localtime()
_.tm_isdst # 0 or 1

mst_epoch_time = time.time() # server in MST/MDT
if time.localtime().tm_isdst: # are we in daylight saving time?
utc_epoch_time = mst_epoch_time + time.altzone
else:
utc_epoch_time = mst_epoch_time + time.timezone

Format current time: [http://docs.python.org/2/library/time.html#time.strftime]
time.strftime('%d%b%y') # '12May11'
time.strftime('%Y.%m.%d %H:%M:%S') # '2013.08.09 14:01:28'

Time conversions:
<pre>
# epoch to str
epoch_time = time.time() # 1375992088.7867229
dt = datetime.datetime.fromtimestamp( epoch_time ) # datetime.datetime(2013, 8, 8, 14, 1, 28, 786723)
date_str = dt.strftime("%Y-%m-%d %H:%M:%S") # '2013-08-08 14:01:28'
tz_offset = - time.altzone / 3600
if tz_offset < 0:
tz_offset_str = "-%02d00" % abs(tz_offset)
else:
tz_offset_str = "+%02d00" % abs(tz_offset)
print date_str + " " + tz_offset_str # '2013-08-08 14:01:28 -0600'
</pre>

<pre>
# UTC epoch to str
gm_epoch_time = calendar.timegm(time.gmtime()) # 1375992114
dt = datetime.utcfromtimestamp( gm_epoch_time ) # datetime.datetime(2013, 8, 8, 20, 1, 54)
dt.strftime("%Y-%m-%d %H:%M:%S +0000") # '2013-08-08 20:01:54 +0000'
dt.strftime("%Y-%m-%dT%H:%M:%SZ") # '2013-08-08 20:01:54 +0000'
</pre>

<pre>
# Local time str to epoch
iso_string='2013-08-08T14:01:28'
epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S' ) ) # 1375992088.0

iso_string='2013-08-08T14:01:28 MDT'
epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S %Z' ) ) # 1375992088.0

# not sure how to specify numerical time zone and pull it in!

# UTC (Zulu) str to epoch, and back again
iso_string = '2013-08-08T20:01:54Z'
epoch_time = calendar.timegm( time.strptime( iso_string.replace('Z', 'GMT'), '%Y-%m-%dT%H:%M:%S%Z' ) )
print epoch_time # 1375992114
print "%sZ" % datetime.utcfromtimestamp(timestamp).isoformat() # 2013-08-08T20:01:54Z
</pre>

<pre>
# Convert epoch from Local to UTC
epoch_time = 1375992114
tz_offset = - time.altzone / 3600
gm_epoch_time = epoch_time - tz_offset * 3600 # 1376013714
</pre>

<pre>
# Convert epoch from UTC to Local
gm_epoch_time = 1376013714
tz_offset = - time.altzone / 3600
epoch_time = gm_epoch_time + tz_offset * 3600 # 1375992114
</pre>

struct_time:
time.localtime() # local zime zone
time.gmtime() # UTC time

today = time.localtime()
today_year = today.tm_year
today_month = today.tm_mon
today_day = today.tm_mday

time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime())

Zulu time to epoch time: [http://stackoverflow.com/questions/127803/how-to-parse-iso-formatted-date-in-python]
s = '2008-09-03T20:56:35.450686Z'
event_time_struct = time.strptime(s.split('.')[0]+ "UTC", "%Y-%m-%dT%H:%M:%S%Z")
epoch_time = calendar.timegm(event_time_struct)

Parse string to time.struct_time (like gmtime()): [http://docs.python.org/2/library/time.html#time.strptime]
time.strptime(time.ctime()) # current time - default format is ctime format: "%a %b %d %H:%M:%S %Y"
time.strptime(time.ctime(time.time())) # default format is ctime format: "%a %b %d %H:%M:%S %Y"
time.strptime("30 Nov 00", "%d %b %y")

Build from ticks: [http://www.tutorialspoint.com/python/python_date_time.htm]
localtime = time.localtime( time.time() ) # build time.struct_time from ticks
print time.asctime( localtime ) # Tue Jan 13 10:17:09 2009

dt = datetime.datetime.fromtimestamp( time.time() ) # build datetime.datetime struct from ticks
dt.strftime("%Y-%m-%d %H:%M:%S") # '2013-07-31 23:07:48'
time.strptime( time.time(), ")

<pre>
from datetime import date
now = date.today() # now = datetime.date(2003, 12,2)
now.strftime("%m-%d-%y") # '12-02-03'
# dates support calendar arthimetic
birthday = date(1964, 7, 31)
age = now - birthday
age.days # 14368
</pre>

ts = (datetime.datetime.now().strftime("%Y%m%d%H%M")) # 201105191118
tstxt = (datetime.datetime.now().strftime("%m-%d-%Y %H:%M")) # 05-19-2011 11:18

Seconds since epoch:
int(time.time()) # 1305842304

Zulu and back again: ([http://aboutsimon.com/2013/06/05/datetime-hell-time-zone-aware-to-unix-timestamp/ Datetime hell])
<pre>
import time
from calendar import timegm
from datetime import datetime

if __name__ == '__main__':
iso_string = '2013-06-05T15:19:10Z'

timestamp = timegm(
time.strptime(
iso_string.replace('Z', 'GMT'),
'%Y-%m-%dT%H:%M:%S%Z'
)
)

print iso_string
print timestamp
print '-> %sZ' % datetime.utcfromtimestamp(timestamp).isoformat()
</pre>

http://wiki.python.org/moin/TimeTransitionsImage?action=AttachFile&do=get&target=v1.png [http://wiki.python.org/moin/WorkingWithTime]

References:
* WorkingWithTime - PythonInfo Wiki - http://wiki.python.org/moin/WorkingWithTime
* Datetime hell. Time zone aware to UNIX timestamp. | About me and the things I do - http://aboutsimon.com/2013/06/05/datetime-hell-time-zone-aware-to-unix-timestamp/
* WorkingWithTime - Python Wiki - http://wiki.python.org/moin/WorkingWithTime

---

Something I cooked up...

<pre>
def utc_time(self, years_offset=60):
mst_epoch_time = time.time() # server is in MST
if time.localtime().tm_isdst: # are we daylight saving time?
utc_epoch_time = mst_epoch_time + time.altzone
else:
utc_epoch_time = mst_epoch_time + time.timezone
return utc_epoch_time

def tz_time(self, tz_offset, daylight_adjust=True):
""" return the future time based on timezone offset
@tz_offset is timezone offset, eg -7 for MST
@daylight_adjust is flag to auto adjust for daylight time"""
if daylight_adjust:
if time.localtime().tm_isdst: # are we daylight saving time?
_tz_offset = tz_offset - 1
else:
_tz_offset = tz_offset
_utc_epoch_time = self.utc_time()
_adj_time = _utc_epoch_time + _tz_offset * 60 * 60
return _adj_time

def future_time(self, tz_offset=-7, daylight_adjust=True, years_offset=60):
""" Calculate some future year """
_tz_time = self.tz_time(tz_offset)
future_utc_epoch_time = _tz_time + years_offset * 365 * 24 * 60 * 60
# account for leap years
cur_year = datetime.datetime.now().year
cur_year_leap = cur_year / 4
future_year = cur_year + years_offset
future_year_leap = future_year / 4
leap_days = future_year_leap - cur_year_leap
future_utc_epoch_time += leap_days * 24 * 60 * 60
return future_utc_epoch_time
</pre>

=== Math ===

Math Functions:
import math
math.ceil(x)
math.floor(x)
math.trunc(x)
math.pow(x,y)

abs(x) # built in

=== Random ===

import random
random.choice(['a', 'b', 'c'])
random.sample(range(100), 10)
random.random() # random float 0 < x < 1
random.randrange(3) # integer in [0, 1, 2]
random.randrange(1, 3) # integer in [1, 2]
random.randint(1, 3) # integer in [1, 2, 3]

=== Operating System Operations ===

Import OS Library:
import os

Common Functions:
# directory functions
os.getcwd() # get current directory
os.listdir(p) # get list of directory contents
os.chdir(p) # change directory
os.mkdir(p) # make directory
os.makedirs(p) # make directory and all intermediate
os.rmdir(p) # remove empty directory, see shutil.rmtree()
os.removedirs(p) # remove empty directories, see shutil.rmtree()
os.path.exists(p) # does file/directory exist?
os.path.isdir(p) # does directory exist
os.path.join(folder1, folder2, file, etc) # portable path join

# file functions
os.remove(p) # remove file
os.unlink(p) # same as os.remove()
os.path.exists(p) # does file/directory exist?
os.path.isfile(p) # does file exist
os.stat(fname) # get file stats
os.stat(fname).st_size # file size
os.link(src, dst) # create hard link
os.readlink(p) # read symbolic link
os.rename(old, new) # rename/move file or directory
os.tempnam(p) # temporary unique name for target path
os.tmpfile() # create temporary file
os.tmpnam() # temporary file name
os.path.getsize(filename) # get file size

# system functions
os.fork() # fork process
os.getpid() # get process id
os.getppid() # get parent's process id
os.kill(pid, sig) # kill a process
os.getlogin() # get user's login name

# execution
os.system(cmd) # output to stdout, return exit_status
os.popen(cmd) # returns a pipe to command

Note: to get path to script:
sys.path[0]

Get directory and file names:
os.path.dirname(path) # '/root/test.txt' -> '/root'
os.path.basename(path) # '/root/test.txt' -> 'test.txt'
os.path.splitext(path) # '/root/test.test.txt' -> ('/root/test.test', '.txt') - extract extention
filename.split(".")[-1] # '/root/test.test.txt' -> 'txt' - extract extention

Normalize path, eliminating double slashes, etc.
os.path.normpath(path) # '/root/..' -> '/' or 'test/..' -> '.'
os.path.abspath(path) # '/root/..' -> '/' or 'test/..' -> '/root/test'

List dir: (returns list)
os.listdir( os.getcwd() )

List of files:
[p for p in os.listdir(os.getcwd())
if os.path.isfile(p)]

List of folders:
[p for p in os.listdir(os.getcwd())
if os.path.isdir(p)]

Build path for local OS: (using appropriate slash)
pathstr = os.path.join("folder", "folder", "file_or_folder")

File and directory management with shutil
import shutil
shutil.copyfile('file1', 'file2')
shutil.move('file1', 'folder2/file1')

File Wildcards:
import glob
glob.glob('*.py') # return list of files

Walk directory tree: [http://stackoverflow.com/questions/120656/directory-listing-in-python]
<pre>
import os
for dirname, dirnames, filenames in os.walk('.'):
for subdirname in dirnames:
print os.path.join(dirname, subdirname)
for filename in filenames:
print os.path.join(dirname, filename)
</pre>

ugly 'deltree' (by kenneth)
<pre>
import os
def deltree(path):
files = os.listdir(path)
for file in files:
file_path = os.path.join(path, file)
if os.path.isdir(file_path):
deltree(file_path)
os.rmdir(file_path)
else:
os.remove(file_path)
</pre>

Cleaner 'deltree' (by Kep)
<pre>
import shutil
# Blow away (if needed), and reclone to proper version
if os.path.isdir("xxx"):
shutil.rmtree("xxx")
</pre>

== System Platform ==

<pre>
import platform

if platform.system() == "Linux":
# ...
if platform.system() == "Windows":
# ...
else:
# ...
</pre>

=== Temp Files ===

import tempfile
filename = tempfile.mktemp()
f = open(filename, 'w')
...
f.close()

import os
os.remove(filename) # delete temp file

=== Pickling ===

Pickling (serializing, making objects persistent):
import pickle
grades = {'bob' : [4, 5, 6],
'sue' : [3, None, 7, 7]}
outfile = open(fname, 'wb')
pickle.dump(grades, outfile)

infile = open(fname, 'rb')
grades = pickle.load(infile)

=== [http://docs.python.org/library/logging.html Logging] ===

* logging — Logging facility for Python — Python 3 documentation - https://docs.python.org/3/library/logging.html
* Logging HOWTO — Python Dev Version documentation - http://docs.python.org/dev/howto/logging.html
* Logging Cookbook — Python Dev Version documentation - http://docs.python.org/dev/howto/logging-cookbook.html

Setup logger:
import logging
logging.basicConfig(level=logging.DEBUG,format='[%(asctime)s.%(msecs)03d] %(levelname)s %(filename)s(line:%(lineno)d): %(message)s')

Log Levels:
import logging
msg = "i broke"
logging.debug(msg)
logging.info(msg)
logging.warning(msg) # WARNING:root:i broke
logging.error(msg) # ERROR:root:i broke
logging.critical(msg) # CRITICAL:root:i broke

log.setLevel(level) # debug, info, warn/warning, error, critical
log.setLevel(logging.DEBUG)

A good convention to use when naming loggers is to use a module-level logger, in each module which uses logging, named as follows: [http://docs.python.org/dev/howto/logging.html]
logger = logging.getLogger(__name__)

Simple example: [http://stackoverflow.com/questions/345991/python-no-handlers-could-be-found-for-logger-opengl-error]
<pre>
log_format='%(asctime)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
logging.basicConfig(format=log_format, datefmt=log_date_format)
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)

log.info('hi')
</pre>

Formatter function:
<pre>
def makeFormatter(format_str="", date_format="%Y-%m-%d %H:%M:%S"):
if not format_str:
format_str = "%(asctime)s | %(levelname)s | %(filename)s:%(lineno)d:%(funcName)s | %(message)s"
return logging.Formatter(fmt=format_str, datefmt=date_format)
</pre>

Medium example:
<pre>
# Create logger
log_format='%(asctime)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
log_formatter = logging.Formatter(fmt=log_format, datefmt=log_date_format)
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)

# Log to console
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(log_formatter)
log.addHandler(ch)

# Log to file
fh = logging.FileHandler('my_logger.log')
fh.setLevel(logging.DEBUG)
fh.setFormatter(log_formatter)
log.addHandler(fh)
</pre>

Log all loggers to file - attach to root logger
<pre>
import logging

logger = logging.getLogger(__name__)

# Configure Logger
log_format='%(asctime)s | %(name)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
log_formatter = logging.Formatter(fmt=log_format, datefmt=log_date_format)
logging.basicConfig(level=logging.DEBUG, format=log_format, datefmt=log_date_format)

# Setup Log to File handler, and attach to root logger
root_logger = logging.getLogger() # Calling getLogger() without a name returns the root logger
fh = logging.FileHandler('logger.txt')
fh.setLevel(logging.DEBUG)
fh.setFormatter(log_formatter)
root_logger.addHandler(fh)
</pre>

Application Setup (not basic log configuration carries across imported modules):
<pre>
import logging
def main():
logging.basicConfig(filename='myapp.log', level=logging.INFO)
logging.info('Started')
...
logging.info('Finished')

if __name__ == '__main__':
main()
</pre>

Log multiple lines:
<pre>
for line in output.splitlines():
log.debug(line)
</pre>

Set log level: (warning is default)
logging.basicConfig(level=logging.DEBUG)

Set log level from string value: [http://docs.python.org/dev/howto/logging.html]
<pre>
# assuming loglevel is bound to the string value obtained from the
# command line argument. Convert to upper case to allow the user to
# specify --log=DEBUG or --log=debug
numeric_level = getattr(logging, loglevel.upper(), None)
if not isinstance(numeric_level, int):
raise ValueError('Invalid log level: %s' % loglevel)
logging.basicConfig(level=numeric_level, ...)
</pre>

Log to file:
logging.basicConfig(filename='example.log',level=logging.DEBUG)
logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG) # do not append

Format output: (once set, can't be changed?)
# WARNING:MESSAGE
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)

# 05/23/2011 12:50:02 PM MESSAGE
logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')

# 05/23/2011 12:55:28 PM : WARNING : MESSAGE
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')

# 2011-05-23 13:05:30 : WARNING : MESSAGE
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

# 2011-05-23 13:05:30 | WARNING | FILE:FUNC | MESSAGE
logging.basicConfig(format='%(asctime)s | %(levelname)s | %(filename)s:%(funcName)s | %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

Log record attributes - http://docs.python.org/library/logging.html#logrecord-attributes

Format Example:
<pre>
FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' # doesn't work!
logging.basicConfig(format=FORMAT)
d = { 'clientip' : '192.168.0.1', 'user' : 'fbloggs' }
logger = logging.getLogger('tcpserver')
logger.warning('Protocol problem: %s', 'connection reset', extra=d)
</pre>

<pre>
import logging

# create logger with 'spam_application'
logger = logging.getLogger('spam_application')
logger.setLevel(logging.DEBUG)

# create file handler which logs even debug messages
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)

# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)

# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(fh)
logger.addHandler(ch)

logger.info('Message to loggers and handlers')
</pre>

=== Templating ===

from string import Template
t = Template('${name}'s fav dish is $food')
t.substitute(name = 'ken', food = 'beans') # throws exception if missing one, but can have more
t.safe_substitute(name = 'ken') # will fill in what it can

Change delimiter:
class MYT(Template):
delimiter = '%'
minput = input("File name format (%d - date, %s - size): ")
myt = MYT(minput)
myt.substitute(%d=date, %s=size)

=== [http://docs.python.org/library/threading.html Threading] ===

Basic thread: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import thread

def someFunc():
print "someFunc was called"

thread.start_new_thread(someFunc, ())
</pre>

More complete basic thread:
<pre>
import thread
import threading
import time

def run_thread (threadname, sleeptime):
print "Thread start: " + threadname
time.sleep(sleeptime)
print "Thread end: " + threadname
thread.interrupt_main() # throw keyboard interrupt to kill main program

if __name__ == "__main__":
thread.start_new_thread(run_thread, ("Thread1", 3))
try:
while 1:
pass
except:
print "Thread1 exited...."
</pre>

Can use a global counter instead of exception to track number of threads:
<pre>
import thread
import threading
import time

threadcount = 3

def run_thread (threadname, sleeptime):
global threadcount
print "Thread start: " + threadname
time.sleep(sleeptime)
print "Thread end: " + threadname
threadcount -= 1

if __name__ == "__main__":
thread.start_new_thread(run_thread, ("Thread1", 3))
thread.start_new_thread(run_thread, ("Thread2", 4))
thread.start_new_thread(run_thread, ("Thread3", 2))
while 1:
if threadcount == 0:
break
else:
pass
print "completed"
</pre>

---

Thread object: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import threading

t1 = threading.Thread(target=someFunc)
# with parameters: Thread(target=myfunction, args=('MyStringHere',1)).start()
t1.start()
t1.join()
</pre>

Basic threading with parameters:
<pre>
import threading
t = threading.Thread(None, someFunc, None, (param1, param2))
t.start()
t.join()
</pre>

<pre>
t = threading.Thread(target=client_recv, args=(s, name))
while True:
t.join(timeout=2)
if t.isAlive():
print "Thread is dead..."
break
</pre>

Threading as a class: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import threading

class FuncThread(threading.Thread):
def __init__(self, target, *args):
self._target = target
self._args = args
threading.Thread.__init__(self)

def run(self):
self._target(*self._args)

# Example usage
def someOtherFunc(data, key):
print "someOtherFunc was called : data=%s; key=%s" % (str(data), str(key))

t1 = FuncThread(someOtherFunc, [1,2], 6)
t1.start()
t1.join()
</pre>

Threading as a class #2:
<pre>
import threading
import time
import sys

class MyT(threading.Thread):
def __init__(self, name, time):
threading.Thread.__init__(self)
self.name = name
self.time = time
def run(self):
print("Thread Start: " + self.name)
for i in range(self.time):
print("[" + self.name + "]", end='')
sys.stdout.flush()
time.sleep(1)
print("Thread Stop: " + self.name)

myt1 = MyT('1', 10)
myt1.start()

myt2 = MyT('2', 15)
myt2.start()

myt3 = MyT('3', 5)
myt3.start()

myt4 = MyT('4', 10)
myt4.start()

myt1.join()
myt2.join()
myt3.join()
myt4.join()
</pre>

Locking:
<pre>
lock=thread.allocate_lock()

# pass lock to thread, and use in thread as such:
lock.acquire()
#some operation
lock.release()
</pre>

Returning data from a thread (using a [http://docs.python.org/library/queue.html queue]): [http://stackoverflow.com/questions/1886090/return-value-from-thread] [http://stackoverflow.com/questions/2829329/catch-a-threads-exception-in-the-caller-thread-in-python]
<pre>
# Pass exception back to parent
import sys
import threading
import Queue

class ExcThread(threading.Thread):
def __init__(self, bucket):
threading.Thread.__init__(self)
self.bucket = bucket
def run(self):
try:
raise Exception('An error occured here.')
except Exception:
self.bucket.put(sys.exc_info())

def main():
bucket = Queue.Queue()
thread_obj = ExcThread(bucket)
thread_obj.start()
while True:
try:
exc = bucket.get(block=False)
except Queue.Empty:
pass
else:
exc_type, exc_obj, exc_trace = exc
# deal with the exception
print "Exception Caught:", exc_type, exc_obj
print "Exception Trace:", exc_trace
thread_obj.join(0.1)
if thread_obj.isAlive():
continue
else:
break

if __name__ == '__main__':
main()
</pre>

<pre>
Exception Caught: <type 'exceptions.Exception'> An error occured here.
Exception Trace: <traceback object at 0x2b2230a285f0>
</pre>

Another Exception catching queue example: (kenneth's example)
<pre>
# thread function
def run_thread(queue, item):
...
queue.put( Exception("Failure on item:" + item) )

threads = []
queue = Queue.Queue()

# start threads
for item in items:
t = Thread(None, run_thread, None, (queue, item))
t.start()
threads.append(t)

# wait for threads
for t in threads:
t.join()

# catch and reraise exceptions
while not queue.empty():
raise(queue.get())
</pre>

Returning data from a thread (using a callback function): [http://stackoverflow.com/questions/1886090/return-value-from-thread]
:"Another approach is to pass a callback function to the thread. This gives a simple, safe and flexible way to return a value to the parent, anytime from the new thread."
<pre>
import threading
import time
import sys

class MyThread(threading.Thread):
def __init__(self, cb):
threading.Thread.__init__(self)
self.callback = cb
def run(self):
for i in range(10):
self.callback(i)
time.sleep(1)

def count(x):
print x
sys.stdout.flush()

t = MyThread(count)
t.start()
</pre>

References:
* 16.2. threading — Higher-level threading interface — Python v2.7.1 documentation - http://docs.python.org/library/threading.html
* An Introduction to Thread Programming - http://www.prasannatech.net/2008/08/introduction-to-thread-programming.html
* Python - Multithreaded Programming - http://www.tutorialspoint.com/python/python_multithreading.htm
* Running Functions as Threads in Python « Software Ramblings - http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html
* Understanding Threading in Python LG #107 - http://linuxgazette.net/107/pai.html
* multithreading - Simple threading in Python 2.6 using thread.start_new_thread() - Stack Overflow - http://stackoverflow.com/questions/849674/simple-threading-in-python-2-6-using-thread-start-new-thread
* python - Return value from thread - Stack Overflow - http://stackoverflow.com/questions/1886090/return-value-from-thread

---

Interrupt main thread:
<pre>
# source: http://stackoverflow.com/questions/4591917/python-how-to-quit-cli-when-stuck-in-blocking-raw-input

import thread
import time

def main():
try:
m = thread.start_new_thread(killable_input, tuple())
while 1:
time.sleep(0.1)
except KeyboardInterrupt:
print "exception"

def killable_input():
w = thread.start_new_thread(normal_input, tuple())
i = thread.start_new_thread(wait_sometime, tuple())

def normal_input():
s = raw_input("input:")

def wait_sometime():
time.sleep(4) # or any other condition to kill the thread
print "too slow, killing imput"
thread.interrupt_main()

if __name__ == '__main__':
main()
</pre>

=== urllib ===

See [[Python/urllib]]

=== Command line arguments ===

<pre>
import sys
for x in sys.argv:
print x
</pre>

=== Execute other python script ===

import sys
sys.argv = ['', 'my', 'args', 'here']
execfile('myscript.py') # does not support arguments directly

=== Exit Code ===

sys.exit(n) http://docs.python.org/library/sys.html:
import sys
sys.exit(1) # exit code of 1 (default is 0)
sys.exit("some error") # exit code of 1, and message displayed

=== [http://docs.python.org/library/syslog.html syslog] ===

<pre>
import syslog
syslog.openlog('[WATER]')
syslog.syslog('Processing started')
if error:
syslog.syslog(syslog.LOG_ERR, 'Processing started')
</pre>

<pre>
import syslog
syslog.openlog('[WATER]')
def log(msg):
syslog.syslog(str(msg))
log("Starting...")
</pre>

=== Subprocess ===

Subprocess - http://docs.python.org/library/subprocess.html

Methods of subprocess (or external command execution):
* subprocess (only method to rule/replace all others)
* os.system
* os.spawn*
* os.popen*
* popen2.*
* commands.*

My New Favorite:
<pre>
import subprocess
class Run(object):
class RC(int):
pass
def __new__(cls, cmd, *args, **kwargs):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
rc = cls.RC(p.wait())
rc.output = "\n".join(p.stdout.readlines())
rc.error = "\n".join(p.stderr.readlines())
return rc

if Run("some command"): print "do something" # on rc of not 0 (negative or positive)
</pre>

My Favorite:
<pre>
def exe(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
#output = " | ".join(p.stdout.readlines())
output = "\n".join(p.stdout.readlines())
rc = p.wait()
return (rc, output)

def exe_checked(cmd, checked=True):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
#output = " | ".join(p.stdout.readlines())
output = "\n".join(p.stdout.readlines())
rc = p.wait()
if rc and checked:
raise Exception("Command returned non zero [{0}]: {1}".format(rc, cmd)
return (rc, output)

def exe_with_stderr(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subproc
ess.PIPE)
output = " | ".join(p.stdout.readlines())
outputerr = " | ".join(p.stderr.readlines())
rc = p.wait()
return (rc, output, outputerr)
</pre>

<pre>
def exec_program(cmd):
s = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = s.communicate()
result = s.wait()
return stdout, stderr, result
</pre>

Pass along output and return true error code:
import subprocess
retcode = subprocess.call(["ls", "-l"]) # true return code, output sent to STDOUT

retcode = subprocess.call(["foo.sh","args"],shell=True) # Execute in shell, with shell processing
retcode = subprocess.call(['C:\\Temp\\a b c\\Notepad.exe'])
retcode = subprocess.call(["D:/Python24/Python.exe", "Skripta.py", "arg1", "arg2"])

Note: shell=True is equivalent to: subprocess.Popen(['/bin/sh', '-c', args[0], args[1], ...])

Process STDOUT and STDERR and get return code:
<pre>
#!/usr/bin/env python
import subprocess
import string

cmd = "cat /etc/passwd | grep ken"
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) # stderr=subprocess.STDOUT to combine

out = string.join(p.stdout.readlines())
outerr = string.join(p.stderr.readlines())
retcode = str(p.wait())

print("STDOUT: " + out)
print("STDERR: " + outerr)
print("RETCODE: " + retcode)
</pre>

Above with stderr redirected to stdout (combined):
<pre>
import subprocess
p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
print line,
retval = p.wait()
</pre>

Replacing shell backtick: [http://docs.python.org/library/subprocess.html]
# shell code:
# OUTPUT=`mycmd myarg`
from subprocess import Popen
output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]

Replacing shell pipeline [http://docs.python.org/library/subprocess.html]
# shell code:
# OUTPUT=`dmesg | grep hda`
from subprocess import Popen
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

Replacing os.system(): [http://docs.python.org/library/subprocess.html]
# old:
sts = os.system("mycmd" + " myarg")

# new:
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)[1]

Control an executed process:
<pre>
cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

# cycle while process continues alive
while p.poll() == None:

# check if input
if select.select([p.stdout.fileno()], [], [], 1)[0]:
while select.select([p.stdout.fileno()], [], [], .1)[0]:
line = p.stdout.readline()
# ... process

# write something to process
p.stdin.write("tp %s %s\n" % (user1, user2))

...

# time to force a quit: (choose which you need)
p.kill() # SIGKILL
p.terminate() # SIGTERM
os.kill(p.pid, signal.SIGTERM) # same as first, but not as clean
</pre>

==== Other Command Execution ====

Old method with os.system(): [http://docs.python.org/library/os.html]
import os
encoded_wait_code = os.system("some command") # output sent to STDOUT
# Note: returns code of 0 if command returns 0, but wait() encoded value for all other values

Quick and dirty old method (prints stderr, captures stdout)
<pre>
import os
out = string.join( os.popen(cmd).readlines() )
</pre>

Quick and dirty old method (prints stderr, captures stdout, captures ugly wait() encoded return code)
<pre>
import os
p = os.popen(cmd)
stdout = "".join( p.readlines() )
encoded_wait_code = p.close() # None if 0
rcode = os.WEXITSTATUS(ugly_rcode) # fix ugly wait() encoded rcode
print(stdout)
print(rcode)

if os.popen(cmd).close():
print "error with cmd: %s" % cmd
sys.exit(1)
</pre>

???:
<pre>
os.spawnvp(os.P_WAIT,"/usr/bin/mplayer",["","-ao","pcm",file])
# or
os.spawnvp(os.P_NOWAIT,"/usr/bin/mplayer",["","-ao","pcm",file])
</pre>

Replacing os.system(): [http://docs.python.org/library/subprocess.html]
# old:
sts = os.system("mycmd" + " myarg")

# new:
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)[1]

More realistic example replacing os.system(): [http://docs.python.org/library/subprocess.html]
<pre>
try:
retcode = call("mycmd" + " myarg", shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e
</pre>

==== select ====

Select:

select.select(rlist, wlist, xlist[, timeout])
: This is a straightforward interface to the Unix select() system call. The first three arguments are sequences of ‘waitable objects’: either integers representing file descriptors or objects with a parameterless method named fileno() returning such an integer
: The optional timeout argument specifies a time-out as a floating point number in seconds. When the timeout argument is omitted the function blocks until at least one file descriptor is ready. A time-out value of zero specifies a poll and never blocks.

if select.select([sys.stdin.fileno()], [], [], .1)[0]:
line = p.stdout.readline()

==== full wrapped execution ====

Process timeout (by Kenneth):
<pre>
import subprocess
import time
import os
import signal

def cmd(cmd, timeout=300, retry=0):
# Execute command, time progress, and kill on timeout

# (status, output) = commands.getstatusoutput(cmd)
start_time = None
status = None
output = None
for retry_count in range(retry+1):
if retry_count > 0:
print("RETRY: " + str(retry_count) + " " + cmd)
start_time = time.time()
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
timeout_fail = False
while p.poll() == None:
if time.time() - start_time > timeout:
# kill process
print("TIMEOUT: " + str(timeout) + "sec - killing (" + str(p.pid) + ") - " + cmd)
try:
os.kill(p.pid, signal.SIGTERM)
except OSError as e:
print("Kill exception: " + str(e))
timeout_fail = True
break
#print(".")
time.sleep(1)
status = p.poll()
output = p.stdout.read()
if timeout_fail == False:
break
seconds = int(time.time() - start_time)
return (status, output, seconds)
</pre>

---

Process input and output wrapper (from minecraft):
<pre>
import subprocess
import select
import re
import sys

cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)

# cycle while process continues alive
while p.poll() == None:

if select.select([p.stdout.fileno()], [], [], 1)[0]:
while select.select([p.stdout.fileno()], [], [], .1)[0]:
line = p.stdout.readline()
# <kiloforce> -tp joe
match = re.findall("<(.*)> -tp (.*)", line)
if match:
user1 = match[0][0]
user2 = match[0][1]
print ">> ACTION: Teleporting %s to %s" % (user1, user2)
p.stdin.write("tp %s %s\n" % (user1, user2))
if line:
print ">> SERVER: %s" % line.strip()
else:
print "The server died!"
sys.exit(1)

if select.select([sys.stdin.fileno()], [], [], .1)[0]:
line = sys.stdin.readline()
print ">> CONSOLE: %s" % line.strip()
p.stdin.write("%s" % line)
</pre>

==== Command With Timeout ====

python - Using module 'subprocess' with timeout - Stack Overflow - http://stackoverflow.com/questions/1191374/using-module-subprocess-with-timeout

<pre>
import subprocess, threading

class Command(object):
def __init__(self, cmd):
self.cmd = cmd
self.process = None

def run(self, timeout):
def target():
print 'Thread started'
self.process = subprocess.Popen(self.cmd, shell=True)
self.process.communicate()
print 'Thread finished'

thread = threading.Thread(target=target)
thread.start()

thread.join(timeout)
if thread.is_alive():
print 'Terminating process'
self.process.terminate()
thread.join()
print self.process.returncode

command = Command("echo 'Process started'; sleep 2; echo 'Process finished'")
command.run(timeout=3)
command.run(timeout=1)
</pre>

My version:
<pre>
import subprocess
import threading

class Command(object):
def __init__(self, cmd, capture=False, shell=True):
self.cmd = cmd
self.process = None
self.thread = None
self.shell = shell
if type(cmd) == list:
self.shell = False
self.rc = None
self.pid = None
self.stdout = None
self.stderr = None
self.capture = capture # capture stdout?

def run(self, timeout=None, wait=True):
def target():
print '>> Thread started (%s)' % self.cmd
if self.capture:
self.process = subprocess.Popen(self.cmd, shell=self.shell, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
else:
self.process = subprocess.Popen(self.cmd, shell=self.shell)
print '>> Thread PID: %d' % self.process.pid
self.pid = self.process.pid
self.stdout, self.stderr = self.process.communicate()
self.rc = self.process.returncode
print '>> Thread finished: %d' % self.rc

self.thread = threading.Thread(target=target)
self.thread.start()

if wait:
self.wait(timeout)
return self # so you can do: c = Command().run()

def wait(self, timeout=None):
print '>> Waiting for thread...'
if timeout:
self.thread.join(timeout)
else:
self.thread.join()
if self.thread.is_alive():
print '>> Terminating process'
self.process.terminate()
self.rc = self.process.returncode
self.thread.join()
if self.thread.is_alive():
print '>> Failed to terminate'
else:
print '>> Thread completed'
#print self.process.returncode

def is_alive(self ):
return self.thread.is_alive()

command = Command('do something').run()

command2 = Command(['do', 'something', 'long']).run(wait=False)
command2.wait(timeout=10)

command3 = Command('want output', capture=True).run()
print command3.stdout
</pre>

=== System call ===

os.system("some_command with args")
os.system("some_command < input_file | another_command > output_file")
stream = os.popen("some_command with args")

Here's a summary of the ways to call external programs and the advantages and disadvantages of each:

1.
: os.system("some_command with args") passes the command and arguments to your system's shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example,
: os.system("some_command < input_file | another_command > output_file")
: However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs.
: http://docs.python.org/lib/os-process.html

2.
: stream = os.popen("some_command with args") will do the same thing as os.system except that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything.
: http://docs.python.org/lib/os-newstreams.html

3.
: The Popen class of the subprocess module. This is intended as a replacement for os.popen but has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you'd say
: print Popen("echo Hello World", stdout=PIPE, shell=True).stdout.read()
: instead of
: print os.popen("echo Hello World").read()
: but it is nice to have all of the options there in one unified class instead of 4 different popen functions.
: http://docs.python.org/lib/node528.html

4.
: The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply wait until the command completes and gives you the return code. For example:
: return_code = call("echo Hello World", shell=True)
: http://docs.python.org/lib/node529.html

5.
: The os module also has all of the fork/exec/spawn functions that you'd have in a C program, but I don't recommend using them directly.

: The subprocess module should probably be what you use.

Source: [http://stackoverflow.com/questions/89228/how-to-call-external-command-in-python]

== Advanced Code ==

=== Executing Code from Command Line ===

python -c [cmd]
# -c cmd : program passed in as string (terminates option list)

<pre>
python -c "import sys;exec(sys.stdin.read())" << "EOF"
def test1():
print "test1"
test1()
print "test"
EOF
</pre>

=== Integer Return Code Class ===

Python3 version:
<pre>
class RC(int):
def __new__(cls, rc, stdout='', stderr=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, stdout='', stderr=''):
super().__init__()
self.rc = rc # redundant
self.stdout = stdout
self.stderr = stderr
def __repr__(self):
"""Optional string output"""
return "(%s, '%s', '%s')" % (self.rc, self.stdout, self.stderr)
</pre>

Python2 version:
<pre>
class RC(int):
def __new__(cls, rc, stdout='', stderr=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, stdout='', stderr=''):
super(self.__class__, self).__init__(rc)
self.rc = rc # redundant
self.stdout = stdout
self.stderr = stderr
def __repr__(self):
"""Optional string output"""
return "(%s, '%s', '%s')" % (self, self.stdout, self.stderr)
</pre>

sample:
<pre>
>>> rc1 = RC(5)
>>> rc2 = RC(0, 'out', 'err')
>>> rc1
(5, '', '')
>>> rc2
(5, 'out', 'err')
>>> rc2.stdout
'out'
>>> int(rc1)
5
>>> 0 + rc1
5
>>> if rc1 <= 5: print "yes"
...
yes
>>> if rc2: print "non-zero"
...
</pre>

=== Smart Execute ===

<pre>
class RC(int):
def __new__(cls, rc, output=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, output=''):
super().__init__()
self.rc = rc # redundant
self.output = output
def __repr__(self):
"""Optional string output"""
return "(%s, '%s')" % (self.rc, self.output)

def smart_execute(cmd, display=True, checked=False):
output = ""
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, shell=True, encoding='utf-8')
for c in iter(lambda: process.stdout.read(1), ""):
if display:
sys.stdout.buffer.write(c.encode('utf-8'))
output += c
process.stdout.close()
rc = process.wait()
if checked and rc:
raise subprocess.CalledProcessError(ret, cmd)
return RC(rc, output.strip())
</pre>

Not smart version:
<pre>
def execute(cmd, return_output=False):
out = ""
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, shell=True, encoding='utf-8')
for c in iter(lambda: process.stdout.read(1), ""):
sys.stdout.buffer.write(c.encode('utf-8'))
if return_output:
out += c
process.stdout.close()
ret = process.wait()
if ret:
raise subprocess.CalledProcessError(ret, cmd)
if return_output:
return out.strip()
</pre>
</pre>

=== select ===

usage: select.select(rlist, wlist, xlist[, timeout]) [http://docs.python.org/2/library/select.html]
* rlist: wait until ready for reading
* wlist: wait until ready for writing
* xlist: wait for an “exceptional condition” (see the manual page for what your system considers such a condition)
* The optional timeout argument specifies a time-out as a floating point number in seconds.

if select.select([p.stdout.fileno()], [], [], 1)[0]:

=== ANSI Color ===

* ANSI escape code - http://en.wikipedia.org/wiki/ANSI_escape_code

---

unicode - Print in terminal with colors using Python? - Stack Overflow - https://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python

<pre>
class bcolors:
HEADER = '\033[95m'
OKBLUE = '\033[94m'
OKGREEN = '\033[92m'
WARNING = '\033[93m'
FAIL = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'

print bcolors.WARNING + "Warning: No active frommets remain. Continue?" + bcolors.ENDC
</pre>

<pre>
class colors:
black = 0
red = 1
green = 2
yellow = 3
blue = 4
magenta = 5
cyan = 6
white = 7
fg = 30 # foreground add
bg = 40 # background add
bfg = 90 # bold foreground add
bbg = 100 # bold background add

print '\033[%dm red foreground \033[0m' % (colors.red + colors.fg)
print '\033[%dm bold red foreground \033[0m' % (colors.red + colors.bfg)
print '\033[%dm blue background \033[0m' % (colors.blue + colors.bg)
print '\033[%dm bold blue background \033[0m' % (colors.blue + colors.bbg)
</pre>

---

Before you start spouting escape sequences, you should check that stdout is a tty. You can do this with sys.stdout.isatty()

<pre>
sys.stdout.isatty()
</pre>

<pre>
def hilite(string, status, bold):
attr = []
if status:
# green
attr.append('32')
else:
# red
attr.append('31')
if bold:
attr.append('1')
return '\x1b[%sm%s\x1b[0m' % (';'.join(attr), string)
</pre>

Python | change text color in shell - Stack Overflow - https://stackoverflow.com/questions/2330245/python-change-text-color-in-shell

---

For reference, a small bash script which prints a table of escape sequences: colortable16.sh: ([http://www.frexx.de/xterm-256-notes/ The 256 color mode of xterm], [http://plug.org/pipermail/plug/2008-March/016821.html The 90s called; they want their 256 colors back])
<pre>
#!/bin/bash

# prints a color table of 8bg * 8fg * 2 states (regular/bold)
echo
echo Table for 16-color terminal escape sequences.
echo Replace ESC with \\033 in bash.
echo
echo "Background | Foreground colors"
echo "---------------------------------------------------------------------"
for((bg=40;bg<=47;bg++)); do
for((bold=0;bold<=1;bold++)) do
echo -en "\033[0m"" ESC[${bg}m | "
for((fg=30;fg<=37;fg++)); do
if [ $bold == "0" ]; then
echo -en "\033[${bg}m\033[${fg}m [${fg}m "
else
echo -en "\033[${bg}m\033[1;${fg}m [1;${fg}m"
fi
done
echo -e "\033[0m"
done
echo "--------------------------------------------------------------------- "
done

echo
echo
</pre>

Python version: (modified by me)
<pre>
#!/usr/bin/env python

import sys

# prints a color table of 8bg * 8fg * 2 states (regular/bold)
print
print "Table for 16-color terminal escape sequences."
print "Replace ESC with \\033 in bash."
print
print "Background | Foreground colors"
print "---------------------------------------------------------------------"
for bg in range(40, 47+1):
for bold in range(0, 1+1):
sys.stdout.write("\033[0m"" ESC[{bg}m | ".format(bg=bg))
for fg in range(30, 37+1):
if bold == 0:
sys.stdout.write("\033[{bg}m\033[{fg}m [{fg}m ".format(bg=bg, fg=fg))
else:
sys.stdout.write("\033[{bg}m\033[1;{fg}m [1;{fg}m".format(bg=bg, fg=fg))
sys.stdout.write("\033[0m\n")
print "--------------------------------------------------------------------- "

print
print
</pre>

=== Execute External Application ===

Execute program: (and send output to STDOUT)
encoded_wait_code = os.system('mkdir mydir') # output sent to STDOUT

Get output and ugly return code: (STDERR goes to STDOUT)
p = os.popen(cmd)
print p.readlines()
encoded_wait_code = p.close()

Execute other python script
import sys
sys.argv = [, 'my', 'args', 'here']
execfile('myscript.py') # does not support arguments directly

Subprocess:
* See [[#Subprocess]]

=== Program Layout ===

<pre>
PackageName/
README
setup.py
bin/
script
docs/
test/ # some include in package_name
pacakgename/
__init__.py
code1.py
subpackage/
__init__.py
</pre>

=== __init__.py ===

Purpose: Makes the folder into a package library

NOTE: This file is most commonly blank

Good place to put your package functions.
<pre>
# ken/__init__.py
def load():
...
</pre>

<pre>
import ken
ken.load()
</pre>

Can import other modules to then use as package modules:
<pre>
# ken/__init__.py
from ken.client import Server
</pre>

<pre>
import ken
ken.Server()
</pre>

Good place to put your Exceptions.
<pre>
class MyException(Exception):
""" My Error Exception """
</pre>

Use the __all__ to specify what gets imported on 'from package import *'
__all__ = ['func1', 'func2']

=== Metadata ===

VERSION = (0, 2, 1)
or
__version__ = '0.2.1'

Also See [[#pydoc]]

---

__version__ = '1.10'
__author__ = 'Joshua Roesslein'
__license__ = 'MIT'

=== setup.py ===

See [[#Distributing Python Modules with Distutils]]

=== Distributing Python Modules with Distutils ===

Distributing Python Modules — http://docs.python.org/distutils/index.html

Uses setup.py for package installation.
python setup.py install

==== Simple example ====

To install foo.py:
<pre>
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
</pre>

Source: http://docs.python.org/distutils/introduction.html

==== klib example ====

klib setup.py:
<pre>
from distutils.core import setup

setup(name = "klib",
version = "101",
description = "klib library",
url = "http://www.oeey.com/klib",
author = "Kenneth",
author_email = "kenneth@oeey.com",
license = "Kenneth License",
long_description = """Really long text here.""",
platforms = "Any Platform",

packages = ['klib','klib.messenger'],
)
</pre>

Installed to:
/opt/python26/lib/python2.6/site-packages/klib/
/opt/python26/lib/python2.6/site-packages/klib-100-py2.6.egg-info

Metadata: (/opt/python26/lib/python2.6/site-packages/klib-101-py2.6.egg-info)
<pre>
Metadata-Version: 1.0
Name: klib
Version: 101
Summary: klib library
Home-page: http://www.oeey.com/klib
Author: Kenneth
Author-email: kenneth@oeey.com
License: Kenneth License
Description: Really long text here.
Platform: Any platform
</pre>

Any fields not included will be marked:
Platform: UNKNOWN

References:
* 2. Writing the Setup Script — Python v2.7.2 documentation - http://docs.python.org/distutils/setupscript.html
* http://wiki.python.org/moin/Distutils/Tutorial

==== Uninstall Package ====

Collect files during install:
python setup.py install --record install-record.txt
sudo rm $(cat install-record.txt)

References:
* installation - python setup.py uninstall - Stack Overflow - http://stackoverflow.com/questions/1550226/python-setup-py-uninstall

=== Distributing Python Modules with Setuptools ===

<pre>
#!/usr/bin/env python
import setuptools

setuptools.setup(name="tweepy",
version="1.10",
description="Twitter library for python",
license="MIT",
author="Joshua Roesslein",
author_email="tweepy@googlegroups.com",
url="http://github.com/tweepy/tweepy",
packages = setuptools.find_packages(),
keywords= "twitter library",
zip_safe = True)

</pre>

=== Decorators ===

Decorators wrap functions

Function Counter: [http://stackoverflow.com/questions/739654/understanding-python-decorators]
<pre>
def fcount(func):
def wrapper(*args, **kw):
wrapper.count = wrapper.count + 1
return func(*args, **kw)
wrapper.count = 0
return wrapper

@fcount
def f(n):
return n*n

print "f.count = ", f.count
print "f(10) = ", f(10)
print "f.count = ", f.count
print "f(3) = ", f(3)
print "f.count = ", f.count
print "f(12) = ", f(12)
print "f.count = ", f.count
print "f(15) = ", f(15)
print "f.count = ", f.count

# Output:
# f.count = 0
# f(10) = 100
# f.count = 1
# f(3) = 9
# f.count = 2
# f(12) = 144
# f.count = 3
# f(15) = 225
# f.count = 4
</pre>

Great reference: Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

Example 1:
@dec1
def func(arg1, arg2, ...): pass

# equivalent to:
def func(arg1, arg2, ...): pass
func = dec1(func)

Example 2:
<pre>
@dec2
@dec1
def func(arg1, arg2, ...):
pass
</pre>

<pre>
# equivalent to:
def func(arg1, arg2, ...):
pass
func = dec2(dec1(func))
</pre>

Example 3:
@decomaker(argA, argB, ...)
def func(arg1, arg2, ...):
pass

# equivalent to:
func = decomaker(argA, argB, ...)(func)

# example:
@accepts(int,int)
@returns(float)
def bar(low,high):
pass

Example 4: (overrides and ignores passed in function - bad, unless desired)
<pre>
def spamrun(fn):
def sayspam(*args):
print "spam, spam, spam"
return sayspam

@spamrun
def useful(a, b):
print a**2 + b**2
</pre>

Example 5: (Decorator that modifies behavior of undecorated func)
<pre>
def addspam(fn):
def new(*args):
print "spam, spam, spam"
return fn(*args)
return new

@addspam
def useful(a, b):
print a**2 + b**2
</pre>

My Example 1: (print args and continue to function)
<pre>
def showargs(fn):
def doshowargs(*args, **kw):
print "Args for:", fn.__name__, args, kw
return fn(*args, **kw)
return doshowargs

@showargs
def useful(a, b):
print a**2 + b**2

</pre>

Another Example: [http://stackoverflow.com/questions/739654/understanding-python-decorators]
<pre>
def makebold(fn):
def wrapped(*args, **kw):
return "" + fn(*args, **kw) + ""
return wrapped

def makeitalic(fn):
def wrapped(*args, **kw):
return "" + fn(*args, **kw) + ""
return wrapped

@makebold
@makeitalic
def hello(msg):
return msg

print hello("hello world")
# output: hello world
</pre>

References:
* PEP 318 -- Decorators for Functions and Methods - http://www.python.org/dev/peps/pep-0318/
* Charming Python: Decorators make magic easy - http://www.ibm.com/developerworks/linux/library/l-cpdecor/index.html
* What are some common uses for Python decorators? - Stack Overflow - http://stackoverflow.com/questions/489720/what-are-some-common-uses-for-python-decorators
* Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

---

Utah Python July 2013 Meeting - Decorator Examples: [https://www.youtube.com/watch?v=uh9uhzitdvw]

Example of uses:
* Count function calls
* Add additional output (debugging)
* Timing code
* Thread synchronization (locks)
* Type checking

<pre>
## Simple wrapping call
def shout(f):
def wrapper(*args):
print('This is before {name}'.format(name=f.__name__))
ret = f(*args)
print('This is after {name}'.format(name=f.__name__))
return ret
wrapper.__name__ = f.__name__
return wrapper

@shout
def echo(text):
print(text)

echo('this is a test')
</pre>

<pre>
## Counting function calls
def count(f):
def wrapper(*args):
wrapper.counter += 1
print("Function called {count} times".format(count=wrapper.counter))
return f(*args)
wrapper.counter = 0
# change wrapper name, or it will just be 'wrapper' (for inspection)
wrapper.__name__ = f.__name__
return wrapper

@count
def donothing():
pass

donothing()
donothing()
print("Function called a total of {count} times".format(count=donothing.counter))
</pre>

<pre>
## Timing your code
import time

def timer(f):
def wrapper(*arg):
t = time.time()
ret = f(*arg)
print("Took {elapsedtime} seconds".format(elapsedtime=time.time()-t))
return ret
return wrapper

@timer
def fact(n):
prod = 1
while n > 0:
prod *= n
n -= 1
return prod

total = fact(10000)
</pre>

<pre>
## Synchronization
import threading

def synchronized(lock):
def wrap(f):
def wrapper(*args, **kwargs):
lock.acquire()
try:
return f(*args, **kwargs)
finally:
lock.release()
return wrapper
return wrap

mylock = threading.Lock()

@synchronized(mylock)
def myfunc():
pass
</pre>

<pre>
## Type checking
def myMethod(ID, name):
''' Old way '''
if not (isinstance(ID, 'uint') and isinstance(name, 'utf8string')):
raise Exception()

def accepts(*typeargs):
''' Decorator way '''
def wrap(f):
def wrapper(*args):
for a, t in zip(args, typeargs):
if not isinstance(a, t):
raise Exception('{a} not of type {t}'.format(a=a, t=t))
return f(*args)
return wrapper
return wrap

@accepts(int)
def add1(n):
return n + 1
</pre>

<pre>
## Memoize
def memoize(f):
def wrapper(*args):
if not wrapper.ret:
wrapper.ret = f(*args)
return wrapper.ret
wrapper.ret = False
wrapper.__name__ = f.__name__
return wrapper

import time

@memoize
def sumto10():
count = 0
sum = 0
while count < 10:
sum += count
time.sleep(1)
count += 1
return sum

# first time will count slow
print sumto10()
# second time will return cached value instantly
print sumto10()
</pre>

Memoize: "Caches a function's return value each time it is called. If called later with the same arguments, the cached value is returned (not reevaluated)" [http://wiki.python.org/moin/PythonDecoratorLibrary#Memoize]

<pre>
## Nested decorators
import time
@shout
@memoize
def test1():
print "calculating"
time.sleep(3)
return 100
</pre>

Order is important! For tested decorators the execution order is from the top to the bottom. If you reverse @shout and @memoize, the @shout will not run because of the way @memoize was written.

<pre>
## registry
registry = []
# can also decorate decorators:
@shout
def registry(f):
registry.append(f)
return f

@register
def test2():
pass
</pre>

Decorator is just a function that returns a function.

Can use functools to automatically update decorator wrappers with __name__, etc [http://docs.python.org/2/library/functools.html]
<pre>
## Auto syntax sygar the wrapper for you
import functools
def my_decorator(f):
@functools.wraps(f)
def wrapper(*args, **kwds):
print 'Calling decorated function'
return f(*args, **kwds)
return wrapper
</pre>

=== Class Method Decorators ===

Static methods: [http://docs.python.org/library/functions.html#staticmethod]
: Similar to C++ and Java static methods
<pre>
class C:
@staticmethod
def say(msg):
print(msg)

C.say('hi')
</pre>

Class method: [http://docs.python.org/library/functions.html#classmethod]
: A class method receives the class as implicit first argument, just like an instance method receives the instance.
: Class methods are different than C++ or Java static methods. If you want those, see staticmethod() in this section.
<pre>
@classmethod
def setup_parser(cls, parser):
</pre>

See also [[#Class Properties]]

=== Class Properties ===

Properties: (not using decorators)
<pre>
class C(object):
def __init__(self):
self._x = None
def getx(self):
print("get x")
return self._x
def setx(self, value):
print("set x")
self._x = value
def delx(self):
print("del x")
del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")
</pre>

<pre>
#usage
c = C()
c.x = 5
print(c.x)
del(c.x)
</pre>

Getter and Setter Property Decorators: [http://docs.python.org/library/functions.html#property]
<pre>
class C(object): # make sure to inherit from object for these properties to work!
def __init__(self):
self._x = None

@property
def x(self):
"""I'm the 'x' property."""
print("get x")
return self._x

@x.setter # python >= 2.6
def x(self, value):
print("set x")
self._x = value

@x.deleter # python >= 2.6
def x(self):
print("del x")
del self._x
</pre>

<pre>
#usage
c = C()
c.x = 5
print(c.x)
del(c.x)
</pre>

Note: If you don't add the @x.setter, the property will be read only, and will throw a "AttributeError: can't set attribute" if attempted to be set.

Using x.getter example: [http://wiki.python.org/moin/AlternativeDescriptionOfProperty]
<pre>
class Cls(object):
def __init__(self):
self._x = 100

x = property()

@x.getter
def x(self):
"""I'm the 'x' property."""
return self._x

@x.setter
def x(self, value):
self._x = value

@x.deleter
def x(self):
del self._x
</pre>

=== Custom Decorators ===

decorator
@dec1
def func(arg1, arg2, ...): pass

equivalent to:
def func(arg1, arg2, ...): pass
func = dec1(func)

Basic layout (names are arbitrary):
<pre>
def my_decorator(function_name):
# update wrapper.__doc__ and .func_name
# or @functools.wraps(my_wrapper)
def mywrapper(*args, **kw):
# do something before
result = function_name(*args, **kw)
# do something after
return result
return my_wrapper
</pre>

Example:
<pre>
def limit(length):
def decorator(function):
def wrapper(*args, **kw):
result = function(*args, **kw)
result = result[:length]
return result
return wrapper
return decorator
</pre>

<pre>
@limit(3)
def echo(foo): return foo

echo('123456') # '123'
</pre>

=== assert ===

assert True
assert False # exception
assert 1 == 1
assert 1 == 2, "some message" # AssertionError: some message

=== Snake Case ===

Convert Camel Case to Snake Case: [http://stackoverflow.com/questions/1175208/elegant-python-function-to-convert-camelcase-to-snake-case]

<pre>
def snake_case(name):
s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name)
return re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1).lower()
</pre>

<pre>
>>> convert('CamelCase')
'camel_case'
</pre>

=== Camel Case ===

Camel Case: [http://stackoverflow.com/questions/8347048/camelcase-every-string-any-standard-library]

>>> ''.join(x for x in 'make IT camel CaSe'.title() if not x.isspace())
'MakeItCamelCase'

To titleize a sentence:
>>> "this is a test".title()
"This Is A Test"

=== Lambda ===

Simple functions
mul = lambda a, b: a * b

# equivalent to:
def mul(a, b):
return a * b

Sort usage:
data = [dict(number=x) for x in '036149']
data.sort(key=lambda x: float(x['number']))

Compare usage:
data = [dict(number=x) for x in '036149']
data.sort(cmp=lambda x,y: cmp(x['number'], y['number']))

Boolean:
lambda x: x % 3 == 0

Multiple parameter:
lambda x, y: x + y

Call another function:
lambda word: len(word)

Strings:
lambda line: line.split()[2]

Embedded:
def transform(n):
return lambda x: x + n

No parameters:
lambda: 1

Default parameters:
lambda x=1: x+1

References:
* Python: Lambda Functions - http://www.secnetix.de/olli/Python/lambda_functions.hawk

=== Map ===

Apply a function to items of a sequence:
map(str, range(3)) # ['0', '1', '2']

Note: list comprehensions can replace map and filter

=== Reduce ===

Apply a function to pairs of the sequence:
import operator
reduce(operator.mul, [1,2,3,4]) # 24 - ((1*2)*3)*4)

Note: sum or for loop can replace reduce

=== Filter ===

Return a sequence of items for which function(item) is True
filter(lambda x: x >= 0, [0, -1, 3]) # [0, 3]

Note: list comprehensions can replace map and filter

Find matching items in two lists:
a = ['a','b','c']
b = ['c','d']
filter(lambda x: x in a, b) # returns only 'c' as it is in both lists

=== List comprehension ===

Nothing more than Syntactic Sugar

5. Data Structures — Python v2.7.5 documentation - http://docs.python.org/2/tutorial/datastructures.html#list-comprehensions
: "List comprehensions provide a concise way to create lists."
: "A list comprehension consists of '''brackets''' containing an '''expression''' followed by a '''for clause''', then '''zero or more for or if clauses'''. "

Remove newlines from each item in a list:
mylist = [x.strip() for x in mylist]

Simple:
doubled = [x*2 for x in mylist]
squares = [x**2 for x in range(10)]

If:
[ 2*x for x in range(1,10) if x >= 5 ]

Complex:
[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]

Expanding PI:
[str(round(pi, i)) for i in range(1, 6)]

Return list of words
my_list = [current_word.lower() for current_word in words]

Utah Python July 2013 Meeting: (http://basepi.net)
<pre>
# Iterative
squares = []
for x in range(10):
squares.append(x**2)
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

## Functional
squares = map(lambda x: x**2, range(10))

## List comprehension
squares = [x**2 for x in range(10)]
even_squares = [x**2 for x in range(10) if x%2 == 0]

# get hash of strings in list:
import sha
strings = ['Python is cool!', 'I love python']
hashes = [sha.sha(x).digest() for x in strings]

# flatten matrix:
matrix = [[1, 2, 3], [4, 5, 6]]
[x for row in matrix for x in row]
# [1, 2, 3, 4, 5, 6]

# build matrix:
[(x,y) for x in range(10) for y in range(10)]
# matrix of not both odd and not both even
[(x,y) for x in range(10) for y in range(10) if x%2 != y%2]

# suck lines of files into list, ignoring commented lines
[line.strip().split("\t") for line in open("my_file.tab")
if not line.startswith('#')]
</pre>

References:
* 5.1.4. List Comprehensions - Data Structures — Python v2.7.2 documentation - http://docs.python.org/tutorial/datastructures.html#list-comprehensions
* Utah Python July 2013 Meeting - YouTube - https://www.youtube.com/watch?v=uh9uhzitdvw [http://basepi.net]

=== [http://docs.python.org/reference/expressions#calls Mass Parameter Passing] ===

Using: *args and **kw
* *args a tuple of remaining unnamed parameters
* **kw a dictionary of remaining named parameters

def param_func(a, b='b', *args, **kw):
print( [a, b, args, kw] )

WARNING: If not careful, this can result in this error: "TypeError: param_func() got multiple values for keyword argument 'myvar'". Might be better to just use:

def param_func(*args, **kw):
print( [args, kw] )

param_func(1, 2, 3, 4, 5) # [1, 2, (3, 4, 5), {}]
param_func(1, 2, 3, 4, 5, joe='test') # [1, 2, (3, 4, 5), {'joe': 'test'}]
mylist = ("a", "c", "g", "h", 5)
param_func(mylist) # [('a', 'c', 'g', 'h', 5), 'b', (), {}]
param_func(*mylist) # ['a', 'c', ('g', 'h', 5), {}] *tricky*
Note: *mydict will strip out just keys and treat like list
Note: **mydict will treat like named parameters

def mywrapper(*args, **kw):
# do something before
result = function_name(*args, **kw)

Passing arbitrary parameters:
<pre>
def fun2(**kw):
print kw # {'a': '1', 'b': '2', 'c': '3'}

def fun1(c='3', **kw):
print kw # {'a': '1', 'b': '2'}
fun2(c='3', **kw)

fun1(a='1', b='2')
</pre>

=== [http://www.python.org/dev/peps/pep-0234/ Iteration Protocol] ===

[http://www.python.org/dev/peps/pep-0234/ PEP 234 -- Iterators]

Making a class iterable:
<pre>
class mycount(object):
def __init__(self, max):
self.x = 0
self.max = max
def __iter__(self): # return object that implements __next__() or next()
return self
def next(self):
if self.x < self.max:
self.x += 1
return self.x
else:
self.x = 0
raise StopIteration()
</pre>

for i in mycount(10): print i # 1 ... 10
print [x for x in mycount(10) ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10)
for i in c: print i # 1 ... 10
[ x for x in c ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

=== Generators ===

Generators are functions that return to their previous state (using 'yield') when iterated over, which can be used in for loops.

==== Yield ====

Improve on Iterators:
* Infinite data (infinite iterators)
* Memory-intensive iteration
* Cooperative Multi-threading

Generators are functions that return to their previous state when iterated over, which can be used in for loops.

Note: Generators throw StopIteration when completed

Note: the 'yield' keyword will stash away the state of the function and return the value. (Continuations)

<pre>
def gen_range(end):
cur = 0
while cur < end:
yield cur
# return here next time
cur += 1

# for loop iteration
for i in gen_range(10): print i

# manual iteration
my_gen = gen_range(10)
try:
while True:
print my_gen.next()
except StopIteration:
pass

# list comprehension
numbers = [x for x in gen_range(10)]
</pre>

Note: one can place an empty 'return' in a generator if needed. A 'return' with argument is invalid.

Making a class generate:
<pre>
class mycount(object):
def __init__(self, max):
self.max = max
def __iter__(self):
return self.next() # change here
def next(self): # change here
x = 0
while x < self.max:
x += 1
yield x

for i in mycount(10): print i # 1 ... 10
print [x for x in mycount(10) ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10)
for i in c: print i # 1 ... 10
[ x for x in c ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

itr = mycount(10).next()
itr.next() # 1
itr.next() # 2 - eventually throws StopIteration exception
</pre>

Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
<pre>
* Iterators are used to traverse a list
* Generators are more powerful Iterators

# simple 'n' number in functional form
def firstn(n):
num = 0
nums = []
while num < n:
nums.append(num)
num += 1
return nums

# simple 'n' number in generator form
def firstn(n):
num = 0
while num < n:
yield num
num += 1

-

# Infinite list generator
def counter():
num = 0
while True:
yield num
num += 1

# Fibbonacci generator
def fib():
num1 = 0
num2 = 1
yield num1
while True:
num1, num2 = num2, num1 + num2
yield num1

# Fibbonacci generator 2
def fib(n):
num1 = 0
num2 = 1
count = 0
yield num1
while count < n:
count += 1
num1, num2 = num2, num1 + num2
yield num1
</pre>

Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
<pre>
## Cooperative Multi-Threading:
# Cooperative multithreading relies on the threads themselves to relinquish control once they are at a stopping point.

def mythread(name):
print('Starting Thread: {name}...'.format(name=name))
counter = 10
while counter > 0:
print('Thread: {name} doing some work!'.format(name=name))
counter -= 1
yield

thread1 = mythread('Alice')
thread2 = mythread('Bob')
thread3 = mythread('Trudy')

try:
while True:
thread1.next()
thread2.next()
thread3.next()
except StopIteration:
pass
</pre>

References:
* Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
** http://files.basepi.net/talks/pyarmory.zip

=== dunder methods ===

Some dunder (double under) methods determine what will happen when + (__add__) or other similar numerical symbols used.

<pre>
class str_add(object):
def __init__(self, name):
self.name = name
def __add__(self, other):
self.name += other.name
return self
# note: only need to return self if you want to go beyond 'a + b'

a = str_add('a_')
b = str_add('b_')
c = str_add('c_')
d = a + b + c
print(a.name) # a_b_c_
print(b.name) # b_
print(c.name) # c_
print(d.name) # a_b_c_
</pre>

Other common methods to override:
* __add__ for +
* __sub__ for -
* __mul__ for *
* __div__ for /
* __mod__ for %

References:
* Emulating numeric types - http://docs.python.org/2/reference/datamodel.html#emulating-numeric-types

=== [http://docs.python.org/library/doctest.html doctest] ===

test.py:
<pre>
#!/usr/bin/env python
"""
This is the "example" module.

>>> double(5)
10
"""

def double(n):
"""
This is double function

>>> double(3)
6
"""
return n * 2

if __name__ == "__main__":
import doctest
doctest.testmod()
</pre>

=== unittest ===

unittest.TestCase

Here is a short script to test three functions from the random module:
<pre>
import random
import unittest

class TestSequenceFunctions(unittest.TestCase):

def setUp(self):
self.seq = range(10)

def test_shuffle(self):
# make sure the shuffled sequence does not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, range(10))

# should raise an exception for an immutable sequence
self.assertRaises(TypeError, random.shuffle, (1,2,3))

def test_choice(self):
element = random.choice(self.seq)
self.assertTrue(element in self.seq)

def test_sample(self):
with self.assertRaises(ValueError):
random.sample(self.seq, 20)
for element in random.sample(self.seq, 5):
self.assertTrue(element in self.seq)

if __name__ == '__main__':
unittest.main()
</pre>

The unittest module can be used from the command line to run tests from modules, classes or even individual test methods:
python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method
python -m unittest -v test_module # verbose
python -m unittest -h # help

References:
* 25.3. unittest — Unit testing framework — Python v2.7.4 documentation - http://docs.python.org/2/library/unittest.html
* Python Debugging Tips - Google Drive - https://docs.google.com/presentation/d/130Wl--IzbDh9ymF6KHMu0WrsIwz6k32pav1gFbsw7NI/edit?usp=sharing

=== Internet Access ===

<pre>
# Python 3
from urllib.request import urlopen
for line in urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'):
line = line.decode('utf-8')
if 'EST' in line or 'EDT' in line: # look for eastern time
print(line)
</pre>

=== Simple SMTP ===

Requires local STMP server.

<pre>
import smtplib
server = smtplib.SMTP("localhost")
server.sendmail('mrken@demo.oeey.com', 'kenneth@demo.oeey.com',
"""To: kenneth@demo.oeey.com
From: mrken@demo.oeey.com
Subject: This is fun

Beware of the dog.
""")
server.quit()
</pre>

<pre>
s = smtplib.SMTP(smtp_server)
</pre>

with TLS:
<pre>
s = smtplib.SMTP("smtp.gmail.com",587)
s.ehlo()
s.starttls()
s.ehlo()
</pre>

SSL:
<pre>
s = smtplib.SMTP_SSL('smtp.mail.yahoo.com', 465)
</pre>

=== Data Compress ===

zlib, gzip, bz2, zipfile, tarfile

<pre>
import zlib
s = b'hello world' # only in python >3
len(s)
t = zlib.compress(s)
zlib.decompress(t)
zlib.crc32(s)
</pre>

=== gzip ===

<pre>
import gzip
content = "Lots of content here"
f = gzip.open('file.txt.gz', 'wb')
f.write(content)
f.close()
</pre>

<pre>
import gzip
f_in = open('file.txt', 'rb')
f_out = gzip.open('file.txt.gz', 'wb')
f_out.writelines(f_in)
f_out.close()
f_in.close()
</pre>

=== Performance Timer ===

Time a task repeated 10 times:
<pre>
from timeit import Timer
Timer('t=a; a=b; b=t', 'a=1; b=2').timeit()
Timer('print("hi")').timeit(10) # repeat 10 times and time
</pre>

Simple method:
<pre>
start_time = int(time.time())
... task ...
compute_time = int(time.time()) - start_time
</pre>

=== Python Version Check ===

Solution:

__init__.py:
<pre>
# Check Python Version:
import sys
assert sys.version_info >= (2, 6, 0), 'requires Python 2.6 or higher'
del sys
</pre>

resulting error if not > 2.6:
<pre>
>>> import nucleon
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "nucleon/__init__.py", line 9, in ?
assert sys.version_info >= (2, 6, 0), 'nucleon requires Python 2.6 or higher'
AssertionError: nucleon requires Python 2.6 or higher
</pre>

Other solutions:

Note: this will throw syntax error on 2.4 due to ternary
<pre>
import sys
if sys.version_info < (2, 4):
raise "must use python 2.5 or greater"
else:
# syntax error in 2.4, ok in 2.5
x = 1 if True else 2
print x
</pre>

maybe solution:
<pre>
try:
eval("1 if True else 2")
except SyntaxError:
# doesn't have ternary
</pre>

<pre>
>>> try:
... from my_module import twoPointSixCode
... except Exception:
... print "can't import, probably because your python is too old!"
>>>
</pre>

<pre>
#!/usr/bin/env/python
import sys
if sys.version_info<(2,6,0):
sys.stderr.write("You need python 2.6 or later to run this script\n")
exit(1)
</pre>

<pre>
>>> import sys
>>> sys.hexversion # hex printed as integer
33817584
>>> '%x' % sys.hexversion # output as hex, not zero fill 2.4.3
'20403f0'
>>> sys.hexversion < 0x02060000 # test if version less than 2.6
True
>>> print "0x%08x" % sys.hexversion # print as hex and zero fill and prepend '0x'
0x020403f0
</pre>

<pre>
import sys
if sys.hexversion < 0x02060500:
print >> sys.stderr, "This script requires python 2.6.5 or higher."
exit(1)
</pre>

<pre>
>>> import sys
>>> sys.version_info
(2, 6, 4, 'final', 0)
>>> if not sys.version_info[:2] == (2, 6):
... print "Error, I need python 2.6"
... else:
... from my_module import twoPointSixCode
>>>
</pre>

References:
* http://stackoverflow.com/questions/446052/python-best-way-to-check-for-python-version-in-program-that-uses-new-language-fe
* http://stackoverflow.com/questions/1093322/how-do-i-check-what-version-of-python-is-running-my-script

=== [http://docs.python.org/library/struct.html Struct - Binary Data Record Layout] ===

Format Characters:
<pre>
Format C Type Python type Standard size
x pad byte no value
c char str of len 1 1
b signed char integer 1
B unsigned char integer 1
? _Bool bool 1
h short integer 2
H unsigned short integer 2
i int integer 4
I unsigned int integer 4
l long integer 4
L unsigned long integer 4
q long long integer 8
Q unsigned long long integer 8
f float float 4
d double float 8
s char[] string
p char[] string
P void * integer
</pre>

H # two byte unassigned number
I # four byte unassigned number
< # standard size and in little-endian byte order

import struct
data = struct.pack('<IIIHH', 1, 2, 3, 4, 5)
# data = '\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x05\x00'

fields = struct.unpack('<IIIHH', data)
# fields = (1L, 2L, 3L, 4, 5)
# using splice: fields = struct.unpack('<IIIHH', data[start:start+16])

f1, f2, f3, f4, f5 = fields

hex(f1) # hex a field:
# '0x1L'

>>> struct.pack("<5sh", "helloo", 11)
'hello\x0b\x00'
>>> struct.unpack("<5sh", 'hello\x0b\x00')
('hello', 11)

=== pdb interactive debugger ===

# uncomment the following to use the interactive debugger:
# import pdb; pdb.set_trace()

=== [http://docs.python.org/library/telnetlib.html Telnet] ===

create:
class telnetlib.Telnet([host[, port[, timeout]]])

Note: The instance is initially not connected by default; the open() method must be used to establish a connection. Alternatively, the host name and optional port number can be passed to the constructor, to, in which case the connection to the server will be established before the constructor returns.

open if not initially done so:
Telnet.open(host[, port[, timeout]])

Functions:
Telnet.read_until(expected[, timeout]) # Read until a given string, expected, is encountered or until timeout seconds have passed.
# on timeout return whatever is in queue, possibly empty string
Telnet.close() # close connection
Telnet.read_all() # Read all data until EOF; block until connection closed.
Telnet.write(buffer) # Write a string to the socket, doubling any IAC characters. This can block if the connection is blocked.
Telnet.expect(list[, timeout]) # Read until one from a list of a regular expressions matches.
# Return a tuple of three items: the index in the list of matches; the match object; and the text read to match.

Read until end of line:
line = tn.read_until("\n")

<pre>
import getpass
import sys
import telnetlib

HOST = "localhost"
user = raw_input("Enter your remote account: ")
password = getpass.getpass()

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ")
tn.write(user + "\n")
if password:
tn.read_until("Password: ")
tn.write(password + "\n")

tn.write("ls\n")
tn.write("exit\n")

print tn.read_all()
</pre>

<pre>
#!/usr/bin/env python

import sys
import telnetlib

HOST = "216.119.193.252"
USER = "admin"
PASS = "@lt12345"

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ")
tn.write(USER + "\n")
tn.read_until("Password: ")
tn.write(PASS + "\n")
tn.read_until("# ")

tn.write("terminal length 0\n")
tn.read_until("# ")

tn.write("show running-config\n")
configlines = tn.read_until("# ")

tn.write("exit\n")
tn.read_all()

print(configlines)
</pre>

=== SSH ===

Cheap method:
os.system('ssh user@server command')

SecureShell - PythonInfo Wiki - http://wiki.python.org/moin/SecureShell
: There are several ways to use Secure Shell (SSH) in Python
* pyssh - http://pyssh.sourceforge.net/
* paramiko - Paramiko is a native Python implementation of SSH. - http://www.lag.net/paramiko/
* conch - conch is another native implementation of SSH and part of the Twisted Matrix project - http://twistedmatrix.com/projects/conch
* SSH wrapper - Some tools just wrap around existing ssh/sftp implementations
** keyphrene - http://www.keyphrene.com/products/org.keyphrene/
** keyphrene SSH wrapper - http://membres.lycos.fr/fredp/python/pyscp.html
* Fabric - http://docs.fabfile.org/en/1.4.0/index.html
* pexpect - Noah.org - http://www.noah.org/wiki/Pexpect

SFTP:
* SFTP in Python: Really Simple SSH | The best of Zeth - http://zeth.net/post/332/

pexpect:
wget http://pexpect.sourceforge.net/pexpect-2.3.tar.gz
tar xzf pexpect-2.3.tar.gz
cd pexpect-2.3
sudo python ./setup.py install

=== SFTP ===

SFTP in Python? (platform independent) - Stack Overflow - http://stackoverflow.com/questions/432385/sftp-in-python-platform-independent

paramiko: ssh2 protocol for python - http://www.lag.net/paramiko/

pip install paramiko

mkdir -p ~/.src ; cd ~/.src
wget http://www.lag.net/paramiko/download/paramiko-1.7.7.1.zip
unzip paramiko-1.7.7.1.zip
cd paramiko-1.7.7.1
python setup.py install

SFTP Methods:
* chdir
* chmod
* chown
* close
* file
* from_transport
* get
* get_channel
* getcwd
* listdir
* listdir_attr
* logger
* lstat
* mkdir
* normalize
* open
* put
* readlink
* remove
* rename
* request_number
* rmdir
* sock
* stat
* symlink
* truncate
* ultra_debug
* unlink
* utime

<pre>
import paramiko

host = "THEHOST.com" #hard-coded
port = 22
transport = paramiko.Transport((host, port))

password = "THEPASSWORD" #hard-coded
username = "THEUSERNAME" #hard-coded
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

import sys
path = './THETARGETDIRECTORY/' + sys.argv[1] #hard-coded
localpath = sys.argv[1]
sftp.put(localpath, path)

sftp.close()
transport.close()
print 'Upload done.'
</pre>

VMware Example:
<pre>
#!/usr/bin/env python

import paramiko
import sys

host = "sftp2.engx.vmware.com"
port = 443
transport = paramiko.Transport((host, port))

username = "xxxx"
password = "xxx"
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

print dir(sftp)

print sftp.chdir("build-uploader")
print sftp.listdir()

sftp.close()
transport.close()
</pre>

=== paramiko ===

See [[Python/paramiko]]

=== [http://docs.python.org/library/getpass.html Get Password] ===

Prompt for password:
getpass.getpass(prompt='Password: ')

import getpass
user = raw_input("Enter your remote account: ")
password = getpass.getpass()

=== [http://docs.python.org/library/configparser.html ini files] ===

<pre>
ini_filename = 'my.ini'

import ConfigParser
parser = ConfigParser.SafeConfigParser()
parser.read(ini_filename)

sections = parser.sections() # get sections
for section in sections:
print "[" + section + "]"
for name, value in parser.items(section):
print name + " = " + value
</pre>

<pre>
if parser.has_option(section, option):
print( parser.get(section, option) )
</pre>

To have the option names not convert to lower case:
parser.optionxform = str

Ignore section: [http://stackoverflow.com/questions/2885190/using-pythons-configparser-to-read-a-file-without-section-name]
ini_str = '[root]\n' + open(ini_path, 'r').read()
ini_fp = StringIO.StringIO(ini_str)
config = ConfigParser.RawConfigParser()
config.readfp(ini_fp)

Get option:
config.has_option('root', 'some_key')
config.get('root', 'some_key')

config.items('root') # list of Key Value tuples

=== optparse - Option Parser ===

[http://docs.python.org/library/optparse.html - Option Parser

Note: Deprecated since version 2.7: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

<pre>
import optparse
parser = optparse.OptionParser()

parser.add_option('--name', default=None)

parser.add_option(
'-l', '--loglevel',
help='Log Level (default: info)',
default="info", action='store', type='string')

(options, args) = parser.parse_args()

# if no parameters passed show help
if len(sys.argv) == 1:
parser.print_help()
sys.exit(1)

# if a stand alone parameter passed, override option
if args:
options.target = args[0]

# show option:
print(options.loglevel)
</pre>

Note: Order is important when listing add_options!

Pretty Description:
parser = optparse.OptionParser(description="Purpose: to do something")
# or
parser.description = "Purpose: to do something"

Parse arguments:
(options, args) = parser.parse_args()
Note: If you like, you can pass a custom argument list to parse_args(), but that’s rarely necessary: by default it uses sys.argv[1:].

Change Usage Message:
<pre>
usage = "%prog [options]" # default message
# Usage: <prog> [options]" # where <prog> is replaced with the script name

parser = OptionParser(usage=usage)
# or
parser.usage = usage # default message
# or
parser.set_usage(usage)
</pre>

Remove option: (does not remove default value from 'options' variable!!! - stupid!)
parser.remove_option('--loglevel')

Modify an option: (changes are saved to parser automatically, unless in an option group)
option = parser.get_option('--loglevel')
option.help = 'something else'
option.default = 'debug'

# or
parser.remove_option("--loglevel")
parser.add_option("--loglevel", help="something else", default="info")

Modify an option in an option group:
opt_group = parser.get_option_group("--loglevel")
opt_group.remove_option("--loglevel")
opt_group.add_option("--loglevel", help="something else", default="info")

Types: "string", "int", "long", "choice", "float" and "complex" [http://docs.python.org/library/optparse.html#optparse-standard-option-types]
Defaults:
* Default actions is 'store'.
* Default type is 'string'.
* Default dest is the long name of the parameter (or short name if no long name provided)
* Default value is None

<pre>
from optparse import OptionParser
[...]
parser = OptionParser()

# string
parser.add_option("-f", "--file", dest="filename",
help="write report to FILE", metavar="FILE")

# boolean - action is what to do when flag found
parser.add_option("-q", "--quiet",
action="store_true", dest="verbose", default=False,
help="don't print status messages to stdout")

# integer
parser.add_option(
"-m", "--minutes",
help="Minutes (default: 60)",
default=60, type='int')

(options, args) = parser.parse_args()

print "Filename: " + options.filename
</pre>

fake handling of required option:
<pre>
if not option.filename:
parser.print_help()
parser.error("filename is required")
</pre>

change default usage:
parser = OptionParser(usage="Usage: %prog [args] arg1 arg2")

Really ugly way to print help usage :-(
<pre>
def Usage():
pname = sys.argv[0]
cmd='%s -h' % (pname)
os.system(cmd)

if (len(sys.argv) < 2):
Usage()
sys.exit(1)
</pre>

Groups:
<pre>
# OptionGroup(parser, title, description=None)
group = OptionGroup(parser, "Dangerous Options",
"Caution: use these options at your own risk. "
"It is believed that some of them bite.")
group.add_option("-g", action="store_true", help="Group option.")
parser.add_option_group(group)
</pre>

Hide an option:
* Option.help
: "Help text to print for this option when listing all available options after the user supplies a help option (such as --help). If no help text is supplied, the option will be listed without help text. To hide this option, use the special value optparse.SUPPRESS_HELP."
<pre>
parser.add_option("--secret", help=SUPPRESS_HELP)
</pre>

Issues with Option Parser:
* No handling for positional arguments
* No adding help for positional arguments
* No handling for required field

=== argparse ===

argparse - http://docs.python.org/library/argparse.html

argparse — Parser for command-line options, arguments and sub-commands

NOTE: MUCH MORE POWERFUL THAN OPTION PARSER!

Note: New in version 2.7.

Installation on <= 2.6
<pre>
# Install argparse: - http://code.google.com/p/argparse/
# REQUIRES SETUPTOOLS
mkdir -p ~/.src ; cd ~/.src
wget http://argparse.googlecode.com/files/argparse-1.2.1.tar.gz
tar -zvxf argparse-1.2.1.tar.gz
cd argparse-1.2.1
sudo python setup.py install
</pre>

The argparse module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse will figure out how to parse those out of sys.argv. The argparse module also automatically generates help and usage messages and issues errors when users give the program invalid arguments.

<pre>
import argparse

parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
const=sum, default=max,
help='sum the integers (default: find the max)')

args = parser.parse_args()
print args.accumulate(args.integers)
</pre>

<pre>
import argparse

argparser = demoproject.util.argparse.ArgumentParser(
usage='Kick off demoproject benchmarks.')
argparser.add_argument(
'--card',
help='What cards to run on. Depends on Jenkins NODE_LABEL environment'
'variable for correct mappings.')
argparser.add_argument(
'--suite', metavar='SUITE',
help='Specify the appropriate benchmark suite as defined in '
'the benchmark ini file. Defaults to standard_perf',
default='standard_perf')
args = argparser.parse_args()

print args.card
</pre>

Remove an argument: [http://stackoverflow.com/questions/32807319/disable-remove-argument-in-argparse]
<pre>
def remove_options(parser, options):
""" remove_options(parser, ['--some-option', '--some-option2']) """
for option in options:
for action in parser._actions:
if vars(action)['option_strings'][0] == option:
parser._handle_conflict_resolve(None,[(option,action)])
break
</pre>

=== [http://docs.python.org/library/commands.html commands] ===

35.16. commands — Utilities for running commands — Python v2.7.1 documentation - http://docs.python.org/library/commands.html

Note: Deprecated since version 2.6: The commands module has been removed in Python 3.0. Use the subprocess module instead.

The commands module contains wrapper functions for os.popen() which take a system command as a string and return any output generated by the command and, optionally, the exit status.

Get status and output:
status, output = commands.getstatusoutput(cmd)

Get output ignoring status:
output = commands.getoutput(cmd)

NOTE: DO NOT USE getstatus() AS IT DOES NOT DO WHAT YOU THINK IT DOES!
status = commands.getstatus(file)

<pre>
>>> import commands

>>> commands.getstatusoutput('ls /bin/ls')
(0, '/bin/ls')

>>> commands.getstatusoutput('cat /bin/junk')
(256, 'cat: /bin/junk: No such file or directory')

>>> commands.getstatusoutput('/bin/junk')
(256, 'sh: /bin/junk: not found')

>>> commands.getoutput('ls /bin/ls')
'/bin/ls'

>>> commands.getstatus('/bin/ls')
'-rwxr-xr-x 1 root 13352 Oct 14 1994 /bin/ls'
</pre>

=== [http://docs.python.org/library/_winreg.html Windows Registry] ===

Note: The _winreg module has been renamed to winreg in Python 3.0.

* 32.3. winreg – Windows registry access — Python v3.2 documentation - http://docs.python.org/py3k/library/winreg.html
* 34.3. _winreg – Windows registry access — Python v2.7.1 documentation - http://docs.python.org/library/_winreg.html

Read from registry:
<pre>
from _winreg import *
key = OpenKey(HKEY_LOCAL_MACHINE, r'Software\Microsoft\Outlook Express', 0, KEY_ALL_ACCESS)
QueryValueEx(key, "InstallRoot")
</pre>

Write to registry:
<pre>
keyVal = r'Software\Microsoft\Internet Explorer\Main'
try:
key = OpenKey(HKEY_CURRENT_USER, keyVal, 0, KEY_ALL_ACCESS)
except:
key = CreateKey(HKEY_CURRENT_USER, keyVal)
SetValueEx(key, "Start Page", 0, REG_SZ, "http://www.blog.pythonlibrary.org/")
CloseKey(key)
</pre>

Enumerate registry keys:
<pre>
from _winreg import EnumKey, HKEY_USERS

try:
i = 0
while True:
subkey = EnumKey(HKEY_USERS, i)
print subkey
i += 1
except WindowsError:
# WindowsError: [Errno 259] No more data is available
pass
</pre>

Another enumeration example:
<pre>
# File: winreg-example-1.py

import _winreg

explorer = _winreg.OpenKey(
_winreg.HKEY_CURRENT_USER,
"Software\\Microsoft\\Windows\\CurrentVersion\\Explorer"
)

# list values owned by this registry key
try:
i = 0
while 1:
name, value, type = _winreg.EnumValue(explorer, i)
print repr(name),
i += 1
except WindowsError:
print

value, type = _winreg.QueryValueEx(explorer, "Logon User Name")

print
print "user is", repr(value)
</pre>

Enable Remote Desktop Example: (by Kenneth)
<pre>
from _winreg import *
key = OpenKey(HKEY_LOCAL_MACHINE, r'SYSTEM\CurrentControlSet\Control\Terminal Server', 0, KEY_ALL_ACCESS)
SetValueEx(key, "fDenyTSConnections", 0, REG_DWORD, 0)
ValueTypeTuple = QueryValueEx(key, "fDenyTSConnections") # (0, 4)
print("Please reboot the system to complete the changes")
</pre>

References:
* Python’s _winreg: Editing the Windows Registry « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/03/20/pythons-_winreg-editing-the-windows-registry/

=== JSON ===

JSON - http://docs.python.org/library/json.html

ENCODING TO JSON STRING (type str)

import json
json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
# '["foo", {"bar": ["baz", null, 1.0, 2]}]'

compact encoding:
import json
json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',',':'))
# '[1,2,3,{"4":5,"6":7}]'

Pretty printing:
import json
print json.dumps({'4': 5, '6': 7}, sort_keys=True, indent=4)
# {
# "4": 5,
# "6": 7
# }

DECODING JSON STRING TO PYTHON LIST OR DICTIONARY OBJECT

import json
json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]')
# [u'foo', {u'bar': [u'baz', None, 1.0, 2]}]

Load from file:
f = open(filename)
j = json.load(f)
print j["kind"] # "dual-controller-adapter"
#{
# "kind": "dual-controller-adapter",
#}

From Command Line:
echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool

# get value
echo '{"test":1,"test2":2}' | python -c 'import sys,json;data=json.loads(sys.stdin.read()); print data["test"]'

# read large file
python -c "import json;f=open('file.json');data=json.load(f);print json.dumps(data,indent=2)"
python -c "import json;print json.dumps(json.load(open('file.json')),indent=2)"

Load from web:
<pre>
import httplib
import json

def get_http_json(server, path):
""" This performs an HTTP GET against a server and path and returns the
resulting page data.

server The server we're hitting, e.g. demo.com
path The path we're hitting, e.g. api/part/MA12345/
"""
connection = httplib.HTTPConnection(server)
try:
url = 'http://%s/%s' % (server, path)
print 'Querying url %s' % url
connection.request('GET', url)
response = connection.getresponse()
if response.status == httplib.OK:
return json.loads(response.read())
else:
print 'Unable to get %s. %s %s' % (url, response.reason,
response.status)
return None
finally:
connection.close()

j = get_http_json(server, path)
</pre>

Pretty printing from the command line:
echo '{"json":"obj"}' | python -mjson.tool
#{
# "json": "obj"
#}

References:
* 18.2. json — JSON encoder and decoder — Python v2.7.1 documentation - http://docs.python.org/library/json.html

=== Sockets ===

See [[Python/Sockets]]

=== [http://docs.python.org/library/base64.html base64] ===

import base64

Encode:
encoded = base64.b64encode('data to be encoded')

Decode:
base64.b64decode(encoded)

=== daemonize ===

Creating a daemon the Python way « Python recipes « ActiveState Code - http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/

<pre>
import os
import sys

try:
pid = os.fork()
except OSError, e:
raise Exception, "%s [%d]" % (e.strerror, e.errno)

if (pid != 0): # parent
os._exit(0)

# create new process session
os.setsid()

# redirect stdin, stdout, stderr
os.open('/dev/null', os.O_RDWR)
os.dup2(0, 1)
os.dup2(0, 2)

#... tasks ...
</pre>

My Solution:
<pre>
import os
import sys
import signal
import syslog

HUP = False

def log(msg):
syslog.syslog(str(msg))
syslog.openlog('[WATER]')

def kill_signal_handler(signal, frame):
log("Being killed off: {0} {1}".format(signal, frame))
log("Shutting Down!")
sys.exit(0)
signal.signal(signal.SIGTERM, kill_signal_handler)
signal.signal(signal.SIGALRM, kill_signal_handler)

def hup_signal_handler(signal, frame):
global HUP
log("Signal reload received: {0} {1}".format(signal, frame))
HUP=True
signal.signal(signal.SIGHUP, hup_signal_handler)

def main_loop():
global HUP

while True:
# ... handle HUP=True ...

# ... main code ...

# wait 60 seconds before we start the next loop
time.sleep(60) # signals will interrupt this

if __name__ == "__main__":
log("Starting Daemon...")
child_id = os.fork()
if child_id:
log("Daemonized!")
sys.exit(0)
os.setsid()
# redirect stdin, stdout, stderr
os.open('/dev/null', os.O_RDWR)
os.dup2(0, 1)
os.dup2(0, 2)

# write pid
with open("water.pid", "w") as f:
f.write(str(os.getpid()))

while True:
try:
main_loop()
except Exception as e:
log("Critical Failure Caught: %s" % str(e))
log("Critical Failure - Restarting!")
time.sleep(30)
#raise e
</pre>

ln -s /opt/pyton27/bin/python pyton-water
./python-water water.py
killall python-water

==== fork ====

<pre>
import os
import sys
import time

child_id = os.fork()
if child_id:
print "parent exiting"
sys.exit(0)

while True:
time.sleep(3)
print "child"
</pre>

For windows, to hide the console, see [[#Hide Console]]

=== csv ===

13.1. csv — CSV File Reading and Writing — Python v2.7.2 documentation - http://docs.python.org/library/csv.html

Read:
<pre>
import csv
spamReader = csv.reader(open('eggs.csv', 'rb'), delimiter=' ', quotechar='|')
for row in spamReader:
print ', '.join(row)
</pre>

Write:
<pre>
import csv
spamWriter = csv.writer(open('eggs.csv', 'wb'), delimiter=' ',
quotechar='|', quoting=csv.QUOTE_MINIMAL)
spamWriter.writerow(['Spam'] * 5 + ['Baked Beans'])
spamWriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam'])
</pre>

=== SQLite ===

11.13. sqlite3 — DB-API 2.0 interface for SQLite databases — Python v2.7.2 documentation - http://docs.python.org/library/sqlite3.html

sqlite3 - New in version 2.5.

<pre>
import sqlite3

conn = sqlite3.connect('/tmp/example')

c = conn.cursor()

# Create table
c.execute('''create table stocks
(date text, trans text, symbol text,
qty real, price real)''')

# Insert a row of data
c.execute("""insert into stocks
values ('2006-01-05','BUY','RHAT',100,35.14)""")

# Show affected row count
print c.rowcount

# Save (commit) the changes
conn.commit()

# We can also close the cursor if we are done with it
c.close()
</pre>

parameter substitution:
<pre>
# Never do this -- insecure!
symbol = 'IBM'
c.execute("... where symbol = '%s'" % symbol)

# Do this instead
t = (symbol,)
c.execute('select * from stocks where symbol=?', t)
c.execute('select * from stocks where symbol=?', (symbol,))

# Larger example
for t in [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00),
('2006-04-06', 'SELL', 'IBM', 500, 53.00),
]:
c.execute('insert into stocks values (?,?,?,?,?)', t)
</pre>

select results:
<pre>
c = conn.cursor()
c.execute('select * from stocks order by price')

for row in c:
print row # row is a tuple

# OR (but not both)

rows = c.fetchall() # rows is list of tuples
print len(rows)
for row in rows:
print row
</pre>

Show which version of sqlite3: [http://zetcode.com/db/sqlitepythontutorial/]
<pre>
#!/usr/bin/python
# -*- coding: utf-8 -*-

import sqlite3 as lite
import sys

con = None

try:
con = lite.connect('test.db')

cur = con.cursor()
cur.execute('SELECT SQLITE_VERSION()')

data = cur.fetchone()

print "SQLite version: %s" % data

except lite.Error, e:

print "Error %s:" % e.args[0]
sys.exit(1)

finally:

if con:
con.close()
</pre>

Or simply: [http://stackoverflow.com/questions/1545479/force-python-to-forego-native-sqlite3-and-use-the-installed-latest-sqlite3-ver]
# echo -e "import sqlite3\nprint sqlite3.sqlite_version_info" | python
(3, 3, 6)

=== MySQL Database ===

Ubuntu packages:
* python-mysqldb - Python interface to MySQL
* python-mysql.connector - pure Python implementation of MySQL Client/Server protocol

apt-get install python-mysqldb
>>> import _mysql

apt-get install python-mysql.connector

PIP:
* https://pypi.python.org/pypi/MySQL-python
* https://pypi.python.org/pypi/mysqlclient

pip install MySQL-python

pip install mysqlclient

---

_mysql - http://mysql-python.sourceforge.net/MySQLdb.html

apt-get install python-mysqldb

<pre>
import _mysql

db=_mysql.connect(host="localhost",user="user",
passwd="pass",db="db")

db.query("""SELECT spam, eggs, sausage FROM breakfast
WHERE price < 5""")

r=db.store_result()
# ...or...
r=db.use_result()

row = r.fetch_row()
</pre>

=== couchdb ===

Getting started with couchdb-python — couchdb-python v0.8 documentation - http://packages.python.org/CouchDB/getting-started.html

<pre>
couch = couchdb.Server('http://127.0.0.1:5984/')

# database
db = couch.create('test') # create new
db = couch['mydb'] # or get existing

# print databases:
for db in couch:
print db

doc = {'foo': 'bar'}
doc_id, doc_rev = db.save(doc)
# (u'9278f7691c3421227a9f8436540008f9', u'1-4c6114c65e295552ab1019e2b046b10e')
# note: this also updates doc with _rev and _id
doc = db['9278f7691c3421227a9f8436540008f9']

# better to create a document using an id you want, instead of the randomly generated one
db[doc.id] = doc # this will write immediately to the database

# print all docs:
for id in db:
print id

# delete doc
db.delete(doc)

# delete database
couch.delete('test')
</pre>

Search database:
<pre>
# show all records
map_fun = '''function(doc) { emit(doc.name, null); }'''

# row = <Row id=u'12341234123412341234', key=None, value=None>
# emit(key, value) - returns the values for key and value

map_fun = '''
function(doc) {
if (doc.username == 'virt-01')
emit(doc.username, null);
}
'''

# row = <Row id=u'12341234123412341234', key=None, value=None>
for row in db.query(map_fun):
print row.id

for row in db.query(map_fun, descending=True):
print row.id
</pre>

couchdb-python - CouchDB Python Library - Google Project Hosting - http://code.google.com/p/couchdb-python/

Installation:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/C/CouchDB/CouchDB-0.8.tar.gz#md5=e32b39e459e1fa2586ea252712a11a59
tar -zvxf CouchDB-0.8.tar.gz
cd CouchDB-0.8
python setup.py install
</pre>

Test:
echo "import couchdb" | python

=== nose ===

nose: nose extends unittest to make testing easier

nose : Python Package Index - http://pypi.python.org/pypi/nose

Install:
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/n/nose/nose-1.1.2.tar.gz#md5=144f237b615e23f21f6a50b2183aa817
tar -zvxf nose-1.1.2.tar.gz
cd nose-1.1.2
sudo python setup.py install

skeleton:
mkdir skeleton
cd skeleton
mkdir bin NAME tests docs
touch NAME/__init__.py
touch tests/__init__.py
touch tests/NAME_tests.py

nose test skeleton:
<pre>
# tests/NAME_tests.py
from nose.tools import *
import klib

def setup():
print "SETUP!"

def teardown():
print "TEAR DOWN!"

def test_basic():
print "I RAN!"

# def test_something_else():
</pre>

Usage:
nosetests

See Also:
* [[#doctest]]
* [[#unittest]]

=== pydoc ===

25.1. pydoc — Documentation generator and online help system — Python v2.7.2 documentation - http://docs.python.org/library/pydoc.html

pydoc - The pydoc module automatically generates documentation from Python modules.

pydoc - same as 'import [library], help([library])'
$ pydoc [library]

Note: lists some special 'module metadata variables' separately (e.g. version, author).

special pydoc recognized metadata sections:
__version__
__author__
__credits__
__date__

==== pydoc function module ====

<pre>
Help on package klib:

NAME
klib - This is the name

FILE
/root/klib/klib/__init__.py

DESCRIPTION
These are module doc strings
that give helpful details

PACKAGE CONTENTS
markone

FUNCTIONS
func1()
This is a doc string
for func1

DATA
PI = 3.141
__all__ = ['func1', 'PI']
__author__ = 'Kenneth <kenneth@oeey.com>'
__copyright__ = 'Copyright 2007, The Project'
__credits__ = 'Credits go to me'
__date__ = '26 February 2001'
__email__ = 'ken@oeey.com'
__license__ = 'GPL'
__maintainer__ = 'Ken'
__status__ = 'Production'
__version__ = '$Revision: 83492 $'

VERSION
83492

DATE
26 February 2001

AUTHOR
Kenneth <kenneth@oeey.com>

CREDITS
Credits go to me
</pre>

Source:
<pre>
"""
This is the name

These are module doc strings
that give helpful details
"""

#__version__ = "1.0"
__version__ = "$Revision: 83492 $"
__author__ = "Kenneth <kenneth@oeey.com>"

__date__ = "26 February 2001"
__credits__ = "Credits go to me"

__all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license'
__license__ = "GPL"
__copyright__ = "Copyright 2007, The Project"
__maintainer__ = "Ken"
__email__ = "ken@oeey.com"
__status__ = "Production"

def func1():
"""
This is a doc string
for func1
"""
</pre>

==== pydoc class module ====

Sample:
<pre>
Help on module class_pydoc:

NAME
class_pydoc - This is the name

FILE
/root/class_pydoc.py

DESCRIPTION
These are module doc strings
that give helpful details

CLASSES
Toad

class Toad
| doc string for object
|
| Methods defined here:
|
| __init___(self, name)
| Constructor
|
| instance_function(self)
| Instance Function
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| class_function(cls) from __builtin__.classobj
| Class Function
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| my_class_attribute = 'sample class attribute'

DATA
PI = 3.141
__author__ = 'Kenneth <kenneth@oeey.com>'
__copyright__ = 'Copyright 2007, The Project'
__credits__ = 'Credits go to me'
__date__ = '26 February 2001'
__email__ = 'ken@oeey.com'
__license__ = 'GPL'
__maintainer__ = 'Ken'
__status__ = 'Production'
__version__ = '1.0'

VERSION
1.0

DATE
26 February 2001

AUTHOR
Kenneth <kenneth@oeey.com>

CREDITS
Credits go to me
</pre>

Code:
<pre>
"""
This is the name

These are module doc strings
that give helpful details
"""

__version__ = "1.0"
#__version__ = "$Revision: 83492 $"
__author__ = "Kenneth <kenneth@oeey.com>"

__date__ = "26 February 2001"
__credits__ = "Credits go to me"

#__all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license'
__license__ = "GPL"
__copyright__ = "Copyright 2007, The Project"
__maintainer__ = "Ken"
__email__ = "ken@oeey.com"
__status__ = "Production"

class Toad:
""" doc string for object """

my_class_attribute = 'sample class attribute'

def __init___(self, name):
""" Constructor """

def instance_function(self):
""" Instance Function """

@classmethod
def class_function(cls):
""" Class Function """
</pre>

=== username ===

User ID:
os.getuid()

User Name:
# fails it stdin piped to program
os.getlogin()

User Name:
import sys
import pwd
def get_username():
return pwd.getpwuid( os.getuid() )[ 0 ]

Environment:
os.environ['USER']

=== hostname ===

import socket
print(socket.gethostname())

os.uname()[1]

hostname = os.system('hostname')

=== Networking ===

==== Get Hostname ====

import socket
print(socket.gethostname())

==== Convert hostname to ip address ====

>>> socket.gethostbyaddr('demo')
('demo.oeey.com', [], ['10.10.10.10'])

/usr/local/bin/whatip: (or nametoip)
<pre>
#!/usr/bin/env python

#
# whatip - convert hostname to ip address
# author: Kenneth Burgener <kenneth@oeey.com> (FEB 2012)
#

import socket
import sys

if len(sys.argv) != 2:
print "Convert Hostname to IP Address"
print "Usage: %s HOSTNAME" % sys.argv[0].split('/')[-1]
sys.exit(1)

hostname = sys.argv[1]
try:
result = socket.gethostbyaddr(hostname)
print result[2][0]
except (socket.gaierror, socket.herror) as e:
print "Error: Unable to resolve '%s'" % hostname
sys.exit(1)
</pre>

==== Get Public IP Address ====

<pre>
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("gmail.com",80))
print(s.getsockname()[0])
s.close()
</pre>

networking - Finding local IP addresses using Python's stdlib - Stack Overflow - http://stackoverflow.com/questions/166506/finding-local-ip-addresses-using-pythons-stdlib

==== Python Network Server ====

<pre>
#!/usr/bin/python # This is server.py file

import socket # Import socket module

s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.
s.bind((host, port)) # Bind to the port

s.listen(5) # Now wait for client connection.
while True:
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
c.send('Thank you for connecting')
c.close() # Close the connection
</pre>

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
* 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

==== Python Network Client ====

<pre>
#!/usr/bin/python # This is client.py file

import socket # Import socket module

s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.

s.connect((host, port))
print s.recv(1024)
s.close # Close the socket when done
</pre>

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
* 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

=== Run Library Module as Script ===

Run python with -m option:
-m mod : run library module as a script (terminates option list)

Example:
cat [json_file] | python -m json.tool
# same as:
cat [json_file] | python /opt/python26/lib/python2.6/json/tool.py

Good for when library is buried deep in the library directories

=== Hex ===

<pre>
>>> hex(10) # print integer as hex string
'0xa'
>>> sys.hexversion # python version as integer
33817584
>>> hex(sys.hexversion) # python version as hex
'0x20403f0'
>>> print "0x%08x" % sys.hexversion # python version as hex
'0x20403f0'
>>> int('0xa', 16) # hex string to integer
10
>>> int('0a', 16) # hex string to integer
10
>>> int('0xa', 0) # hex string to integer auto detected du to '0x'
10
</pre>

=== Binary ===

Integer to Binary String with Negative Number Support: (2's complement)
<pre>
# source: http://www.gossamer-threads.com/lists/python/python/645216
def bin(x):
if x < -32768 or x > 65535:
raise Exception("Out of Range Error: %d" % x)
if x < 0:
x = 65535 + x + 1
bin_str = ''
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(15, 11, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(11, 7, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(7, 3, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(3, -1, -1))
return bin_str
</pre>

Integer to Binary String:
<pre>
# source: http://www.gossamer-threads.com/lists/python/python/645216
def bin(x):
if x < 0 or x > 65535:
raise Exception("Out of Range Error: %d" % x)
bin_str = ''
bin_str = ''.join(x & (1 << i) and '1' or '0' for i in range(15, 7, -1))
bin_str = bin_str + ' '
bin_str = bin_str + ''.join(x & (1 << i) and '1' or '0' for i in range(7, -1, -1))
return bin_str

print bin(0) # '00000000 00000000'
print bin(1) # '00000000 00000001'
print bin(255) # '00000000 11111111'
print bin(256) # '00000001 00000000'
print bin(43690) # '10101010 10101010'
print bin(65535) # '11111111 11111111'
print bin(65536) # '00000000 00000000' - rolled over to next set!
</pre>

Integer to Binary:
bin(170) # '0b10101010'

Binary to Integer:
int('10101010', 2) # 170

=== bitwise ===

See [[#Binary]]

BitwiseOperators - Python Wiki - http://wiki.python.org/moin/BitwiseOperators

<pre>
The Operators:

x << y
Returns x with the bits shifted to the left by y places (and new bits on the right-hand-side are zeros). This is the same as multiplying x by 2**y.
x >> y
Returns x with the bits shifted to the right by y places. This is the same as //'ing x by 2**y.
x & y
AND - Does a "bitwise and". Each bit of the output is 1 if the corresponding bit of x AND of y is 1, otherwise it's 0.
x | y
OR - Does a "bitwise or". Each bit of the output is 0 if the corresponding bit of x AND of y is 0, otherwise it's 1.
~ x
Returns the complement of x - the number you get by switching each 1 for a 0 and each 0 for a 1. This is the same as -x - 1.
x ^ y
XOR - Does a "bitwise exclusive or". Each bit of the output is the same as the corresponding bit in x if that bit in y is 0, and it's the complement of the bit in x if that bit in y is 1.
</pre>

=== Script Path ===

<pre>
if __name__ == "__main__":
# Change directory to where this script lives so we can find relative
# paths from here regardless of where the working directory was when we
# ran the script.
# APP_DIR = os.path.dirname(sys.argv[0])
APP_DIR = os.path.dirname(os.path.abspath(sys.argv[0))
if APP_DIR:
os.chdir(APP_DIR)
</pre>

normpath(path) # Normalize path, eliminating double slashes, etc.
abspath(path) # Return an absolute path

Path to a module: [http://stackoverflow.com/questions/247770/retrieving-python-module-path] (couldn't fine a_module)
import a_module
import os
path = os.path.dirname(a_module.__file__)

Path to module: [http://stackoverflow.com/questions/50499/in-python-how-do-i-get-the-path-and-name-of-the-file-that-is-currently-executin]
import inspect, os
print inspect.getfile(inspect.currentframe()) # script filename (usually with path)
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))) # script directory

Even better:
print os.path.realpath(__file__)
print os.path.dirname(__file__)
print os.path.basename(__file__)

references:
* http://stackoverflow.com/questions/595305/python-path-of-script

keywords: scriptpath script path program path pathname

=== Singleton ===

GLOBAL OBJECT!

None is a singleton (Python only has one copy of None in the interpreter).

Make your own singleton:

override the new method like this:
<pre>
class Singleton(object):
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Singleton, cls).__new__(
cls, *args, **kwargs)
return cls._instance

if __name__ == '__main__':
s1=Singleton()
s2=Singleton()
if(id(s1)==id(s2)):
print "Same"
else:
print "Different"
</pre>

Using decorator -implementation from PEP318? Implementing the singleton pattern with a decorator:
<pre>
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance

@singleton
class MyClass:
...
</pre>

* Python and the Singleton Pattern - Stack Overflow - http://stackoverflow.com/questions/42558/python-and-the-singleton-pattern

=== XML ===

<pre>
from xml.dom.minidom import parse, parseString

dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name

datasource = open('c:\\temp\\mydata.xml')
dom2 = parse(datasource) # parse an open file

dom3 = parseString('<myxml>Some data<empty/> some more data</myxml>')
</pre>

Pretty Print: [http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python]
<pre>
import xml.dom.minidom

xml = xml.dom.minidom.parse(xml_fname) # or xml.dom.minidom.parseString(xml_string)
pretty_xml_as_string = xml.toprettyxml()
</pre>

Pretty Print with etree [http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python]
<pre>
import lxml.etree as etree

x = etree.parse("filename")
print etree.tostring(x, pretty_print = True)
</pre>

References:
* 19.7. xml.dom.minidom — Lightweight DOM implementation — Python v2.7.3 documentation - http://docs.python.org/library/xml.dom.minidom.html
* Pretty printing XML in python - Stack Overflow - http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python

---

Simply don't include self.output.startElement and self.output.endElement for elements you don't want:

simple_filter_parser.py: (my version of [http://freegeographytools.com/2010/removin-time-data-out-of-a-gpx-or-kml-file-to-make-it-work-right-in-google-earth])
<pre>
# this parses out the time element of a gpx for Google Earth
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class my_filter_xml(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self.indent = 0
self.ignore = False
self.output.startDocument()

def startElement(self, name, attrs):
print " " * self.indent + "start: " + name
if attrs.keys():
keys = " " * (self.indent + 2) + "attrs:"
for key in attrs.keys():
keys = keys + " " + key + "=" + attrs[key]
print keys
self.indent = self.indent + 2
#self.output.startElement(name, {})
if name == "time":
self.ignore = True
#if not self.ignore:
self.output.startElement(name, attrs)

def characters(self, ch):
if ch.strip():
print " " * self.indent + "ch: " + ch.strip()
if not self.ignore:
self.output.characters(ch)
else:
if not self.ignore:
self.output.characters("\n")

def endElement(self, name):
self.indent = self.indent - 2
print " " * self.indent + "/end: " + name
#if not self.ignore:
self.output.endElement(name)
if name == "time":
self.ignore = False

if __name__ == '__main__':
source = open("tracks.gpx")
dest = open("tracks_out.gpx", 'w+')
xml.sax.parse(source, my_filter_xml(dest, True))
</pre>

---

fixdescriptor.py:
<pre>
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self._outfile = outfile
self._verbose = verbose
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self._filter = [ 'payload',
'payloads',
'file-list',
'file',
'checksum' ]
self._keep = [ 'vib',
'maintenance-mode' ]
self._inside = None

def startElement(self, name, attrs):
# Certain elements are kept unchanged
if name in self._keep:
if self._verbose:
print '%s element start found' % (name)
self.output.startElement(name, {})
if name == 'maintenance-mode':
if self._verbose:
print 'inside maintenance-mode'
self._inside = name
return
# Other elements are filtered (removed)
elif name in self._filter:
if self._verbose:
print '%s element start found' % (name)
self._inside = name
return
self.output.startElement(name, attrs)

def characters(self, ch):
if self._inside == 'maintenance-mode':
if self._verbose:
print 'new content for maintenance-mode'
self.output.characters('false')
elif self._inside not in self._filter:
self.output.characters(ch)

def endElement(self, name):
# If it's not a checksum element, only print it
if name in self._filter + self._keep:
self._inside = None
if self._verbose:
print '%s element end found' % (name)
if name in self._filter:
return
self.output.endElement(name)

if __name__ == '__main__':
#
parser = optparse.OptionParser(usage='%prog ...')
parser.add_option('-v', '--verbose',
action='store_true', default=False,
help='Print more information.')
parser.add_option('--input-xml',
type='str', default='descriptor.xml',
help='Name of input XML [default: %default]')
parser.add_option('--output-xml',
type='str', default='fixed_descriptor.xml',
help='Name of output XML [default: %default]')
opts, args = parser.parse_args(argv[1:])
source = open(opts.input_xml)
dest = open(opts.output_xml, 'w+')
xml.sax.parse(source, fix_descriptor(dest, opts.verbose))
</pre>

--

Simple read and output parsing:
<pre>
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self._outfile = outfile
self._verbose = verbose
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self._inside = None

def startElement(self, name, attrs):
# attrs.items() is a list of tuples (name, value)
print "start:", name, "/", attrs.items()
self.output.startElement(name, attrs)

def characters(self, ch):
print "char:", self._inside, "/", ch
self.output.characters(ch)

def endElement(self, name):
print "end:", name
self.output.endElement(name)

if __name__ == '__main__':
parser = optparse.OptionParser(usage='%prog ...')
parser.add_option('-v', '--verbose',
action='store_true', default=False,
help='Print more information.')
parser.add_option('--input-xml',
type='str', default='descriptor.xml',
help='Name of input XML [default: %default]')
parser.add_option('--output-xml',
type='str', default='fixed_descriptor.xml',
help='Name of output XML [default: %default]')
opts, args = parser.parse_args(argv[1:])
source = open(opts.input_xml)
dest = open(opts.output_xml, 'w+')
xml.sax.parse(source, fix_descriptor(dest, opts.verbose))
</pre>

=== Unicode ===

Unicode string:
u'Hello World !'

Raw Unicode string:
>>> ur'Hello\u0020World !'
u'Hello World !'

Convert:
u = u'hello\xe4'
str(u) # UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 5: ordinal not in range(128)
s = u.encode('utf8') # 's' is now type str

References:
* Introduction to Unicode Strings - http://docs.python.org/tutorial/introduction.html#unicode-strings

=== Zip ===

See [[Python/Zip]]

=== signals ===

signal.SIGTERM : kill [pid]
signal.SIGALRM : kill -ALRM [pid]
signal.SIGHUP : kill -HUP [pid]

<pre>
import signal, os

def handler(signum, frame):
print 'Signal handler called with signal', signum
raise IOError("Couldn't open device!")

# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)

# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)

signal.alarm(0) # Disable the alarm
</pre>

Note: use signal.SIGTERM to handle "kill" command

References:
* 17.4. signal — Set handlers for asynchronous events — Python v2.7.5 documentation - http://docs.python.org/2/library/signal.html

==== Ctrl+C ====

<pre>
#!/usr/bin/env python
import signal
import sys
def signal_handler(signal, frame):
print 'You pressed Ctrl+C!'
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
print 'Press Ctrl+C'

# linux only:
signal.pause()

# windows alternative:
#import time
#time.sleep(60)
</pre>

References:
* control - How do I capture SIGINT in Python? - Stack Overflow - http://stackoverflow.com/questions/1112343/how-do-i-capture-sigint-in-python

==== Non-blocking raw_input for Python ====

Non-blocking raw_input for Python:
<pre>
# source: http://www.garyrobinson.net/2009/10/non-blocking-raw_input-for-python.html

import signal

class AlarmException(Exception):
pass

def alarmHandler(signum, frame):
raise AlarmException

def nonBlockingRawInput(prompt='', timeout=20):
signal.signal(signal.SIGALRM, alarmHandler)
signal.alarm(timeout)
try:
text = raw_input(prompt)
signal.alarm(0)
return text
except AlarmException:
print '\nPrompt timeout. Continuing...'
signal.signal(signal.SIGALRM, signal.SIG_IGN)
return ''
</pre>

=== email ===

<pre>
import smtplib
import email.mime.text

msg_text = "Hello World"
msg = email.mime.text.MIMEText(msg_text)

# me == the sender's email address
# you == the recipient's email address
msg['Subject'] = 'The contents of %s' % textfile
msg['From'] = me
msg['To'] = you

# Send the message via our own SMTP server, but don't include the
# envelope header.
s = smtplib.SMTP('localhost')
s.sendmail(me, [you], msg.as_string())
s.quit()
</pre>

my nagios fake /bin/mail:
<pre>
#!/usr/bin/env python

import smtplib
import email.mime.text
import optparse
import os
import sys
import pwd

os.system('logger mail called')

parser = optparse.OptionParser()
parser.add_option(
'-s', '--subject',
help='Subject',
default="")
parser.add_option(
'-t', '--to',
help='To',
dest="send_to",
default=None)
parser.add_option(
'-f', '--from',
help='From',
dest="send_from",
default=None)
parser.add_option(
'-S', '--server',
help='server',
default="prime")
(options, args) = parser.parse_args()

def get_username():
return pwd.getpwuid( os.getuid() )[ 0 ]

if options.send_to:
send_to = options.send_to
else:
if len(args) == 1:
send_to = args[0]
else:
parser.print_help()
sys.exit(1)

if options.send_from:
send_from = options.send_from
else:
username = get_username()
hostname = os.uname()[1]
send_from = username + "@" + hostname

msg_text = sys.stdin.read()
msg = email.mime.text.MIMEText(msg_text)

msg['Subject'] = options.subject
msg['From'] = send_from
msg['To'] = send_to

s = smtplib.SMTP(options.server)
s.sendmail(send_from, [send_to], msg.as_string())
s.quit()
</pre>

Authenticated (with gmail):
<pre>
import smtplib
import email.mime.text

# from_email = "kenneth@demo.oeey.com"
# to_email = "kenneth@demo.oeey.com"
from_email = "Kenneth Burgener <kenneth@demo.oeey.com>"
to_emails = ["Kenneth Burgener <kenneth@demo.oeey.com>", "Kenneth Cell <xxx@xxx.com>"]

msg_text = "hello world"
msg = email.mime.text.MIMEText(msg_text)
msg['Subject'] = 'Test text'
msg['From'] = from_email
# msg['To'] = to_email
msg['To'] = ", ".join(to_emails)
# s = smtplib.SMTP('localhost')
print "sending..."
s = smtplib.SMTP_SSL('smtp.gmail.com', '465')
s.login('xxx', 'xxx')
# s.sendmail(from_email, [to_email], msg.as_string())
s.sendmail(from_email, to_emails, msg.as_string())
s.quit()
print "sent!"
</pre>

Source:
* 20.12. smtplib — SMTP protocol client — Python v2.7.5 documentation - http://docs.python.org/2/library/smtplib.html
* 18.1.11. email: Examples — Python v2.7.5 documentation - http://docs.python.org/2/library/email-examples.html

=== pprint ===

8.18. pprint — Data pretty printer — Python v2.7.5 documentation - http://docs.python.org/2/library/pprint.html

import pprint
pprint.pprint(myVar)

import pprint
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(myVar)

Note: Useless on objects!

Dump object:
<pre>
def dumpobject(obj):
print "<<<---"
print type(obj), obj.__class__.__name__
for attr in dir(obj):
if attr[0] == "_": continue # ignore under/dunder properties
if str(type(getattr(obj, attr))) == "<type 'instancemethod'>":
print attr, ": METHOD"
else:
print attr, ":", type(getattr(obj, attr)), ":", getattr(obj, attr)
print "END --->>>"
</pre>

=== Curses ===

Curses Programming with Python — Python v3.3.2 documentation - http://docs.python.org/3/howto/curses.html

"The curses library supplies a terminal-independent screen-painting and keyboard-handling facility for text-based terminals; such terminals include VT100s, the Linux console, and the simulated terminal provided by various programs. Display terminals support various control codes to perform common operations such as moving the cursor, scrolling the screen, and erasing areas. Different terminals use widely differing codes, and often have their own minor quirks.

In a world of graphical displays, one might ask “why bother”? It’s true that character-cell display terminals are an obsolete technology, but there are niches in which being able to do fancy things with them are still valuable. One niche is on small-footprint or embedded Unixes that don’t run an X server. Another is tools such as OS installers and kernel configurators that may have to run before any graphical support is available."

---

Thread from Utah Python User's Group mailing list:

-

Hello, I am trying to learn more about curses in python for a desktop application. Does anybody have any recommendations on tutorials or documentation?

-Corban

-

We've use urwid at work. Seems to be a good approach if you aren't already tied to something. Learning curve can be a bit steep (well, compared to what you'd think it should be). Good luck!

http://excess.org/urwid/

Gabriel Gunderson

-

This video series looks pretty good:

http://www.youtube.com/user/pythoncursestutorial?feature=watch

Official HOWTOs from the python.org site:

http://docs.python.org/2/howto/curses.html
http://docs.python.org/3/howto/curses.html

As Gabe mentioned, there are other libraries that provide similar or
greater functionality, so if you aren't doing this specifically to
learn curses, it's worth at least looking at a few of them to compare.

There's a brief comparison of three possibilities here:

http://stackoverflow.com/questions/8349085/python-ncurses-cdk-urwid-difference

Daniel Fackrell

-

Here's another option that looks promising:

https://pypi.python.org/pypi/blessings/

Daniel Fackrell

=== dialog ===

pythondialog - http://pythondialog.sourceforge.net/
: "pythondialog is a Python wrapper for the dialog utility originally written by Savio Lam, and later rewritten by Thomas E. Dickey. Its purpose is to provide an easy to use, pythonic and comprehensive Python interface to dialog. This allows one to make simple text-mode user interfaces on Unix-like systems (including Linux). "

Note: The latest versions (2.12 and later) only support Python 3; users who really want to stick to Python 2 should use version 2.11 for now.

Install:
# latest:
# wget http://sourceforge.net/projects/pythondialog/files/latest/download?source=dlp

wget http://sourceforge.net/projects/pythondialog/files/pythondialog/2.11/python2-pythondialog-2.11.tar.gz/download
tar -zvxf python2-pythondialog-2.11.tar.gz
cd pythondialog-2.11

import dialog
d = dialog.Dialog()
if d.yesno("Continue?") == d.DIALOG_OK:
print "Continuing..."

---

My solution

kiloforce / dialog / source / — Bitbucket - https://bitbucket.org/kiloforce/dialog

<pre>
#!/usr/bin/env python

""" Wrapper for Linux 'dialog' command """

__author__ = "Kenneth Burgener <kenneth@k.ttak.org>"
__copyright__ = "(c) 2013"
__date__ = "$Sep 27, 2013$"
__version__ = "1.0"
__credits__ = r"""
XXXXXXXXXXXXXXXXXX
X Dialog Wrapper #
##################
"""

import subprocess
import time
import sys

class Dialog():
""" Python wrapper for Linux 'dialog' command

dialog - display dialog boxes from shell scripts

>>> import dialog
>>> dlg = dialog.Dialog()
>>> rc = dlg.msgbox('hello world')

Return codes:
YES = 0
OK = 0
EXIT = 0
NO = 1
CANCEL = 1
ESC = 255
"""

YES = 0
OK = 0
EXIT = 0
NO = 1
CANCEL = 1
ESC = 255

def __init__(self):
rc = self._exe('which dialog > /dev/null')[0]
if rc != 0:
print "Error: failed to find Linux dialog command"
sys.exit(1)

def _exe(self, cmd, strip=True):
p = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE)
output = "\n".join(p.stderr.readlines())
if strip:
output = output.strip()
rc = p.wait()
return (rc, output)

def rc_to_str(self, rc):
if rc == 0:
return "YES"
elif rc == 1:
return "NO"
elif rc == 255:
return "ESX"
else:
raise Exception("Unknown return code: %s" % rc)

def clear(self):
"""
Clear screen

Example:
>>> dialog.clear()"""
rc, output = self._exe('clear')
assert rc in (0,), "clear rc"
return rc

def msgbox(self, msg):
"""
Simple message box

@param msg (str) = display message
@return: rc (int)

Example:
>>> rc = dialog.msgbox('hello world')"""
rc, output = self._exe('dialog --msgbox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.ESC), "msgbox rc"
return rc

def yesno(self, msg):
"""
Yes/No dialog

@param msg (str) = display message
@return: rc (int)

Example:
>>> rc = dialog.yesno('continue')"""
rc, output = self._exe('dialog --yesno "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.NO, self.ESC), "yesno rc"
return rc

def infobox(self, msg):
"""
Dialog that does not wait for user's response
(do your own pause)

@param msg (str) = display message
@return: rc (int) always zero

Example:
>>> for i in range(10):
>>> dialog.infobox('counting: ' + str(i))
>>> time.sleep(3)"""
rc, output = self._exe('dialog --infobox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES,), "infobox rc"
return rc

def inputbox(self, msg):
"""
Get user input

@param msg (str) = display message
@return: rc (int), output (str/None)

Example:
>>> rc, output = dialog.inputbox('name:')"""
rc, output = self._exe('dialog --inputbox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.NO, self.ESC), "inputbox rc"
if not output:
output = None
return rc, output

def textbox(self, filename):
"""
Display a file

@param filename (str) = file to display
@return: rc (int)

Example:
>>> dialog.textbox('/etc/passwd')"""
rc, output = self._exe('dialog --textbox {filename} 0 0'.format(filename=filename))
assert rc in (self.YES, self.ESC), "textbox rc"
return rc

def menu(self, msg, item_list):
"""
Single selection checkbox menu

@param msg (str) = display message
@param item_list (str list) = list of items to display
@return: rc (int), selection (int/None)

Example:
>>> rc, selection = dialog.menu('Select multiple options:',
... ('option 1', 'option 2'))"""
item_str = ""
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
rc, output = self._exe('dialog --menu "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
if output:
selection = int(output)
else:
selection = None
assert rc in (self.YES, self.NO, self.ESC), "menu rc"
return rc, selection

def checklist(self, msg, item_list, checked_list=None):
"""
Multiple selection checkbox menu

@param msg (str) = display message
@param item_list (str list) = list of items
@param checked_list (int list) = default items to check
@return: rc (int), selections (int list/None)

Example:
>>> rc, selections = dialog.checklist('Select multiple options:',
... ('option 1', 'option 2')
"""
item_str = ""
assert type(item_list) in (type(None), type(()), type([])), "checklist: item_list must be a list"
assert type(checked_list) in (type(None), type(()), type([])), "checklist: checked_list must be a list"
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
if checked_list and i+1 in checked_list:
item_str += " on"
else:
item_str += " off"
rc, output = self._exe('dialog --checklist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
print item_str
selections = []
for choice in output.split('"'):
if choice.strip():
selections.append(int(choice))
if selections:
selections = tuple(selections)
else:
selections = None
assert rc in (self.YES, self.NO, self.ESC), "checklist rc"
return rc, selections

def radiolist(self, msg, item_list, checked_item=None):
"""
Single selection radio menu, with default

@param msg (str) = display message
@param item_list (str list) = list of items
@param checked_item (int) = default item to select
@return: rc (int), selection (int)

Example:
>>> rc, selection = dialog.radiolist('Select one option:',
>>> ('option 1', 'option 2'), 2)"""
item_str = ""
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
if checked_item and i+1 == int(checked_item):
item_str += " on"
else:
item_str += " off"
rc, selection = self._exe('dialog --radiolist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
if not selection:
selection = None
assert rc in (self.YES, self.NO, self.ESC), "radiolist rc"
return rc, selection

def gauge(self, msg, percent):
"""
Progress bar (does not wait for user's input)

@param msg (str) = display message
@param percent (int) = progress
@return: rc (int) always zero

Example:
>>> for i in range(0, 100+1, 2):
>>> dialog.gauge('Progress:', i)
>>> time.sleep(.1)"""
rc, output = self._exe('echo {percent} | dialog --gauge "{msg}" 0 0'.format(percent=percent, msg=msg))
assert rc in (self.YES,), "guage rc"
return rc

def progress(self, msg, percent):
"""
Alias for self.guage()
"""
return self.gauge(msg=msg, percent=percent)

def form(self, msg, item_list):
"""
Multiple item entry form

@param msg (str) = display message
@param item_list (str list) = list of items
@return: rc (int), answers (str list)

Example:
>>> rc, answers = dialog.form('Enter details',
... ('first name', 'last name'))"""
item_str = ""
name_len = 1
for i in range(len(item_list)):
if len(item_list[i])+3 > name_len:
name_len = len(item_list[i])+3
for i in range(len(item_list)):
item_str += " \"" + item_list[i] + ":\" " + str(i+1) + " 1 \"\" " + str(i+1) +\
" " + str(name_len) + " 40 0"
rc, answers = self._exe('dialog --form "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str),
strip=False)
answers = answers.split('\n\n')
if not answers:
answers = None
assert rc in (self.YES, self.NO, self.ESC), "form rc"
return rc, answers

def test(self):
"""
Test features

>>> import dialog
>>> dlg = dialog.Dialog()
>>> dlg.test()
"""

def wait_user(msg):
self.msgbox("%s" % msg)

# rc_to_str
wait_user("Testing return code...")
print self.rc_to_str(self.YES) # YES
print self.rc_to_str(self.NO) # NO
print self.rc_to_str(self.ESC) # ESX
try:
print self.rc_to_str(111) # exception
raise Exception("Did not throw exception")
except Exception as rc:
pass

# msgbox
wait_user("Testing msgbox...")
rc = self.msgbox("msgbox")
if rc not in (self.YES, self.ESC):
raise Exception("Unknown return code for msgbox: %s" % rc)

# yesno
wait_user("Testing yesno...")
rc = self.yesno("yesno")
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for yesno: %s" % rc)

# infobox
wait_user("Testing infobox...")
rc = self.infobox("infobox sleep 1 sec")
if rc not in (self.YES,):
raise Exception("Unknown return code for infobox %s" % rc)
time.sleep(1)

# inputbox
wait_user("Testing inputbox...")
rc, output = self.inputbox("inputbox")
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for inputbox: %s" % rc)
wait_user("Result: %s" % output)

# textbox
wait_user("Testing textbox...")
rc = self.textbox("/etc/passwd")
if rc not in (self.YES, self.ESC):
raise Exception("Unknown return code for textbox: %s" % rc)

# menu
wait_user("Testing menu...")
rc, selection = self.menu("menu", ("option 1", "option 2"))
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for menu: %s" % rc)
wait_user("Result: %s" % selection)

# checklist
wait_user("Testing checklist...")
rc, selection = self.checklist("menu", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for checklist: %s" % rc)
wait_user("Result: %s" % str(selection))
wait_user("Testing checklist with default selection...")
rc, selection = self.checklist("menu", ["option 1", "option 2"], [1, 2])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for checklist: %s" % rc)
wait_user("Result: %s" % str(selection))

# radiolist
wait_user("Testing radiolist...")
rc, selection = self.radiolist("menu", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for radiolist: %s" % rc)
wait_user("Result: %s" % str(selection))
wait_user("Testing radiolist with default selection...")
rc, selection = self.radiolist("menu", ["option 1", "option 2"], 2)
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for radiolist: %s" % rc)
wait_user("Result: %s" % str(selection))

# guage
wait_user("Testing guage...")
for i in range(4+1):
rc = self.gauge("test guage", i*25)
# rc = self.progress("test progress", i*25)
if rc not in (self.YES,):
raise Exception("Unknown return code for guage: %s" % rc)
time.sleep(1)

# form
wait_user("Testing form...")
rc, answers = self.form("form", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for form: %s" % rc)
wait_user("Result: %s" % str(answers))

# done with test, cleanup
wait_user("Done with tests...")
self.clear()

if __name__ == "__main__":
# dialog = Dialog()
# dialog.test()
pass
</pre>

=== Busy Loop and Thread.yield() ===

"I want to tell my Python threads to yield, and so avoid hogging the CPU unnecessarily. In Java, you could do that using the Thread.yield() function"

Solution:
time.sleep(0)
time.sleep(0.00001)
time.sleep(epsilon)

References:
* multithreading - In there something similar to Java's Thread.yield() in Python? Does that even make sense? - Stack Overflow - http://stackoverflow.com/questions/1908206/in-there-something-similar-to-javas-thread-yield-in-python-does-that-even-ma
* multithreading - How does a threading.Thread yield the rest of its quantum in Python? - Stack Overflow - http://stackoverflow.com/questions/787803/how-does-a-threading-thread-yield-the-rest-of-its-quantum-in-python

=== USB ===

Programming with PyUSB 1.0 - http://pyusb.sourceforge.net/docs/1.0/tutorial.html

PyUSB - http://pyusb.sourceforge.net/

git clone https://github.com/walac/pyusb

import usb.core
dev = usb.core.find()

# find our device
dev = usb.core.find(idVendor=0xfffe, idProduct=0x0001)

"'''Dealing with multiple identical devices''' - Sometimes you may have two identical devices connected to the computer. How can you differentiate them? Device objects come with two additional attributes which are not part of the USB Spec, but are very useful: bus and address attributes. First of all, it is worth to say that these attributes come from the backend and a backend is free to not support them, in which case they are set to None. That said, these attributes represent the bus number and bus address of the device and, as you might already have imagined, can be used to differentiate two devices with the same idVendor and idProduct attributes."

=== Tkinter ===

Python GUI Bitcoin: [http://bitcoin.stackexchange.com/questions/10594/simplest-way-of-obtaining-mtgox-exchange-rate-via-an-api]
<pre>
import json
import urllib2
from Tkinter import *

url='http://api.bitcoincharts.com/v1/markets.json'
req=urllib2.Request(url)
response=urllib2.urlopen(req).read()
output=json.loads(response)

mtusd_price = output[-10]['avg']
mteur_price = output[31]['avg']
mtgbp_price = output[44]['avg']

MTUSD= 'USD' + ' ' + str(mtusd_price)
MTEUR= 'EUR' + ' ' + str(mteur_price)
MTGBP = 'GBP' + ' ' + str(mtgbp_price)

root= Tk()
root.title("Bitcoin Price")
root.geometry("250x100")

USD = Label(root, text=MTUSD)
EUR = Label(root, text=MTEUR)
GBP = Label(root, text=MTGBP)
USD.pack()
EUR.pack()
GBP.pack()
mainloop()
</pre>

=== ctypes ===

"ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python."
:-- 15.17. ctypes — A foreign function library for Python — Python v2.7.7 documentation - https://docs.python.org/2/library/ctypes.html

Use "c scope" to build a database of symbols to see what options are available.

---

testlib.c
<pre>
#include <stdio.h>

void myprint(void);

void myprint()
{
printf("hello world\n");
}
</pre>

Compile:
gcc -shared -Wl,-soname,testlib -o testlib.so -fPIC testlib.c

testlibwrapper.py

Python:
<pre>
import ctypes

testlib = ctypes.CDLL('/full/path/to/testlib.so')
testlib.myprint()
</pre>

Source: python - ctypes - Beginner - Stack Overflow - http://stackoverflow.com/questions/5081875/ctypes-beginner

---

nzjrs's Gists - https://gist.github.com/nzjrs

---

<pre>
#include <stdio.h>

int sum(double *x,int n)
{
int i;
double counter;
counter = 0;
for(i=0;i<n;i++)
{
counter=counter+x[i];

}
return counter;
}
</pre>

<pre>
gcc -c sum.c
gcc -shared -o sum.so sum.o
</pre>

<pre>
from ctypes import *
my_sum=CDLL('sum.so')
a=numpy.array(range(10),dtype=float)
my_sum.sum(a.ctypes.data_as(c_void_p),int(10))
</pre>

<pre>
a=(c_double*10)()
for i in range(10):
a[i]=i
my_sum.sum(a,int(10))
</pre>

Source: Ctypes — Numerical Computing with Sage v6.2 - http://www.sagemath.org/doc/numerical_sage/ctypes.html

---

=== Warnings ===

if sys.version_info >= (1, 0, 0):
warnings.warn("Danger Will Robinson: trying Phi on untested Python 3.0!")

=== enum ===

Python 3: https://docs.python.org/3/library/enum.html

<pre>
from enum import Enum

# class syntax
class Color(Enum):
RED = 1
GREEN = 2
BLUE = 3

# functional syntax
Color = Enum('Color', ['RED', 'GREEN', 'BLUE'])
</pre>

---

Python 2: [http://stackoverflow.com/questions/36932/how-can-i-represent-an-enum-in-python]
<pre>
class Enum(set):
""" Basic enumerated type

Example
>>> Animals = phi.Enum(["DOG", "CAT", "HORSE"])
>>> Animals.DOG
'DOG'
>>> 'HORSE' in Animals
True
"""
def __getattr__(self, name):
if name in self:
return name
raise AttributeError("Missing Enum property: %s" % name)
def __setattr__(self, name, value): # this makes it read-only
raise AttributeError("Unable to modify Enum property: %s (%s)" % (name, value))

Animals = Enum(["DOG", "CAT", "HORSE"])

print(Animals.DOG) # 'DOG'
</pre>

----------

Alternative:
<pre>
class Stationery:
Pen, Pencil, Eraser = range(0, 3)
</pre>

One liner:
class Enum(tuple): __getattr__ = tuple.index

<pre>
>>> State = Enum(['Unclaimed', 'Claimed'])
>>> State.Claimed
1
>>> State[1]
'Claimed'
>>> State
('Unclaimed', 'Claimed')
>>> range(len(State))
[0, 1]
>>> [(k, State[k]) for k in range(len(State))]
[(0, 'Unclaimed'), (1, 'Claimed')]
>>> [(k, getattr(State, k)) for k in State]
[('Unclaimed', 0), ('Claimed', 1)]
</pre>

Python 3 supports enum natively with the enum.Enum library:

=== pySerial ===

https://pythonhosted.org/pyserial/

"This module encapsulates the access for the serial port. It provides backends for Python running on Windows, OSX, Linux, BSD (possibly any POSIX compliant system) and IronPython. The module named “serial” automatically selects the appropriate backend."

https://github.com/pyserial/

Intro - https://pythonhosted.org/pyserial/shortintro.html

---

<pre>
import serial
import time

ser = serial.Serial(
port='COM4',\
baudrate=9600,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)

print("connected to: " + ser.portstr)

#this will store the line
seq = []
count = 1

while True:
for c in ser.read():
seq.append(chr(c)) #convert from ANSII
joined_seq = ''.join(str(v) for v in seq) #Make a string from array

if chr(c) == '\n':
print("Line " + str(count) + ': ' + joined_seq)
seq = []
count += 1
break

ser.close()
</pre>

Ref: http://stackoverflow.com/questions/16077912/python-serial-how-to-use-the-read-or-readline-function-to-read-more-than-1-char

---

I modified the code to monitor a COM port: (Windows)

Monitor Serial:

<pre>
import serial, time, sys

logger = open("serial.txt", "w")
timer = time.time()

def log(msg, head=False):
msg = msg.strip()
if head:
msg = ">>> " + msg
logger.write(msg + "\n")
sys.stdout.write(msg + "\n")
newtime = time.time()
if newtime > timer + 1:
logger.flush()
sys.stdout.flush()

while True:

ser = None
try:
ser = serial.Serial(
port='COM1',\
baudrate=115200,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)
except serial.serialutil.SerialException as e:
log(str(e), True)
time.sleep(1)
continue

log("Connected to: " + ser.portstr, True)
count=1

try:
line = []
while True:
for c in ser.readline():
line.append(c)
if c == "\n":
log("".join(line))
line = []
except serial.serialutil.SerialException as e:
log(str(e), True)
time.sleep(1)
ser.close()
</pre>

=== MyBool ===

<pre>
>>> class NewBool(int):
... def __new__(cls, value):
... return int.__new__(cls, bool(value))

>>> y = NewBool(2)
>>> y == 1
True
</pre>

https://jfine-python-classes.readthedocs.io/en/latest/subclass-int.html

Also check out the __cmp__

== md5 ==

import md5
md5.new("Nobody inspects the spammish repetition").hexdigest()
# 'bb649c83dd1ea5c9d9dec9a18df0ffe9'

<pre>
$ echo -n "Nobody inspects the spammish repetition" | md5sum
bb649c83dd1ea5c9d9dec9a18df0ffe9 -
<pre>

== External Libraries ==

=== execnet ===

Python Package Index - execnet - https://pypi.python.org/pypi/execnet

Examples - http://codespeak.net/execnet/examples.html
* basic local and remote communication - http://codespeak.net/execnet/example/test_info.html

Execute source code in subprocess, communicate through a channel:
<pre>
>>> import execnet
>>> gw = execnet.makegateway()
>>> channel = gw.remote_exec("channel.send(channel.receive()+1)")
>>> channel.send(1)
>>> channel.receive()
2
</pre>

Compare current working directories
<pre>
>>> import execnet, os
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec("import os; channel.send(os.getcwd())")
>>> res = ch.receive()
>>> assert res == os.getcwd()
</pre>

Get information from remote ssh account
<pre>
>>> import execnet, os
>>> gw = execnet.makegateway("ssh=codespeak.net")
>>> channel = gw.remote_exec("""
... import sys, os
... channel.send((sys.platform, sys.version_info, os.getpid()))
... """)
>>> platform, version_info, remote_pid = channel.receive()
>>> platform
'linux2'
>>> version_info
(2, 4, 2, 'final', 0)
</pre>

remote-exec a function (avoiding inlined source part I)
<pre>
import execnet

def multiplier(channel, factor):
while not channel.isclosed():
param = channel.receive()
channel.send(param * factor)

gw = execnet.makegateway()
channel = gw.remote_exec(multiplier, factor=10)

for i in range(5):
channel.send(i)
result = channel.receive()
assert result == i * 10

gw.exit()
</pre>

remote-exec a module (avoiding inlined source part II)
:"You can pass a module object to remote_exec in which case its source code will be sent. No dependencies will be transferred so the module must be self-contained or only use modules that are installed on the “other” side. Module code can detect if it is running in a remote_exec situation by checking for the special __name__ attribute."

<pre>
# content of a module remote1.py

if __name__ == '__channelexec__':
channel.send('initialization complete')
</pre>

<pre>
>>> import execnet, remote1
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec(remote1)
>>> print (ch.receive())
initialization complete
</pre>

Use a callback instead of receive() and wait for completion
<pre>
>>> import execnet
>>> gw = execnet.makegateway()
>>> channel = gw.remote_exec("for i in range(10): channel.send(i)")
>>> l = []
>>> channel.setcallback(l.append, endmarker=None)
>>> channel.waitclose() # waits for closing, i.e. remote exec finish
>>> l
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, None]
</pre>

a simple command loop pattern
<pre>
# contents of: remotecmd.py
import os

def simple(arg):
return arg + 1

def listdir(path):
return os.listdir(path)

if __name__ == '__channelexec__':
for item in channel:
channel.send(eval(item))
</pre>

<pre>
>>> import execnet, remotecmd
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec(remotecmd)
>>> ch.send('simple(10)') # execute func-call remotely
>>> ch.receive()
11
</pre>

=== evdev ===

Install:
apt-get install python-setuptools python-dev
easy_install pip
pip install evdev

Simple read usage:
<pre>
from evdev import InputDevice
from evdev import ecodes

# look for a /dev/input/by-id/usb...kbd or similar
DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd"

dev = InputDevice(DEVICE)

while True:
r, w, x = select([dev], [], [], 5)
for event in dev.read():
if event.type == 1 and event.value == 1:
print event.code
</pre>

List LED states:
dev.leds()
dev.leds(verbose=True)

Inject usage: (does not appear to really work)
<pre>
from evdev import InputDevice
from evdev import ecodes
from evdev import UInput

DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd"
dev = InputDevice(DEVICE)

ui = UInput()

ui.write(ecodes.EV_KEY, ecodes.KEY_4, 1)
ui.write(ecodes.EV_KEY, ecodes.KEY_4, 0)
ui.syn()

ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 1)
ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 0)
ui.syn()

ui.close()
</pre>

Tutorial:
* evdev documentation — python-evdev 0.4.3 documentation - http://pythonhosted.org/evdev/index.html
References:
* python-evdev/doc/tutorial.rst at master · gvalkov/python-evdev · GitHub - https://github.com/gvalkov/python-evdev/blob/master/doc/tutorial.rst

=== ldap-python ===

python-ldap: LDAP client API for Python - http://www.python-ldap.org/

Index of Packages : Python Package Index - https://pypi.python.org/pypi/python-ldap/

yum install python-devel openldap-devel
pip install python-ldap

Articles:
* Python LDAP Applications: Part 1 - Installing and Configuring the Python-LDAP Library and Binding to an LDAP Directory | Packt Publishing - http://www.packtpub.com/article/installing-and-configuring-the-python-ldap-library-and-binding-to-an-ldap-directory

=== xmmp ===

PDF:
* XMPP: The Definitive Guide - Oreilly - http://oriolrius.cat/blog/wp-content/uploads/2009/10/Oreilly.XMPP.The.Definitive.Guide.May.2009.pdf

xmmp is the Jabber protocol

Best options:

MIT libraries.
* sleekxmpp (was: sleekxmpp) is pretty popular and is used for examples in Peter Saint-Andre's XMPP book from O'Reilly. It reportedly works well, and finally got an email list in April 2010, and has a chat room at sleek@conference.jabber.org.

GPL libraries.
* xmpppy is used by gajim.

LPGL libraries.
* pyxmpp is pretty good and uses libxml2 internally for xml parsing.
* pyxmpp2 is the next version of pyxmpp, runs on python 2.7 and 3.2, and removes the libxml2 requirement. Like many, it requires dnspython.
* jabberpy is the original and still works for a lot of tasks, but is thoroughly unmaintained.

Other libraries.
* Wokkel, mentioned in another post. That's a new one for me, based on twisted.

chat - Best python XMPP / Jabber client library? - Stack Overflow - http://stackoverflow.com/questions/1901828/best-python-xmpp-jabber-client-library

=== xmpppy ===

xmpppy: the jabber python project - http://xmpppy.sourceforge.net/

"xmpppy is a Python library that is targeted to provide easy scripting with Jabber. Similar projects are Twisted Words and jabber.py.

This library was not designed from scratch. It inherits some code from jabberpy and have very similar API in many places. Though it is separate project since it have almost completely different architecture and primarily aims to work with jabberd2 - the new Open Source Jabber Server.

xmpppy is distributed under the terms of GNU General Public License and can be freely redistributed without any charge."

wget http://sourceforge.net/projects/xmpppy/files/xmpppy/0.4.1/xmpppy-0.4.1.tar.gz/download -O xmpppy-0.4.1.tar.gz
tar -zvxf xmpppy-0.4.1.tar.gz
pip install pydns

--- basic connection ---

<pre>
#!/usr/bin/env python

import xmpp

user="username@gmail.com"
password="password"
server="gmail.com"

jid = xmpp.JID(user)
connection = xmpp.Client(server,debug=[])
connection.connect()
result = connection.auth(jid.getNode(), password,"LFY-client")

connection.sendInitPresence()

while connection.Process(1):
pass
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- basic receive ---

Print messages when received:
<pre>
#!/usr/bin/env python

import xmpp

user="username@gmail.com"
password="password"
server="gmail.com"

def message_handler(connect_object, message_node):
message = "Welcome to my first Gtalk Bot :)"
connect_object.send( xmpp.Message( message_node.getFrom() ,message))

jid = xmpp.JID(user)
connection = xmpp.Client(server)
connection.connect()
result = connection.auth(jid.getNode(), password, "LFY-client")
connection.RegisterHandler('message', message_handler)

connection.sendInitPresence()

while connection.Process(1):
pass
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- remote-control shell bot ---

Replace the simple bot’s message_handler() function with this new one:

<pre>
def message_handler(connect_object,message_node):
command = str(message_node.getBody())

process = subprocess.Popen(command,shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
message = process.stdout.read()
if message=="":
message=process.stderr.read()

connect_object.send( xmpp.Message( message_node.getFrom() ,message))
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- roster ---

<pre>
import xmpp
import sys

userID = 'myname@gmail.com'
password = 'mypassword'
ressource = 'Script'

jid = xmpp.protocol.JID(userID)
jabber = xmpp.Client(jid.getDomain(), debug=[])

connection = jabber.connect(('talk.google.com',5222))
auth = jabber.auth(jid.getNode(), password, ressource)

jabber.sendInitPresence(requestRoster=1)
myroster = jabber.getRoster()
</pre>

To get status, you have to wait for the other clients to send you their status:
<pre>
jabber.RegisterHandler('presence', myPresenceHandler)

def myPresenceHandler(con, event):
fromjid = event.getFrom().getStripped()
status = myroster.getStatus(fromjid)
</pre>

References:
python - XMPP chat: accessing contacts' status messages with xmppPy's Roster - Stack Overflow - http://stackoverflow.com/questions/2381597/xmpp-chat-accessing-contacts-status-messages-with-xmpppys-roster

--- full roster script ---

kparal/jabber-roster · GitHub - https://github.com/kparal/jabber-roster

A simple Python tool for listing your Jabber roster contacts. You can use it to easily backup list of your buddies.

This program is maintained, but not further developed. Bugs will be fixed, but no new features will be added. If you want to work on this program, don't hesitate to contact me, I will gladly assign you to the development team.

git clone https://github.com/kparal/jabber-roster
python jabber_roster.py

https://pypi.python.org/pypi/jabber-roster

--- vcard ---

This information is not in the roster. You will need to query the clients individually and get their vCard by sending this IQ :

<pre>
<iq from='stpeter@jabber.org/roundabout'
id='v1'
type='get'>
<vCard xmlns='vcard-temp'/>
</iq>
</pre>

My solution:
<pre>
def vcard(disp, jid):
msg = xmpp.protocol.Iq()
msg.setType('get')
msg.setTo(jid)
qc = msg.addChild('vCard')
qc.setAttr('xmlns', 'vcard-temp')
rep = disp.SendAndWaitForResponse(msg)
# to see what other fields are available in the XML output:
# print rep
userid=fname=lname=title=department=region=None
for i in rep.getChildren():
for j in i.getChildren():
if j.getName() == "TITLE":
title = j.getData().encode('utf-8')
for k in j.getChildren():
if k.getName() == "GIVEN":
fname = k.getData().encode('utf-8')
if k.getName() == "FAMILY":
lname = k.getData().encode('utf-8')
if k.getName() == "ORGUNIT":
department = k.getData().encode('utf-8')
if k.getName() == "REGION":
region = k.getData().encode('utf-8')
return fname, lname, title, department, region
</pre>

References:
* xmpppy - Retrieve gtalk nickname in python xmpp - Stack Overflow - http://stackoverflow.com/questions/4539477/retrieve-gtalk-nickname-in-python-xmpp
* XEP-0054: vcard-temp - http://xmpp.org/extensions/xep-0054.html

=== google-api-python-client ===

Google APIs Client Library for Python, which includes:
* google-api-python-client: The core Python library for accessing Google APIs.
* oauth2client: A Python client library for OAuth 2.0.
* Sample applications using google-api-python-client and oauth2client

-

https://developers.google.com/api-client-library/python/start/installation
* Calendar API - Python/Command Line

-
Download:
wget https://code.google.com/p/google-api-python-client/downloads/detail?name=google-api-python-client-1.2.zip
wget https://code.google.com/p/google-api-python-client/downloads/detail?name=oauth2client-1.2.zip

Git:
git clone https://github.com/google/google-api-python-client
git clone https://github.com/google/oauth2client

References:
* google-api-python-client - Google APIs Client Library for Python - Google Project Hosting - https://code.google.com/p/google-api-python-client/

== Documentation Tools ==

=== pydoc ===

pydoc - http://docs.python.org/library/pydoc.html
: Documentation generator and online help system

=== Doxygen and Doxypy ===

See [[Python/Doxygen]]

=== Sphinx ===

Sphinx - http://sphinx.pocoo.org/

"Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license.

It was originally created for the new Python documentation, and it has excellent facilities for the documentation of Python projects, but C/C++ is already supported as well, and it is planned to add special support for other languages as well. Of course, this site is also created from reStructuredText sources using Sphinx!"

=== epydoc ===

epydoc - http://epydoc.sourceforge.net/

"Epydoc is a tool for generating API documentation for Python modules, based on their docstrings. For an example of epydoc's output, see the API documentation for epydoc itself (html, pdf). A lightweight markup language called epytext can be used to format docstrings, and to add information about specific fields, such as parameters and instance variables. Epydoc also understands docstrings written in reStructuredText, Javadoc, and plaintext. For a more extensive example of epydoc's output, see the API documentation for Python 2.5. "

== Python 3 ==

=== Python 3 Wall of Superpowers ===

Python 3 Wall of Superpowers - https://python3wos.appspot.com/

"As listed on PyPI - packages in red don't support python 3, packages in green do. Hopefully one day everything will be greener. "

=== New in Python 3 ===

==== qualnames ====

New in 3.3
<pre>
>>> def f():
... def g():
... class H: pass
... return H
... return g
...

>>> H=f()()

>>> # 3.3:
>>> H
<class '__main__.f.<locals>.g.<locals>.H'>
>>> H.__qualname__
'f.<locals>.g.<locals>.H'

>>> # 2.6:
>>> H
<class __main__.H at 0x2b885044b890>
>>> H.__qualname__
AttributeError: class H has no attribute '__qualname__'
</pre>

== Python 2 or Python 3 ==

Python2orPython3 - PythonInfo Wiki - http://wiki.python.org/moin/Python2orPython3
:Short version: Python 2.x is the status quo, Python 3.x is the shiny new thing.
:At the time of writing (July 4, 2010), the final 2.7 release is out, with a statement of extended support for this end-of-life release. The 2.x branch will see no new major releases after that. 3.x is under active and continued development, with 3.1 already available and 3.2 due for release around the turn of the year.

Python 2.7.1 Release - http://www.python.org/download/releases/2.7.1/

What’s New In Python 3.0 — Python v3.0.1 documentation - http://docs.python.org/release/3.0.1/whatsnew/3.0.html
* print() requires parenthesis
* raw_input() is now input()

== Python for Windows ==

Python for Windows - http://www.python.org/download/windows/

=== Windows IDEs ===

==== PyCharm ====

http://www.pycharm.com/

=== Hide Console ===

fork() does not work on windows. So what else can we do?

How to hide console window in python? - http://stackoverflow.com/questions/764631/how-to-hide-console-window-in-python
* "Simply save it with a .pyw extension. This will prevent the console window from opening. Explanation at the bottom of section [http://docs.python.org/2/tutorial/interpreter.html#executable-python-scripts]"

=== 32bit vs 64bit ===

IDLE:

32bit:
<pre>
C:\Python27>python
Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
</pre>

64bit: (note: default path is C:\Python27)
<pre>
C:\Python27-64bit>python.exe
Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
</pre>

Task Manager:
python.exe *32

python.exe

Platform:
>>> import platform
>>> platform.python_compiler()
'MSC v.1500 32 bit (Intel)'

>>> import platform
>>> platform.python_compiler()
'MSC v.1500 64 bit (AMD64)'

Code: [http://stackoverflow.com/questions/2208828/detect-64bit-os-windows-in-python]
import struct
def is_python_64bit():
return (struct.calcsize("P") == 8)

=== Stand-Alone Executable ===

==== Python ZIP Application ====

Python can be run from a single bundled zip file - similar to a Java WAR! Python has been able to (Allow interpreter to execute a zip file http://bugs.python.org/issue1739468)

"Python quietly added a new feature in 2.6 that makes it possible to bundle up a directory full of Python code into a single executable file." [http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html]

When a python zip application is executed, it will execute '__main__.py' [http://docs.python.org/using/cmdline.html#interface-options]

mkdir app
touch app/__main__.py
cd app
zip -r ../app.zip *
cd ..
python app.zip

Load a resource - The easiest way to find and load a program bundle like this is to use the pkg_resources module. - http://packages.python.org/distribute/pkg_resources.html

<pre>
# file: app/__main__.py

import pkg_resources

def main():
print('The rain in Spain falls mainly in the plain.')
print(pkg_resources.resource_string('resources', 'inFrance.txt'))

if __name__ == '__main__':
main()
</pre>

Now use a bit of UNIX magic to turn app2.zip into an executable:
echo '#!/usr/bin/env python' | cat - app2.zip > app2
chmod +x app2

"The zip file format is designed to allow a small executable program to be inserted at the front (that's how self-extracting zip files are created), so this is kosher and doesn't corrupt the zip file." [http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html]

Now you can simply run app2 like any executable.
$ ./app2

References:
* Able Pear Software: Bundling Python files into a stand-alone executable - http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html
* PEP 441 -- Improving Python ZIP Application Support - http://www.python.org/dev/peps/pep-0441/
* Issue 1739468: Allow interpreter to execute a zip file - Python tracker - http://bugs.python.org/issue1739468
* Package Discovery and Resource Access using pkg_resources — Distribute documentation - http://packages.python.org/distribute/pkg_resources.html

==== Shell Python Loader ====

Came before the Python ZIP Application support.

Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe

"This takes your Python application and packages it into a single executable shell script. You application can be made of up multiple python scripts. The executable shell script is also compressed, so the resulting script is smaller than the original. This should work on any system that has Python and a Bourne shell (works on Cygwin too). Note that this does not package the Python interpreter into the executable. This packages only you python code into a zip file, but this still allows you to distribute your application as a single shell script that will run on any system with Python installed. In theory, you could zip the .pyc files, but the byte code is guaranteed to run only on exactly the same version Python interpreter (including minor version). It is safer to just package the .py files."

First you need to zip all your python files:
zip main.zip main.py spam.py eggs.py

zipheader.sh:
<pre>
#!/bin/sh
# This is a self-extracting executable.
# Execute this like any normal executable.
# You may need to "chmod a+x" this file.
# This is a binary ZIP file with a Python loader header.
#
# Bourne shell loader:
PYTHON=$(which python 2>/dev/null)
if [ ! -x "$PYTHON" ] ; then
echo "Python not found!"
exit 1
fi
exec $PYTHON -c "
# Python loader:
import sys, os
if int(sys.version[0])<2:
print 'Python version 2.3 final or greater is required.'
print 'Your version is', sys.version
os._exit(1)
major = sys.version_info[0]
minor = sys.version_info[1]
releaselevel = sys.version_info[3]
if (major==2 and minor<3) or (major==2 and minor==3 and releaselevel!='final'):
print 'Python version 2.3 final or greater is required.'
print 'Your version is', sys.version
os._exit(1)
sys.path.insert(0, sys.argv[1])
del sys.argv[0:1]
import main
main.main()
" $0 $@
# Zip file:
</pre>

Concatinate zipheader.sh with the zip file.
cat zipheader.sh main.zip > main

Set main as executable.
chmod +x main

"This is an improvement on Joerg Raedler's Python recipe, [http://code.activestate.com/recipes/215301-binding-main-skript-and-modules-to-one-executable-/ 215301]. The original closes stdin. The herefile in the shell script redirects stdin before python gets a chance to start. This disables raw_input() and anything else that reads sys.stdin. Unfortunately, Raedler's boot script closes stdin, which is a fairly big limitation. The herefile in the shell script (END_OF_PYTHON_CODE) redirects stdin before python starts; stdin is closed at the end of the herefile. This disables raw_input() and anything else that reads sys.stdin. "

References:
* Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe
* Build a compressed self-extracting executable script on UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/497000-build-a-compressed-self-extracting-executable-scri/
* binding main skript and modules to one executable with python-2.3 under UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/215301-binding-main-skript-and-modules-to-one-executable-/

==== py2exe ====

py2exe - http://www.py2exe.org/
: "py2exe is a Python Distutils extension which converts Python scripts into executable Windows programs, able to run without requiring a Python installation. "

Tutorial - http://www.py2exe.org/index.cgi/Tutorial

Note: Install the version that matches your version of python

Note: run installer "as administrator"

setup.py
<pre>
from distutils.core import setup
import py2exe

setup(console=['hello.py'])
</pre>

No console window version: [http://ubuntuforums.org/showthread.php?t=728170] [http://stackoverflow.com/questions/3275293/hiding-console-window-of-python-gui-app-with-py2exe]
<pre>
from distutils.core import setup
import py2exe

setup(windows=['hello.py'])
</pre>

Build:
python setup.py py2exe

pygame is more complex: http://www.pygame.org/wiki/Pygame2exe

==== py2app ====

==== pyinstaller ====

==== cx_Freeze ====

=== Windows Libraries ===

==== Python for Windows Extensions - PyWin32 ====

Python for Windows Extensions - http://starship.python.net/~skippy/win32/
* There is a build for each python version and architecture (32bit vs 64bit)

Follow event viewer
<pre>
import win32evtlog # install Python for Windows Extensions - http://starship.python.net/~skippy/win32/

# open event viewer
server = "localhost"
source = "System"
handle = win32evtlog.OpenEventLog(server, source)

# seek to end of log
flags = win32evtlog.EVENTLOG_BACKWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
events = win32evtlog.ReadEventLog(handle, flags, 0)

# return to normal reading direction
flags = win32evtlog.EVENTLOG_FORWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
win32evtlog.ReadEventLog(handle, flags, 0) # ignore initial logs

# loop through events
while True:
events = win32evtlog.ReadEventLog(handle, flags, 0)
if events:
for event in events:
data = event.StringInserts
if data:
# Example: 07/03/14 12:34:35 - MYEVENTSOURCE - 0 - 1 - 4 "HELLO"
print event.TimeGenerated, '-', event.SourceName, '-', event.EventCategory, '-', event.EventID, '-', event.EventType, '-', '"%s"' % " ".join(data)
time.sleep(.5) # yield cpu
</pre>

References:
* PyWin32: Getting Windows Event Logs « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/07/27/pywin32-getting-windows-event-logs/

==== PyCrypto for Windows ====

Download Install: http://www.voidspace.org.uk/python/modules.shtml#pycrypto

Installation: [http://stackoverflow.com/questions/11405549/how-do-i-install-pycrypto-on-windows]
easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win32-py2.7.exe
easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win-amd64-py2.7.exe

==== Paramiko for Windows ====

# Install PyCrypto (above)
# pip install paramiko

==== PyGame for Windows ====

Download - http://www.pygame.org/download.shtml
* http://pygame.org/ftp/pygame-1.9.1.win32-py2.7.msi

== Indentation ==

Python: Myths about Indentation - http://www.secnetix.de/olli/Python/block_indentation.hawk
:There are quite some prejudices and myths about Python's indentation rules among people who don't really know Python. I'll try to address a few of these concerns on this page.
*"Whitespace is significant in Python source code."
*"Python forces me to use a certain indentation style."
*"You cannot safely mix tabs and spaces in Python."
*"I just don't like it."
*"How does the compiler parse the indentation?"

== PEP - Python Enhancement Proposals ==

Python Enhancement Proposals (PEPs)

* Index of Python Enhancement Proposals (PEPs) - http://www.python.org/dev/peps/

== Convert Tabs to Spaces ==

Convert tabs to 4 spaces:
sed -i 's/\t/ /g' script.py

== Linting ==

* pep8 - python style guide checker
* pychecker - more verbose, imports code, slower
* pylint - most verbose, configurable, "rates" code
* pyflakes - least verbose (dead/redundant code)

rpmforge:
yum install pyflakes pychecker
# pylint - not sure which pacakge includes

=== pep8 ===

See [[Python/pep8]]

=== pychecker ===

See [[Python/pychecker]]

=== pylint ===

See [[Python/pylint]]

== PEP8 Style Guidlines ==

PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/

* Use spaces - Use 4 spaces per indentation level.
* Never mix tabs and spaces.
* Limit all lines to a maximum of 79 characters.
* Line wrapping - The preferred way of wrapping long lines is by using Python's implied line continuation inside parentheses, brackets and braces. Long lines can be broken over multiple lines by wrapping expressions in parentheses.
* Encoding - For Python 3.0 and beyond, UTF-8 is preferred over Latin-1, see PEP 3120.
* Imports should usually be on separate lines
* Avoid extraneous whitespace in most situations
* Always use 'self' for the first argument to instance methods.

=== PEP8 Style Checker ===

See [[#pep8]]

== Benefits of Python ==

What is the return on investment for this group by learning Python? The list of benefits below sums up what the language has to offer:
* Python is designed from the ground up to encourage maintainable, readable and reusable code. It has great support for particular paradigms such as object-oriented programming.
* Python code is usually around a quarter of the length of code generated by C++ or Java - less typing, less debugging and less maintenance.
* Python does not need to be compiled or linked, which allows for a more iterative programming style and a quicker development cycle.
* Python is extremely portable. Code can be easily exchanged between Windows, Mac OS and Linux with little to no modification.
* Python comes with a staggering array of standard libraries and mature third-party projects. Web development, scientific computing, operator system access and network scripting are all supported.
* Python can talk with many other languages easily. It has support for invoking C and C++ code and can itself be called from libraries written in these languages.

Source: http://quantstart.com/articles/Quant-Reading-List-Python-Programming/ - Quant Reading List Python Programming

Benefits of Python:
* Python is available for most operating systems, including Windows, UNIX, Linux, and Mac OS.
* Python's clean object-oriented design and extensive support libraries offer two to ten fold the programmer productivity seen with languages like C, C++, C#, Java, VB, and Perl.

Source: Python Development - Benefits of Python - Digital Mesh - http://www.digitalmesh.com/offshore-development-center/python-development/benefits-of-python.html

== Python Package Management ==

=== Install Module ===

Installing Python Modules — Python v2.7.1 documentation - http://docs.python.org/install/index.html

The new standard: Distutils
python setup.py install

Download and extract target package, then:
sudo python setup.py install

Note: Running setup.py install builds and installs all modules in one run

Help:
python setup.py --help

Incremental method:
python setup.py build
python setup.py install
python setup.py build --help # help

Force rebuild and install:
python setup.py build --force # if have to repeat
python setup.py install --force # if have to repeat

Default installation paths:
<pre>
Platform Standard installation location Default value
Unix (pure) prefix/lib/pythonX.Y/site-packages /usr/local/lib/pythonX.Y/site-packages
Unix (non-pure) exec-prefix/lib/pythonX.Y/site-packages /usr/local/lib/pythonX.Y/site-packages
Windows prefix C:\Python
</pre>

Alternate install path:
python setup.py install --home=<dir>

Alternate base folders:
<pre>
Type of file Installation Directory Override option
pure module distribution home/lib/python --install-purelib
non-pure module distribution home/lib/python --install-platlib
scripts home/bin --install-scripts
data home/share --install-data
</pre>

Modifying Python’s Search Path:
import sys
sys.path.append('/www/python/')

=== Python Package Index ===

user-submitted Python packages:
http://pypi.python.org/pypi

sudo python setup.py install

Package Site:
http://pypi.python.org/pypi/[PACKAGE]
http://pypi.python.org/pypi/test

If they are on http://pypi.python.org/:
pip install foo

[http://stackoverflow.com/questions/410163/for-python-programmers-is-there-anything-equivalent-to-perls-cpan For Python programmers, is there anything equivalent to Perl's CPAN? - Stack Overflow]

=== Easy Installation ===

EasyInstall - The PEAK Developers' Center - http://peak.telecommunity.com/DevCenter/EasyInstall
: Easy Install is a python module (easy_install) bundled with setuptools that lets you automatically download, build, install, and manage Python packages.

CheeseShopTutorial - PythonInfo Wiki - http://wiki.python.org/moin/CheeseShopTutorial
: EasyInstall (easy_install) gives you a quick and painless way to install packages remotely by connecting to the Package Index or even other websites via HTTP. It is somewhat analogous to the CPAN and PEAR tools for Perl and PHP, respectively.

=== pip ===

See [[Python/pip]]

==== Create PIP Package ====

See [[Python/pip#Create PIP Package]]

=== Distribute ===

distribute - Easily download, build, install, upgrade, and uninstall Python packages

distribute 0.6.24 : Python Package Index - http://pypi.python.org/pypi/distribute

Installation:
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/d/distribute/distribute-0.6.24.tar.gz#md5=17722b22141aba8235787f79800cc452
tar -zvxf distribute-0.6.24.tar.gz
cd distribute-0.6.24
sudo python setup.py install

Test:
echo "import distribute" | grep python

== Named Parameters ==

http://diveintopython.org/power_of_introspection/optional_arguments.html

Python allows function arguments to have default values; if the function is called without the argument, the argument gets its default value. Futhermore, arguments can be specified in any order by using named arguments. Stored procedures in SQL Server Transact/SQL can do this, so if you're a SQL Server scripting guru, you can skim this part.

Here is an example of info, a function with two optional arguments:

def info(object, spacing=10, collapse=1):

spacing and collapse are optional, because they have default values defined. object is required, because it has no default value. If info is called with only one argument, spacing defaults to 10 and collapse defaults to 1. If info is called with two arguments, collapse still defaults to 1.

Say you want to specify a value for collapse but want to accept the default value for spacing. In most languages, you would be out of luck, because you would need to call the function with three arguments. But in Python, arguments can be specified by name, in any order.

http://www.wellho.net/mouth/1871_Optional-and-named-parameters-in-Python.html

If you add a parameter with two asterixes (**) on the end of a function definition, that parameter is taken as being the name of a dictionary into which all otherwise unidentified parameters are stored.

<pre>
def getnet(gross, taxrate=17.5, **others):
net = gross / (1.0 + taxrate*0.01)
for more in others.keys():
print "GETNET:",more," - ",others[more]
return neto
</pre>

== Python vs other languages ==

[http://www.strombergers.com/python/ Python is Cool (and Perl is not), Especially for C/C++ Programmers]

==Experimenting with Memory Remanence==
[http://citp.princeton.edu/memory/exp/ Experimenting with Memory Remanence]

<pre>
#!/usr/bin/env python

# a pirate's favorite chemical element
a = ""
while 1: a += "ARGON"
</pre>

sudo strings /dev/mem | less

==UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)==
See [[UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)]]

== Program Layout Best Practices Presentation ==

Program layout best practices - Seth House <seth@eseth.com> - Utah Django User Group - 2011-06-23
* Slides - https://github.com/whiteinge/presentations/tree/master/upyug_2011-06-09_program-layout
* Video - http://blip.tv/utah-open-source/python-layout-best-practices-for-modules-django-cli-utah-django-user-group-5315808

Module skeleton:
<pre>
#!/usr/bin/env python
# coding: utf-8

"""Module docstring"""

__author__ = ’Mr. Me <me@example.net>’
__version__ = ’1.2.3’

if __name__==’__main__’:
main()
</pre>

Executing modules as scripts
# PEP 338:
python -m SimpleHTTPServer

== CLI package ==

cli — command line tools — pyCLI v1.1.1 documentation - http://packages.python.org/pyCLI/

== Python IDE ==

IntegratedDevelopmentEnvironments - PythonInfo Wiki - http://wiki.python.org/moin/IntegratedDevelopmentEnvironments
* IDEs with introspection-based code completion and integrated debugger
* IDEs with introspection-based code completion /or/ integrated debugger
* IDEs with integrated gui builder
* Editors
* Reviews

Is there a good, free Python IDE for Windows? - Stack Overflow - http://stackoverflow.com/questions/126753/is-there-a-good-free-python-ide-for-windows
* Eclipse with PyDev
* Aptana Studio 3
* Active State also offers Komodo Edit
* WingWare - http://www.wingware.com/
* Eric IDE - http://www.die-offenbachs.de/eric/index.html
* MMM-Experts - Products - http://mmm-experts.com/Products.aspx?ProductId=4
** PyScripter is a Python IDE built in Python for Delphi(P4D) components and created with the ambition to become competitive in functionality with commercial Windows-based IDEs available for other languages. Being built in a compiled language is rather snappier than some of the other Python IDEs and provides an extensive blend of features that make it a productive Python development environment.
* ActiveState Python comes with PythonWin which is reasonably good
* PyCharm from JetBrains is not free in general, but has free early access program.
* SPE - Stani's Python Editor, is pretty awesome. See http://pythonide.blogspot.com/.
* You may want to take a look at the pdb module - http://docs.python.org/lib/module-pdb.html
* Geany (http://www.geany.org/) is a fairly light text editor that does a lot of IDE-like stuff.
* SciTE: http://www.scintilla.org/SciTE.html
* Editra: http://editra.org/
* Netbeans has support for python development in their Python Early Access version. - http://www.netbeans.org/features/python/
* I use Notepad++ with the appropriate settings.
* The Eric Python IDE - http://www.die-offenbachs.de/eric/index.html
* BOa Constructor www.boa-constructor.sourceforge.net really nice! drag and drop stuff!
* DrPtyhon is also very nice, and has many plug-ins. http://drpython.sourceforge.net/
* Vim
* I will make a plug for my preference of Python development environments: SPE. SPE is an integrated development environment that provides code completion, integrated Python shell, calltips, and various helper tools like notes and todo lists. It also includes wxGlade so you can make wxPython applications easily. Plus, it includes the python debugger WinPdb.
* http://groups.google.com.tr/group/python_opengl this software is my sample visual python ide.its demo version.
* I use PyScripter, and find it pretty awesome.
* There is one Python IDE written in Java: jHepWork: http://jwork.org/jhepwork/ it can detect Python's indentation rules rather nicely (+ detects structure of a code and syntax)
* gvim built with your python version support and with this plugins:
** NERDTree (don't read he's blog ;)
** matchit (it's builtin in new versions just put a symlink or junction)
** pyflakes
** pysmell
* Try Python Tools for Visual Studio. Supports CPython and IronPython. - http://pytools.codeplex.com/

PythonEditors - PythonInfo Wiki - http://wiki.python.org/moin/PythonEditors
* List of editors

Python Ide - http://c2.com/cgi/wiki?PythonIde
: There are a lot of Ides for PythonLanguage springing up. Many of them written in Python

Is there a better Python IDE? « Lennart Regebro: Python, Plone, Web - http://regebro.wordpress.com/2010/10/09/is-there-a-better-python-ide/
: I’m currently a user of WingIDE, but sometimes I feel the need for something better, mainly something that is a bit more stable. WingIDE often hangs when switching projects, sometimes the file listing decides that you must double-click on the expand arrow instead of just click on it, etc. It still doesn’t have macros despite this being a much requested feature (how hard can it be to add, really?) and I never really get the hang of the SVN integration, so I just ignore it. And they keep recommending me to use WingIDE’s separate instance of GTK instead of the System GTK, which is daft. Their designs are ugly, and it should work with the system GTK so it looks the same. Anything else is just silly.

=== Open Komodo ===

Open Komodo - http://www.openkomodo.com/
: Open Komodo is the source code repository for Komodo Edit - a free multi-language editor for dynamic languages, based on the award-winning Komodo IDE. ActiveState created the Open Komodo code repository in August 2007 and it has been used since then for building all Komodo Edit releases.

=== Eclipse ===

Build an Eclipse development environment for Perl, Python, and PHP - http://www.eclipse.org/resources/resource.php?id=485
: This tutorial shows how Eclipse's DLTK makes it possible to build development tools for scripting languages. In particular, it explains how to implement syntax coloring, user preferences, and interpreter integration in a plug-in-based project.

See [[#PyDev]]

=== EasyEclipse ===

EasyEclipse - http://www.easyeclipse.org/
:EasyEclipse packages together Eclipse, the open-source software
development platform, and selected open source plugins.
:We select, assemble, test, patch, build installers and document
a full IDE, offered as reliable distributions and plugins.
:EasyEclipse is:
* Free and open-source,
* Easy to download and install, and
* Simple to maintain, without version and dependency issues.

EasyEclipse for Python - http://www.easyeclipse.org/site/distributions/python.html

EclipsePythonIntegration - PythonInfo Wiki - http://wiki.python.org/moin/EclipsePythonIntegration
: Using Eclipse as a Python editor

=== PyDev ===

PyDev - http://pydev.org/
: PyDev is a Python IDE for Eclipse, which may be used in Python, Jython and IronPython development.

---

Pydev - http://pydev.sourceforge.net/ (old link)

How to Configure Eclipse for Python - http://www.rose-hulman.edu/class/csse/resources/Eclipse/eclipse-python-configuration.htm

Python Development with PyDev and Eclipse - Tutorial - http://www.vogella.de/articles/Python/article.html
: This article describes how to write and debug Python programs with Eclipse

PyDev Getting Started - http://pydev.org/manual_101_root.html

Running a program - http://pydev.org/manual_101_run.html

---

Supported Task Tags:
* TODO:
* TASK:
* FIXME:

---

Installation:
# Platform Runtime Binary (download around 45-50 MB) - http://download.eclipse.org/eclipse/downloads/ or http://www.eclipse.org/downloads/
#* Example: Eclipse Classic 4.2.2
# Eclipse Software Add - PyDev: http://pydev.org/updates
# Eclipse Software Add - MercurialEclipse: http://cbes.javaforge.com/update [http://javaforge.com/project/HGE] [http://mercurial.selenic.com/wiki/MercurialEclipse]
# Switch perspectives to PyDev

=== The Eric Python IDE ===

The Eric Python IDE - http://eric-ide.python-projects.org/
:Eric is a full featured Python and Ruby editor and IDE, written in python. It is based on the cross platform Qt gui toolkit, integrating the highly flexible Scintilla editor control. It is designed to be usable as everdays' quick and dirty editor as well as being usable as a professional project management tool integrating many advanced features Python offers the professional coder. eric4 includes a plugin system, which allows easy extension of the IDE functionality with plugins downloadable from the net.

Eric Python IDE - Wikipedia - http://en.wikipedia.org/wiki/Eric_Python_IDE

=== Wingware Python IDE ===

The Intelligent Development Environment for Python Programmers - Wingware Python IDE - http://www.wingware.com/
:Wingware's Python IDE is an Integrated Development Environment designed specifically for the Python programming language. Since 1999, Wingware has focused on Python.

=== VIM ===

Python .vimrc:
<pre>
" Kenneth's Preferences

set nobackup " don't create backup files
"set nohlsearch " don't highlight search results
set hlsearch " do highlight search results
set incsearch " incremental search
set ignorecase " ignore case during search
set smartcase " do case sensitive search when upper case is typed
set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
set showmatch " briefly highlight matching bracket
set matchtime=1 " highlight match for 0.1 seconds
set background=dark " black background
set shiftwidth=4 " shift blocks by 4 spaces
set shortmess=at " use short messages so we don't have to hit enter
set expandtab " expand tabs into spaces
set scrolloff=5 " always show 5 lines before & after cursor
set shell=bash " use bash for the command shell
set shellcmdflag=-lc " make bash startup as a login shell for ~/.bashrc
"set nowrap " do not wrap long lines
set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set wildmenu " enable filename tab completion
set ruler " display the ruler: shows cursor position
set cpoptions+=$ " display '$' at end of cw sequence
set title " display filename and vim info in window title bar
set modelines=5 " search the first and last 5 lines for "vim:" modeline
set backspace=2 " backspace deletes

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

"if v:version > 700
" set cursorline " underline the current line
"endif

if &t_Co > 2 || has( "gui_running" )
syntax on
endif

if has( 'mouse' )
set mouse=a " enable mouse
endif

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

"folding settings
set foldmethod=indent "fold based on indent
set foldnestmax=10 "deepest fold is 10 levels
set nofoldenable "dont fold by default
set foldlevel=10 "this is just what i use

" Auto clear trailing white space in Python files
func! DeleteTrailingWS()
exec "normal mz"
%s/\s\+$//ge
exe "normal `z"
endfunc
autocmd BufWrite *.py :call DeleteTrailingWS()
</pre>

----

Older versions:
<pre>
" KENNETH SETTINGS
set nocp " turn off compatibility mode
set nomodeline " do not allow files to dictate vim behavior

set incsearch " incremental search
set ignorecase " ignore case during search

set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set background=dark " black background
syntax on " turn on syntax highlighting
set hlsearch " highligh last search pattern

set nowrap " do not wrap long lines
set scrolloff=5 " always show 5 lines before & after cursor
set ruler " show the cursor position all the time
set backspace=2 " allow backspace deletion
set bs=indent,eol,start " allow backspacing over everything in insert mode

set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set expandtab " expand tab into spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
filetype indent on " turn on auto indent for specific file types
</pre>

Other options:
<pre>
set number " show line numbers
set numberwidth=1 " remove space before largest line numbers

set smartindent " ?? - maybe included in 'filetype indent on'

set ai " always auto indent

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

</pre>

Kenneth's fio .vimrc:
<pre>

"source $VIM/mswin.vim

set nobackup " don't create backup files
"set nohlsearch " don't highlight search results
set incsearch " incremental search
set ignorecase " ignore case during search
set smartcase " do case sensitive search when upper case is typed
set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
set showmatch " briefly highlight matching bracket
set matchtime=1 " highlight match for 0.1 seconds
set background=dark " black background
set shiftwidth=4 " shift blocks by 4 spaces
set shortmess=at " use short messages so we don't have to hit enter
set expandtab " expand tabs into spaces
set scrolloff=5 " always show 5 lines before & after cursor
set shell=bash " use bash for the command shell
set shellcmdflag=-lc " make bash startup as a login shell for ~/.bashrc
"set nowrap " do not wrap long lines
set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set wildmenu " enable filename tab completion
set ruler " display the ruler: shows cursor position
set cpoptions+=$ " display '$' at end of cw sequence
set title " display filename and vim info in window title bar
set modelines=5 " search the first and last 5 lines for "vim:" modeline

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

if v:version > 700
set cursorline " underline the current line
endif

if &t_Co > 2 || has( "gui_running" )
syntax on
endif

if has( 'mouse' )
set mouse=a " enable mouse
endif

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

"folding settings
set foldmethod=indent "fold based on indent
set foldnestmax=10 "deepest fold is 10 levels
set nofoldenable "dont fold by default
set foldlevel=10 "this is just what i use
</pre>

== Interpreter vs Script ==

Interpreter:
>>> import os
>>> os.uname()
('Linux', 'demo.oeey.com', '2.6.18-194.8.1.el5', '#1 SMP Thu Jul 1 19:07:06 EDT 2010', 'i686')

Script:
#!/usr/bin/env python
import os
print os.uname()

== Web Development ==

WebProgramming - PythonInfo Wiki - http://wiki.python.org/moin/WebProgramming
: This topic guide attempts to cover every aspect of programming Web applications (both clients and servers) using Python.

HOWTO Use Python in the web — Python v2.7.2 documentation - http://docs.python.org/howto/webservers.html
: This document shows how Python fits into the web. It presents some ways to integrate Python with a web server, and general practices useful for developing web sites.

=== Plug Simple Python Web App ===

<pre>
-------- Original Message --------
Subject: Re: Writing a simple python web app
Date: Mon, 25 Oct 2010 19:03:31 -0400
From: Roberto Mello <roberto.mello@gmail.com>
To: Provo Linux Users Group <plug@plug.org>

On Sat, Sep 11, 2010 at 5:40 PM, Dave Smith <dave@thesmithfam.org> wrote:
> I'm creating a very simple web application, and I want to use python. My first guess was that mod_python
would provide the easiest entry point. Boy, was I wrong. All the mod_python tutorials spend 80% of their
time extolling the virtues of mod_python over CGI, but they are quite lean on specific examples. It's quite
disappointing. A natural progression then leads me to mod_wsgi, and since that has equally poor
documentation, to Django. However, Django is overkill for my app (I don't even have a database).

For something as simple as you have described, and to run on an
embedded platform, I would probably stay away from frameworks designed
to run on full hardware, and go with something more minimalistic.

Take a look at CherryPy [1] or web.py [2].

[1] http://www.cherrypy.org/
[2] http://webpy.org/

Roberto Mello
</pre>

---

See also [[web.py]]

See also [[CherryPy]]

See also [[Django]]

=== Apache CGI ===

See [[CGI]]

Great tutorial: https://www.tutorialspoint.com/python/python_cgi_programming.htm

My favorite option, make all .cgi files executable: (or .py)
<pre>
DirectoryIndex index.html index.php index.py
<Location />
# AddHandler cgi-script .cgi
AddHandler cgi-script .py
Options +ExecCGI
</Location>
</pre>

If you want to allow .htaccess:
* see [[Apache#.htaccess]]

To enable on Ubuntu:
sudo a2enmod cgi
service apache2 restart

Directory index:
DirectoryIndex index.py

Or if you really want to go crazy, this will try to run *every* file: [http://blog.kapsobor.de/archives/2006/09/05/apache_addhandler_multiple_extensions/]
<pre>
# this has problems with / directory being run as a script, instead of index too!
#<Location />
# SetHandler cgi-script
# Options +ExecCGI
#</Location>

<Files ~ ".">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>

or:
<pre>
<Files ~ "^.[a-zA-Z]*$">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>

Or just have index file:
<Files "index">

Note: Place within a <VirtualHost> section if needed.

Note: Make sure to 'chmod +x' the files.

Start all HTML output with:
print "Content-Type: text/html\n"

If you want just text output:
print "Content-Type: text/plain\n"

GET (from Environment):
<pre>
print "<pre>"
for env in os.environ:
print "%s = %s" % (env, os.environ[env])
print "</pre>"
</pre>

<pre>
# sample for: http://oeey.com/py/test.py/test?var1=ken&var2=barbie

REQUEST_URI = /py/index.py/test?var1=ken&var2=barbie
QUERY_STRING = var1=ken&var2=barbie
PATH_INFO = /test
REQUEST_METHOD = GET

SCRIPT_NAME = /py/test.py
SERVER_NAME = oeey.com
REMOTE_ADDR = 216.51.42.66
SERVER_ADDR = 192.168.108.4
DOCUMENT_ROOT = /www/py
SCRIPT_FILENAME = /www/py/test.py
HTTP_HOST = cgi.oeey.com
</pre>

POST (from stdin):
<pre>
post = sys.stdin.readline()
if post:
print post
</pre>

<pre>
CONTENT_TYPE = application/x-www-form-urlencoded 

yin=one+two+three&yang=four+five+six
</pre>

cgi helper class:
<pre>
import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage()

# field keys
print "fields keys: ", form.keys() # list of field keys

# fild values
if "name" in form: # if form.has_key('name'):
print "field 'name' is: ", form["name"].value
print "field 'name' is: ", form.getvalue('name')

# headers
print "dictionary of headers: ", form.headers
print form.type
print form.headers['content-type']
</pre>

To prevent XSS escape the characters:
print cgi.escape(form.getvalue('name'))

<pre>
import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage()
if (form.has_key("action") and form.has_key("name") \
and form.has_key("age")):
if (form["action"].value == "display"):
print "%s = %s" % (form["name"].value form["age"].value)
</pre>

Note: cgi helper class also does sessions and cookies.

References:
* CGI Tutorial | Web Python - http://webpython.codepoint.net/cgi_tutorial
* Python CGI: An Interactive Instruction - http://www.cs.virginia.edu/~lab2q/

See also:
* Web Python - http://webpython.codepoint.net/

---

REST Example:

<pre>
import os, re
def get_rest():
if not "PATH_INFO" in os.environ:
return ()
# filter for garbage
#print os.environ["PATH_INFO"]
path_info = re.findall('[/A-Za-z0-9_-]*', os.environ["PATH_INFO"])
if not path_info:
return ()
path_info = ''.join(path_info)
# collapse empty paths
paths = path_info.split('/')
final_paths = []
for path in paths:
if path:
final_paths.append(path)
return tuple(final_paths)
rest = get_rest();
</pre>

=== mod_python ===

AVOID! AVOID! AVOID! - if you want this simple, just use CGI!

See [[Apache mod_python]]

mod_python - http://docs.python.org/howto/webservers.html#mod-python

Install:
yum install mod_python

Config:
/etc/httpd/conf.d/python.conf

Simple:
<pre>

</pre>

:"People coming from PHP often find it hard to grasp how to use Python in the web. Their first thought is mostly mod_python, because they think that this is the equivalent to mod_php. Actually, there are many differences. What mod_python does is embed the interpreter into the Apache process, thus speeding up requests by not having to start a Python interpreter for each request. On the other hand, it is not “Python intermixed with HTML” in the way that PHP is often intermixed with HTML. The Python equivalent of that is a template engine. mod_python itself is much more powerful and provides more access to Apache internals. It can emulate CGI, work in a “Python Server Pages” mode (similar to JSP) which is “HTML intermingled with Python”, and it has a “Publisher” which designates one file to accept all requests and decide what to do with them.

:mod_python does have some problems. Unlike the PHP interpreter, the Python interpreter uses caching when executing files, so changes to a file will require the web server to be restarted. Another problem is the basic concept – Apache starts child processes to handle the requests, and unfortunately every child process needs to load the whole Python interpreter even if it does not use it. This makes the whole web server slower. Another problem is that, because mod_python is linked against a specific version of libpython, it is not possible to switch from an older version to a newer (e.g. 2.4 to 2.5) without recompiling mod_python. mod_python is also bound to the Apache web server, so programs written for mod_python cannot easily run on other web servers.

:These are the reasons why mod_python should be avoided when writing new programs. In some circumstances it still might be a good idea to use mod_python for deployment, but WSGI makes it possible to run WSGI programs under mod_python as well."

=== WSGI ===

WSGI - http://docs.python.org/howto/webservers.html#mod-wsgi

: "The Web Server Gateway Interface, or WSGI for short, is defined in PEP 333 and is currently the best way to do Python web programming. While it is great for programmers writing frameworks, a normal web developer does not need to get in direct contact with it. When choosing a framework for web development it is a good idea to choose one which supports WSGI."

See [[mod_wsgi]]

=== Django ===

See [[Django]]

=== CherryPy ===

See [[CherryPy]]

=== web.py ===

web.py: makes web apps

Think about the ideal way to write a web app. Write the code to make it happen.

web.py : Python Package Index - http://pypi.python.org/pypi/web.py

Home Page: http://webpy.org/

"Django lets you write web apps in Django. TurboGears lets you write web apps in TurboGears. Web.py lets you write web apps in Python."
:— Adam Atlas

NOTE: Can modify source code without having to restart app! :-)

---

PIP Install:
pip install web.py

---

Hello World: [http://webpy.org/]
<pre>
import web

urls = (
'/(.*)', 'hello'
)
app = web.application(urls, globals())

class hello:
def GET(self, name):
if not name:
name = 'World'
return 'Hello, ' + name + '!'

if __name__ == "__main__":
app.run()
</pre>

Note: app.py can be renamed.

Running:
python app.py

Note: You can add an IP address/port after the "app.py" bit to control where web.py launches the server. You can also tell it to run a fastcgi or scgi server.
python app.py 1234

---

Templates

<pre>
# templates/hello.html
$def with (name)

$if name:
I just wanted to say hello to $name.
$else:
Hello, world!
</pre>

app = ...
render = web.template.render('templates/')

index.GET:
name = 'Bob'
return render.hello(name) # render.[TEMPLATE_NAME](vars)

get variable: (/?name=Joe)
i = web.input(name=None)
return render.hello(i.name)

variable in url:
urls = ('/(.*)', 'hello')

class index:
def GET(self, name):
return render.hello(name)

# visit: /Joe

multiple variables in url:
urls = ('/(.*)/(.*)', 'hello')

class hello:
def GET(self, fname, lname):
return render.hello(fname, lname)

# visit: /Joe/Smith

=== lpthw.web ===

lpthw.web: a fork of web.py for Learn Python The Hard Way

Locks web.py at a specific version and removes some magic. Thanks to Aaron Swartz for making it originally.

lpthw.web: a fork of web.py for Learn Python The Hard Way - http://pypi.python.org/pypi/lpthw.web

Home Page: http://learnpythonthehardway.org/

This conflicts with web.py, so only install one.

=== web ===

web - Web modules for CGI and WSGI web programming

web : Python Package Index - http://pypi.python.org/pypi/web

Warning: This is an alpha release for those who are interested in
checking progress. No documentation and not for production use.

Auth, session, mail, form and templating tools for writing CGI scripts or WSGI web applications.

This version incompatible with previous versions becuase large portions of the code have been refactored into separate components.

== Issues ==

=== Hide DeprecationWarning ===

==== Import DeprecationWarning ====

Error:
<pre>
/opt/xmpppy-0.4.1/xmpp/auth.py:24: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import sha,base64,random,dispatcher,re
/opt/xmpppy-0.4.1/xmpp/auth.py:26: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5
</pre>

Hide deprecation warnings with code: [http://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python]
<pre>
import warnings

with warnings.catch_warnings():
warnings.filterwarnings("ignore",category=DeprecationWarning)
import md5, sha
</pre>

Hide with header:
#!/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line:
python -W ignore::DeprecationWarning script.py

References:
* How to ignore deprecation warnings in Python - Stack Overflow - http://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python

==== Funciton DeprecationWarning ====

Error:
<pre>
/opt/xmpppy-0.4.1/xmpp/transports.py:307: DeprecationWarning: socket.ssl() is deprecated. Use ssl.wrap_socket() instead.
tcpsock._sslObj = socket.ssl(tcpsock._sock, None, None)
</pre>

Hide with header:
#!/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line:
python -W ignore::DeprecationWarning script.py

==== Ubuntu 24 pip packages ====

<pre>
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
</pre>

== keywords ==

[[Category:Programming]]
[[Category:Python]]

Python

2026-04-27T20:28:18Z

Kenneth: /* doc strings */

{{subtop}}

== Python ==

Python (programming language) - Wikipedia - http://en.wikipedia.org/wiki/Python_(programming_language)
: Python is an interpreted, general-purpose high-level programming language whose design philosophy emphasizes code readability. Python aims to combine "remarkable power with very clear syntax", and its standard library is large and comprehensive. Its use of indentation for block delimiters is unique among popular programming languages.
: Python supports multiple programming paradigms, primarily but not limited to object-oriented, imperative and, to a lesser extent, functional programming styles. It features a fully dynamic type system and automatic memory management, similar to that of Scheme, Ruby, Perl, and Tcl. Like other dynamic languages, Python is often used as a scripting language, but is also used in a wide range of non-scripting contexts.
: The reference implementation of Python ([http://en.wikipedia.org/wiki/CPython CPython]) is free and open source software and has a community-based development model, as do all or nearly all of its alternative implementations. CPython is managed by the non-profit Python Software Foundation.
: Python interpreters are available for many operating systems, and Python programs can be packaged into stand-alone executable code for many systems using various tools.

Author:
:"Python was conceived in the late 1980s and its implementation was started in December 1989 by '''[http://www.python.org/~guido/ Guido van Rossum]''' at CWI in the Netherlands as a successor to the ABC programming language (itself inspired by SETL) capable of exception handling and interfacing with the Amoeba operating system. Van Rossum is Python's principal author, and his continuing central role in deciding the direction of Python is reflected in the title given to him by the Python community, Benevolent Dictator for Life (BDFL)." [http://en.wikipedia.org/wiki/Python_(programming_language)]

Name:
:"The language is named after the BBC show “Monty Python’s Flying Circus” and has nothing to do with reptiles. Making references to Monty Python skits in documentation is not only allowed, it is encouraged!" [http://docs.python.org/py3k/tutorial/appetite.html]

We as people are "Pythonites" [http://raspi.tv/2013/how-to-use-interrupts-with-python-on-the-raspberry-pi-and-rpi-gpio-part-2]

== [http://www.python.org/dev/peps/pep-0020/ The Zen of Python] ==

"The core philosophy of the language is summarized by the document 'PEP 20 (The Zen of Python)'" [http://en.wikipedia.org/wiki/Python_%28programming_language%29]
* PEP 20 -- The Zen of Python - http://www.python.org/dev/peps/pep-0020/

<pre>
Abstract

Long time Pythoneer Tim Peters succinctly channels the BDFL's
guiding principles for Python's design into 20 aphorisms, only 19
of which have been written down.

The Zen of Python

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Easter Egg

>>> import this
</pre>

Other Sources:
* http://www.python.org/doc/humor/
* The Zen of Python – as a Poster « CODE POETRY - http://codepoetry.wordpress.com/2010/03/23/the-zen-of-python-as-a-poster/
** Zen of Python (PDF) http://codepoetry.files.wordpress.com/2010/03/zen-of-python.pdf

== Documentation ==

* Overview — Python v2.7.1 documentation - http://docs.python.org/index.html
** Download — Python v2.7.1 documentation - http://docs.python.org/download.html

* Overview — Python v3.2 documentation - http://docs.python.org/py3k/
** Download — Python v3.2 documentation - http://docs.python.org/py3k/download.html

* PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/

== Tutorials ==

* The Python Tutorial — Python v2.7.1 documentation - http://docs.python.org/tutorial/

* Python 101 -- Introduction to Python - http://www.rexx.com/~dkuhlman/python_101/python_101.html

* 4. More Control Flow Tools — Python v2.7.1 documentation - http://docs.python.org/tutorial/controlflow.html

== Linux Installation ==

RHEL Distribution installation:
yum -y install python python-setuptools

Ubuntu Distribution installation: [https://help.ubuntu.com/community/Mercurial]
# enable 'universe' in the sources.list
# "Mercurial is included in the Universe repository - activate this repo first"
sudo apt-get install mercurial

CentOS 5 Option Recommended Dependencies:
yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel

# this leaves: 2.6.6: _bsddb, bsddb185, dl, imageop, sunaudiodev
# this leaves: 2.7.1: _bsddb, bsddb185, dl, imageop, sunaudiodev
# this leaves: 3.2: none

CentOS 9 Option Recommended Dependencies:
# gdbm-devel not found in centos9?
yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel sqlite-devel bzip2-devel tk-devel tcl-devel

Ubuntu 12.04 LTS 64bit dependencies:
apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev

# maybe: lib64readline-dev libsqlite3-dev tk-dev

# this leaves: 2.7: _bsddb, bsddb185, dl, imageop, sunaudiodev

=== 2.x ===

CentOS dependencies:
sudo yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel

Ubuntu 12.04 LTS 64bit dependencies:
sudo apt-get update
sudo apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev

Ubuntu 24.04 LTS 64bit dependencies:
sudo apt-get update
sudo apt-get install gcc make sudo libbz2-dev tcl-dev libncurses-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev libffi-dev
# optional install
sudo apt-get install checkinstall

2.7.15:
<pre>
# PYVER=2.6.6
# PYVER=2.7.5
# PYVER=2.7.6
# PYVER=2.7.9
# PYVER=2.7.10
# PYVER=2.7.15
PYVER=2.7.18
mkdir -p ~/.src ; cd ~/.src
wget --no-check-certificate http://www.python.org/ftp/python/$PYVER/Python-$PYVER.tgz
tar -zvxf Python-$PYVER.tgz
cd Python-$PYVER
# ./configure --prefix=/opt/python-$PYVER
# make clean && make
./configure --prefix=/opt/python-$PYVER && make clean && make
sudo make install

sudo /opt/python-$PYVER/bin/python -m ensurepip --upgrade
sudo /opt/python-$PYVER/bin/pip install --upgrade pip
sudo /opt/python-$PYVER/bin/pip install requests

export PATH=/opt/python-$PYVER/bin:$PATH
echo -e "\n\nexport PATH=/opt/python-$PYVER/bin:\$PATH\n" >> ~/.bash_profile
# echo -e "\n\nexport PATH=/opt/python27/bin:\$PATH\n" >> ~/.bashrc

# NOTE: remove any old versions from your .bash_profile
</pre>

Bootstrapping the pip installer - https://docs.python.org/2/library/ensurepip.html

=== 3.x ===

3.8.16:
<pre>
#VER=3.7.0
#VER=3.8.15
#VER=3.10.9
VER=3.8.16
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/${VER}/Python-${VER}.tgz
tar -zvxf Python-${VER}.tgz
cd Python-${VER}
./configure --prefix=/opt/python${VER}
make

sudo make install

## OPTIONAL alternative install path:
sudo checkinstall
## note: 'checkinstall' makes a .deb, which you can uninstall with easier, but if you are using --prefix, you do need to
## premake the folders
sudo mkdir -p /opt/python${VER}/bin
sudo mkdir -p /opt/python${VER}/lib
shortVER=`echo $VER | cut -d '.' -f 1-2` # get short version eg. 3.10 from 3.10.9
sudo mkdir -p /opt/python${VER}/lib/python${shortVER}
sudo mkdir -p /opt/python${VER}/lib/python${shortVER}/test
# sudo checkinstall # hit Y, and enter twice
## note: older versions have problems with the virtualization of the file system, so do FS Translation = no
checkinstall --fstrans=no

export PATH=/opt/python${VER}/bin:$PATH
echo -e "\n\nexport PATH=/opt/python${VER}/bin:\$PATH\n" >> ~/.bash_profile

# have to manually update version
rm /opt/python${VER}/bin/python
ln -sfn python3.8 /opt/python${VER}/bin/python
#cp /opt/python${VER}/bin/python3.8 /opt/python${VER}/bin/python

# DEPENDENCIES
sudo apt install ...
yum install libffi-devel openssl-devel
</pre>

Note, if you want the libraries to be shared (like python-dev):
./configure --enable-shared

---

VER=3.6.9

VER=3.6.8

---

3.2.0:
<pre>
mkdir -p ~/src ; cd ~/src
wget http://www.python.org/ftp/python/3.2/Python-3.2.tgz
tar -zvxf Python-3.2.tgz
cd Python-3.2
./configure --prefix=/opt/python32
make
sudo make install

export PATH=/opt/python32/bin:$PATH
echo -e "\n\nexport PATH=/opt/python32/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.3.0:
<pre>
mkdir -p ~/src ; cd ~/src
wget http://www.python.org/ftp/python/3.3.0/Python-3.3.0.tgz
tar -zvxf Python-3.3.0.tgz
cd Python-3.3.0
./configure --prefix=/opt/python3.3.0
make
sudo make install

export PATH=/opt/python3.3.0/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.3.0/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.3.4:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/3.3.4/Python-3.3.4.tgz
tar -zvxf Python-3.3.4.tgz
cd Python-3.3.4
./configure --prefix=/opt/python3.3.4
make clean
make
sudo make install

export PATH=/opt/python3.3.4/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.3.4/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.4.0:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz
tar -zvxf Python-3.4.0.tgz
cd Python-3.4.0
./configure --prefix=/opt/python3.4.0
make
sudo make install

export PATH=/opt/python3.4.0/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.4.0/bin:\$PATH\n" >> ~/.bash_profile
</pre>

=== setuptools ===

Purpose: to install pip

Note: required for several 'python setup.py install' installations.

Python Package Index : setuptools 0.6c11 - http://pypi.python.org/pypi/setuptools

By Package Manager:
yum install python-setuptools

By Source:
#VER=2.0.1
#VER=2.2
VER=3.4.4
mkdir -p ~/.src ; cd ~/.src
wget --no-check-certificate http://pypi.python.org/packages/source/s/setuptools/setuptools-$VER.tar.gz
tar -zvxf setuptools-$VER.tar.gz
cd setuptools-$VER
# sudo python setup.py install
sudo /opt/python-2.7.9/bin/python setup.py install

Fix header:
# Only needed on older versions
sed -i 's#!/usr/bin/python#!/usr/bin/env python#g' /opt/python279/bin/easy_install

Install pip:
sudo /opt/python-2.7.9/bin/easy_install pip

Sample pip usage:
sudo /opt/python-2.7.9/bin/pip install paramiko

== IDLE and Interactive Mode ==

Interactive Mode - "When commands are read from a tty, the interpreter is said to be in interactive mode. In this mode it prompts for the next command with the primary prompt, usually three greater-than signs (>>>); for continuation lines it prompts with the secondary prompt, by default three dots (...)." [http://docs.python.org/2/tutorial/interpreter.html#invoking-the-interpreter]

Can use '_' to reference last output:
>>> 5 + 6
11
>>> print _
11

References:
* 2. Using the Python Interpreter — Python v2.7.3 documentation - http://docs.python.org/2/tutorial/interpreter.html

== Python 2.7 EOL ==

<blockquote>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
</blockquote>

== Show All Python in Path ==

compgen -c | grep -i "python"

== Code ==

=== Hello World ===

[http://en.wikipedia.org/wiki/Shebang_%28Unix%29 shebang] (also called a hashbang) of Python:
#!/usr/bin/env python
print "Hello World"

NOTE: can also just use "#!/usr/bin/python"

=== Standard Script ===

Standard Script:
<pre>
def main(args):
# do logic

if __name__ == "__main__":
sys.exit(main(sys.argv))
</pre>

Needed for things like pylint

=== White Space ===

Python uses white space to indicate code blocks. Instead of ''{'' use a '':'' and indent consistently (4 spaces).

if a == b:
print 'equal'

Convert tabs to spaces:
sed -i 's/\t/ /g' *.py

Remove trailing whitespace:
sed -i 's/\s*$//' *.py
sed --in-place 's/[[:space:]]\+$//' fileName

Clean project:
# find . -name "*py" -exec dos2unix {} \;
# find . -name "*py" -exec sed -i 's/\t/ /g' {} \;
# find . -name "*py" -exec sed -i 's/\s*$//' {} \;
find . -name "*py" -exec dos2unix {} \; -exec sed -i 's/\t/ /g' {} \; -exec sed -i 's/\s*$//' {} \;

==== Trim Tailing White Space ====

<pre>
sed -e 's/[[:space:]]*$//'
</pre>

<pre>
find . -name "*.py" -exec sed -i -e 's/[[:space:]]*$//' {} \;
</pre>

ref: [https://stackoverflow.com/questions/369758/how-to-trim-whitespace-from-a-bash-variable]

=== Print ===

print - http://docs.python.org/tutorial/inputoutput.html

Print: # include new line
print("Hello World")
print "Hello World" # python 2 only

Suppress new line: [http://stackoverflow.com/questions/493386/how-to-print-in-python-without-newline-or-space]
<pre>
print('#', end='') # python 3
print "Hello World", # python 2 only (does produce a single separating white space though)

import sys
sys.stdout.write('.') # no separating white space
</pre>

# flush output
# optional, if new line won't come for some time
import sys
sys.stdout.flush()

Multiple with separator:
print("hello", "world") # separated by spaces
print("hello", "world", sep = '.') # separated by period

Formatting:
<pre>
mystr = "abc"
myint = 5
myflt = 3.141592
print( "%s %d %.3f" % (mystr, myint, myflt) )
print( "{0} {1} {2}".format(mystr, myint, myflt) ) # string's format function (python > 2.7)
print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name
print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding
</pre>

Padded floating point decimal number: (rounds)
print "%06.2f" % 2.319 # "002.32"
print "%6.2f" % 2.311 # " 2.31"
print "{:06.2f}".format(2.319) # "002.32"
print "{:6.2f}".format(2.319) # " 2.32"

# Python 3 version where var=2.321234
print(f"{var:.2f}") # "2.32"
print(f"{var:6.2f}") # " 2.32"
print(f"{var:06.2f}") # "002.32"
print(f"{var:08.2f}") # "00002.32"
print(f"{var:08.4f}") # "002.3212"
# Python 3 version where var=2.329 - rounding
print(f"{var:06.2f}") # "002.33"

=== Comments ===

# this is a comment
# this is a comment
x = y # this is a comment

Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended)
"""
multi line comment
"""

'''
multi line comment can comment out """
'''

This can be used like a C "#if 0" concept to comment out whole sections of code.

Using false block to comment out a block code: (does not ignore bad syntax though)
if False:
# code in this block is ignored

=== doc strings ===

Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended)
"""
multi line comment
"""

== Variable Type ===

print( type(myvar).__name__ )

Used to be able to just print the type(myvar), but this doesn't seem to work anymore :-(

== Variable Directory ===

All of the properties and functions of a variable or object

print( dir(myvar) )

=== Help ===

dir - list functions and properties of objects:
import math
dir(math) # module
dir(dir) # function
dir("") # object - "", [], {}, etc...
dir(__builtins__) # built in functions
dir() # list imported objects, and dunder methods

Note: dunder (double under) methods determine special functionality (eg. __add__ for +)

help - manual printout: (class doc string, function doc string, data, etc)
import math
help(math) # full module info
help(len) # full function info
help(["foo"]) # full object info

Pydoc: (command line, same as 'import math, help(math)'
$ pydoc math

Documentation String: (aka docstring, doc string)
print(math.pow.__doc__) # module doc string
print(len.__doc__) # function doc string

Object or function type:
type(x)

mylist = []
isinstance(mylist, list) # True
type(s) is list # True

Note: isinstance() and type() not equivalent when objects and subclass get involved

IDLE interactive help:
help()

=== Variables ===

Basic Data Types
str = "string" # string
num = 5 # integer
flt = 5.0 # float
nothing = None # None (not the same as undefined)

Undefined. Variables start off in an 'undefined' state. It is always better to initially set the variable to None to avoid this state. To check if undefined: [http://code.activestate.com/recipes/59892-testing-if-a-variable-is-defined/] [http://code.activestate.com/recipes/59892/]
<pre>
try:
x
except NameError:
x = None
</pre>

Rules: [http://www.python.org/dev/peps/pep-0008/ PEP8 - Method Names and Instance Variables]
* any length
* letters, numbers, underscore
* first character can't be number
* case sensitive
* cannot use keywords

Multiple assignment:
x = y = z = 0 # Zero x, y and z
x, y, z = 1, "two", 3.0 # x = 1, y = "two", z = 3.0

Swap variables: (done in parallel)
a, b = b, a

Unpack variables: (requires equal items on each side)
a, b, c = [1, 2, 3]

Other Data Types:
* tuple
* list
* dictionary
* set
* objects

Everything in Python is an object that has:
* identity: id(obj)
* type: type(obj)
* value: obj
** mutable - id() stays the same on modification (dictionary, lists, objects)
** immutable - id() changes on modification as new object is created (string, integer, tuple)

==== Check if Variable Exists ====

<pre>
try:
a # does a exist in the current namespace
except NameError:
a = 10 # nope
</pre>

'a' in vars() or 'a' in globals()
'a' in vars(__builtins__)

<pre>
if hasattr(a, 'property'):
doStuff(a.property)
else:
otherStuff()
</pre>

Ref: [https://stackoverflow.com/questions/1592565/determine-if-variable-is-defined-in-python] [https://stackoverflow.com/questions/610883/how-to-know-if-an-object-has-an-attribute-in-python]

=== Numbers ===

num = 5
print "num: %d" % num

>>> print "Today's stock price: %f" % 50.4625 1
50.462500
>>> print "Today's stock price: %.2f" % 50.4625 2
50.46
>>> print "Change since yesterday: %+.2f" % 1.5 3
+1.50

Arithmetic:
+ - * /
// # integer division
% # modulus
** # exponent

Compact Arithmetic:
i+=1 # Increment - Note, there is no 'i++'
i*=2 # Multiply
i%=2 # modulus
...

Math Functions:
import math
math.ceil(x)
math.floor(x)
math.trunc(x)
math.pow(x,y)

abs(x) # built in

String interpolation:
%% % character
%d integer
%x hex
%X hex (upper)
%f float
%s string

x = 1/81
print('value: %.2f' % x) # 0.01
print('value: %.5f' % x) # 0.012345
print('%d %d %d' % (1, 2, 3))

Type check:
if not isinstance(myvar, (int, float)): ...

=== Strings ===

STRINGS ARE IMMUTABLE! (to modify convert to a list)

Strings: (String Constants)
str = "Hello"
str = 'Hello'
str = """
Hello
World
""" # can also use '''...'''

String literals can span multiple lines: (no wrapping when \ used)
hello = "hello\
world"

Length:
len("str")

Raw string: (back slashes are not escaped)
myrawstr = r"hello\nworld\
how are you"
myrawstr = r"C:\" # ERROR - limitation of what won't be escaped

Concatenation:
str = 'hot' + 'dog'
print "Hello" "World"

Can't concat numbers
str = 5 + 'test' # error
num = 5
str = num + 'test' # error
str = str(5) + str(num) + 'test' # ok

Repeated Concatenation:
str = 10 * 'ha'
str = 'ha' * 10 # same
print '-'*60 # --------------------

String Indexing:
s = 'hello'
print( s[0] ) # is 'h'
print( s[-1] ) # is 'o'
print( s[len(s) - 1 ] ) # is also 'o'

Modify Strings: [http://stackoverflow.com/questions/1228299/change-one-character-in-a-string-in-python]
s[1] = 'x' # ERROR!
ERROR: can't modify string by index

# solution:
s = list("Hello World")
s[0] = 'C'
print( "".join(s) )

String Slicing:
s = 'hello'
print( s[0:3] ) # 'hel'
i = 1
print( s[i:i+1] ) # 'e'
print( s[:3] ) # 0:3 'hel'
print( s[4:] ) # 4:end 'o'
print( s[:] ) # start:end 'hello'
print( s[-2:] ) # -2:end 'lo'

For loop:
s = 'hello'
for c in s:
print c # c = 'h' first time

Escape characters:
\\ \' \" \n \r \t

Alternate loop:
s = 'hello'
for i in range( len(s) ):
print s[i]

Ord and Char (ASCII):
ord('a') # 97
chr(97) # 'a'

0-9 : 48-57
A-Z : 65-90
a-z : 97-122

List Functions: (dir on any string)
dir('')
dir(str)

Help:
help('')
help(str)

Padding:
"a".rjust(2, '0') # '0a'
"1".ljust(4, '0') # '1000'

String Test Functions:
"HELLO".isupper() # True
"hello".islower() # True
"hello".isalpha() # True
"12.22".isdigit() # False (because of '.')
"12.22".isnumeric() # False (because of '.')
"hel2".isalnum() # True
"e" in "hello" # True
"hello".startswith('he') # True
"hello".endswith('lo') # True

Note: To check for floats: [http://stackoverflow.com/questions/354038/how-do-i-check-if-a-string-is-a-number-in-python]
<pre>
def is_number(s):
try:
float(s)
return True
except ValueError:
return False
</pre>

String Modify Functions: (sort of... Strings are immutable)
"HELLO".lower() # 'hello'
"hello".upper() # 'HELLO'
" hello ".strip() # 'hello' remove leading/tailing whitespace
" hello ".lstrip() # 'hello ' remove left whitespace
" hello ".rstrip() # ' hello' remove right whitespace
"www.com".split('.') # ['www', 'com'] <- list
"www.com".partition('.') # ('www', '.', 'com') <- tuple
"".join( ('a','b','c') ) # "abc" join strings, list or tuple
"-".join( ('a','b','c') ) # "a-b-c" join with separator
"23".zfill(4) # "0023"

String Search Functions:
"hello".find('lo') # 3 (-1 on fail)
"hello".index('lo') # 3 (exception on fail)
"hello".rfind('lo') # 3 right to left (-1 on fail)
"hello".rindex('lo') # 3 right to left (exception on fail)

Formatting: (form of concatenation)
<pre>
mystr = "abc"
myint = 5
myflt = 3.141592
print( "%s %d %.3f" % (mystr, myint, myflt) )
print( "{0} {1} {2}".format(mystr, myint, myflt) ) # pep 3101 style - string's format function (python > 2.7)
print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name
print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding
</pre>

Split multi line string:
for line in str.split('\n'): print(line)
lines = str.splitlines() # returns list
lines = str.splitlines(True) # returns list (each includes new line characters)

Input a string:
name = input('What is your name: ').strip() # python 3
name = raw_input('What is your name: ').strip() # python 2

Leet Speak:
leet_table = ''.maketrans('EIOBT', '31087')
'BE COOL. SPEAK LEET!'.translate(leet_table)

String Functions:
newstr = mystr.strip() # remove white space on both sides
newstr = mystr.rstrip() # remove white space on right side
newstr = mystr.lstrip() # remove white space on left side
newstr = mystr.replace('\n', ' ') # replace string with another
newstr = mystr.replace('jello', 'hello') # replace string with another

==== String Formatting ====

PyFormat: Using % and .format() for great good! - https://pyformat.info/

===== Zero Padding =====

3 zero padding:
"{:03}".format(1) # 003

3 zero padding, specify position:
"{1:03}".format(1, 2) # 002

Old method:
"%03d" % 2 # 002

=== chr and ord ===

Ord and Char (ASCII):
ord('a') # 97
chr(97) # 'a'

=== Range ===

Creates a list of numbers. Remember: range always ends one less than you want!

Count:
ten = 10
one = 1
range(ten) # [0,1,...,9] starts with zero by default
range(one, ten + 1) # [1,...,10] actually get 1 to 10
range(ten, one - 1, -1) # [10,9,...,1] count down
range(2, 10 + 1, 2) # [2,4,...,10] even numbers

=== drange ===

Solution for decimal range() step value:
<pre>
def drange(start, stop, step):
r = start
while r < stop:
yield r
r += step

sub_ten = drange(0.0, 1.0, 0.1)
</pre>

References:
* floating point - Python decimal range() step value - Stack Overflow - http://stackoverflow.com/questions/477486/python-decimal-range-step-value

=== Casting ===

Convert:
float(3)
str(85)
int(8.6)
round(5.5)

Check Type:
type(5) # <class 'int'>
type(5.0) # <class 'float'>
type('5') # <class 'str'>
type(None) # <class 'NoneType'>
type(print) # <class 'builtin_function_or_method'>
type(()) # <class 'tuple'>
type([]) # <class 'list'>

mytype = int
if type(1) is mytype: ...

=== Boolean Logic ===

True and False:
bol = True
bol = False

Boolean Operators: (in order precedence)
a == b
a != b
not a
a and b
a or b

Note: Short-circuit evaluated - 'and' and 'or'

Return Boolean:
return myint >= 16

Get Boolean from string: [http://stackoverflow.com/questions/715417/converting-from-a-string-to-boolean-in-python]
# Note: no built in way to do this

myString = "false"
val = (myString == "true")

True if myString=="True" else False

s in ['true', '1', 't', 'y', 'yes', 'yeah', 'yup', 'certainly', 'uh-huh']

def str2bool(v):
return v.lower() in ("yes", "true", "t", "1")

=== None ===

None is Python's version of NULL or NIL.

None is a singleton (Python only has one copy of None in the interpreter).

if myvar is None:
...

=== If Conditionals ===

Operators:
==, !=, not, and, or, <, >, <>, <=, >=, in, not in, is, is not

If statement:
<pre>
if a == b:
...
elif c < age < d:
...
else:
...
</pre>

Shorthand conditional: (ternary operator)
val = True if a == b else False

is - checks for identical objects - same id()
# None is a singleton, so this works great for None
if myvar is None:
...

Other shorthand:
<pre>
def mycmp(x, y):
if x > y: return True
else: return False
</pre>

Switch/Case statement:
* there is no built in switch/case statement, use "if ... elif ... elif ..." sequence instead

=== For Loop ===

''for'' is used for iterating over an iterable sequence (eg. lists, dictionaries, tuples). (for each)

# range(start, end + 1, skip)

for i in range(10):
print(i) # 0 .. 9

for i in range(5, 10):
print(i) # 5 .. 9

for i in range(10, 0, -1):
print(i) # 10 .. 1

for i in [1, 2, 3]: # iterate over list
print(i) # 1 2 3

Discard variable: [http://bytes.com/topic/python/answers/757444-loop-without-variable]
for _ in range(10): # repeat and discard variable (good for pep8 check)
print "hello world"

for key in my_dict.keys(): ...
for value in my_dict.values(): ...
for key, value in my_dict.items(): ...

Compact version:
for i in range(10): print(i) # 0 .. 9

Access index: [http://stackoverflow.com/questions/522563/accessing-the-index-in-python-for-loops]
# using enumeration
for idx, val in enumerate(mylist):
print idx, val

# using range
for ix in range(len(mylist)):
print mylist[ix]

# list comprehension
[ (ix, mylist[ix]) for ix in range(len(mylist))]

=== While Loop ===

i = 0
while i < 10:
i = i + 1

while True:
... # infinite loop

=== Breaking Loops ===

break statement:
while True:
if s == 'done':
break # jump out of the loop
...

continue statement: (works with while too)
for i in range(10):
if i == 2:
continue # skip 2
...

=== Functions ===

No return value: (None)
x = print('')
# x is 'None'

Note: specifying no return value is the same as this:
return None

No Operation - 'pass':
def noop():
pass

Function:
<pre>
def area(radius):
""" This is a doc string """ # can be " ... " also...
import math
return math.pi * radius ** 2
</pre>

Function Doc string:
print(area.__doc__)
foo.func_doc

Function name:
foo.func_name

Local variable:
name = "old"
def cname():
name = "new"
print(name) # is still 'old'

Global variable:
name = "old"
def cname():
global name
name = "new"
cname()
print(name) # is now 'new'

By convention starting point is: (not required)
def main():

Parameters: (with default)
def foo(x, y, z = "default"):

Note: don't use mutable for defaults, as the object is remembered across runs! (unless that is what you want)
<pre>
# each call to t() will increase list
def t(foo=[]):
foo.append('1')
print(foo)
</pre>

<pre>
# fix mutable issue
def named_param(a, foo=None):
foo = foo or []
if not foo:
foo.append(a)
</pre>

Parameters by keyword:
def foo(x = '1', y = '2', z = '3'):
...
foo(y = 'a', x = 'b') # order is not important with keywords

Trick for optional parameter:
def do(x = None):
if x == None:

Function tests - functions are callable
callable(myfunc)

=== Type Hints ===

: "Note The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers, IDEs, linters, etc."

def surface_area_of_cube(edge_length: float) -> str:
return f"The surface area of the cube is {6 * edge_length ** 2}."

Without the hints:
def surface_area_of_cube(edge_length):
...

ref: https://docs.python.org/3/library/typing.html

=== Pass ===

Pass is a null operation (noop):

Do nothing, noop, take no action:
pass

Great for creating minimal classes:
class MyEmptyClass:
pass

Greate for making stub functions/methods:
def initlog(*args):
pass # remember to implement this!

=== Slice ===

Take take a slice of strings, lists, tuples, etc...
a = [0,1,2,3,4,5]
a[0] # 0 - not a slice
a[-1] # 5 - not a slice
a[0:1] # [0]
a[1:3] # [1, 2]
a[:2] # [0, 1]
a[:-1] # [0, 1, 2, 3, 4] - all but last
a[2:] # [2, 3, 4, 5]
a[::2] # [0, 2, 4] - stride
a[::-1] # [5, 4, 3, 2, 1, 0] - reverse
range(0,10)[::2] # [0, 2, 4, 6, 8] - stride

a = '012345'
a[1:3] # '12'
a[::2] # '024'

=== [http://docs.python.org/reference/compound_stmts.html Compound Statements] ===

Compound statements: (generally discouraged)
fun1(); fun2(); fun3()

Compound with statements: [http://docs.python.org/reference/compound_stmts.html#the-with-statement]
with A() as a, B() as b: ...

#is equivalent to
with A() as a:
with B() as b:
...

[http://www.python.org/dev/peps/pep-0343 PEP 0343] - The “with” statement

f = open('file')
with f: ...

Try: (quick ignore all exceptions, very dangerous)
try: my_broken_function()
except: pass

=== Passing Command Line Arguments ===

Parameters are stored in the 'sys.argv' variable:
import sys
print( sys.argv[0] ) # name of script
print( sys.argv[1] ) # first argument
for arg in sys.argv: print( arg ) # list all arguments

Print script name:
os.path.basename(sys.argv[0])

Print path to script:
print sys.path[0]

Paths: [http://code.activestate.com/recipes/474083-get-the-path-of-the-currently-executing-python-scr/]
<pre>
import os,sys
print "CWD: ",os.getcwd()
print "Script: ",sys.argv[0]
print ".EXE: ",os.path.dirname(sys.executable)
print "Script dir: ", os.path.realpath(os.path.dirname(sys.argv[0]))
pathname, scriptname = os.path.split(sys.argv[0])
print "Relative script dir: ",pathname
print "Script dir: ", os.path.abspath(pathname)
</pre>

os.path.realpath(__file__)

print 'sys.argv[0] =', sys.argv[0]
pathname = os.path.dirname(sys.argv[0])
print 'path =', pathname
print 'full path =', os.path.abspath(pathname)

=== Environment Variables ===

import os
a = os.environ.get('MYVAR') # get environment var
a = os.environ.get('MYVAR','default') # get environment var
a = os.environ['MYVAR'] # get environment var
os.environ['MYVAR'] = 'something' # set environment var
os.environ.update({'MYVAR':'something'}) # set environment var

for env in os.environ:
print "%s = %s" % (env, os.environ[env])

with open("environment.txt", "w") as f:
for env in os.environ:
f.write("%s = %s\n" % (env, os.environ[env]))

=== PYTHONPATH ===

Environment Search path:
PYTHONPATH=...

From Code:
sys.path.append('...')

Get site-package directory from shell: [http://stackoverflow.com/questions/122327/how-do-i-find-the-location-of-my-python-site-packages-directory]
python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

=== Import and Modules ===

Import and Modules - http://docs.python.org/tutorial/modules.html

Import module:
import package # standard import
import package.code # nested name space
from package import code # import specifics from module
import module # standard import
from math import sin # import specifics from module
from module import myfunc as fun1 # alias function/object
import math as other_math # alias module
import longname as ln # alias module
from module import * # BAD BAD BAD!

import math
math.pow(x,y)

Group and organize by:
* stdlib libraries
* local libraries
* 3rd party libraries

Remove import:
del math

from math import *
pow(x,y)

from math import pow
pow(x,y)

import math as math_lib # rename
math_lib.pow(x,y)

List functions in module:
import math
dir(math)

List all built in functions:
dir(__builtins__)

Note: Modules have the extension .py

Simply module Example:

# mymod.py:
def hello():
""" this is a doc string """
print "Hello World"

# Use the module:
import mymod
mymod.hello():

# Use everything in the the module, except for names beginning with '_': (bad practice!)
from mymod import *
hello()

# Use specific functions in the module:
from mymod import hello, hi
hello()

# Function assignment:
h = mymod.hello
h()

# module name
mymod.__name__ # mymod
__name__ # used inside of module will print current module or "__main__" if directly called

Note: careful when building own modules, as the import literally imports and executes the code, so any code not in functions is executed!

Check if being called directly or indirectly:
if __name__ == "__main__":
# execute tests...

Standard Script Setup:
<pre>
def main(args):
# do logic
if __name__ == "__main__":
sys.exit(main(sys.argv))
</pre>

Easter egg:
# The Zen of Python
import this

Search path:
* environment variable 'PYTHONPATH' (on Unix, this is usually .:/usr/local/lib/python)
* sys.path (list of paths, initialized to PYTHONPATH)
** sys.path.append('path')

Module folder: [http://stackoverflow.com/questions/391879/organising-my-python-project] [http://stackoverflow.com/questions/279237/python-import-a-module-from-a-folder] [http://stackoverflow.com/questions/72852/how-to-do-relative-imports-in-python]
/kenlib/kenmod.py
# so you can do "import kenlib.kenmod"
# create file '__init__.py' in kenlib/

Check if package exists:
<pre>
try:
import argparse
except ImportError:
print >> sys.stderr, "Please install argparse from http://code.google.com/p/
argparse."
exit(1)
</pre>

=== ifmain ===

if __name__ == '__main__':
sys.exit(main(sys.argv))

if __name__ == '__main__':
sys.exit(main(sys.argv[1:]) or 0)

=== Lists ===

* Lists can be modified (mutable)
* Tuples are constant (immutable)

List: (mutable or modifiable)
x = [] # empty list
x = [5] # singleton list

s = list("Hello World")
print(s) # s = ['H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd']
s[0] = 'X'

s.append(x) # append x
s.insert(i, x) # insert x at i
del(s[i]) # delete item at i
s.sort() # sort (modify in place!)
s.reverse() # reverse
s.index(x) # index of x in s
new = old[:] # create copy of list
new = list(old) # create copy of list

Check if item in list:
if x in s: ...

Filter list of strings based on contents: [https://stackoverflow.com/questions/2152898/filtering-a-list-of-strings-based-on-contents]
res = [k for k in lst if 'ab' in k]

List comprehension
[n * n for n in range(1, 11)] # squares of numbers 1 to 10
[c for c in 'pizza'] # list of string 'pizza'
[c.upper() for c in 'pizza'] # list of string 'pizza' uppercase
[n for n in nums if n > 0] # filters out only positive numbers in nums list
[fname for fname in os.listdir(path) # on multiple lines
if os.path.isfile(fname)
if fname.endswith('.py')]

Enumeration
for index, value in enumerate(mylist): print index, value

Length:
len(x)

Concatenation:
x + s
x * 2

Convert list to string:
"".join(mylist) # no separator
" ".join(mylist) # space separator
",".join(mylist) # comma separator

Test Membership
x in s

Multi dimensional list:
mylist = [ [0,"hi"], [1,"bye"] ]
print( mylist[0][1] )

Sorting: (see lambda)
data.sort() # alpha numerica sort
data.reverse() # alpha numerica sort

<pre>
# custom sort
data = [1, 5, 3, 9]
def mylistcmp(x, y):
if x == y: return 0
if x < y:
return -1
else: # swap 1 and -1 here to reverse
return 1

data.sort(cmp=mylistcmp)
</pre>

Slices:
lst = ['1a', '2b', '3c', '4d', '5e', '6f']
lst[0] # '1a'
lst[-1] # '6f' last item
lst[0:2] # ['1a', '2b']
lst[0:2] = ['11', '22'] # lst = ['11', '22', '3c', '4d', '5e', '6f'] - assignment to slice
lst[0:2] = [] # remove items
lst[0:0] = ['1a', '2b'] # insert items (at beginning)
lst[-1:-1] = ['7g', '8h'] # insert items (at item BEFORE last), not really what you want, use append()
lst[-1:] = ['7g', '8h'] # replace last item
lst[:] = [] # clear entire list
del(lst[0:2]) # delete slice items

Note: it is not safe to modify a sequence while being iterated over. Create a copy first:
for x in mylst[:]: # make a slice copy of the entire list
if len(x) > 6: a.insert(0,x)

Make list from tuple:
list( range(10) )

Merge multiple lists: [http://www.testingreflections.com/node/view/4930]
<pre>
def merge(seq):
merged = []
for s in seq:
for x in s:
merged.append(x)
return merged

# sample usage:
foo = [['a', 'b'],['c'],['d', 'e', 'f']]
print merge(foo)
>>>['a', 'b', 'c', 'd', 'e', 'f']
</pre>

=== Tuple ===

Tuple: (immutable or constant)
x = () # empty tuple
x = (5,) # singleton tuple
x = (5) # NOTE: integer with parenthesis, not tuple

s = tuple("Hello World")
print(s) # s = ('H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd')
# ERROR: s[0] = 'X'

Make list from tuple:
list( range(10) )

Use in conditionals:
if answer in ('y', 'ye', 'yes'): ...

Convert tuple string to tuple: [http://stackoverflow.com/questions/3945856/converting-string-to-tuple-and-adding-to-tuple]
x = "(1,2,3)"
t = tuple(int(v) for v in re.findall("[0-9]+", x))

Another Convert tuple string to tuple: [http://stackoverflow.com/questions/3945856/converting-string-to-tuple-and-adding-to-tuple]
import ast
ast.literal_eval("(1,2,3,4)") # (1,2,3,4)

=== Named Tuple ===

What are "named tuples" in Python? - Stack Overflow - http://stackoverflow.com/questions/2970608/what-are-named-tuples-in-python

<pre>
from collections import namedtuple
Point = namedtuple('Point', 'x y')
pt1 = Point(1.0, 5.0)
pt2 = Point(2.5, 1.5)
print p1 # point(x=1, y=5)
print p1.x # 1.0
print p1[0] # 1.0
</pre>

=== Dictionary ===

NOTE: Dictionaries are also known as associative arrays, maps or hash tables.

Dictionary: (mutable or modifiable)

color = {} # empty dictionary
color = {'red' : 1, 'blue', 2}
print( color['red'] ) # 1
print( color.get('red') ) # 1
print( color.get('red', 'default') ) # red if found, or return 'default'
color['green'] = 3 # able to add new item on the fly!
'red' in color # True - check if index exists
del(color["red"]) # delete item
len(color) # 2 - count of items

List in dictionary
<pre>
d.setdefault(key,[]).append(member) # tricky!

# more tricky: set default type as list
from from collections import defaultdict
d = defaultdict(list)
d[key].append(member)
</pre>

Enumeration:
for index, key in enumerate(mylist): print index, key

functions:
d.clear()
d.copy()
d.get(key)
d.get(key, default_val)
d.update(e) # update with (key, value) pairs in e
d.setdefault(key, value) # only update if not exist, and return set value

Views: (adjust on the fly as dictionary changes)
d.items() (key, values)
for k, v in d.items(): print(k, v)

d.keys()
for k in d.keys(): print(k)

d.values()
for v in d.values(): print(v)

Sorting: (see also lambda)
<pre>
# custom sort of dictionaries in list
data = [dict(number=x) for x in '036149'] # create list of dictionaries
def mydictcmp(x, y):
if x['number'] == y['number']: return 0
if x['number'] < y['number']:
return -1
else: # swap 1 and -1 here to reverse
return 1

# data.sort(cmp=mydictcmp) # modifies data

new_data = sorted(data, cmp=mydictcmp) # better solution
</pre>

Sorting:
<pre>
my_dict = {'a':1, 'b':4, 'c':3, 'd':9}
my_dict_keys = my_dict.keys()

def my_dict_sort(x, y):
if my_dict[x] == my_dict[y]: return 0
if my_dict[x] < my_dict[y]: return -1
if my_dict[x] > my_dict[y]: return 1

#sorted_my_dict = sorted(my_dict_keys, cmp=my_dict_sort, reverse=True)
sorted_tuple = sorted(my_dict_keys, cmp=my_dict_sort)
print "Sorted Dictionary:"
for key in sorted_tuple:
print "%s: %d" % (key, my_dict[key])

# a: 1, c: 3, b: 4, d: 9,
# reverse: d: 9, b: 4, c: 3, a: 1,
</pre>

Sort and keys:
keys = sorted(mydict.keys())

Sort by value: [http://stackoverflow.com/questions/613183/python-sort-a-dictionary-by-value]
import operator
x = {1: 2, 3: 4, 4:3, 2:1, 0:0}
sorted_x = sorted(x.iteritems(), key=operator.itemgetter(1))

Sort options:
# LIST.sort(cmp=None, key=None, reverse=False) -- stable sort *IN PLACE*;
# sorted(iterable, cmp=None, key=None, reverse=False)

Another: (works well!)
sorted(my_dict.items(), key=lambda x: x[1])

See: http://wiki.python.org/moin/HowTo/Sorting/

=== Sets ===

Sets are special lists with unique values (good for removing duplicates)
lst = [1, 1, 6, 8, 1, 5]
s = set(lst)
print(s) # set([8, 1, 5, 6])
s.add(item)
item in s # true
s.remove(item)

Union (return combination of all):
s.union(t)

Intersection (return matching):
s.intersection(t)

Symmetric Difference (return only items not matching either, inverse of intersection):
s.symmetric_difference(t)
list( set(mylist).symmetric_difference(set(mylist2)) )

<pre>
Operation Equivalent Result
len(s) cardinality of set s
x in s test x for membership in s
x not in s test x for non-membership in s
s.issubset(t) s <= t test whether every element in s is in t
s.issuperset(t) s >= t test whether every element in t is in s
s.union(t) s | t new set with elements from both s and t
s.intersection(t) s & t new set with elements common to s and t
s.difference(t) s - t new set with elements in s but not in t
s.symmetric_difference(t) s ^ t new set with elements in either s or t but not both
s.copy() new set with a shallow copy of s
</pre>

References:
* sets — Unordered collections of unique elements - http://docs.python.org/library/sets.html

=== Files ===

Open file, get file descriptor:
fd = open('/dev/null')

File Modes:
'r' read (default)
'w' write (truncate and create if needed)
'a' append (create if needed)

'b' binary
't' text (default)

'+' read and write

Common Functions: (fd)

f.read() # read whole file to string
f.read(n) # read n bytes from file to string
f.readline() # read single line as string (while loop)
f.readlines() # read all lines to list

f.write(str) # write string to file (does not auto include new line)
f.writelines(list) # write list to file (does not auto include new lines)

f.seek(0) # seek to start of file
f.seek(pos) # seek to position
f.truncate() # truncate file

f.close() # close file (fd)
f.closed # True/False flag that reports if file is closed

Help:
help(open('/dev/null'))
pydoc file

Read Text File Line by Line
f = open(fname, 'r') # ('r' optional)
for line in f: # or f.readlines()
print(line)
f.close() # optional

Write list to file (with new lines):
f.writelines('\n'.join(mylist))

One liner read whole file:
print( open( fname, 'r' ).read() )

Write text file:
# f = open(fname, 'a') # append text file
f = open(fname, 'w')
f.write("Hello\n")
f.write( str(10) ) # convert numbers to strings first
f.writelines(["line one\n", "line two\n"])
f.close

Read binary file:
def is_gif(fname):
f = open(fname, 'br') # not sure 'b' is correct
first4 = tuple(f.read(4))
return first4 == (0x47, 0x49, 0x46, 0x38)

Read all characters of file:
c = f.read(1)
while c:
# if c == ... logic
c = f.read(1)

Reading file in while loop: [http://ubuntuforums.org/showthread.php?t=1092300]
<pre>
f=open("file")
while True:
line=f.readline()
if not line: break
print line
f.close()
</pre>

with - implicit close - [http://www.python.org/dev/peps/pep-0343/ PEP 343 -- The "with" Statement]: [http://docs.python.org/reference/compound_stmts.html#the-with-statement] (python >= 2.5)
with open("/etc/passwd") as f:
# ..., f.read(), etc

With statement: (promptly close open handle at end of block)
with open(fname, 'r') as f:
for ...

<pre>
# eqivalent to:
f = open("/etc/passwd")
f.__enter__()
try:
...
finally:
f.__exit__()
</pre>

cat: (for grabbing a configuration line)
<pre>
def cat(filename):
with open(filename) as f:
return f.readline().strip()
</pre>

=== With ===

PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/

'with' structure:
<pre>
with EXPR as VAR:
BLOCK
</pre>

translates to:
<pre>
VAR = EXPR
VAR.__enter__()
try:
BLOCK
finally:
VAR.__exit__()
</pre>

Note: The "as VAR" part is optional. Good for locking:
<pre>
with locking(myLock):
BLOCK
</pre>

with - implicit close - [http://www.python.org/dev/peps/pep-0343/ PEP 343 -- The "with" Statement]: [http://docs.python.org/reference/compound_stmts.html#the-with-statement] (python >= 2.5)
with open("/etc/passwd") as f:
...

With statement: (promptly close open handle at end of block)
with open(fname, 'r') as f:
for ...

See also [[#Files]]

References:
* PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/
* PEP 340 -- Anonymous Block Statements - http://www.python.org/dev/peps/pep-0340/

=== Try Exception Handling ===

Exception Handling - http://docs.python.org/release/2.6.6/tutorial/errors.html

throw execption:
raise [EXCEPTION]
raise IOError("This is a test!")

Catch exceptions:
try:
except ( [EXCEPTION], [EXCEPTION] ):
except [EXCEPTION]:
except [EXCEPTION] as err: # use "[EXCEPTION], err" in python < 2.6
print("Error: {}".format(err))
except:
raise # reraise caught error that is not handled
else: # if no exception caught
finally: # always executed

Note: Usually a good idea to re-raise exception if you don't handle it (just 'raise')

General Exception: (all exceptions inherit from Exception)
try:
raise Exception('spam', 'eggs')
except Exception as e:
pass

Custom Exception Class:
<pre>
class MyError(Exception):
"""My Error Exception"""

class MyException(Exception):
pass

rasie MyError("My Error")
</pre>

<pre>
class ValidationError(Exception):
def __init__(self, message, Errors):

# Call the base class constructor with the parameters it needs
Exception.__init__(self, message)

# Now for your custom code...
self.Errors = Errors
def __str__(self):
return "MyErrors - %s" % self.Errors
</pre>

Rename standard exception:
<pre>
class Failure(StandardError):
"""Exception for unexpected failures."""
</pre>

Note: a good place to hide the exception classes in the __init__.py file.

==== exception line number ====

Python When I catch an exception, how do I get the type, file, and line number? - Stack Overflow [http://stackoverflow.com/questions/1278705/python-when-i-catch-an-exception-how-do-i-get-the-type-file-and-line-number]
<pre>
import sys, os

try:
raise NotImplementedError("No error")
except Exception as e:
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print(exc_type, fname, exc_tb.tb_lineno)
</pre>

==== traceback ====

<pre>
import traceback
try:
...
except Exception as E:
...
print 'Traceback: %s', traceback.format_exc()
</pre>

Catching multiple exceptions and getting properties from the exception:
<pre>
import traceback
import sys

class MyError(Exception):
"""My Error Exception"""

try:
#raise ValueError("a value error") # <type 'exceptions.ValueError'>
raise MyError("a my error") # <class '__main__.MyError'>
raise Exception("won't be caught") # <type 'exceptions.Exception'>
except (ValueError, MyError) as e:
print type(e) # <class '__main__.MyError'>
print e # __str__() = a my error
msg = str(e) # msg = 'a my error'
print e.__doc__ # My Error Exception
print e.__str__() # a my error
print e.__repr__() # MyError('a my error',)
print e.args # ('a my error',)
x = e # __getitem__() allows args to be unpacked directly
# x, y = e # if two args were passed, use len(e.args) to see count

# manually print stack trace (import traceback, sys)
print "Exception in user code:"
print '-'*60
traceback.print_exc(file=sys.stdout)
# stack_trace = traceback.format_exc()
print '-'*60
</pre>

=== Object Oriented Programming ===

Terms:
* encapsulates
* inheritance
* polymorphism
* method overriding

==== class ====

class animal.py:
<pre>
class Animal(object): # default "(object)" not required
" Animal class "
def __init__(self, name):
self.name = name # name is private variable
self.__iam = "animal" # __iam is private variable
def __str__(self): # string representation
return "name = '%s'" % (self.name)
def __repr__(self): # object representation (just typing instance in IDLE, or repr())
return 'Animal(%s)' % str(self) # returns __str__
def speak(self, msg='Nothing to say'):
print(msg)
</pre>

Note: if no __str__ defined, but __repr__ is, then: __str__ = __repr__

Usage:
<pre>
from animal import Animal
a = Animal('good')
print(a.__doc__) # Animal class
print(a) # name = 'good'
a # Animal(name = 'good')
print(a.name) # good
a.speak('hi') # hi
print(a._Animal__iam) # animal - access to private variable (shouldn't do)
</pre>

subclass dog.py:
<pre>
from animal import Animal
class Dog(Animal):
" Dog class "
def __init__(self, name):
self.name = name # name is private variable
self.__iam = "dog" # __iam is private variable
# NOTICE: no __str__, inherited from super class
def __repr__(self): # override to reflect "dog"
return 'Dog(%s)' % str(self) # returns __str__
def speak(self, msg='Nothing to say'): # override method
super(Dog, self).speak('Dog says: ' + msg) # call super class method
</pre>

Subclass usage:
<pre>
from dog import Dog
d = Dog('bad')
print(d.__doc__) # Dog class
print(d) # name = 'bad'
d # Dog(name = 'bad')
print(d.name) # bad
d.speak('hi') # Dog says: hi
print(d._Dog__iam) # dog - access to private variable (shouldn't do)
</pre>

Super: [http://stackoverflow.com/questions/576169/understanding-python-super-with-init-methods]
# python 3.0
super().__init__()

# python 2.7
super(self.__class__, self).__init__()
super([CLASSNAME], self).__init__()

<pre>
class Rectangle(Polygon):
def __init__(self, id, width, height):
# super(Rectangle, self).__init__(id)
super(self.__class__, self).__init__(id)
self.shape = (width, height)
</pre>

Get class name:
self.__class__.__name__
obj.__class__.__name__

Full name:
obj.__module__ + "." + obj.__class__.__name__

Private variables: (uses name mangling)
self.__age
# direct access: _classname__varname (eg. p._Person__age = 44)

Tests:
isinstance(myobj, myclass) # check if myobj is a class or subclass of myclass
issubclass(A, B) # check if A is subclass of B
issubclass(Exception, BaseException) # True

Multiple Inheritance: (methods searched in order)
class DerviedClass(Base1, Base2, Base3):
pass

Standard startup script:
class MyClass():
...
@classmethod
def main(cls, argv=None):
...
# return cls() # creator
if __name__ == "__main__";
sys.exit(MyClass.main(sys.argv))

Raise an error if a method is not overridden (not implemented)
def _get_dist(self):
"""Find linux disto. Implemented in subclasses."""
raise NotImplementedError

==== Multiple Constructors ====

<pre>
class Animal(object):

def __init__(self, name):
self.name = name

@classmethod
def create_fido(cls):
return cls("fido")
</pre>

==== Descriptor Classes ====

"In general, a descriptor is an object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol: __get__(), __set__(), and __delete__(). If any of those methods are defined for an object, it is said to be a descriptor." [http://docs.python.org/2/reference/datamodel.html#implementing-descriptors]

<pre>
object.__get__(self, instance, owner)

Called to get the attribute of the owner class (class attribute access) or of an instance of that
class (instance attribute access). owner is always the owner class, while instance is the instance
that the attribute was accessed through, or None when the attribute is accessed through the owner.
This method should return the (computed) attribute value or raise an AttributeError exception.

object.__set__(self, instance, value)

Called to set the attribute on an instance instance of the owner class to a new value, value.
</pre>

Example: [http://stackoverflow.com/questions/3798835/understanding-get-and-set-and-python-descriptors]
<pre>
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)

class Temperature(object):
celsius = Celsius()

t = Temperature()
t.celsius = 5 # calls Celsius.__set__
print t.celsius # calls Celsius.__get__
</pre>

Note: Also appears in 'pydoc' under "Data descriptors defined here" with object's docstring. Trick: the __doc__ can be overridden in the __init__ method, which makes for dynamic docstrings!

References:
* Descriptor HowTo Guide — Python v2.7.5 documentation - http://docs.python.org/2/howto/descriptor.html
* 3.4.2.2. Implementing Descriptors¶ - 3. Data model — Python v2.7.5 documentation - http://docs.python.org/2/reference/datamodel.html#implementing-descriptors

=== [http://docs.python.org/reference/simple_stmts.html#the-exec-statement exec code] ===

DANGEROUS!

Execute code in string:
exec( 'print("hi")' )

Example of executing a startup script for IDLE:
if os.path.isfile('.pythonrc.py'): exec(open('.pythonrc.py').read())

import os
filename = os.environ.get('PYTHONSTARTUP')
if filename and os.path.isfile(filename):
exec(open(filename).read())

=== eval ===

VERY DANGEROUS

Will execute a string as though it were pure python code!

eval("print 'hello'")

=== Common Header Format ===

<pre>
__author__ = "software team"
__date__ ="$Feb 15, 2011 1:42:17 PM$"
__version__ = "1.1.0.441 bandelier@66d9fe1edf3e"
</pre>

<pre>
Next should be authorship information. This information should follow this format:

__author__ = "Rob Knight, Gavin Huttley, and Peter Maxwell"
__copyright__ = "Copyright 2007, The Cogent Project"
__credits__ = ["Rob Knight", "Peter Maxwell", "Gavin Huttley",
"Matthew Wakefield"]
__license__ = "GPL"
__version__ = "1.0.1"
__maintainer__ = "Rob Knight"
__email__ = "rob@spot.colorado.edu"
__status__ = "Production"
</pre>
Source: Python Coding Guidelines - http://bayes.colorado.edu/PythonGuidelines.html

These will show up in the help() like such:
<pre>
Help on module test:

NAME
test

FILE
/home/kenneth/test.py

FUNCTIONS
test()

DATA
__author__ = 'test author'
__version__ = '1.0'

VERSION
1.0

AUTHOR
test author
</pre>

References:
* Python: What is the common header format? - Stack Overflow - http://stackoverflow.com/questions/1523427/python-what-is-the-common-header-format

=== Common Environment Variables ===

Command line and environment — Python v2.7.2 documentation - http://docs.python.org/using/cmdline.htm

PYTHONUNBUFFERED
: If this is set to a non-empty string it is equivalent to specifying the -u option. [http://docs.python.org/using/cmdline.html#envvar-PYTHONUNBUFFERED]
: -u - Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
* See also stdout - Python output buffering - Stack Overflow - http://stackoverflow.com/questions/107705/python-output-buffering
export PYTHONUNBUFFERED=1
set PYTHONUNBUFFERED=1

PYTHONVERBOSE
:If this is set to a non-empty string it is equivalent to specifying the -v option. If set to an integer, it is equivalent to specifying -v multiple times.
: -v - Print a message each time a module is initialized, showing the place (filename or built-in module) from which it is loaded.
export PYTHONVERBOSE=1
set PYTHONVERBOSE=1

PYTHONPATH
: Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored. [http://docs.python.org/using/cmdline.html#envvar-PYTHONPATH]
export PYTHONPATH=/path1/:path2/
set PYTHONPATH=C:\path1\;C:\path2\

=== Queue ===

Good for threading.

"The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics." [http://docs.python.org/2/library/queue.html]

<pre>
import Queue

bucket = Queue.Queue()
bucket.put("something")

while True:
try:
item = bucket.get(block=False)
except Queue.Empty:
pass
else:
# do stuff with queue
</pre>

Important methods:
Queue.qsize()
Queue.empty()
Queue.put(item[, block[, timeout]])
Queue.get([block[, timeout]])

-

Example: http://docs.python.org/2/library/queue.html
<pre>
def worker():
while True:
item = q.get()
do_work(item)
q.task_done()

q = Queue()
for i in range(num_worker_threads):
t = Thread(target=worker)
t.daemon = True
t.start()

for item in source():
q.put(item)

q.join() # block until all tasks are done
</pre>

-

"The module implements three types of queue, which differ only in the order in which the entries are retrieved. In a FIFO queue, the first tasks added are the first retrieved. In a LIFO queue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the heapq module) and the lowest valued entry is retrieved first."

Queue Types:
class Queue.Queue(maxsize=0) # FIFO Queue
class Queue.LifoQueue(maxsize=0) # LIFO Queue
class Queue.PriorityQueue(maxsize=0) # Priority Queue

References:
* 8.10. Queue — A synchronized queue class — Python v2.7.5 documentation - http://docs.python.org/2/library/queue.html

== Recipes ==

=== Remove Numbers from String ===

Using list comprehension: [http://stackoverflow.com/questions/12851791/removing-numbers-from-string]
<pre>
result = ''.join([i for i in s if not i.isdigit()])
</pre>

Using pattern matching:
<pre>
>>> re.compile(r'\D+').findall("test 1234 test 1234")
['test ', ' test ']

>>> re.compile(r"\d").sub("", "test 1234 test 1234")
'test test '
</pre>

=== Retry Wrapper ===

<pre>
import functool
class SomethingRetry:

def retry_with_backoff(retries: int, base_delay: float, exceptions: tuple[type[Exception], ...] = (Exception,)) -> callable:
"""
Retries a function with exponential backoff, doubling on each attempt.
Arguments:
retries: Number of retry attempts.
base_delay: Initial delay between retries in seconds.
exceptions: Tuple of exceptions to catch and retry.
"""

def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(1, retries + 1):
try:
return func(*args, **kwargs)
except exceptions as e:
print(f"Warning: Attempt {attempt} failed with error: {e}")
if attempt == retries:
print("Error: All retry attempts failed.")
raise
sleep_time = base_delay * (2 ** (attempt - 1))
print(f"Notice: Retrying in {sleep_time} seconds...")
time.sleep(sleep_time)
return wrapper
return decorator

@retry_with_backoff(retries=6, base_delay=10, exceptions=(Exception,))
def do_something(self, sample_param: str):
</pre>

== Standard Libraries ==

=== user input ===

<pre>
line = raw_input() # Python 2
# line = input() # Python 2
line = input('What is your name: ')
print(": " + line.strip() + " :")
</pre>

=== sys - stdout stderr stdin ===

STDOUT:
import sys
sys.stdout.write("Hello ") # does not write new line
sys.stdout.write("World\n")

STDERR:
import sys
sys.stderr.write("ERROR!\n")

STDIN:
import sys
line = sys.stdin.readline()

line = input() # use raw_input() in Python 2
line = input('What is your name: ')
print(": " + line.strip() + " :")

Read piped stdin line by line: [http://stackoverflow.com/questions/1450393/how-do-you-read-from-stdin-in-python]
# cat /etc/passwd | ./readlines.py # linux
# type file.txt | python readlines.py # windows

# fileinput.input() - returns fileinput object which is iterative
import fileinput
for line in fileinput.input():
print(": " + line + " :")

# sys.stdin.readlines() - returns list of strings
import sys
for line in sys.stdin.readlines():
print(": " + line + " :")

# sys.stdin - returns object which is iterative
import sys
for line in sys.stdin:
print(": " + line + " :")

# ugly while loop: [http://ubuntuforums.org/showthread.php?t=1092300]
while True:
try:
s = raw_input("Say something ")
except EOFError:
break
print ": %s" % s

Check if stdin has any waiting data: [http://stackoverflow.com/questions/3762881/how-do-i-check-if-stdin-has-some-data]
<pre>
import sys
import select

# if select.select([sys.stdin,],[],[],0.0)[0]: # alternative old method

if not sys.stdin.isatty():
print "Have data!"
print sys.stdin.readlines()
else:
print "No data"
</pre>

<pre>
if len(sys.argv) > 1:
message = ' '.join(sys.argv[1:])
elif not sys.stdin.isatty():
# read from pipe in
message = sys.stdin.readline()
else:
# read from user
sys.stdout.write("Message: ")
message = raw_input().strip()
</pre>

=== [http://docs.python.org/library/re.html Regular Expressions] ===

Import:
import re # user regular expressions

Match vs Search:
re.match("cde", "abcdef") # no match (match searches beginning of string, first match)
re.match("abc", "abcdef") # match (match searches beginning of string, first match)
re.search("cd", "abcdef") # match (matches any location, first match)

Both re.match() and re.search() return objects that are not intuitive to use. You can check the object's group() method for what matched, but I prefer re.findall().

Match:
s = 'done'
if re.match('done|quit', s) != None # True

Search:
s = 'done'
if re.search('do', s) != None # True

Replace:
msg = 'jello world'
msg = re.sub('jello', 'hello', msg)

For case intensive searches append 3rd parameter "re.I".

Find all:
re.findall("[a-z]+", "aaa111bbb222") # ['aaa', 'bbb']
re.findall("111([a-z]+)", "aaa111bbb222") # ['bbb']
re.findall("hello", "HELLO", re.I) # ['HELLO'] - Case Insensitive

Split:
>>> re.split('[a-z]+','111aaa222bbb')
['111', '222', '']

>>> re.split('([a-z]+)','111aaa222bbb') # parenthesis cause return of all
['111', 'aaa', '222', 'bbb', '']

Replacement Substitution:
# re.sub(pattern, repl, string[, count, flags])
re.sub(r'\\', r'/', r'\\data\files')

Compiled regular express for repeated use: (performance)
valid = re.compile(r"[0-9]+")
valid.match("aaa111bbb") # not valid
match = valid.search("aaa111bbb222") # valid
print(match.group()) # what matched: '111'

=== Date and Time ===

Sleep in seconds:
import time
time.sleep(seconds)

Use the following functions to convert between time representations:
<table border="1">
<tr><th>From</th><th>To</th><th>Use</th></tr>
<tr><td>seconds since the epoch</td><td>struct_time in UTC</td><td>gmtime()</td></tr>
<tr><td>seconds since the epoch</td><td>struct_time in local time</td><td>localtime()</td></tr>
<tr><td>struct_time in UTC</td><td>seconds since the epoch</td><td>calendar.timegm()</td></tr>
<tr><td>struct_time in local time</td><td>seconds since the epoch</td><td>mktime()</td></tr>
</table>

Date formats:
<pre>
>>> time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
'2014-04-04 12:22:43'

>>> time.time()
1396635763.0581281

>>> datetime.datetime.now()
datetime.datetime(2014, 4, 4, 12, 22, 43, 58456)
# datetime(year, month, day, hour, minute, second, microsecond)

>>> time.localtime()
time.struct_time(tm_year=2014, tm_mon=4, tm_mday=4, tm_hour=12, tm_min=22, tm_sec=43, tm_wday=4, tm_yday=94, tm_isdst=1)
</pre>

Current epoch time:
int(time.time())

Time zone offset:
standard_tz_offset = - time.timezone / 3600 # 7
daylight_tz_offset = - time.altzone / 3600 # 6

Is daylight saving time: [http://stackoverflow.com/questions/2881025/python-daylight-savings-time]
import time
time.localtime()
_.tm_isdst # 0 or 1

mst_epoch_time = time.time() # server in MST/MDT
if time.localtime().tm_isdst: # are we in daylight saving time?
utc_epoch_time = mst_epoch_time + time.altzone
else:
utc_epoch_time = mst_epoch_time + time.timezone

Format current time: [http://docs.python.org/2/library/time.html#time.strftime]
time.strftime('%d%b%y') # '12May11'
time.strftime('%Y.%m.%d %H:%M:%S') # '2013.08.09 14:01:28'

Time conversions:
<pre>
# epoch to str
epoch_time = time.time() # 1375992088.7867229
dt = datetime.datetime.fromtimestamp( epoch_time ) # datetime.datetime(2013, 8, 8, 14, 1, 28, 786723)
date_str = dt.strftime("%Y-%m-%d %H:%M:%S") # '2013-08-08 14:01:28'
tz_offset = - time.altzone / 3600
if tz_offset < 0:
tz_offset_str = "-%02d00" % abs(tz_offset)
else:
tz_offset_str = "+%02d00" % abs(tz_offset)
print date_str + " " + tz_offset_str # '2013-08-08 14:01:28 -0600'
</pre>

<pre>
# UTC epoch to str
gm_epoch_time = calendar.timegm(time.gmtime()) # 1375992114
dt = datetime.utcfromtimestamp( gm_epoch_time ) # datetime.datetime(2013, 8, 8, 20, 1, 54)
dt.strftime("%Y-%m-%d %H:%M:%S +0000") # '2013-08-08 20:01:54 +0000'
dt.strftime("%Y-%m-%dT%H:%M:%SZ") # '2013-08-08 20:01:54 +0000'
</pre>

<pre>
# Local time str to epoch
iso_string='2013-08-08T14:01:28'
epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S' ) ) # 1375992088.0

iso_string='2013-08-08T14:01:28 MDT'
epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S %Z' ) ) # 1375992088.0

# not sure how to specify numerical time zone and pull it in!

# UTC (Zulu) str to epoch, and back again
iso_string = '2013-08-08T20:01:54Z'
epoch_time = calendar.timegm( time.strptime( iso_string.replace('Z', 'GMT'), '%Y-%m-%dT%H:%M:%S%Z' ) )
print epoch_time # 1375992114
print "%sZ" % datetime.utcfromtimestamp(timestamp).isoformat() # 2013-08-08T20:01:54Z
</pre>

<pre>
# Convert epoch from Local to UTC
epoch_time = 1375992114
tz_offset = - time.altzone / 3600
gm_epoch_time = epoch_time - tz_offset * 3600 # 1376013714
</pre>

<pre>
# Convert epoch from UTC to Local
gm_epoch_time = 1376013714
tz_offset = - time.altzone / 3600
epoch_time = gm_epoch_time + tz_offset * 3600 # 1375992114
</pre>

struct_time:
time.localtime() # local zime zone
time.gmtime() # UTC time

today = time.localtime()
today_year = today.tm_year
today_month = today.tm_mon
today_day = today.tm_mday

time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime())

Zulu time to epoch time: [http://stackoverflow.com/questions/127803/how-to-parse-iso-formatted-date-in-python]
s = '2008-09-03T20:56:35.450686Z'
event_time_struct = time.strptime(s.split('.')[0]+ "UTC", "%Y-%m-%dT%H:%M:%S%Z")
epoch_time = calendar.timegm(event_time_struct)

Parse string to time.struct_time (like gmtime()): [http://docs.python.org/2/library/time.html#time.strptime]
time.strptime(time.ctime()) # current time - default format is ctime format: "%a %b %d %H:%M:%S %Y"
time.strptime(time.ctime(time.time())) # default format is ctime format: "%a %b %d %H:%M:%S %Y"
time.strptime("30 Nov 00", "%d %b %y")

Build from ticks: [http://www.tutorialspoint.com/python/python_date_time.htm]
localtime = time.localtime( time.time() ) # build time.struct_time from ticks
print time.asctime( localtime ) # Tue Jan 13 10:17:09 2009

dt = datetime.datetime.fromtimestamp( time.time() ) # build datetime.datetime struct from ticks
dt.strftime("%Y-%m-%d %H:%M:%S") # '2013-07-31 23:07:48'
time.strptime( time.time(), ")

<pre>
from datetime import date
now = date.today() # now = datetime.date(2003, 12,2)
now.strftime("%m-%d-%y") # '12-02-03'
# dates support calendar arthimetic
birthday = date(1964, 7, 31)
age = now - birthday
age.days # 14368
</pre>

ts = (datetime.datetime.now().strftime("%Y%m%d%H%M")) # 201105191118
tstxt = (datetime.datetime.now().strftime("%m-%d-%Y %H:%M")) # 05-19-2011 11:18

Seconds since epoch:
int(time.time()) # 1305842304

Zulu and back again: ([http://aboutsimon.com/2013/06/05/datetime-hell-time-zone-aware-to-unix-timestamp/ Datetime hell])
<pre>
import time
from calendar import timegm
from datetime import datetime

if __name__ == '__main__':
iso_string = '2013-06-05T15:19:10Z'

timestamp = timegm(
time.strptime(
iso_string.replace('Z', 'GMT'),
'%Y-%m-%dT%H:%M:%S%Z'
)
)

print iso_string
print timestamp
print '-> %sZ' % datetime.utcfromtimestamp(timestamp).isoformat()
</pre>

http://wiki.python.org/moin/TimeTransitionsImage?action=AttachFile&do=get&target=v1.png [http://wiki.python.org/moin/WorkingWithTime]

References:
* WorkingWithTime - PythonInfo Wiki - http://wiki.python.org/moin/WorkingWithTime
* Datetime hell. Time zone aware to UNIX timestamp. | About me and the things I do - http://aboutsimon.com/2013/06/05/datetime-hell-time-zone-aware-to-unix-timestamp/
* WorkingWithTime - Python Wiki - http://wiki.python.org/moin/WorkingWithTime

---

Something I cooked up...

<pre>
def utc_time(self, years_offset=60):
mst_epoch_time = time.time() # server is in MST
if time.localtime().tm_isdst: # are we daylight saving time?
utc_epoch_time = mst_epoch_time + time.altzone
else:
utc_epoch_time = mst_epoch_time + time.timezone
return utc_epoch_time

def tz_time(self, tz_offset, daylight_adjust=True):
""" return the future time based on timezone offset
@tz_offset is timezone offset, eg -7 for MST
@daylight_adjust is flag to auto adjust for daylight time"""
if daylight_adjust:
if time.localtime().tm_isdst: # are we daylight saving time?
_tz_offset = tz_offset - 1
else:
_tz_offset = tz_offset
_utc_epoch_time = self.utc_time()
_adj_time = _utc_epoch_time + _tz_offset * 60 * 60
return _adj_time

def future_time(self, tz_offset=-7, daylight_adjust=True, years_offset=60):
""" Calculate some future year """
_tz_time = self.tz_time(tz_offset)
future_utc_epoch_time = _tz_time + years_offset * 365 * 24 * 60 * 60
# account for leap years
cur_year = datetime.datetime.now().year
cur_year_leap = cur_year / 4
future_year = cur_year + years_offset
future_year_leap = future_year / 4
leap_days = future_year_leap - cur_year_leap
future_utc_epoch_time += leap_days * 24 * 60 * 60
return future_utc_epoch_time
</pre>

=== Math ===

Math Functions:
import math
math.ceil(x)
math.floor(x)
math.trunc(x)
math.pow(x,y)

abs(x) # built in

=== Random ===

import random
random.choice(['a', 'b', 'c'])
random.sample(range(100), 10)
random.random() # random float 0 < x < 1
random.randrange(3) # integer in [0, 1, 2]
random.randrange(1, 3) # integer in [1, 2]
random.randint(1, 3) # integer in [1, 2, 3]

=== Operating System Operations ===

Import OS Library:
import os

Common Functions:
# directory functions
os.getcwd() # get current directory
os.listdir(p) # get list of directory contents
os.chdir(p) # change directory
os.mkdir(p) # make directory
os.makedirs(p) # make directory and all intermediate
os.rmdir(p) # remove empty directory, see shutil.rmtree()
os.removedirs(p) # remove empty directories, see shutil.rmtree()
os.path.exists(p) # does file/directory exist?
os.path.isdir(p) # does directory exist
os.path.join(folder1, folder2, file, etc) # portable path join

# file functions
os.remove(p) # remove file
os.unlink(p) # same as os.remove()
os.path.exists(p) # does file/directory exist?
os.path.isfile(p) # does file exist
os.stat(fname) # get file stats
os.stat(fname).st_size # file size
os.link(src, dst) # create hard link
os.readlink(p) # read symbolic link
os.rename(old, new) # rename/move file or directory
os.tempnam(p) # temporary unique name for target path
os.tmpfile() # create temporary file
os.tmpnam() # temporary file name
os.path.getsize(filename) # get file size

# system functions
os.fork() # fork process
os.getpid() # get process id
os.getppid() # get parent's process id
os.kill(pid, sig) # kill a process
os.getlogin() # get user's login name

# execution
os.system(cmd) # output to stdout, return exit_status
os.popen(cmd) # returns a pipe to command

Note: to get path to script:
sys.path[0]

Get directory and file names:
os.path.dirname(path) # '/root/test.txt' -> '/root'
os.path.basename(path) # '/root/test.txt' -> 'test.txt'
os.path.splitext(path) # '/root/test.test.txt' -> ('/root/test.test', '.txt') - extract extention
filename.split(".")[-1] # '/root/test.test.txt' -> 'txt' - extract extention

Normalize path, eliminating double slashes, etc.
os.path.normpath(path) # '/root/..' -> '/' or 'test/..' -> '.'
os.path.abspath(path) # '/root/..' -> '/' or 'test/..' -> '/root/test'

List dir: (returns list)
os.listdir( os.getcwd() )

List of files:
[p for p in os.listdir(os.getcwd())
if os.path.isfile(p)]

List of folders:
[p for p in os.listdir(os.getcwd())
if os.path.isdir(p)]

Build path for local OS: (using appropriate slash)
pathstr = os.path.join("folder", "folder", "file_or_folder")

File and directory management with shutil
import shutil
shutil.copyfile('file1', 'file2')
shutil.move('file1', 'folder2/file1')

File Wildcards:
import glob
glob.glob('*.py') # return list of files

Walk directory tree: [http://stackoverflow.com/questions/120656/directory-listing-in-python]
<pre>
import os
for dirname, dirnames, filenames in os.walk('.'):
for subdirname in dirnames:
print os.path.join(dirname, subdirname)
for filename in filenames:
print os.path.join(dirname, filename)
</pre>

ugly 'deltree' (by kenneth)
<pre>
import os
def deltree(path):
files = os.listdir(path)
for file in files:
file_path = os.path.join(path, file)
if os.path.isdir(file_path):
deltree(file_path)
os.rmdir(file_path)
else:
os.remove(file_path)
</pre>

Cleaner 'deltree' (by Kep)
<pre>
import shutil
# Blow away (if needed), and reclone to proper version
if os.path.isdir("xxx"):
shutil.rmtree("xxx")
</pre>

== System Platform ==

<pre>
import platform

if platform.system() == "Linux":
# ...
if platform.system() == "Windows":
# ...
else:
# ...
</pre>

=== Temp Files ===

import tempfile
filename = tempfile.mktemp()
f = open(filename, 'w')
...
f.close()

import os
os.remove(filename) # delete temp file

=== Pickling ===

Pickling (serializing, making objects persistent):
import pickle
grades = {'bob' : [4, 5, 6],
'sue' : [3, None, 7, 7]}
outfile = open(fname, 'wb')
pickle.dump(grades, outfile)

infile = open(fname, 'rb')
grades = pickle.load(infile)

=== [http://docs.python.org/library/logging.html Logging] ===

* logging — Logging facility for Python — Python 3 documentation - https://docs.python.org/3/library/logging.html
* Logging HOWTO — Python Dev Version documentation - http://docs.python.org/dev/howto/logging.html
* Logging Cookbook — Python Dev Version documentation - http://docs.python.org/dev/howto/logging-cookbook.html

Setup logger:
import logging
logging.basicConfig(level=logging.DEBUG,format='[%(asctime)s.%(msecs)03d] %(levelname)s %(filename)s(line:%(lineno)d): %(message)s')

Log Levels:
import logging
msg = "i broke"
logging.debug(msg)
logging.info(msg)
logging.warning(msg) # WARNING:root:i broke
logging.error(msg) # ERROR:root:i broke
logging.critical(msg) # CRITICAL:root:i broke

log.setLevel(level) # debug, info, warn/warning, error, critical
log.setLevel(logging.DEBUG)

A good convention to use when naming loggers is to use a module-level logger, in each module which uses logging, named as follows: [http://docs.python.org/dev/howto/logging.html]
logger = logging.getLogger(__name__)

Simple example: [http://stackoverflow.com/questions/345991/python-no-handlers-could-be-found-for-logger-opengl-error]
<pre>
log_format='%(asctime)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
logging.basicConfig(format=log_format, datefmt=log_date_format)
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)

log.info('hi')
</pre>

Formatter function:
<pre>
def makeFormatter(format_str="", date_format="%Y-%m-%d %H:%M:%S"):
if not format_str:
format_str = "%(asctime)s | %(levelname)s | %(filename)s:%(lineno)d:%(funcName)s | %(message)s"
return logging.Formatter(fmt=format_str, datefmt=date_format)
</pre>

Medium example:
<pre>
# Create logger
log_format='%(asctime)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
log_formatter = logging.Formatter(fmt=log_format, datefmt=log_date_format)
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)

# Log to console
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(log_formatter)
log.addHandler(ch)

# Log to file
fh = logging.FileHandler('my_logger.log')
fh.setLevel(logging.DEBUG)
fh.setFormatter(log_formatter)
log.addHandler(fh)
</pre>

Log all loggers to file - attach to root logger
<pre>
import logging

logger = logging.getLogger(__name__)

# Configure Logger
log_format='%(asctime)s | %(name)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
log_formatter = logging.Formatter(fmt=log_format, datefmt=log_date_format)
logging.basicConfig(level=logging.DEBUG, format=log_format, datefmt=log_date_format)

# Setup Log to File handler, and attach to root logger
root_logger = logging.getLogger() # Calling getLogger() without a name returns the root logger
fh = logging.FileHandler('logger.txt')
fh.setLevel(logging.DEBUG)
fh.setFormatter(log_formatter)
root_logger.addHandler(fh)
</pre>

Application Setup (not basic log configuration carries across imported modules):
<pre>
import logging
def main():
logging.basicConfig(filename='myapp.log', level=logging.INFO)
logging.info('Started')
...
logging.info('Finished')

if __name__ == '__main__':
main()
</pre>

Log multiple lines:
<pre>
for line in output.splitlines():
log.debug(line)
</pre>

Set log level: (warning is default)
logging.basicConfig(level=logging.DEBUG)

Set log level from string value: [http://docs.python.org/dev/howto/logging.html]
<pre>
# assuming loglevel is bound to the string value obtained from the
# command line argument. Convert to upper case to allow the user to
# specify --log=DEBUG or --log=debug
numeric_level = getattr(logging, loglevel.upper(), None)
if not isinstance(numeric_level, int):
raise ValueError('Invalid log level: %s' % loglevel)
logging.basicConfig(level=numeric_level, ...)
</pre>

Log to file:
logging.basicConfig(filename='example.log',level=logging.DEBUG)
logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG) # do not append

Format output: (once set, can't be changed?)
# WARNING:MESSAGE
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)

# 05/23/2011 12:50:02 PM MESSAGE
logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')

# 05/23/2011 12:55:28 PM : WARNING : MESSAGE
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')

# 2011-05-23 13:05:30 : WARNING : MESSAGE
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

# 2011-05-23 13:05:30 | WARNING | FILE:FUNC | MESSAGE
logging.basicConfig(format='%(asctime)s | %(levelname)s | %(filename)s:%(funcName)s | %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

Log record attributes - http://docs.python.org/library/logging.html#logrecord-attributes

Format Example:
<pre>
FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' # doesn't work!
logging.basicConfig(format=FORMAT)
d = { 'clientip' : '192.168.0.1', 'user' : 'fbloggs' }
logger = logging.getLogger('tcpserver')
logger.warning('Protocol problem: %s', 'connection reset', extra=d)
</pre>

<pre>
import logging

# create logger with 'spam_application'
logger = logging.getLogger('spam_application')
logger.setLevel(logging.DEBUG)

# create file handler which logs even debug messages
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)

# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)

# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(fh)
logger.addHandler(ch)

logger.info('Message to loggers and handlers')
</pre>

=== Templating ===

from string import Template
t = Template('${name}'s fav dish is $food')
t.substitute(name = 'ken', food = 'beans') # throws exception if missing one, but can have more
t.safe_substitute(name = 'ken') # will fill in what it can

Change delimiter:
class MYT(Template):
delimiter = '%'
minput = input("File name format (%d - date, %s - size): ")
myt = MYT(minput)
myt.substitute(%d=date, %s=size)

=== [http://docs.python.org/library/threading.html Threading] ===

Basic thread: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import thread

def someFunc():
print "someFunc was called"

thread.start_new_thread(someFunc, ())
</pre>

More complete basic thread:
<pre>
import thread
import threading
import time

def run_thread (threadname, sleeptime):
print "Thread start: " + threadname
time.sleep(sleeptime)
print "Thread end: " + threadname
thread.interrupt_main() # throw keyboard interrupt to kill main program

if __name__ == "__main__":
thread.start_new_thread(run_thread, ("Thread1", 3))
try:
while 1:
pass
except:
print "Thread1 exited...."
</pre>

Can use a global counter instead of exception to track number of threads:
<pre>
import thread
import threading
import time

threadcount = 3

def run_thread (threadname, sleeptime):
global threadcount
print "Thread start: " + threadname
time.sleep(sleeptime)
print "Thread end: " + threadname
threadcount -= 1

if __name__ == "__main__":
thread.start_new_thread(run_thread, ("Thread1", 3))
thread.start_new_thread(run_thread, ("Thread2", 4))
thread.start_new_thread(run_thread, ("Thread3", 2))
while 1:
if threadcount == 0:
break
else:
pass
print "completed"
</pre>

---

Thread object: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import threading

t1 = threading.Thread(target=someFunc)
# with parameters: Thread(target=myfunction, args=('MyStringHere',1)).start()
t1.start()
t1.join()
</pre>

Basic threading with parameters:
<pre>
import threading
t = threading.Thread(None, someFunc, None, (param1, param2))
t.start()
t.join()
</pre>

<pre>
t = threading.Thread(target=client_recv, args=(s, name))
while True:
t.join(timeout=2)
if t.isAlive():
print "Thread is dead..."
break
</pre>

Threading as a class: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import threading

class FuncThread(threading.Thread):
def __init__(self, target, *args):
self._target = target
self._args = args
threading.Thread.__init__(self)

def run(self):
self._target(*self._args)

# Example usage
def someOtherFunc(data, key):
print "someOtherFunc was called : data=%s; key=%s" % (str(data), str(key))

t1 = FuncThread(someOtherFunc, [1,2], 6)
t1.start()
t1.join()
</pre>

Threading as a class #2:
<pre>
import threading
import time
import sys

class MyT(threading.Thread):
def __init__(self, name, time):
threading.Thread.__init__(self)
self.name = name
self.time = time
def run(self):
print("Thread Start: " + self.name)
for i in range(self.time):
print("[" + self.name + "]", end='')
sys.stdout.flush()
time.sleep(1)
print("Thread Stop: " + self.name)

myt1 = MyT('1', 10)
myt1.start()

myt2 = MyT('2', 15)
myt2.start()

myt3 = MyT('3', 5)
myt3.start()

myt4 = MyT('4', 10)
myt4.start()

myt1.join()
myt2.join()
myt3.join()
myt4.join()
</pre>

Locking:
<pre>
lock=thread.allocate_lock()

# pass lock to thread, and use in thread as such:
lock.acquire()
#some operation
lock.release()
</pre>

Returning data from a thread (using a [http://docs.python.org/library/queue.html queue]): [http://stackoverflow.com/questions/1886090/return-value-from-thread] [http://stackoverflow.com/questions/2829329/catch-a-threads-exception-in-the-caller-thread-in-python]
<pre>
# Pass exception back to parent
import sys
import threading
import Queue

class ExcThread(threading.Thread):
def __init__(self, bucket):
threading.Thread.__init__(self)
self.bucket = bucket
def run(self):
try:
raise Exception('An error occured here.')
except Exception:
self.bucket.put(sys.exc_info())

def main():
bucket = Queue.Queue()
thread_obj = ExcThread(bucket)
thread_obj.start()
while True:
try:
exc = bucket.get(block=False)
except Queue.Empty:
pass
else:
exc_type, exc_obj, exc_trace = exc
# deal with the exception
print "Exception Caught:", exc_type, exc_obj
print "Exception Trace:", exc_trace
thread_obj.join(0.1)
if thread_obj.isAlive():
continue
else:
break

if __name__ == '__main__':
main()
</pre>

<pre>
Exception Caught: <type 'exceptions.Exception'> An error occured here.
Exception Trace: <traceback object at 0x2b2230a285f0>
</pre>

Another Exception catching queue example: (kenneth's example)
<pre>
# thread function
def run_thread(queue, item):
...
queue.put( Exception("Failure on item:" + item) )

threads = []
queue = Queue.Queue()

# start threads
for item in items:
t = Thread(None, run_thread, None, (queue, item))
t.start()
threads.append(t)

# wait for threads
for t in threads:
t.join()

# catch and reraise exceptions
while not queue.empty():
raise(queue.get())
</pre>

Returning data from a thread (using a callback function): [http://stackoverflow.com/questions/1886090/return-value-from-thread]
:"Another approach is to pass a callback function to the thread. This gives a simple, safe and flexible way to return a value to the parent, anytime from the new thread."
<pre>
import threading
import time
import sys

class MyThread(threading.Thread):
def __init__(self, cb):
threading.Thread.__init__(self)
self.callback = cb
def run(self):
for i in range(10):
self.callback(i)
time.sleep(1)

def count(x):
print x
sys.stdout.flush()

t = MyThread(count)
t.start()
</pre>

References:
* 16.2. threading — Higher-level threading interface — Python v2.7.1 documentation - http://docs.python.org/library/threading.html
* An Introduction to Thread Programming - http://www.prasannatech.net/2008/08/introduction-to-thread-programming.html
* Python - Multithreaded Programming - http://www.tutorialspoint.com/python/python_multithreading.htm
* Running Functions as Threads in Python « Software Ramblings - http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html
* Understanding Threading in Python LG #107 - http://linuxgazette.net/107/pai.html
* multithreading - Simple threading in Python 2.6 using thread.start_new_thread() - Stack Overflow - http://stackoverflow.com/questions/849674/simple-threading-in-python-2-6-using-thread-start-new-thread
* python - Return value from thread - Stack Overflow - http://stackoverflow.com/questions/1886090/return-value-from-thread

---

Interrupt main thread:
<pre>
# source: http://stackoverflow.com/questions/4591917/python-how-to-quit-cli-when-stuck-in-blocking-raw-input

import thread
import time

def main():
try:
m = thread.start_new_thread(killable_input, tuple())
while 1:
time.sleep(0.1)
except KeyboardInterrupt:
print "exception"

def killable_input():
w = thread.start_new_thread(normal_input, tuple())
i = thread.start_new_thread(wait_sometime, tuple())

def normal_input():
s = raw_input("input:")

def wait_sometime():
time.sleep(4) # or any other condition to kill the thread
print "too slow, killing imput"
thread.interrupt_main()

if __name__ == '__main__':
main()
</pre>

=== urllib ===

See [[Python/urllib]]

=== Command line arguments ===

<pre>
import sys
for x in sys.argv:
print x
</pre>

=== Execute other python script ===

import sys
sys.argv = ['', 'my', 'args', 'here']
execfile('myscript.py') # does not support arguments directly

=== Exit Code ===

sys.exit(n) http://docs.python.org/library/sys.html:
import sys
sys.exit(1) # exit code of 1 (default is 0)
sys.exit("some error") # exit code of 1, and message displayed

=== [http://docs.python.org/library/syslog.html syslog] ===

<pre>
import syslog
syslog.openlog('[WATER]')
syslog.syslog('Processing started')
if error:
syslog.syslog(syslog.LOG_ERR, 'Processing started')
</pre>

<pre>
import syslog
syslog.openlog('[WATER]')
def log(msg):
syslog.syslog(str(msg))
log("Starting...")
</pre>

=== Subprocess ===

Subprocess - http://docs.python.org/library/subprocess.html

Methods of subprocess (or external command execution):
* subprocess (only method to rule/replace all others)
* os.system
* os.spawn*
* os.popen*
* popen2.*
* commands.*

My New Favorite:
<pre>
import subprocess
class Run(object):
class RC(int):
pass
def __new__(cls, cmd, *args, **kwargs):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
rc = cls.RC(p.wait())
rc.output = "\n".join(p.stdout.readlines())
rc.error = "\n".join(p.stderr.readlines())
return rc

if Run("some command"): print "do something" # on rc of not 0 (negative or positive)
</pre>

My Favorite:
<pre>
def exe(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
#output = " | ".join(p.stdout.readlines())
output = "\n".join(p.stdout.readlines())
rc = p.wait()
return (rc, output)

def exe_checked(cmd, checked=True):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
#output = " | ".join(p.stdout.readlines())
output = "\n".join(p.stdout.readlines())
rc = p.wait()
if rc and checked:
raise Exception("Command returned non zero [{0}]: {1}".format(rc, cmd)
return (rc, output)

def exe_with_stderr(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subproc
ess.PIPE)
output = " | ".join(p.stdout.readlines())
outputerr = " | ".join(p.stderr.readlines())
rc = p.wait()
return (rc, output, outputerr)
</pre>

<pre>
def exec_program(cmd):
s = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = s.communicate()
result = s.wait()
return stdout, stderr, result
</pre>

Pass along output and return true error code:
import subprocess
retcode = subprocess.call(["ls", "-l"]) # true return code, output sent to STDOUT

retcode = subprocess.call(["foo.sh","args"],shell=True) # Execute in shell, with shell processing
retcode = subprocess.call(['C:\\Temp\\a b c\\Notepad.exe'])
retcode = subprocess.call(["D:/Python24/Python.exe", "Skripta.py", "arg1", "arg2"])

Note: shell=True is equivalent to: subprocess.Popen(['/bin/sh', '-c', args[0], args[1], ...])

Process STDOUT and STDERR and get return code:
<pre>
#!/usr/bin/env python
import subprocess
import string

cmd = "cat /etc/passwd | grep ken"
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) # stderr=subprocess.STDOUT to combine

out = string.join(p.stdout.readlines())
outerr = string.join(p.stderr.readlines())
retcode = str(p.wait())

print("STDOUT: " + out)
print("STDERR: " + outerr)
print("RETCODE: " + retcode)
</pre>

Above with stderr redirected to stdout (combined):
<pre>
import subprocess
p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
print line,
retval = p.wait()
</pre>

Replacing shell backtick: [http://docs.python.org/library/subprocess.html]
# shell code:
# OUTPUT=`mycmd myarg`
from subprocess import Popen
output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]

Replacing shell pipeline [http://docs.python.org/library/subprocess.html]
# shell code:
# OUTPUT=`dmesg | grep hda`
from subprocess import Popen
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

Replacing os.system(): [http://docs.python.org/library/subprocess.html]
# old:
sts = os.system("mycmd" + " myarg")

# new:
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)[1]

Control an executed process:
<pre>
cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

# cycle while process continues alive
while p.poll() == None:

# check if input
if select.select([p.stdout.fileno()], [], [], 1)[0]:
while select.select([p.stdout.fileno()], [], [], .1)[0]:
line = p.stdout.readline()
# ... process

# write something to process
p.stdin.write("tp %s %s\n" % (user1, user2))

...

# time to force a quit: (choose which you need)
p.kill() # SIGKILL
p.terminate() # SIGTERM
os.kill(p.pid, signal.SIGTERM) # same as first, but not as clean
</pre>

==== Other Command Execution ====

Old method with os.system(): [http://docs.python.org/library/os.html]
import os
encoded_wait_code = os.system("some command") # output sent to STDOUT
# Note: returns code of 0 if command returns 0, but wait() encoded value for all other values

Quick and dirty old method (prints stderr, captures stdout)
<pre>
import os
out = string.join( os.popen(cmd).readlines() )
</pre>

Quick and dirty old method (prints stderr, captures stdout, captures ugly wait() encoded return code)
<pre>
import os
p = os.popen(cmd)
stdout = "".join( p.readlines() )
encoded_wait_code = p.close() # None if 0
rcode = os.WEXITSTATUS(ugly_rcode) # fix ugly wait() encoded rcode
print(stdout)
print(rcode)

if os.popen(cmd).close():
print "error with cmd: %s" % cmd
sys.exit(1)
</pre>

???:
<pre>
os.spawnvp(os.P_WAIT,"/usr/bin/mplayer",["","-ao","pcm",file])
# or
os.spawnvp(os.P_NOWAIT,"/usr/bin/mplayer",["","-ao","pcm",file])
</pre>

Replacing os.system(): [http://docs.python.org/library/subprocess.html]
# old:
sts = os.system("mycmd" + " myarg")

# new:
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)[1]

More realistic example replacing os.system(): [http://docs.python.org/library/subprocess.html]
<pre>
try:
retcode = call("mycmd" + " myarg", shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e
</pre>

==== select ====

Select:

select.select(rlist, wlist, xlist[, timeout])
: This is a straightforward interface to the Unix select() system call. The first three arguments are sequences of ‘waitable objects’: either integers representing file descriptors or objects with a parameterless method named fileno() returning such an integer
: The optional timeout argument specifies a time-out as a floating point number in seconds. When the timeout argument is omitted the function blocks until at least one file descriptor is ready. A time-out value of zero specifies a poll and never blocks.

if select.select([sys.stdin.fileno()], [], [], .1)[0]:
line = p.stdout.readline()

==== full wrapped execution ====

Process timeout (by Kenneth):
<pre>
import subprocess
import time
import os
import signal

def cmd(cmd, timeout=300, retry=0):
# Execute command, time progress, and kill on timeout

# (status, output) = commands.getstatusoutput(cmd)
start_time = None
status = None
output = None
for retry_count in range(retry+1):
if retry_count > 0:
print("RETRY: " + str(retry_count) + " " + cmd)
start_time = time.time()
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
timeout_fail = False
while p.poll() == None:
if time.time() - start_time > timeout:
# kill process
print("TIMEOUT: " + str(timeout) + "sec - killing (" + str(p.pid) + ") - " + cmd)
try:
os.kill(p.pid, signal.SIGTERM)
except OSError as e:
print("Kill exception: " + str(e))
timeout_fail = True
break
#print(".")
time.sleep(1)
status = p.poll()
output = p.stdout.read()
if timeout_fail == False:
break
seconds = int(time.time() - start_time)
return (status, output, seconds)
</pre>

---

Process input and output wrapper (from minecraft):
<pre>
import subprocess
import select
import re
import sys

cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)

# cycle while process continues alive
while p.poll() == None:

if select.select([p.stdout.fileno()], [], [], 1)[0]:
while select.select([p.stdout.fileno()], [], [], .1)[0]:
line = p.stdout.readline()
# <kiloforce> -tp joe
match = re.findall("<(.*)> -tp (.*)", line)
if match:
user1 = match[0][0]
user2 = match[0][1]
print ">> ACTION: Teleporting %s to %s" % (user1, user2)
p.stdin.write("tp %s %s\n" % (user1, user2))
if line:
print ">> SERVER: %s" % line.strip()
else:
print "The server died!"
sys.exit(1)

if select.select([sys.stdin.fileno()], [], [], .1)[0]:
line = sys.stdin.readline()
print ">> CONSOLE: %s" % line.strip()
p.stdin.write("%s" % line)
</pre>

==== Command With Timeout ====

python - Using module 'subprocess' with timeout - Stack Overflow - http://stackoverflow.com/questions/1191374/using-module-subprocess-with-timeout

<pre>
import subprocess, threading

class Command(object):
def __init__(self, cmd):
self.cmd = cmd
self.process = None

def run(self, timeout):
def target():
print 'Thread started'
self.process = subprocess.Popen(self.cmd, shell=True)
self.process.communicate()
print 'Thread finished'

thread = threading.Thread(target=target)
thread.start()

thread.join(timeout)
if thread.is_alive():
print 'Terminating process'
self.process.terminate()
thread.join()
print self.process.returncode

command = Command("echo 'Process started'; sleep 2; echo 'Process finished'")
command.run(timeout=3)
command.run(timeout=1)
</pre>

My version:
<pre>
import subprocess
import threading

class Command(object):
def __init__(self, cmd, capture=False, shell=True):
self.cmd = cmd
self.process = None
self.thread = None
self.shell = shell
if type(cmd) == list:
self.shell = False
self.rc = None
self.pid = None
self.stdout = None
self.stderr = None
self.capture = capture # capture stdout?

def run(self, timeout=None, wait=True):
def target():
print '>> Thread started (%s)' % self.cmd
if self.capture:
self.process = subprocess.Popen(self.cmd, shell=self.shell, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
else:
self.process = subprocess.Popen(self.cmd, shell=self.shell)
print '>> Thread PID: %d' % self.process.pid
self.pid = self.process.pid
self.stdout, self.stderr = self.process.communicate()
self.rc = self.process.returncode
print '>> Thread finished: %d' % self.rc

self.thread = threading.Thread(target=target)
self.thread.start()

if wait:
self.wait(timeout)
return self # so you can do: c = Command().run()

def wait(self, timeout=None):
print '>> Waiting for thread...'
if timeout:
self.thread.join(timeout)
else:
self.thread.join()
if self.thread.is_alive():
print '>> Terminating process'
self.process.terminate()
self.rc = self.process.returncode
self.thread.join()
if self.thread.is_alive():
print '>> Failed to terminate'
else:
print '>> Thread completed'
#print self.process.returncode

def is_alive(self ):
return self.thread.is_alive()

command = Command('do something').run()

command2 = Command(['do', 'something', 'long']).run(wait=False)
command2.wait(timeout=10)

command3 = Command('want output', capture=True).run()
print command3.stdout
</pre>

=== System call ===

os.system("some_command with args")
os.system("some_command < input_file | another_command > output_file")
stream = os.popen("some_command with args")

Here's a summary of the ways to call external programs and the advantages and disadvantages of each:

1.
: os.system("some_command with args") passes the command and arguments to your system's shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example,
: os.system("some_command < input_file | another_command > output_file")
: However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs.
: http://docs.python.org/lib/os-process.html

2.
: stream = os.popen("some_command with args") will do the same thing as os.system except that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything.
: http://docs.python.org/lib/os-newstreams.html

3.
: The Popen class of the subprocess module. This is intended as a replacement for os.popen but has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you'd say
: print Popen("echo Hello World", stdout=PIPE, shell=True).stdout.read()
: instead of
: print os.popen("echo Hello World").read()
: but it is nice to have all of the options there in one unified class instead of 4 different popen functions.
: http://docs.python.org/lib/node528.html

4.
: The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply wait until the command completes and gives you the return code. For example:
: return_code = call("echo Hello World", shell=True)
: http://docs.python.org/lib/node529.html

5.
: The os module also has all of the fork/exec/spawn functions that you'd have in a C program, but I don't recommend using them directly.

: The subprocess module should probably be what you use.

Source: [http://stackoverflow.com/questions/89228/how-to-call-external-command-in-python]

== Advanced Code ==

=== Executing Code from Command Line ===

python -c [cmd]
# -c cmd : program passed in as string (terminates option list)

<pre>
python -c "import sys;exec(sys.stdin.read())" << "EOF"
def test1():
print "test1"
test1()
print "test"
EOF
</pre>

=== Integer Return Code Class ===

Python3 version:
<pre>
class RC(int):
def __new__(cls, rc, stdout='', stderr=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, stdout='', stderr=''):
super().__init__()
self.rc = rc # redundant
self.stdout = stdout
self.stderr = stderr
def __repr__(self):
"""Optional string output"""
return "(%s, '%s', '%s')" % (self.rc, self.stdout, self.stderr)
</pre>

Python2 version:
<pre>
class RC(int):
def __new__(cls, rc, stdout='', stderr=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, stdout='', stderr=''):
super(self.__class__, self).__init__(rc)
self.rc = rc # redundant
self.stdout = stdout
self.stderr = stderr
def __repr__(self):
"""Optional string output"""
return "(%s, '%s', '%s')" % (self, self.stdout, self.stderr)
</pre>

sample:
<pre>
>>> rc1 = RC(5)
>>> rc2 = RC(0, 'out', 'err')
>>> rc1
(5, '', '')
>>> rc2
(5, 'out', 'err')
>>> rc2.stdout
'out'
>>> int(rc1)
5
>>> 0 + rc1
5
>>> if rc1 <= 5: print "yes"
...
yes
>>> if rc2: print "non-zero"
...
</pre>

=== Smart Execute ===

<pre>
class RC(int):
def __new__(cls, rc, output=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, output=''):
super().__init__()
self.rc = rc # redundant
self.output = output
def __repr__(self):
"""Optional string output"""
return "(%s, '%s')" % (self.rc, self.output)

def smart_execute(cmd, display=True, checked=False):
output = ""
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, shell=True, encoding='utf-8')
for c in iter(lambda: process.stdout.read(1), ""):
if display:
sys.stdout.buffer.write(c.encode('utf-8'))
output += c
process.stdout.close()
rc = process.wait()
if checked and rc:
raise subprocess.CalledProcessError(ret, cmd)
return RC(rc, output.strip())
</pre>

Not smart version:
<pre>
def execute(cmd, return_output=False):
out = ""
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, shell=True, encoding='utf-8')
for c in iter(lambda: process.stdout.read(1), ""):
sys.stdout.buffer.write(c.encode('utf-8'))
if return_output:
out += c
process.stdout.close()
ret = process.wait()
if ret:
raise subprocess.CalledProcessError(ret, cmd)
if return_output:
return out.strip()
</pre>
</pre>

=== select ===

usage: select.select(rlist, wlist, xlist[, timeout]) [http://docs.python.org/2/library/select.html]
* rlist: wait until ready for reading
* wlist: wait until ready for writing
* xlist: wait for an “exceptional condition” (see the manual page for what your system considers such a condition)
* The optional timeout argument specifies a time-out as a floating point number in seconds.

if select.select([p.stdout.fileno()], [], [], 1)[0]:

=== ANSI Color ===

* ANSI escape code - http://en.wikipedia.org/wiki/ANSI_escape_code

---

unicode - Print in terminal with colors using Python? - Stack Overflow - https://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python

<pre>
class bcolors:
HEADER = '\033[95m'
OKBLUE = '\033[94m'
OKGREEN = '\033[92m'
WARNING = '\033[93m'
FAIL = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'

print bcolors.WARNING + "Warning: No active frommets remain. Continue?" + bcolors.ENDC
</pre>

<pre>
class colors:
black = 0
red = 1
green = 2
yellow = 3
blue = 4
magenta = 5
cyan = 6
white = 7
fg = 30 # foreground add
bg = 40 # background add
bfg = 90 # bold foreground add
bbg = 100 # bold background add

print '\033[%dm red foreground \033[0m' % (colors.red + colors.fg)
print '\033[%dm bold red foreground \033[0m' % (colors.red + colors.bfg)
print '\033[%dm blue background \033[0m' % (colors.blue + colors.bg)
print '\033[%dm bold blue background \033[0m' % (colors.blue + colors.bbg)
</pre>

---

Before you start spouting escape sequences, you should check that stdout is a tty. You can do this with sys.stdout.isatty()

<pre>
sys.stdout.isatty()
</pre>

<pre>
def hilite(string, status, bold):
attr = []
if status:
# green
attr.append('32')
else:
# red
attr.append('31')
if bold:
attr.append('1')
return '\x1b[%sm%s\x1b[0m' % (';'.join(attr), string)
</pre>

Python | change text color in shell - Stack Overflow - https://stackoverflow.com/questions/2330245/python-change-text-color-in-shell

---

For reference, a small bash script which prints a table of escape sequences: colortable16.sh: ([http://www.frexx.de/xterm-256-notes/ The 256 color mode of xterm], [http://plug.org/pipermail/plug/2008-March/016821.html The 90s called; they want their 256 colors back])
<pre>
#!/bin/bash

# prints a color table of 8bg * 8fg * 2 states (regular/bold)
echo
echo Table for 16-color terminal escape sequences.
echo Replace ESC with \\033 in bash.
echo
echo "Background | Foreground colors"
echo "---------------------------------------------------------------------"
for((bg=40;bg<=47;bg++)); do
for((bold=0;bold<=1;bold++)) do
echo -en "\033[0m"" ESC[${bg}m | "
for((fg=30;fg<=37;fg++)); do
if [ $bold == "0" ]; then
echo -en "\033[${bg}m\033[${fg}m [${fg}m "
else
echo -en "\033[${bg}m\033[1;${fg}m [1;${fg}m"
fi
done
echo -e "\033[0m"
done
echo "--------------------------------------------------------------------- "
done

echo
echo
</pre>

Python version: (modified by me)
<pre>
#!/usr/bin/env python

import sys

# prints a color table of 8bg * 8fg * 2 states (regular/bold)
print
print "Table for 16-color terminal escape sequences."
print "Replace ESC with \\033 in bash."
print
print "Background | Foreground colors"
print "---------------------------------------------------------------------"
for bg in range(40, 47+1):
for bold in range(0, 1+1):
sys.stdout.write("\033[0m"" ESC[{bg}m | ".format(bg=bg))
for fg in range(30, 37+1):
if bold == 0:
sys.stdout.write("\033[{bg}m\033[{fg}m [{fg}m ".format(bg=bg, fg=fg))
else:
sys.stdout.write("\033[{bg}m\033[1;{fg}m [1;{fg}m".format(bg=bg, fg=fg))
sys.stdout.write("\033[0m\n")
print "--------------------------------------------------------------------- "

print
print
</pre>

=== Execute External Application ===

Execute program: (and send output to STDOUT)
encoded_wait_code = os.system('mkdir mydir') # output sent to STDOUT

Get output and ugly return code: (STDERR goes to STDOUT)
p = os.popen(cmd)
print p.readlines()
encoded_wait_code = p.close()

Execute other python script
import sys
sys.argv = [, 'my', 'args', 'here']
execfile('myscript.py') # does not support arguments directly

Subprocess:
* See [[#Subprocess]]

=== Program Layout ===

<pre>
PackageName/
README
setup.py
bin/
script
docs/
test/ # some include in package_name
pacakgename/
__init__.py
code1.py
subpackage/
__init__.py
</pre>

=== __init__.py ===

Purpose: Makes the folder into a package library

NOTE: This file is most commonly blank

Good place to put your package functions.
<pre>
# ken/__init__.py
def load():
...
</pre>

<pre>
import ken
ken.load()
</pre>

Can import other modules to then use as package modules:
<pre>
# ken/__init__.py
from ken.client import Server
</pre>

<pre>
import ken
ken.Server()
</pre>

Good place to put your Exceptions.
<pre>
class MyException(Exception):
""" My Error Exception """
</pre>

Use the __all__ to specify what gets imported on 'from package import *'
__all__ = ['func1', 'func2']

=== Metadata ===

VERSION = (0, 2, 1)
or
__version__ = '0.2.1'

Also See [[#pydoc]]

---

__version__ = '1.10'
__author__ = 'Joshua Roesslein'
__license__ = 'MIT'

=== setup.py ===

See [[#Distributing Python Modules with Distutils]]

=== Distributing Python Modules with Distutils ===

Distributing Python Modules — http://docs.python.org/distutils/index.html

Uses setup.py for package installation.
python setup.py install

==== Simple example ====

To install foo.py:
<pre>
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
</pre>

Source: http://docs.python.org/distutils/introduction.html

==== klib example ====

klib setup.py:
<pre>
from distutils.core import setup

setup(name = "klib",
version = "101",
description = "klib library",
url = "http://www.oeey.com/klib",
author = "Kenneth",
author_email = "kenneth@oeey.com",
license = "Kenneth License",
long_description = """Really long text here.""",
platforms = "Any Platform",

packages = ['klib','klib.messenger'],
)
</pre>

Installed to:
/opt/python26/lib/python2.6/site-packages/klib/
/opt/python26/lib/python2.6/site-packages/klib-100-py2.6.egg-info

Metadata: (/opt/python26/lib/python2.6/site-packages/klib-101-py2.6.egg-info)
<pre>
Metadata-Version: 1.0
Name: klib
Version: 101
Summary: klib library
Home-page: http://www.oeey.com/klib
Author: Kenneth
Author-email: kenneth@oeey.com
License: Kenneth License
Description: Really long text here.
Platform: Any platform
</pre>

Any fields not included will be marked:
Platform: UNKNOWN

References:
* 2. Writing the Setup Script — Python v2.7.2 documentation - http://docs.python.org/distutils/setupscript.html
* http://wiki.python.org/moin/Distutils/Tutorial

==== Uninstall Package ====

Collect files during install:
python setup.py install --record install-record.txt
sudo rm $(cat install-record.txt)

References:
* installation - python setup.py uninstall - Stack Overflow - http://stackoverflow.com/questions/1550226/python-setup-py-uninstall

=== Distributing Python Modules with Setuptools ===

<pre>
#!/usr/bin/env python
import setuptools

setuptools.setup(name="tweepy",
version="1.10",
description="Twitter library for python",
license="MIT",
author="Joshua Roesslein",
author_email="tweepy@googlegroups.com",
url="http://github.com/tweepy/tweepy",
packages = setuptools.find_packages(),
keywords= "twitter library",
zip_safe = True)

</pre>

=== Decorators ===

Decorators wrap functions

Function Counter: [http://stackoverflow.com/questions/739654/understanding-python-decorators]
<pre>
def fcount(func):
def wrapper(*args, **kw):
wrapper.count = wrapper.count + 1
return func(*args, **kw)
wrapper.count = 0
return wrapper

@fcount
def f(n):
return n*n

print "f.count = ", f.count
print "f(10) = ", f(10)
print "f.count = ", f.count
print "f(3) = ", f(3)
print "f.count = ", f.count
print "f(12) = ", f(12)
print "f.count = ", f.count
print "f(15) = ", f(15)
print "f.count = ", f.count

# Output:
# f.count = 0
# f(10) = 100
# f.count = 1
# f(3) = 9
# f.count = 2
# f(12) = 144
# f.count = 3
# f(15) = 225
# f.count = 4
</pre>

Great reference: Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

Example 1:
@dec1
def func(arg1, arg2, ...): pass

# equivalent to:
def func(arg1, arg2, ...): pass
func = dec1(func)

Example 2:
<pre>
@dec2
@dec1
def func(arg1, arg2, ...):
pass
</pre>

<pre>
# equivalent to:
def func(arg1, arg2, ...):
pass
func = dec2(dec1(func))
</pre>

Example 3:
@decomaker(argA, argB, ...)
def func(arg1, arg2, ...):
pass

# equivalent to:
func = decomaker(argA, argB, ...)(func)

# example:
@accepts(int,int)
@returns(float)
def bar(low,high):
pass

Example 4: (overrides and ignores passed in function - bad, unless desired)
<pre>
def spamrun(fn):
def sayspam(*args):
print "spam, spam, spam"
return sayspam

@spamrun
def useful(a, b):
print a**2 + b**2
</pre>

Example 5: (Decorator that modifies behavior of undecorated func)
<pre>
def addspam(fn):
def new(*args):
print "spam, spam, spam"
return fn(*args)
return new

@addspam
def useful(a, b):
print a**2 + b**2
</pre>

My Example 1: (print args and continue to function)
<pre>
def showargs(fn):
def doshowargs(*args, **kw):
print "Args for:", fn.__name__, args, kw
return fn(*args, **kw)
return doshowargs

@showargs
def useful(a, b):
print a**2 + b**2

</pre>

Another Example: [http://stackoverflow.com/questions/739654/understanding-python-decorators]
<pre>
def makebold(fn):
def wrapped(*args, **kw):
return "" + fn(*args, **kw) + ""
return wrapped

def makeitalic(fn):
def wrapped(*args, **kw):
return "" + fn(*args, **kw) + ""
return wrapped

@makebold
@makeitalic
def hello(msg):
return msg

print hello("hello world")
# output: hello world
</pre>

References:
* PEP 318 -- Decorators for Functions and Methods - http://www.python.org/dev/peps/pep-0318/
* Charming Python: Decorators make magic easy - http://www.ibm.com/developerworks/linux/library/l-cpdecor/index.html
* What are some common uses for Python decorators? - Stack Overflow - http://stackoverflow.com/questions/489720/what-are-some-common-uses-for-python-decorators
* Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

---

Utah Python July 2013 Meeting - Decorator Examples: [https://www.youtube.com/watch?v=uh9uhzitdvw]

Example of uses:
* Count function calls
* Add additional output (debugging)
* Timing code
* Thread synchronization (locks)
* Type checking

<pre>
## Simple wrapping call
def shout(f):
def wrapper(*args):
print('This is before {name}'.format(name=f.__name__))
ret = f(*args)
print('This is after {name}'.format(name=f.__name__))
return ret
wrapper.__name__ = f.__name__
return wrapper

@shout
def echo(text):
print(text)

echo('this is a test')
</pre>

<pre>
## Counting function calls
def count(f):
def wrapper(*args):
wrapper.counter += 1
print("Function called {count} times".format(count=wrapper.counter))
return f(*args)
wrapper.counter = 0
# change wrapper name, or it will just be 'wrapper' (for inspection)
wrapper.__name__ = f.__name__
return wrapper

@count
def donothing():
pass

donothing()
donothing()
print("Function called a total of {count} times".format(count=donothing.counter))
</pre>

<pre>
## Timing your code
import time

def timer(f):
def wrapper(*arg):
t = time.time()
ret = f(*arg)
print("Took {elapsedtime} seconds".format(elapsedtime=time.time()-t))
return ret
return wrapper

@timer
def fact(n):
prod = 1
while n > 0:
prod *= n
n -= 1
return prod

total = fact(10000)
</pre>

<pre>
## Synchronization
import threading

def synchronized(lock):
def wrap(f):
def wrapper(*args, **kwargs):
lock.acquire()
try:
return f(*args, **kwargs)
finally:
lock.release()
return wrapper
return wrap

mylock = threading.Lock()

@synchronized(mylock)
def myfunc():
pass
</pre>

<pre>
## Type checking
def myMethod(ID, name):
''' Old way '''
if not (isinstance(ID, 'uint') and isinstance(name, 'utf8string')):
raise Exception()

def accepts(*typeargs):
''' Decorator way '''
def wrap(f):
def wrapper(*args):
for a, t in zip(args, typeargs):
if not isinstance(a, t):
raise Exception('{a} not of type {t}'.format(a=a, t=t))
return f(*args)
return wrapper
return wrap

@accepts(int)
def add1(n):
return n + 1
</pre>

<pre>
## Memoize
def memoize(f):
def wrapper(*args):
if not wrapper.ret:
wrapper.ret = f(*args)
return wrapper.ret
wrapper.ret = False
wrapper.__name__ = f.__name__
return wrapper

import time

@memoize
def sumto10():
count = 0
sum = 0
while count < 10:
sum += count
time.sleep(1)
count += 1
return sum

# first time will count slow
print sumto10()
# second time will return cached value instantly
print sumto10()
</pre>

Memoize: "Caches a function's return value each time it is called. If called later with the same arguments, the cached value is returned (not reevaluated)" [http://wiki.python.org/moin/PythonDecoratorLibrary#Memoize]

<pre>
## Nested decorators
import time
@shout
@memoize
def test1():
print "calculating"
time.sleep(3)
return 100
</pre>

Order is important! For tested decorators the execution order is from the top to the bottom. If you reverse @shout and @memoize, the @shout will not run because of the way @memoize was written.

<pre>
## registry
registry = []
# can also decorate decorators:
@shout
def registry(f):
registry.append(f)
return f

@register
def test2():
pass
</pre>

Decorator is just a function that returns a function.

Can use functools to automatically update decorator wrappers with __name__, etc [http://docs.python.org/2/library/functools.html]
<pre>
## Auto syntax sygar the wrapper for you
import functools
def my_decorator(f):
@functools.wraps(f)
def wrapper(*args, **kwds):
print 'Calling decorated function'
return f(*args, **kwds)
return wrapper
</pre>

=== Class Method Decorators ===

Static methods: [http://docs.python.org/library/functions.html#staticmethod]
: Similar to C++ and Java static methods
<pre>
class C:
@staticmethod
def say(msg):
print(msg)

C.say('hi')
</pre>

Class method: [http://docs.python.org/library/functions.html#classmethod]
: A class method receives the class as implicit first argument, just like an instance method receives the instance.
: Class methods are different than C++ or Java static methods. If you want those, see staticmethod() in this section.
<pre>
@classmethod
def setup_parser(cls, parser):
</pre>

See also [[#Class Properties]]

=== Class Properties ===

Properties: (not using decorators)
<pre>
class C(object):
def __init__(self):
self._x = None
def getx(self):
print("get x")
return self._x
def setx(self, value):
print("set x")
self._x = value
def delx(self):
print("del x")
del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")
</pre>

<pre>
#usage
c = C()
c.x = 5
print(c.x)
del(c.x)
</pre>

Getter and Setter Property Decorators: [http://docs.python.org/library/functions.html#property]
<pre>
class C(object): # make sure to inherit from object for these properties to work!
def __init__(self):
self._x = None

@property
def x(self):
"""I'm the 'x' property."""
print("get x")
return self._x

@x.setter # python >= 2.6
def x(self, value):
print("set x")
self._x = value

@x.deleter # python >= 2.6
def x(self):
print("del x")
del self._x
</pre>

<pre>
#usage
c = C()
c.x = 5
print(c.x)
del(c.x)
</pre>

Note: If you don't add the @x.setter, the property will be read only, and will throw a "AttributeError: can't set attribute" if attempted to be set.

Using x.getter example: [http://wiki.python.org/moin/AlternativeDescriptionOfProperty]
<pre>
class Cls(object):
def __init__(self):
self._x = 100

x = property()

@x.getter
def x(self):
"""I'm the 'x' property."""
return self._x

@x.setter
def x(self, value):
self._x = value

@x.deleter
def x(self):
del self._x
</pre>

=== Custom Decorators ===

decorator
@dec1
def func(arg1, arg2, ...): pass

equivalent to:
def func(arg1, arg2, ...): pass
func = dec1(func)

Basic layout (names are arbitrary):
<pre>
def my_decorator(function_name):
# update wrapper.__doc__ and .func_name
# or @functools.wraps(my_wrapper)
def mywrapper(*args, **kw):
# do something before
result = function_name(*args, **kw)
# do something after
return result
return my_wrapper
</pre>

Example:
<pre>
def limit(length):
def decorator(function):
def wrapper(*args, **kw):
result = function(*args, **kw)
result = result[:length]
return result
return wrapper
return decorator
</pre>

<pre>
@limit(3)
def echo(foo): return foo

echo('123456') # '123'
</pre>

=== assert ===

assert True
assert False # exception
assert 1 == 1
assert 1 == 2, "some message" # AssertionError: some message

=== Snake Case ===

Convert Camel Case to Snake Case: [http://stackoverflow.com/questions/1175208/elegant-python-function-to-convert-camelcase-to-snake-case]

<pre>
def snake_case(name):
s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name)
return re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1).lower()
</pre>

<pre>
>>> convert('CamelCase')
'camel_case'
</pre>

=== Camel Case ===

Camel Case: [http://stackoverflow.com/questions/8347048/camelcase-every-string-any-standard-library]

>>> ''.join(x for x in 'make IT camel CaSe'.title() if not x.isspace())
'MakeItCamelCase'

To titleize a sentence:
>>> "this is a test".title()
"This Is A Test"

=== Lambda ===

Simple functions
mul = lambda a, b: a * b

# equivalent to:
def mul(a, b):
return a * b

Sort usage:
data = [dict(number=x) for x in '036149']
data.sort(key=lambda x: float(x['number']))

Compare usage:
data = [dict(number=x) for x in '036149']
data.sort(cmp=lambda x,y: cmp(x['number'], y['number']))

Boolean:
lambda x: x % 3 == 0

Multiple parameter:
lambda x, y: x + y

Call another function:
lambda word: len(word)

Strings:
lambda line: line.split()[2]

Embedded:
def transform(n):
return lambda x: x + n

No parameters:
lambda: 1

Default parameters:
lambda x=1: x+1

References:
* Python: Lambda Functions - http://www.secnetix.de/olli/Python/lambda_functions.hawk

=== Map ===

Apply a function to items of a sequence:
map(str, range(3)) # ['0', '1', '2']

Note: list comprehensions can replace map and filter

=== Reduce ===

Apply a function to pairs of the sequence:
import operator
reduce(operator.mul, [1,2,3,4]) # 24 - ((1*2)*3)*4)

Note: sum or for loop can replace reduce

=== Filter ===

Return a sequence of items for which function(item) is True
filter(lambda x: x >= 0, [0, -1, 3]) # [0, 3]

Note: list comprehensions can replace map and filter

Find matching items in two lists:
a = ['a','b','c']
b = ['c','d']
filter(lambda x: x in a, b) # returns only 'c' as it is in both lists

=== List comprehension ===

Nothing more than Syntactic Sugar

5. Data Structures — Python v2.7.5 documentation - http://docs.python.org/2/tutorial/datastructures.html#list-comprehensions
: "List comprehensions provide a concise way to create lists."
: "A list comprehension consists of '''brackets''' containing an '''expression''' followed by a '''for clause''', then '''zero or more for or if clauses'''. "

Remove newlines from each item in a list:
mylist = [x.strip() for x in mylist]

Simple:
doubled = [x*2 for x in mylist]
squares = [x**2 for x in range(10)]

If:
[ 2*x for x in range(1,10) if x >= 5 ]

Complex:
[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]

Expanding PI:
[str(round(pi, i)) for i in range(1, 6)]

Return list of words
my_list = [current_word.lower() for current_word in words]

Utah Python July 2013 Meeting: (http://basepi.net)
<pre>
# Iterative
squares = []
for x in range(10):
squares.append(x**2)
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

## Functional
squares = map(lambda x: x**2, range(10))

## List comprehension
squares = [x**2 for x in range(10)]
even_squares = [x**2 for x in range(10) if x%2 == 0]

# get hash of strings in list:
import sha
strings = ['Python is cool!', 'I love python']
hashes = [sha.sha(x).digest() for x in strings]

# flatten matrix:
matrix = [[1, 2, 3], [4, 5, 6]]
[x for row in matrix for x in row]
# [1, 2, 3, 4, 5, 6]

# build matrix:
[(x,y) for x in range(10) for y in range(10)]
# matrix of not both odd and not both even
[(x,y) for x in range(10) for y in range(10) if x%2 != y%2]

# suck lines of files into list, ignoring commented lines
[line.strip().split("\t") for line in open("my_file.tab")
if not line.startswith('#')]
</pre>

References:
* 5.1.4. List Comprehensions - Data Structures — Python v2.7.2 documentation - http://docs.python.org/tutorial/datastructures.html#list-comprehensions
* Utah Python July 2013 Meeting - YouTube - https://www.youtube.com/watch?v=uh9uhzitdvw [http://basepi.net]

=== [http://docs.python.org/reference/expressions#calls Mass Parameter Passing] ===

Using: *args and **kw
* *args a tuple of remaining unnamed parameters
* **kw a dictionary of remaining named parameters

def param_func(a, b='b', *args, **kw):
print( [a, b, args, kw] )

WARNING: If not careful, this can result in this error: "TypeError: param_func() got multiple values for keyword argument 'myvar'". Might be better to just use:

def param_func(*args, **kw):
print( [args, kw] )

param_func(1, 2, 3, 4, 5) # [1, 2, (3, 4, 5), {}]
param_func(1, 2, 3, 4, 5, joe='test') # [1, 2, (3, 4, 5), {'joe': 'test'}]
mylist = ("a", "c", "g", "h", 5)
param_func(mylist) # [('a', 'c', 'g', 'h', 5), 'b', (), {}]
param_func(*mylist) # ['a', 'c', ('g', 'h', 5), {}] *tricky*
Note: *mydict will strip out just keys and treat like list
Note: **mydict will treat like named parameters

def mywrapper(*args, **kw):
# do something before
result = function_name(*args, **kw)

Passing arbitrary parameters:
<pre>
def fun2(**kw):
print kw # {'a': '1', 'b': '2', 'c': '3'}

def fun1(c='3', **kw):
print kw # {'a': '1', 'b': '2'}
fun2(c='3', **kw)

fun1(a='1', b='2')
</pre>

=== [http://www.python.org/dev/peps/pep-0234/ Iteration Protocol] ===

[http://www.python.org/dev/peps/pep-0234/ PEP 234 -- Iterators]

Making a class iterable:
<pre>
class mycount(object):
def __init__(self, max):
self.x = 0
self.max = max
def __iter__(self): # return object that implements __next__() or next()
return self
def next(self):
if self.x < self.max:
self.x += 1
return self.x
else:
self.x = 0
raise StopIteration()
</pre>

for i in mycount(10): print i # 1 ... 10
print [x for x in mycount(10) ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10)
for i in c: print i # 1 ... 10
[ x for x in c ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

=== Generators ===

Generators are functions that return to their previous state (using 'yield') when iterated over, which can be used in for loops.

==== Yield ====

Improve on Iterators:
* Infinite data (infinite iterators)
* Memory-intensive iteration
* Cooperative Multi-threading

Generators are functions that return to their previous state when iterated over, which can be used in for loops.

Note: Generators throw StopIteration when completed

Note: the 'yield' keyword will stash away the state of the function and return the value. (Continuations)

<pre>
def gen_range(end):
cur = 0
while cur < end:
yield cur
# return here next time
cur += 1

# for loop iteration
for i in gen_range(10): print i

# manual iteration
my_gen = gen_range(10)
try:
while True:
print my_gen.next()
except StopIteration:
pass

# list comprehension
numbers = [x for x in gen_range(10)]
</pre>

Note: one can place an empty 'return' in a generator if needed. A 'return' with argument is invalid.

Making a class generate:
<pre>
class mycount(object):
def __init__(self, max):
self.max = max
def __iter__(self):
return self.next() # change here
def next(self): # change here
x = 0
while x < self.max:
x += 1
yield x

for i in mycount(10): print i # 1 ... 10
print [x for x in mycount(10) ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10)
for i in c: print i # 1 ... 10
[ x for x in c ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

itr = mycount(10).next()
itr.next() # 1
itr.next() # 2 - eventually throws StopIteration exception
</pre>

Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
<pre>
* Iterators are used to traverse a list
* Generators are more powerful Iterators

# simple 'n' number in functional form
def firstn(n):
num = 0
nums = []
while num < n:
nums.append(num)
num += 1
return nums

# simple 'n' number in generator form
def firstn(n):
num = 0
while num < n:
yield num
num += 1

-

# Infinite list generator
def counter():
num = 0
while True:
yield num
num += 1

# Fibbonacci generator
def fib():
num1 = 0
num2 = 1
yield num1
while True:
num1, num2 = num2, num1 + num2
yield num1

# Fibbonacci generator 2
def fib(n):
num1 = 0
num2 = 1
count = 0
yield num1
while count < n:
count += 1
num1, num2 = num2, num1 + num2
yield num1
</pre>

Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
<pre>
## Cooperative Multi-Threading:
# Cooperative multithreading relies on the threads themselves to relinquish control once they are at a stopping point.

def mythread(name):
print('Starting Thread: {name}...'.format(name=name))
counter = 10
while counter > 0:
print('Thread: {name} doing some work!'.format(name=name))
counter -= 1
yield

thread1 = mythread('Alice')
thread2 = mythread('Bob')
thread3 = mythread('Trudy')

try:
while True:
thread1.next()
thread2.next()
thread3.next()
except StopIteration:
pass
</pre>

References:
* Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
** http://files.basepi.net/talks/pyarmory.zip

=== dunder methods ===

Some dunder (double under) methods determine what will happen when + (__add__) or other similar numerical symbols used.

<pre>
class str_add(object):
def __init__(self, name):
self.name = name
def __add__(self, other):
self.name += other.name
return self
# note: only need to return self if you want to go beyond 'a + b'

a = str_add('a_')
b = str_add('b_')
c = str_add('c_')
d = a + b + c
print(a.name) # a_b_c_
print(b.name) # b_
print(c.name) # c_
print(d.name) # a_b_c_
</pre>

Other common methods to override:
* __add__ for +
* __sub__ for -
* __mul__ for *
* __div__ for /
* __mod__ for %

References:
* Emulating numeric types - http://docs.python.org/2/reference/datamodel.html#emulating-numeric-types

=== [http://docs.python.org/library/doctest.html doctest] ===

test.py:
<pre>
#!/usr/bin/env python
"""
This is the "example" module.

>>> double(5)
10
"""

def double(n):
"""
This is double function

>>> double(3)
6
"""
return n * 2

if __name__ == "__main__":
import doctest
doctest.testmod()
</pre>

=== unittest ===

unittest.TestCase

Here is a short script to test three functions from the random module:
<pre>
import random
import unittest

class TestSequenceFunctions(unittest.TestCase):

def setUp(self):
self.seq = range(10)

def test_shuffle(self):
# make sure the shuffled sequence does not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, range(10))

# should raise an exception for an immutable sequence
self.assertRaises(TypeError, random.shuffle, (1,2,3))

def test_choice(self):
element = random.choice(self.seq)
self.assertTrue(element in self.seq)

def test_sample(self):
with self.assertRaises(ValueError):
random.sample(self.seq, 20)
for element in random.sample(self.seq, 5):
self.assertTrue(element in self.seq)

if __name__ == '__main__':
unittest.main()
</pre>

The unittest module can be used from the command line to run tests from modules, classes or even individual test methods:
python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method
python -m unittest -v test_module # verbose
python -m unittest -h # help

References:
* 25.3. unittest — Unit testing framework — Python v2.7.4 documentation - http://docs.python.org/2/library/unittest.html
* Python Debugging Tips - Google Drive - https://docs.google.com/presentation/d/130Wl--IzbDh9ymF6KHMu0WrsIwz6k32pav1gFbsw7NI/edit?usp=sharing

=== Internet Access ===

<pre>
# Python 3
from urllib.request import urlopen
for line in urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'):
line = line.decode('utf-8')
if 'EST' in line or 'EDT' in line: # look for eastern time
print(line)
</pre>

=== Simple SMTP ===

Requires local STMP server.

<pre>
import smtplib
server = smtplib.SMTP("localhost")
server.sendmail('mrken@demo.oeey.com', 'kenneth@demo.oeey.com',
"""To: kenneth@demo.oeey.com
From: mrken@demo.oeey.com
Subject: This is fun

Beware of the dog.
""")
server.quit()
</pre>

<pre>
s = smtplib.SMTP(smtp_server)
</pre>

with TLS:
<pre>
s = smtplib.SMTP("smtp.gmail.com",587)
s.ehlo()
s.starttls()
s.ehlo()
</pre>

SSL:
<pre>
s = smtplib.SMTP_SSL('smtp.mail.yahoo.com', 465)
</pre>

=== Data Compress ===

zlib, gzip, bz2, zipfile, tarfile

<pre>
import zlib
s = b'hello world' # only in python >3
len(s)
t = zlib.compress(s)
zlib.decompress(t)
zlib.crc32(s)
</pre>

=== gzip ===

<pre>
import gzip
content = "Lots of content here"
f = gzip.open('file.txt.gz', 'wb')
f.write(content)
f.close()
</pre>

<pre>
import gzip
f_in = open('file.txt', 'rb')
f_out = gzip.open('file.txt.gz', 'wb')
f_out.writelines(f_in)
f_out.close()
f_in.close()
</pre>

=== Performance Timer ===

Time a task repeated 10 times:
<pre>
from timeit import Timer
Timer('t=a; a=b; b=t', 'a=1; b=2').timeit()
Timer('print("hi")').timeit(10) # repeat 10 times and time
</pre>

Simple method:
<pre>
start_time = int(time.time())
... task ...
compute_time = int(time.time()) - start_time
</pre>

=== Python Version Check ===

Solution:

__init__.py:
<pre>
# Check Python Version:
import sys
assert sys.version_info >= (2, 6, 0), 'requires Python 2.6 or higher'
del sys
</pre>

resulting error if not > 2.6:
<pre>
>>> import nucleon
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "nucleon/__init__.py", line 9, in ?
assert sys.version_info >= (2, 6, 0), 'nucleon requires Python 2.6 or higher'
AssertionError: nucleon requires Python 2.6 or higher
</pre>

Other solutions:

Note: this will throw syntax error on 2.4 due to ternary
<pre>
import sys
if sys.version_info < (2, 4):
raise "must use python 2.5 or greater"
else:
# syntax error in 2.4, ok in 2.5
x = 1 if True else 2
print x
</pre>

maybe solution:
<pre>
try:
eval("1 if True else 2")
except SyntaxError:
# doesn't have ternary
</pre>

<pre>
>>> try:
... from my_module import twoPointSixCode
... except Exception:
... print "can't import, probably because your python is too old!"
>>>
</pre>

<pre>
#!/usr/bin/env/python
import sys
if sys.version_info<(2,6,0):
sys.stderr.write("You need python 2.6 or later to run this script\n")
exit(1)
</pre>

<pre>
>>> import sys
>>> sys.hexversion # hex printed as integer
33817584
>>> '%x' % sys.hexversion # output as hex, not zero fill 2.4.3
'20403f0'
>>> sys.hexversion < 0x02060000 # test if version less than 2.6
True
>>> print "0x%08x" % sys.hexversion # print as hex and zero fill and prepend '0x'
0x020403f0
</pre>

<pre>
import sys
if sys.hexversion < 0x02060500:
print >> sys.stderr, "This script requires python 2.6.5 or higher."
exit(1)
</pre>

<pre>
>>> import sys
>>> sys.version_info
(2, 6, 4, 'final', 0)
>>> if not sys.version_info[:2] == (2, 6):
... print "Error, I need python 2.6"
... else:
... from my_module import twoPointSixCode
>>>
</pre>

References:
* http://stackoverflow.com/questions/446052/python-best-way-to-check-for-python-version-in-program-that-uses-new-language-fe
* http://stackoverflow.com/questions/1093322/how-do-i-check-what-version-of-python-is-running-my-script

=== [http://docs.python.org/library/struct.html Struct - Binary Data Record Layout] ===

Format Characters:
<pre>
Format C Type Python type Standard size
x pad byte no value
c char str of len 1 1
b signed char integer 1
B unsigned char integer 1
? _Bool bool 1
h short integer 2
H unsigned short integer 2
i int integer 4
I unsigned int integer 4
l long integer 4
L unsigned long integer 4
q long long integer 8
Q unsigned long long integer 8
f float float 4
d double float 8
s char[] string
p char[] string
P void * integer
</pre>

H # two byte unassigned number
I # four byte unassigned number
< # standard size and in little-endian byte order

import struct
data = struct.pack('<IIIHH', 1, 2, 3, 4, 5)
# data = '\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x05\x00'

fields = struct.unpack('<IIIHH', data)
# fields = (1L, 2L, 3L, 4, 5)
# using splice: fields = struct.unpack('<IIIHH', data[start:start+16])

f1, f2, f3, f4, f5 = fields

hex(f1) # hex a field:
# '0x1L'

>>> struct.pack("<5sh", "helloo", 11)
'hello\x0b\x00'
>>> struct.unpack("<5sh", 'hello\x0b\x00')
('hello', 11)

=== pdb interactive debugger ===

# uncomment the following to use the interactive debugger:
# import pdb; pdb.set_trace()

=== [http://docs.python.org/library/telnetlib.html Telnet] ===

create:
class telnetlib.Telnet([host[, port[, timeout]]])

Note: The instance is initially not connected by default; the open() method must be used to establish a connection. Alternatively, the host name and optional port number can be passed to the constructor, to, in which case the connection to the server will be established before the constructor returns.

open if not initially done so:
Telnet.open(host[, port[, timeout]])

Functions:
Telnet.read_until(expected[, timeout]) # Read until a given string, expected, is encountered or until timeout seconds have passed.
# on timeout return whatever is in queue, possibly empty string
Telnet.close() # close connection
Telnet.read_all() # Read all data until EOF; block until connection closed.
Telnet.write(buffer) # Write a string to the socket, doubling any IAC characters. This can block if the connection is blocked.
Telnet.expect(list[, timeout]) # Read until one from a list of a regular expressions matches.
# Return a tuple of three items: the index in the list of matches; the match object; and the text read to match.

Read until end of line:
line = tn.read_until("\n")

<pre>
import getpass
import sys
import telnetlib

HOST = "localhost"
user = raw_input("Enter your remote account: ")
password = getpass.getpass()

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ")
tn.write(user + "\n")
if password:
tn.read_until("Password: ")
tn.write(password + "\n")

tn.write("ls\n")
tn.write("exit\n")

print tn.read_all()
</pre>

<pre>
#!/usr/bin/env python

import sys
import telnetlib

HOST = "216.119.193.252"
USER = "admin"
PASS = "@lt12345"

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ")
tn.write(USER + "\n")
tn.read_until("Password: ")
tn.write(PASS + "\n")
tn.read_until("# ")

tn.write("terminal length 0\n")
tn.read_until("# ")

tn.write("show running-config\n")
configlines = tn.read_until("# ")

tn.write("exit\n")
tn.read_all()

print(configlines)
</pre>

=== SSH ===

Cheap method:
os.system('ssh user@server command')

SecureShell - PythonInfo Wiki - http://wiki.python.org/moin/SecureShell
: There are several ways to use Secure Shell (SSH) in Python
* pyssh - http://pyssh.sourceforge.net/
* paramiko - Paramiko is a native Python implementation of SSH. - http://www.lag.net/paramiko/
* conch - conch is another native implementation of SSH and part of the Twisted Matrix project - http://twistedmatrix.com/projects/conch
* SSH wrapper - Some tools just wrap around existing ssh/sftp implementations
** keyphrene - http://www.keyphrene.com/products/org.keyphrene/
** keyphrene SSH wrapper - http://membres.lycos.fr/fredp/python/pyscp.html
* Fabric - http://docs.fabfile.org/en/1.4.0/index.html
* pexpect - Noah.org - http://www.noah.org/wiki/Pexpect

SFTP:
* SFTP in Python: Really Simple SSH | The best of Zeth - http://zeth.net/post/332/

pexpect:
wget http://pexpect.sourceforge.net/pexpect-2.3.tar.gz
tar xzf pexpect-2.3.tar.gz
cd pexpect-2.3
sudo python ./setup.py install

=== SFTP ===

SFTP in Python? (platform independent) - Stack Overflow - http://stackoverflow.com/questions/432385/sftp-in-python-platform-independent

paramiko: ssh2 protocol for python - http://www.lag.net/paramiko/

pip install paramiko

mkdir -p ~/.src ; cd ~/.src
wget http://www.lag.net/paramiko/download/paramiko-1.7.7.1.zip
unzip paramiko-1.7.7.1.zip
cd paramiko-1.7.7.1
python setup.py install

SFTP Methods:
* chdir
* chmod
* chown
* close
* file
* from_transport
* get
* get_channel
* getcwd
* listdir
* listdir_attr
* logger
* lstat
* mkdir
* normalize
* open
* put
* readlink
* remove
* rename
* request_number
* rmdir
* sock
* stat
* symlink
* truncate
* ultra_debug
* unlink
* utime

<pre>
import paramiko

host = "THEHOST.com" #hard-coded
port = 22
transport = paramiko.Transport((host, port))

password = "THEPASSWORD" #hard-coded
username = "THEUSERNAME" #hard-coded
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

import sys
path = './THETARGETDIRECTORY/' + sys.argv[1] #hard-coded
localpath = sys.argv[1]
sftp.put(localpath, path)

sftp.close()
transport.close()
print 'Upload done.'
</pre>

VMware Example:
<pre>
#!/usr/bin/env python

import paramiko
import sys

host = "sftp2.engx.vmware.com"
port = 443
transport = paramiko.Transport((host, port))

username = "xxxx"
password = "xxx"
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

print dir(sftp)

print sftp.chdir("build-uploader")
print sftp.listdir()

sftp.close()
transport.close()
</pre>

=== paramiko ===

See [[Python/paramiko]]

=== [http://docs.python.org/library/getpass.html Get Password] ===

Prompt for password:
getpass.getpass(prompt='Password: ')

import getpass
user = raw_input("Enter your remote account: ")
password = getpass.getpass()

=== [http://docs.python.org/library/configparser.html ini files] ===

<pre>
ini_filename = 'my.ini'

import ConfigParser
parser = ConfigParser.SafeConfigParser()
parser.read(ini_filename)

sections = parser.sections() # get sections
for section in sections:
print "[" + section + "]"
for name, value in parser.items(section):
print name + " = " + value
</pre>

<pre>
if parser.has_option(section, option):
print( parser.get(section, option) )
</pre>

To have the option names not convert to lower case:
parser.optionxform = str

Ignore section: [http://stackoverflow.com/questions/2885190/using-pythons-configparser-to-read-a-file-without-section-name]
ini_str = '[root]\n' + open(ini_path, 'r').read()
ini_fp = StringIO.StringIO(ini_str)
config = ConfigParser.RawConfigParser()
config.readfp(ini_fp)

Get option:
config.has_option('root', 'some_key')
config.get('root', 'some_key')

config.items('root') # list of Key Value tuples

=== optparse - Option Parser ===

[http://docs.python.org/library/optparse.html - Option Parser

Note: Deprecated since version 2.7: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

<pre>
import optparse
parser = optparse.OptionParser()

parser.add_option('--name', default=None)

parser.add_option(
'-l', '--loglevel',
help='Log Level (default: info)',
default="info", action='store', type='string')

(options, args) = parser.parse_args()

# if no parameters passed show help
if len(sys.argv) == 1:
parser.print_help()
sys.exit(1)

# if a stand alone parameter passed, override option
if args:
options.target = args[0]

# show option:
print(options.loglevel)
</pre>

Note: Order is important when listing add_options!

Pretty Description:
parser = optparse.OptionParser(description="Purpose: to do something")
# or
parser.description = "Purpose: to do something"

Parse arguments:
(options, args) = parser.parse_args()
Note: If you like, you can pass a custom argument list to parse_args(), but that’s rarely necessary: by default it uses sys.argv[1:].

Change Usage Message:
<pre>
usage = "%prog [options]" # default message
# Usage: <prog> [options]" # where <prog> is replaced with the script name

parser = OptionParser(usage=usage)
# or
parser.usage = usage # default message
# or
parser.set_usage(usage)
</pre>

Remove option: (does not remove default value from 'options' variable!!! - stupid!)
parser.remove_option('--loglevel')

Modify an option: (changes are saved to parser automatically, unless in an option group)
option = parser.get_option('--loglevel')
option.help = 'something else'
option.default = 'debug'

# or
parser.remove_option("--loglevel")
parser.add_option("--loglevel", help="something else", default="info")

Modify an option in an option group:
opt_group = parser.get_option_group("--loglevel")
opt_group.remove_option("--loglevel")
opt_group.add_option("--loglevel", help="something else", default="info")

Types: "string", "int", "long", "choice", "float" and "complex" [http://docs.python.org/library/optparse.html#optparse-standard-option-types]
Defaults:
* Default actions is 'store'.
* Default type is 'string'.
* Default dest is the long name of the parameter (or short name if no long name provided)
* Default value is None

<pre>
from optparse import OptionParser
[...]
parser = OptionParser()

# string
parser.add_option("-f", "--file", dest="filename",
help="write report to FILE", metavar="FILE")

# boolean - action is what to do when flag found
parser.add_option("-q", "--quiet",
action="store_true", dest="verbose", default=False,
help="don't print status messages to stdout")

# integer
parser.add_option(
"-m", "--minutes",
help="Minutes (default: 60)",
default=60, type='int')

(options, args) = parser.parse_args()

print "Filename: " + options.filename
</pre>

fake handling of required option:
<pre>
if not option.filename:
parser.print_help()
parser.error("filename is required")
</pre>

change default usage:
parser = OptionParser(usage="Usage: %prog [args] arg1 arg2")

Really ugly way to print help usage :-(
<pre>
def Usage():
pname = sys.argv[0]
cmd='%s -h' % (pname)
os.system(cmd)

if (len(sys.argv) < 2):
Usage()
sys.exit(1)
</pre>

Groups:
<pre>
# OptionGroup(parser, title, description=None)
group = OptionGroup(parser, "Dangerous Options",
"Caution: use these options at your own risk. "
"It is believed that some of them bite.")
group.add_option("-g", action="store_true", help="Group option.")
parser.add_option_group(group)
</pre>

Hide an option:
* Option.help
: "Help text to print for this option when listing all available options after the user supplies a help option (such as --help). If no help text is supplied, the option will be listed without help text. To hide this option, use the special value optparse.SUPPRESS_HELP."
<pre>
parser.add_option("--secret", help=SUPPRESS_HELP)
</pre>

Issues with Option Parser:
* No handling for positional arguments
* No adding help for positional arguments
* No handling for required field

=== argparse ===

argparse - http://docs.python.org/library/argparse.html

argparse — Parser for command-line options, arguments and sub-commands

NOTE: MUCH MORE POWERFUL THAN OPTION PARSER!

Note: New in version 2.7.

Installation on <= 2.6
<pre>
# Install argparse: - http://code.google.com/p/argparse/
# REQUIRES SETUPTOOLS
mkdir -p ~/.src ; cd ~/.src
wget http://argparse.googlecode.com/files/argparse-1.2.1.tar.gz
tar -zvxf argparse-1.2.1.tar.gz
cd argparse-1.2.1
sudo python setup.py install
</pre>

The argparse module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse will figure out how to parse those out of sys.argv. The argparse module also automatically generates help and usage messages and issues errors when users give the program invalid arguments.

<pre>
import argparse

parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
const=sum, default=max,
help='sum the integers (default: find the max)')

args = parser.parse_args()
print args.accumulate(args.integers)
</pre>

<pre>
import argparse

argparser = demoproject.util.argparse.ArgumentParser(
usage='Kick off demoproject benchmarks.')
argparser.add_argument(
'--card',
help='What cards to run on. Depends on Jenkins NODE_LABEL environment'
'variable for correct mappings.')
argparser.add_argument(
'--suite', metavar='SUITE',
help='Specify the appropriate benchmark suite as defined in '
'the benchmark ini file. Defaults to standard_perf',
default='standard_perf')
args = argparser.parse_args()

print args.card
</pre>

Remove an argument: [http://stackoverflow.com/questions/32807319/disable-remove-argument-in-argparse]
<pre>
def remove_options(parser, options):
""" remove_options(parser, ['--some-option', '--some-option2']) """
for option in options:
for action in parser._actions:
if vars(action)['option_strings'][0] == option:
parser._handle_conflict_resolve(None,[(option,action)])
break
</pre>

=== [http://docs.python.org/library/commands.html commands] ===

35.16. commands — Utilities for running commands — Python v2.7.1 documentation - http://docs.python.org/library/commands.html

Note: Deprecated since version 2.6: The commands module has been removed in Python 3.0. Use the subprocess module instead.

The commands module contains wrapper functions for os.popen() which take a system command as a string and return any output generated by the command and, optionally, the exit status.

Get status and output:
status, output = commands.getstatusoutput(cmd)

Get output ignoring status:
output = commands.getoutput(cmd)

NOTE: DO NOT USE getstatus() AS IT DOES NOT DO WHAT YOU THINK IT DOES!
status = commands.getstatus(file)

<pre>
>>> import commands

>>> commands.getstatusoutput('ls /bin/ls')
(0, '/bin/ls')

>>> commands.getstatusoutput('cat /bin/junk')
(256, 'cat: /bin/junk: No such file or directory')

>>> commands.getstatusoutput('/bin/junk')
(256, 'sh: /bin/junk: not found')

>>> commands.getoutput('ls /bin/ls')
'/bin/ls'

>>> commands.getstatus('/bin/ls')
'-rwxr-xr-x 1 root 13352 Oct 14 1994 /bin/ls'
</pre>

=== [http://docs.python.org/library/_winreg.html Windows Registry] ===

Note: The _winreg module has been renamed to winreg in Python 3.0.

* 32.3. winreg – Windows registry access — Python v3.2 documentation - http://docs.python.org/py3k/library/winreg.html
* 34.3. _winreg – Windows registry access — Python v2.7.1 documentation - http://docs.python.org/library/_winreg.html

Read from registry:
<pre>
from _winreg import *
key = OpenKey(HKEY_LOCAL_MACHINE, r'Software\Microsoft\Outlook Express', 0, KEY_ALL_ACCESS)
QueryValueEx(key, "InstallRoot")
</pre>

Write to registry:
<pre>
keyVal = r'Software\Microsoft\Internet Explorer\Main'
try:
key = OpenKey(HKEY_CURRENT_USER, keyVal, 0, KEY_ALL_ACCESS)
except:
key = CreateKey(HKEY_CURRENT_USER, keyVal)
SetValueEx(key, "Start Page", 0, REG_SZ, "http://www.blog.pythonlibrary.org/")
CloseKey(key)
</pre>

Enumerate registry keys:
<pre>
from _winreg import EnumKey, HKEY_USERS

try:
i = 0
while True:
subkey = EnumKey(HKEY_USERS, i)
print subkey
i += 1
except WindowsError:
# WindowsError: [Errno 259] No more data is available
pass
</pre>

Another enumeration example:
<pre>
# File: winreg-example-1.py

import _winreg

explorer = _winreg.OpenKey(
_winreg.HKEY_CURRENT_USER,
"Software\\Microsoft\\Windows\\CurrentVersion\\Explorer"
)

# list values owned by this registry key
try:
i = 0
while 1:
name, value, type = _winreg.EnumValue(explorer, i)
print repr(name),
i += 1
except WindowsError:
print

value, type = _winreg.QueryValueEx(explorer, "Logon User Name")

print
print "user is", repr(value)
</pre>

Enable Remote Desktop Example: (by Kenneth)
<pre>
from _winreg import *
key = OpenKey(HKEY_LOCAL_MACHINE, r'SYSTEM\CurrentControlSet\Control\Terminal Server', 0, KEY_ALL_ACCESS)
SetValueEx(key, "fDenyTSConnections", 0, REG_DWORD, 0)
ValueTypeTuple = QueryValueEx(key, "fDenyTSConnections") # (0, 4)
print("Please reboot the system to complete the changes")
</pre>

References:
* Python’s _winreg: Editing the Windows Registry « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/03/20/pythons-_winreg-editing-the-windows-registry/

=== JSON ===

JSON - http://docs.python.org/library/json.html

ENCODING TO JSON STRING (type str)

import json
json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
# '["foo", {"bar": ["baz", null, 1.0, 2]}]'

compact encoding:
import json
json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',',':'))
# '[1,2,3,{"4":5,"6":7}]'

Pretty printing:
import json
print json.dumps({'4': 5, '6': 7}, sort_keys=True, indent=4)
# {
# "4": 5,
# "6": 7
# }

DECODING JSON STRING TO PYTHON LIST OR DICTIONARY OBJECT

import json
json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]')
# [u'foo', {u'bar': [u'baz', None, 1.0, 2]}]

Load from file:
f = open(filename)
j = json.load(f)
print j["kind"] # "dual-controller-adapter"
#{
# "kind": "dual-controller-adapter",
#}

From Command Line:
echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool

# get value
echo '{"test":1,"test2":2}' | python -c 'import sys,json;data=json.loads(sys.stdin.read()); print data["test"]'

# read large file
python -c "import json;f=open('file.json');data=json.load(f);print json.dumps(data,indent=2)"
python -c "import json;print json.dumps(json.load(open('file.json')),indent=2)"

Load from web:
<pre>
import httplib
import json

def get_http_json(server, path):
""" This performs an HTTP GET against a server and path and returns the
resulting page data.

server The server we're hitting, e.g. demo.com
path The path we're hitting, e.g. api/part/MA12345/
"""
connection = httplib.HTTPConnection(server)
try:
url = 'http://%s/%s' % (server, path)
print 'Querying url %s' % url
connection.request('GET', url)
response = connection.getresponse()
if response.status == httplib.OK:
return json.loads(response.read())
else:
print 'Unable to get %s. %s %s' % (url, response.reason,
response.status)
return None
finally:
connection.close()

j = get_http_json(server, path)
</pre>

Pretty printing from the command line:
echo '{"json":"obj"}' | python -mjson.tool
#{
# "json": "obj"
#}

References:
* 18.2. json — JSON encoder and decoder — Python v2.7.1 documentation - http://docs.python.org/library/json.html

=== Sockets ===

See [[Python/Sockets]]

=== [http://docs.python.org/library/base64.html base64] ===

import base64

Encode:
encoded = base64.b64encode('data to be encoded')

Decode:
base64.b64decode(encoded)

=== daemonize ===

Creating a daemon the Python way « Python recipes « ActiveState Code - http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/

<pre>
import os
import sys

try:
pid = os.fork()
except OSError, e:
raise Exception, "%s [%d]" % (e.strerror, e.errno)

if (pid != 0): # parent
os._exit(0)

# create new process session
os.setsid()

# redirect stdin, stdout, stderr
os.open('/dev/null', os.O_RDWR)
os.dup2(0, 1)
os.dup2(0, 2)

#... tasks ...
</pre>

My Solution:
<pre>
import os
import sys
import signal
import syslog

HUP = False

def log(msg):
syslog.syslog(str(msg))
syslog.openlog('[WATER]')

def kill_signal_handler(signal, frame):
log("Being killed off: {0} {1}".format(signal, frame))
log("Shutting Down!")
sys.exit(0)
signal.signal(signal.SIGTERM, kill_signal_handler)
signal.signal(signal.SIGALRM, kill_signal_handler)

def hup_signal_handler(signal, frame):
global HUP
log("Signal reload received: {0} {1}".format(signal, frame))
HUP=True
signal.signal(signal.SIGHUP, hup_signal_handler)

def main_loop():
global HUP

while True:
# ... handle HUP=True ...

# ... main code ...

# wait 60 seconds before we start the next loop
time.sleep(60) # signals will interrupt this

if __name__ == "__main__":
log("Starting Daemon...")
child_id = os.fork()
if child_id:
log("Daemonized!")
sys.exit(0)
os.setsid()
# redirect stdin, stdout, stderr
os.open('/dev/null', os.O_RDWR)
os.dup2(0, 1)
os.dup2(0, 2)

# write pid
with open("water.pid", "w") as f:
f.write(str(os.getpid()))

while True:
try:
main_loop()
except Exception as e:
log("Critical Failure Caught: %s" % str(e))
log("Critical Failure - Restarting!")
time.sleep(30)
#raise e
</pre>

ln -s /opt/pyton27/bin/python pyton-water
./python-water water.py
killall python-water

==== fork ====

<pre>
import os
import sys
import time

child_id = os.fork()
if child_id:
print "parent exiting"
sys.exit(0)

while True:
time.sleep(3)
print "child"
</pre>

For windows, to hide the console, see [[#Hide Console]]

=== csv ===

13.1. csv — CSV File Reading and Writing — Python v2.7.2 documentation - http://docs.python.org/library/csv.html

Read:
<pre>
import csv
spamReader = csv.reader(open('eggs.csv', 'rb'), delimiter=' ', quotechar='|')
for row in spamReader:
print ', '.join(row)
</pre>

Write:
<pre>
import csv
spamWriter = csv.writer(open('eggs.csv', 'wb'), delimiter=' ',
quotechar='|', quoting=csv.QUOTE_MINIMAL)
spamWriter.writerow(['Spam'] * 5 + ['Baked Beans'])
spamWriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam'])
</pre>

=== SQLite ===

11.13. sqlite3 — DB-API 2.0 interface for SQLite databases — Python v2.7.2 documentation - http://docs.python.org/library/sqlite3.html

sqlite3 - New in version 2.5.

<pre>
import sqlite3

conn = sqlite3.connect('/tmp/example')

c = conn.cursor()

# Create table
c.execute('''create table stocks
(date text, trans text, symbol text,
qty real, price real)''')

# Insert a row of data
c.execute("""insert into stocks
values ('2006-01-05','BUY','RHAT',100,35.14)""")

# Show affected row count
print c.rowcount

# Save (commit) the changes
conn.commit()

# We can also close the cursor if we are done with it
c.close()
</pre>

parameter substitution:
<pre>
# Never do this -- insecure!
symbol = 'IBM'
c.execute("... where symbol = '%s'" % symbol)

# Do this instead
t = (symbol,)
c.execute('select * from stocks where symbol=?', t)
c.execute('select * from stocks where symbol=?', (symbol,))

# Larger example
for t in [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00),
('2006-04-06', 'SELL', 'IBM', 500, 53.00),
]:
c.execute('insert into stocks values (?,?,?,?,?)', t)
</pre>

select results:
<pre>
c = conn.cursor()
c.execute('select * from stocks order by price')

for row in c:
print row # row is a tuple

# OR (but not both)

rows = c.fetchall() # rows is list of tuples
print len(rows)
for row in rows:
print row
</pre>

Show which version of sqlite3: [http://zetcode.com/db/sqlitepythontutorial/]
<pre>
#!/usr/bin/python
# -*- coding: utf-8 -*-

import sqlite3 as lite
import sys

con = None

try:
con = lite.connect('test.db')

cur = con.cursor()
cur.execute('SELECT SQLITE_VERSION()')

data = cur.fetchone()

print "SQLite version: %s" % data

except lite.Error, e:

print "Error %s:" % e.args[0]
sys.exit(1)

finally:

if con:
con.close()
</pre>

Or simply: [http://stackoverflow.com/questions/1545479/force-python-to-forego-native-sqlite3-and-use-the-installed-latest-sqlite3-ver]
# echo -e "import sqlite3\nprint sqlite3.sqlite_version_info" | python
(3, 3, 6)

=== MySQL Database ===

Ubuntu packages:
* python-mysqldb - Python interface to MySQL
* python-mysql.connector - pure Python implementation of MySQL Client/Server protocol

apt-get install python-mysqldb
>>> import _mysql

apt-get install python-mysql.connector

PIP:
* https://pypi.python.org/pypi/MySQL-python
* https://pypi.python.org/pypi/mysqlclient

pip install MySQL-python

pip install mysqlclient

---

_mysql - http://mysql-python.sourceforge.net/MySQLdb.html

apt-get install python-mysqldb

<pre>
import _mysql

db=_mysql.connect(host="localhost",user="user",
passwd="pass",db="db")

db.query("""SELECT spam, eggs, sausage FROM breakfast
WHERE price < 5""")

r=db.store_result()
# ...or...
r=db.use_result()

row = r.fetch_row()
</pre>

=== couchdb ===

Getting started with couchdb-python — couchdb-python v0.8 documentation - http://packages.python.org/CouchDB/getting-started.html

<pre>
couch = couchdb.Server('http://127.0.0.1:5984/')

# database
db = couch.create('test') # create new
db = couch['mydb'] # or get existing

# print databases:
for db in couch:
print db

doc = {'foo': 'bar'}
doc_id, doc_rev = db.save(doc)
# (u'9278f7691c3421227a9f8436540008f9', u'1-4c6114c65e295552ab1019e2b046b10e')
# note: this also updates doc with _rev and _id
doc = db['9278f7691c3421227a9f8436540008f9']

# better to create a document using an id you want, instead of the randomly generated one
db[doc.id] = doc # this will write immediately to the database

# print all docs:
for id in db:
print id

# delete doc
db.delete(doc)

# delete database
couch.delete('test')
</pre>

Search database:
<pre>
# show all records
map_fun = '''function(doc) { emit(doc.name, null); }'''

# row = <Row id=u'12341234123412341234', key=None, value=None>
# emit(key, value) - returns the values for key and value

map_fun = '''
function(doc) {
if (doc.username == 'virt-01')
emit(doc.username, null);
}
'''

# row = <Row id=u'12341234123412341234', key=None, value=None>
for row in db.query(map_fun):
print row.id

for row in db.query(map_fun, descending=True):
print row.id
</pre>

couchdb-python - CouchDB Python Library - Google Project Hosting - http://code.google.com/p/couchdb-python/

Installation:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/C/CouchDB/CouchDB-0.8.tar.gz#md5=e32b39e459e1fa2586ea252712a11a59
tar -zvxf CouchDB-0.8.tar.gz
cd CouchDB-0.8
python setup.py install
</pre>

Test:
echo "import couchdb" | python

=== nose ===

nose: nose extends unittest to make testing easier

nose : Python Package Index - http://pypi.python.org/pypi/nose

Install:
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/n/nose/nose-1.1.2.tar.gz#md5=144f237b615e23f21f6a50b2183aa817
tar -zvxf nose-1.1.2.tar.gz
cd nose-1.1.2
sudo python setup.py install

skeleton:
mkdir skeleton
cd skeleton
mkdir bin NAME tests docs
touch NAME/__init__.py
touch tests/__init__.py
touch tests/NAME_tests.py

nose test skeleton:
<pre>
# tests/NAME_tests.py
from nose.tools import *
import klib

def setup():
print "SETUP!"

def teardown():
print "TEAR DOWN!"

def test_basic():
print "I RAN!"

# def test_something_else():
</pre>

Usage:
nosetests

See Also:
* [[#doctest]]
* [[#unittest]]

=== pydoc ===

25.1. pydoc — Documentation generator and online help system — Python v2.7.2 documentation - http://docs.python.org/library/pydoc.html

pydoc - The pydoc module automatically generates documentation from Python modules.

pydoc - same as 'import [library], help([library])'
$ pydoc [library]

Note: lists some special 'module metadata variables' separately (e.g. version, author).

special pydoc recognized metadata sections:
__version__
__author__
__credits__
__date__

==== pydoc function module ====

<pre>
Help on package klib:

NAME
klib - This is the name

FILE
/root/klib/klib/__init__.py

DESCRIPTION
These are module doc strings
that give helpful details

PACKAGE CONTENTS
markone

FUNCTIONS
func1()
This is a doc string
for func1

DATA
PI = 3.141
__all__ = ['func1', 'PI']
__author__ = 'Kenneth <kenneth@oeey.com>'
__copyright__ = 'Copyright 2007, The Project'
__credits__ = 'Credits go to me'
__date__ = '26 February 2001'
__email__ = 'ken@oeey.com'
__license__ = 'GPL'
__maintainer__ = 'Ken'
__status__ = 'Production'
__version__ = '$Revision: 83492 $'

VERSION
83492

DATE
26 February 2001

AUTHOR
Kenneth <kenneth@oeey.com>

CREDITS
Credits go to me
</pre>

Source:
<pre>
"""
This is the name

These are module doc strings
that give helpful details
"""

#__version__ = "1.0"
__version__ = "$Revision: 83492 $"
__author__ = "Kenneth <kenneth@oeey.com>"

__date__ = "26 February 2001"
__credits__ = "Credits go to me"

__all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license'
__license__ = "GPL"
__copyright__ = "Copyright 2007, The Project"
__maintainer__ = "Ken"
__email__ = "ken@oeey.com"
__status__ = "Production"

def func1():
"""
This is a doc string
for func1
"""
</pre>

==== pydoc class module ====

Sample:
<pre>
Help on module class_pydoc:

NAME
class_pydoc - This is the name

FILE
/root/class_pydoc.py

DESCRIPTION
These are module doc strings
that give helpful details

CLASSES
Toad

class Toad
| doc string for object
|
| Methods defined here:
|
| __init___(self, name)
| Constructor
|
| instance_function(self)
| Instance Function
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| class_function(cls) from __builtin__.classobj
| Class Function
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| my_class_attribute = 'sample class attribute'

DATA
PI = 3.141
__author__ = 'Kenneth <kenneth@oeey.com>'
__copyright__ = 'Copyright 2007, The Project'
__credits__ = 'Credits go to me'
__date__ = '26 February 2001'
__email__ = 'ken@oeey.com'
__license__ = 'GPL'
__maintainer__ = 'Ken'
__status__ = 'Production'
__version__ = '1.0'

VERSION
1.0

DATE
26 February 2001

AUTHOR
Kenneth <kenneth@oeey.com>

CREDITS
Credits go to me
</pre>

Code:
<pre>
"""
This is the name

These are module doc strings
that give helpful details
"""

__version__ = "1.0"
#__version__ = "$Revision: 83492 $"
__author__ = "Kenneth <kenneth@oeey.com>"

__date__ = "26 February 2001"
__credits__ = "Credits go to me"

#__all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license'
__license__ = "GPL"
__copyright__ = "Copyright 2007, The Project"
__maintainer__ = "Ken"
__email__ = "ken@oeey.com"
__status__ = "Production"

class Toad:
""" doc string for object """

my_class_attribute = 'sample class attribute'

def __init___(self, name):
""" Constructor """

def instance_function(self):
""" Instance Function """

@classmethod
def class_function(cls):
""" Class Function """
</pre>

=== username ===

User ID:
os.getuid()

User Name:
# fails it stdin piped to program
os.getlogin()

User Name:
import sys
import pwd
def get_username():
return pwd.getpwuid( os.getuid() )[ 0 ]

Environment:
os.environ['USER']

=== hostname ===

import socket
print(socket.gethostname())

os.uname()[1]

hostname = os.system('hostname')

=== Networking ===

==== Get Hostname ====

import socket
print(socket.gethostname())

==== Convert hostname to ip address ====

>>> socket.gethostbyaddr('demo')
('demo.oeey.com', [], ['10.10.10.10'])

/usr/local/bin/whatip: (or nametoip)
<pre>
#!/usr/bin/env python

#
# whatip - convert hostname to ip address
# author: Kenneth Burgener <kenneth@oeey.com> (FEB 2012)
#

import socket
import sys

if len(sys.argv) != 2:
print "Convert Hostname to IP Address"
print "Usage: %s HOSTNAME" % sys.argv[0].split('/')[-1]
sys.exit(1)

hostname = sys.argv[1]
try:
result = socket.gethostbyaddr(hostname)
print result[2][0]
except (socket.gaierror, socket.herror) as e:
print "Error: Unable to resolve '%s'" % hostname
sys.exit(1)
</pre>

==== Get Public IP Address ====

<pre>
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("gmail.com",80))
print(s.getsockname()[0])
s.close()
</pre>

networking - Finding local IP addresses using Python's stdlib - Stack Overflow - http://stackoverflow.com/questions/166506/finding-local-ip-addresses-using-pythons-stdlib

==== Python Network Server ====

<pre>
#!/usr/bin/python # This is server.py file

import socket # Import socket module

s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.
s.bind((host, port)) # Bind to the port

s.listen(5) # Now wait for client connection.
while True:
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
c.send('Thank you for connecting')
c.close() # Close the connection
</pre>

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
* 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

==== Python Network Client ====

<pre>
#!/usr/bin/python # This is client.py file

import socket # Import socket module

s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.

s.connect((host, port))
print s.recv(1024)
s.close # Close the socket when done
</pre>

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
* 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

=== Run Library Module as Script ===

Run python with -m option:
-m mod : run library module as a script (terminates option list)

Example:
cat [json_file] | python -m json.tool
# same as:
cat [json_file] | python /opt/python26/lib/python2.6/json/tool.py

Good for when library is buried deep in the library directories

=== Hex ===

<pre>
>>> hex(10) # print integer as hex string
'0xa'
>>> sys.hexversion # python version as integer
33817584
>>> hex(sys.hexversion) # python version as hex
'0x20403f0'
>>> print "0x%08x" % sys.hexversion # python version as hex
'0x20403f0'
>>> int('0xa', 16) # hex string to integer
10
>>> int('0a', 16) # hex string to integer
10
>>> int('0xa', 0) # hex string to integer auto detected du to '0x'
10
</pre>

=== Binary ===

Integer to Binary String with Negative Number Support: (2's complement)
<pre>
# source: http://www.gossamer-threads.com/lists/python/python/645216
def bin(x):
if x < -32768 or x > 65535:
raise Exception("Out of Range Error: %d" % x)
if x < 0:
x = 65535 + x + 1
bin_str = ''
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(15, 11, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(11, 7, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(7, 3, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(3, -1, -1))
return bin_str
</pre>

Integer to Binary String:
<pre>
# source: http://www.gossamer-threads.com/lists/python/python/645216
def bin(x):
if x < 0 or x > 65535:
raise Exception("Out of Range Error: %d" % x)
bin_str = ''
bin_str = ''.join(x & (1 << i) and '1' or '0' for i in range(15, 7, -1))
bin_str = bin_str + ' '
bin_str = bin_str + ''.join(x & (1 << i) and '1' or '0' for i in range(7, -1, -1))
return bin_str

print bin(0) # '00000000 00000000'
print bin(1) # '00000000 00000001'
print bin(255) # '00000000 11111111'
print bin(256) # '00000001 00000000'
print bin(43690) # '10101010 10101010'
print bin(65535) # '11111111 11111111'
print bin(65536) # '00000000 00000000' - rolled over to next set!
</pre>

Integer to Binary:
bin(170) # '0b10101010'

Binary to Integer:
int('10101010', 2) # 170

=== bitwise ===

See [[#Binary]]

BitwiseOperators - Python Wiki - http://wiki.python.org/moin/BitwiseOperators

<pre>
The Operators:

x << y
Returns x with the bits shifted to the left by y places (and new bits on the right-hand-side are zeros). This is the same as multiplying x by 2**y.
x >> y
Returns x with the bits shifted to the right by y places. This is the same as //'ing x by 2**y.
x & y
AND - Does a "bitwise and". Each bit of the output is 1 if the corresponding bit of x AND of y is 1, otherwise it's 0.
x | y
OR - Does a "bitwise or". Each bit of the output is 0 if the corresponding bit of x AND of y is 0, otherwise it's 1.
~ x
Returns the complement of x - the number you get by switching each 1 for a 0 and each 0 for a 1. This is the same as -x - 1.
x ^ y
XOR - Does a "bitwise exclusive or". Each bit of the output is the same as the corresponding bit in x if that bit in y is 0, and it's the complement of the bit in x if that bit in y is 1.
</pre>

=== Script Path ===

<pre>
if __name__ == "__main__":
# Change directory to where this script lives so we can find relative
# paths from here regardless of where the working directory was when we
# ran the script.
# APP_DIR = os.path.dirname(sys.argv[0])
APP_DIR = os.path.dirname(os.path.abspath(sys.argv[0))
if APP_DIR:
os.chdir(APP_DIR)
</pre>

normpath(path) # Normalize path, eliminating double slashes, etc.
abspath(path) # Return an absolute path

Path to a module: [http://stackoverflow.com/questions/247770/retrieving-python-module-path] (couldn't fine a_module)
import a_module
import os
path = os.path.dirname(a_module.__file__)

Path to module: [http://stackoverflow.com/questions/50499/in-python-how-do-i-get-the-path-and-name-of-the-file-that-is-currently-executin]
import inspect, os
print inspect.getfile(inspect.currentframe()) # script filename (usually with path)
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))) # script directory

Even better:
print os.path.realpath(__file__)
print os.path.dirname(__file__)
print os.path.basename(__file__)

references:
* http://stackoverflow.com/questions/595305/python-path-of-script

keywords: scriptpath script path program path pathname

=== Singleton ===

GLOBAL OBJECT!

None is a singleton (Python only has one copy of None in the interpreter).

Make your own singleton:

override the new method like this:
<pre>
class Singleton(object):
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Singleton, cls).__new__(
cls, *args, **kwargs)
return cls._instance

if __name__ == '__main__':
s1=Singleton()
s2=Singleton()
if(id(s1)==id(s2)):
print "Same"
else:
print "Different"
</pre>

Using decorator -implementation from PEP318? Implementing the singleton pattern with a decorator:
<pre>
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance

@singleton
class MyClass:
...
</pre>

* Python and the Singleton Pattern - Stack Overflow - http://stackoverflow.com/questions/42558/python-and-the-singleton-pattern

=== XML ===

<pre>
from xml.dom.minidom import parse, parseString

dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name

datasource = open('c:\\temp\\mydata.xml')
dom2 = parse(datasource) # parse an open file

dom3 = parseString('<myxml>Some data<empty/> some more data</myxml>')
</pre>

Pretty Print: [http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python]
<pre>
import xml.dom.minidom

xml = xml.dom.minidom.parse(xml_fname) # or xml.dom.minidom.parseString(xml_string)
pretty_xml_as_string = xml.toprettyxml()
</pre>

Pretty Print with etree [http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python]
<pre>
import lxml.etree as etree

x = etree.parse("filename")
print etree.tostring(x, pretty_print = True)
</pre>

References:
* 19.7. xml.dom.minidom — Lightweight DOM implementation — Python v2.7.3 documentation - http://docs.python.org/library/xml.dom.minidom.html
* Pretty printing XML in python - Stack Overflow - http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python

---

Simply don't include self.output.startElement and self.output.endElement for elements you don't want:

simple_filter_parser.py: (my version of [http://freegeographytools.com/2010/removin-time-data-out-of-a-gpx-or-kml-file-to-make-it-work-right-in-google-earth])
<pre>
# this parses out the time element of a gpx for Google Earth
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class my_filter_xml(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self.indent = 0
self.ignore = False
self.output.startDocument()

def startElement(self, name, attrs):
print " " * self.indent + "start: " + name
if attrs.keys():
keys = " " * (self.indent + 2) + "attrs:"
for key in attrs.keys():
keys = keys + " " + key + "=" + attrs[key]
print keys
self.indent = self.indent + 2
#self.output.startElement(name, {})
if name == "time":
self.ignore = True
#if not self.ignore:
self.output.startElement(name, attrs)

def characters(self, ch):
if ch.strip():
print " " * self.indent + "ch: " + ch.strip()
if not self.ignore:
self.output.characters(ch)
else:
if not self.ignore:
self.output.characters("\n")

def endElement(self, name):
self.indent = self.indent - 2
print " " * self.indent + "/end: " + name
#if not self.ignore:
self.output.endElement(name)
if name == "time":
self.ignore = False

if __name__ == '__main__':
source = open("tracks.gpx")
dest = open("tracks_out.gpx", 'w+')
xml.sax.parse(source, my_filter_xml(dest, True))
</pre>

---

fixdescriptor.py:
<pre>
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self._outfile = outfile
self._verbose = verbose
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self._filter = [ 'payload',
'payloads',
'file-list',
'file',
'checksum' ]
self._keep = [ 'vib',
'maintenance-mode' ]
self._inside = None

def startElement(self, name, attrs):
# Certain elements are kept unchanged
if name in self._keep:
if self._verbose:
print '%s element start found' % (name)
self.output.startElement(name, {})
if name == 'maintenance-mode':
if self._verbose:
print 'inside maintenance-mode'
self._inside = name
return
# Other elements are filtered (removed)
elif name in self._filter:
if self._verbose:
print '%s element start found' % (name)
self._inside = name
return
self.output.startElement(name, attrs)

def characters(self, ch):
if self._inside == 'maintenance-mode':
if self._verbose:
print 'new content for maintenance-mode'
self.output.characters('false')
elif self._inside not in self._filter:
self.output.characters(ch)

def endElement(self, name):
# If it's not a checksum element, only print it
if name in self._filter + self._keep:
self._inside = None
if self._verbose:
print '%s element end found' % (name)
if name in self._filter:
return
self.output.endElement(name)

if __name__ == '__main__':
#
parser = optparse.OptionParser(usage='%prog ...')
parser.add_option('-v', '--verbose',
action='store_true', default=False,
help='Print more information.')
parser.add_option('--input-xml',
type='str', default='descriptor.xml',
help='Name of input XML [default: %default]')
parser.add_option('--output-xml',
type='str', default='fixed_descriptor.xml',
help='Name of output XML [default: %default]')
opts, args = parser.parse_args(argv[1:])
source = open(opts.input_xml)
dest = open(opts.output_xml, 'w+')
xml.sax.parse(source, fix_descriptor(dest, opts.verbose))
</pre>

--

Simple read and output parsing:
<pre>
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self._outfile = outfile
self._verbose = verbose
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self._inside = None

def startElement(self, name, attrs):
# attrs.items() is a list of tuples (name, value)
print "start:", name, "/", attrs.items()
self.output.startElement(name, attrs)

def characters(self, ch):
print "char:", self._inside, "/", ch
self.output.characters(ch)

def endElement(self, name):
print "end:", name
self.output.endElement(name)

if __name__ == '__main__':
parser = optparse.OptionParser(usage='%prog ...')
parser.add_option('-v', '--verbose',
action='store_true', default=False,
help='Print more information.')
parser.add_option('--input-xml',
type='str', default='descriptor.xml',
help='Name of input XML [default: %default]')
parser.add_option('--output-xml',
type='str', default='fixed_descriptor.xml',
help='Name of output XML [default: %default]')
opts, args = parser.parse_args(argv[1:])
source = open(opts.input_xml)
dest = open(opts.output_xml, 'w+')
xml.sax.parse(source, fix_descriptor(dest, opts.verbose))
</pre>

=== Unicode ===

Unicode string:
u'Hello World !'

Raw Unicode string:
>>> ur'Hello\u0020World !'
u'Hello World !'

Convert:
u = u'hello\xe4'
str(u) # UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 5: ordinal not in range(128)
s = u.encode('utf8') # 's' is now type str

References:
* Introduction to Unicode Strings - http://docs.python.org/tutorial/introduction.html#unicode-strings

=== Zip ===

See [[Python/Zip]]

=== signals ===

signal.SIGTERM : kill [pid]
signal.SIGALRM : kill -ALRM [pid]
signal.SIGHUP : kill -HUP [pid]

<pre>
import signal, os

def handler(signum, frame):
print 'Signal handler called with signal', signum
raise IOError("Couldn't open device!")

# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)

# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)

signal.alarm(0) # Disable the alarm
</pre>

Note: use signal.SIGTERM to handle "kill" command

References:
* 17.4. signal — Set handlers for asynchronous events — Python v2.7.5 documentation - http://docs.python.org/2/library/signal.html

==== Ctrl+C ====

<pre>
#!/usr/bin/env python
import signal
import sys
def signal_handler(signal, frame):
print 'You pressed Ctrl+C!'
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
print 'Press Ctrl+C'

# linux only:
signal.pause()

# windows alternative:
#import time
#time.sleep(60)
</pre>

References:
* control - How do I capture SIGINT in Python? - Stack Overflow - http://stackoverflow.com/questions/1112343/how-do-i-capture-sigint-in-python

==== Non-blocking raw_input for Python ====

Non-blocking raw_input for Python:
<pre>
# source: http://www.garyrobinson.net/2009/10/non-blocking-raw_input-for-python.html

import signal

class AlarmException(Exception):
pass

def alarmHandler(signum, frame):
raise AlarmException

def nonBlockingRawInput(prompt='', timeout=20):
signal.signal(signal.SIGALRM, alarmHandler)
signal.alarm(timeout)
try:
text = raw_input(prompt)
signal.alarm(0)
return text
except AlarmException:
print '\nPrompt timeout. Continuing...'
signal.signal(signal.SIGALRM, signal.SIG_IGN)
return ''
</pre>

=== email ===

<pre>
import smtplib
import email.mime.text

msg_text = "Hello World"
msg = email.mime.text.MIMEText(msg_text)

# me == the sender's email address
# you == the recipient's email address
msg['Subject'] = 'The contents of %s' % textfile
msg['From'] = me
msg['To'] = you

# Send the message via our own SMTP server, but don't include the
# envelope header.
s = smtplib.SMTP('localhost')
s.sendmail(me, [you], msg.as_string())
s.quit()
</pre>

my nagios fake /bin/mail:
<pre>
#!/usr/bin/env python

import smtplib
import email.mime.text
import optparse
import os
import sys
import pwd

os.system('logger mail called')

parser = optparse.OptionParser()
parser.add_option(
'-s', '--subject',
help='Subject',
default="")
parser.add_option(
'-t', '--to',
help='To',
dest="send_to",
default=None)
parser.add_option(
'-f', '--from',
help='From',
dest="send_from",
default=None)
parser.add_option(
'-S', '--server',
help='server',
default="prime")
(options, args) = parser.parse_args()

def get_username():
return pwd.getpwuid( os.getuid() )[ 0 ]

if options.send_to:
send_to = options.send_to
else:
if len(args) == 1:
send_to = args[0]
else:
parser.print_help()
sys.exit(1)

if options.send_from:
send_from = options.send_from
else:
username = get_username()
hostname = os.uname()[1]
send_from = username + "@" + hostname

msg_text = sys.stdin.read()
msg = email.mime.text.MIMEText(msg_text)

msg['Subject'] = options.subject
msg['From'] = send_from
msg['To'] = send_to

s = smtplib.SMTP(options.server)
s.sendmail(send_from, [send_to], msg.as_string())
s.quit()
</pre>

Authenticated (with gmail):
<pre>
import smtplib
import email.mime.text

# from_email = "kenneth@demo.oeey.com"
# to_email = "kenneth@demo.oeey.com"
from_email = "Kenneth Burgener <kenneth@demo.oeey.com>"
to_emails = ["Kenneth Burgener <kenneth@demo.oeey.com>", "Kenneth Cell <xxx@xxx.com>"]

msg_text = "hello world"
msg = email.mime.text.MIMEText(msg_text)
msg['Subject'] = 'Test text'
msg['From'] = from_email
# msg['To'] = to_email
msg['To'] = ", ".join(to_emails)
# s = smtplib.SMTP('localhost')
print "sending..."
s = smtplib.SMTP_SSL('smtp.gmail.com', '465')
s.login('xxx', 'xxx')
# s.sendmail(from_email, [to_email], msg.as_string())
s.sendmail(from_email, to_emails, msg.as_string())
s.quit()
print "sent!"
</pre>

Source:
* 20.12. smtplib — SMTP protocol client — Python v2.7.5 documentation - http://docs.python.org/2/library/smtplib.html
* 18.1.11. email: Examples — Python v2.7.5 documentation - http://docs.python.org/2/library/email-examples.html

=== pprint ===

8.18. pprint — Data pretty printer — Python v2.7.5 documentation - http://docs.python.org/2/library/pprint.html

import pprint
pprint.pprint(myVar)

import pprint
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(myVar)

Note: Useless on objects!

Dump object:
<pre>
def dumpobject(obj):
print "<<<---"
print type(obj), obj.__class__.__name__
for attr in dir(obj):
if attr[0] == "_": continue # ignore under/dunder properties
if str(type(getattr(obj, attr))) == "<type 'instancemethod'>":
print attr, ": METHOD"
else:
print attr, ":", type(getattr(obj, attr)), ":", getattr(obj, attr)
print "END --->>>"
</pre>

=== Curses ===

Curses Programming with Python — Python v3.3.2 documentation - http://docs.python.org/3/howto/curses.html

"The curses library supplies a terminal-independent screen-painting and keyboard-handling facility for text-based terminals; such terminals include VT100s, the Linux console, and the simulated terminal provided by various programs. Display terminals support various control codes to perform common operations such as moving the cursor, scrolling the screen, and erasing areas. Different terminals use widely differing codes, and often have their own minor quirks.

In a world of graphical displays, one might ask “why bother”? It’s true that character-cell display terminals are an obsolete technology, but there are niches in which being able to do fancy things with them are still valuable. One niche is on small-footprint or embedded Unixes that don’t run an X server. Another is tools such as OS installers and kernel configurators that may have to run before any graphical support is available."

---

Thread from Utah Python User's Group mailing list:

-

Hello, I am trying to learn more about curses in python for a desktop application. Does anybody have any recommendations on tutorials or documentation?

-Corban

-

We've use urwid at work. Seems to be a good approach if you aren't already tied to something. Learning curve can be a bit steep (well, compared to what you'd think it should be). Good luck!

http://excess.org/urwid/

Gabriel Gunderson

-

This video series looks pretty good:

http://www.youtube.com/user/pythoncursestutorial?feature=watch

Official HOWTOs from the python.org site:

http://docs.python.org/2/howto/curses.html
http://docs.python.org/3/howto/curses.html

As Gabe mentioned, there are other libraries that provide similar or
greater functionality, so if you aren't doing this specifically to
learn curses, it's worth at least looking at a few of them to compare.

There's a brief comparison of three possibilities here:

http://stackoverflow.com/questions/8349085/python-ncurses-cdk-urwid-difference

Daniel Fackrell

-

Here's another option that looks promising:

https://pypi.python.org/pypi/blessings/

Daniel Fackrell

=== dialog ===

pythondialog - http://pythondialog.sourceforge.net/
: "pythondialog is a Python wrapper for the dialog utility originally written by Savio Lam, and later rewritten by Thomas E. Dickey. Its purpose is to provide an easy to use, pythonic and comprehensive Python interface to dialog. This allows one to make simple text-mode user interfaces on Unix-like systems (including Linux). "

Note: The latest versions (2.12 and later) only support Python 3; users who really want to stick to Python 2 should use version 2.11 for now.

Install:
# latest:
# wget http://sourceforge.net/projects/pythondialog/files/latest/download?source=dlp

wget http://sourceforge.net/projects/pythondialog/files/pythondialog/2.11/python2-pythondialog-2.11.tar.gz/download
tar -zvxf python2-pythondialog-2.11.tar.gz
cd pythondialog-2.11

import dialog
d = dialog.Dialog()
if d.yesno("Continue?") == d.DIALOG_OK:
print "Continuing..."

---

My solution

kiloforce / dialog / source / — Bitbucket - https://bitbucket.org/kiloforce/dialog

<pre>
#!/usr/bin/env python

""" Wrapper for Linux 'dialog' command """

__author__ = "Kenneth Burgener <kenneth@k.ttak.org>"
__copyright__ = "(c) 2013"
__date__ = "$Sep 27, 2013$"
__version__ = "1.0"
__credits__ = r"""
XXXXXXXXXXXXXXXXXX
X Dialog Wrapper #
##################
"""

import subprocess
import time
import sys

class Dialog():
""" Python wrapper for Linux 'dialog' command

dialog - display dialog boxes from shell scripts

>>> import dialog
>>> dlg = dialog.Dialog()
>>> rc = dlg.msgbox('hello world')

Return codes:
YES = 0
OK = 0
EXIT = 0
NO = 1
CANCEL = 1
ESC = 255
"""

YES = 0
OK = 0
EXIT = 0
NO = 1
CANCEL = 1
ESC = 255

def __init__(self):
rc = self._exe('which dialog > /dev/null')[0]
if rc != 0:
print "Error: failed to find Linux dialog command"
sys.exit(1)

def _exe(self, cmd, strip=True):
p = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE)
output = "\n".join(p.stderr.readlines())
if strip:
output = output.strip()
rc = p.wait()
return (rc, output)

def rc_to_str(self, rc):
if rc == 0:
return "YES"
elif rc == 1:
return "NO"
elif rc == 255:
return "ESX"
else:
raise Exception("Unknown return code: %s" % rc)

def clear(self):
"""
Clear screen

Example:
>>> dialog.clear()"""
rc, output = self._exe('clear')
assert rc in (0,), "clear rc"
return rc

def msgbox(self, msg):
"""
Simple message box

@param msg (str) = display message
@return: rc (int)

Example:
>>> rc = dialog.msgbox('hello world')"""
rc, output = self._exe('dialog --msgbox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.ESC), "msgbox rc"
return rc

def yesno(self, msg):
"""
Yes/No dialog

@param msg (str) = display message
@return: rc (int)

Example:
>>> rc = dialog.yesno('continue')"""
rc, output = self._exe('dialog --yesno "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.NO, self.ESC), "yesno rc"
return rc

def infobox(self, msg):
"""
Dialog that does not wait for user's response
(do your own pause)

@param msg (str) = display message
@return: rc (int) always zero

Example:
>>> for i in range(10):
>>> dialog.infobox('counting: ' + str(i))
>>> time.sleep(3)"""
rc, output = self._exe('dialog --infobox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES,), "infobox rc"
return rc

def inputbox(self, msg):
"""
Get user input

@param msg (str) = display message
@return: rc (int), output (str/None)

Example:
>>> rc, output = dialog.inputbox('name:')"""
rc, output = self._exe('dialog --inputbox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.NO, self.ESC), "inputbox rc"
if not output:
output = None
return rc, output

def textbox(self, filename):
"""
Display a file

@param filename (str) = file to display
@return: rc (int)

Example:
>>> dialog.textbox('/etc/passwd')"""
rc, output = self._exe('dialog --textbox {filename} 0 0'.format(filename=filename))
assert rc in (self.YES, self.ESC), "textbox rc"
return rc

def menu(self, msg, item_list):
"""
Single selection checkbox menu

@param msg (str) = display message
@param item_list (str list) = list of items to display
@return: rc (int), selection (int/None)

Example:
>>> rc, selection = dialog.menu('Select multiple options:',
... ('option 1', 'option 2'))"""
item_str = ""
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
rc, output = self._exe('dialog --menu "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
if output:
selection = int(output)
else:
selection = None
assert rc in (self.YES, self.NO, self.ESC), "menu rc"
return rc, selection

def checklist(self, msg, item_list, checked_list=None):
"""
Multiple selection checkbox menu

@param msg (str) = display message
@param item_list (str list) = list of items
@param checked_list (int list) = default items to check
@return: rc (int), selections (int list/None)

Example:
>>> rc, selections = dialog.checklist('Select multiple options:',
... ('option 1', 'option 2')
"""
item_str = ""
assert type(item_list) in (type(None), type(()), type([])), "checklist: item_list must be a list"
assert type(checked_list) in (type(None), type(()), type([])), "checklist: checked_list must be a list"
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
if checked_list and i+1 in checked_list:
item_str += " on"
else:
item_str += " off"
rc, output = self._exe('dialog --checklist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
print item_str
selections = []
for choice in output.split('"'):
if choice.strip():
selections.append(int(choice))
if selections:
selections = tuple(selections)
else:
selections = None
assert rc in (self.YES, self.NO, self.ESC), "checklist rc"
return rc, selections

def radiolist(self, msg, item_list, checked_item=None):
"""
Single selection radio menu, with default

@param msg (str) = display message
@param item_list (str list) = list of items
@param checked_item (int) = default item to select
@return: rc (int), selection (int)

Example:
>>> rc, selection = dialog.radiolist('Select one option:',
>>> ('option 1', 'option 2'), 2)"""
item_str = ""
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
if checked_item and i+1 == int(checked_item):
item_str += " on"
else:
item_str += " off"
rc, selection = self._exe('dialog --radiolist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
if not selection:
selection = None
assert rc in (self.YES, self.NO, self.ESC), "radiolist rc"
return rc, selection

def gauge(self, msg, percent):
"""
Progress bar (does not wait for user's input)

@param msg (str) = display message
@param percent (int) = progress
@return: rc (int) always zero

Example:
>>> for i in range(0, 100+1, 2):
>>> dialog.gauge('Progress:', i)
>>> time.sleep(.1)"""
rc, output = self._exe('echo {percent} | dialog --gauge "{msg}" 0 0'.format(percent=percent, msg=msg))
assert rc in (self.YES,), "guage rc"
return rc

def progress(self, msg, percent):
"""
Alias for self.guage()
"""
return self.gauge(msg=msg, percent=percent)

def form(self, msg, item_list):
"""
Multiple item entry form

@param msg (str) = display message
@param item_list (str list) = list of items
@return: rc (int), answers (str list)

Example:
>>> rc, answers = dialog.form('Enter details',
... ('first name', 'last name'))"""
item_str = ""
name_len = 1
for i in range(len(item_list)):
if len(item_list[i])+3 > name_len:
name_len = len(item_list[i])+3
for i in range(len(item_list)):
item_str += " \"" + item_list[i] + ":\" " + str(i+1) + " 1 \"\" " + str(i+1) +\
" " + str(name_len) + " 40 0"
rc, answers = self._exe('dialog --form "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str),
strip=False)
answers = answers.split('\n\n')
if not answers:
answers = None
assert rc in (self.YES, self.NO, self.ESC), "form rc"
return rc, answers

def test(self):
"""
Test features

>>> import dialog
>>> dlg = dialog.Dialog()
>>> dlg.test()
"""

def wait_user(msg):
self.msgbox("%s" % msg)

# rc_to_str
wait_user("Testing return code...")
print self.rc_to_str(self.YES) # YES
print self.rc_to_str(self.NO) # NO
print self.rc_to_str(self.ESC) # ESX
try:
print self.rc_to_str(111) # exception
raise Exception("Did not throw exception")
except Exception as rc:
pass

# msgbox
wait_user("Testing msgbox...")
rc = self.msgbox("msgbox")
if rc not in (self.YES, self.ESC):
raise Exception("Unknown return code for msgbox: %s" % rc)

# yesno
wait_user("Testing yesno...")
rc = self.yesno("yesno")
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for yesno: %s" % rc)

# infobox
wait_user("Testing infobox...")
rc = self.infobox("infobox sleep 1 sec")
if rc not in (self.YES,):
raise Exception("Unknown return code for infobox %s" % rc)
time.sleep(1)

# inputbox
wait_user("Testing inputbox...")
rc, output = self.inputbox("inputbox")
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for inputbox: %s" % rc)
wait_user("Result: %s" % output)

# textbox
wait_user("Testing textbox...")
rc = self.textbox("/etc/passwd")
if rc not in (self.YES, self.ESC):
raise Exception("Unknown return code for textbox: %s" % rc)

# menu
wait_user("Testing menu...")
rc, selection = self.menu("menu", ("option 1", "option 2"))
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for menu: %s" % rc)
wait_user("Result: %s" % selection)

# checklist
wait_user("Testing checklist...")
rc, selection = self.checklist("menu", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for checklist: %s" % rc)
wait_user("Result: %s" % str(selection))
wait_user("Testing checklist with default selection...")
rc, selection = self.checklist("menu", ["option 1", "option 2"], [1, 2])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for checklist: %s" % rc)
wait_user("Result: %s" % str(selection))

# radiolist
wait_user("Testing radiolist...")
rc, selection = self.radiolist("menu", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for radiolist: %s" % rc)
wait_user("Result: %s" % str(selection))
wait_user("Testing radiolist with default selection...")
rc, selection = self.radiolist("menu", ["option 1", "option 2"], 2)
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for radiolist: %s" % rc)
wait_user("Result: %s" % str(selection))

# guage
wait_user("Testing guage...")
for i in range(4+1):
rc = self.gauge("test guage", i*25)
# rc = self.progress("test progress", i*25)
if rc not in (self.YES,):
raise Exception("Unknown return code for guage: %s" % rc)
time.sleep(1)

# form
wait_user("Testing form...")
rc, answers = self.form("form", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for form: %s" % rc)
wait_user("Result: %s" % str(answers))

# done with test, cleanup
wait_user("Done with tests...")
self.clear()

if __name__ == "__main__":
# dialog = Dialog()
# dialog.test()
pass
</pre>

=== Busy Loop and Thread.yield() ===

"I want to tell my Python threads to yield, and so avoid hogging the CPU unnecessarily. In Java, you could do that using the Thread.yield() function"

Solution:
time.sleep(0)
time.sleep(0.00001)
time.sleep(epsilon)

References:
* multithreading - In there something similar to Java's Thread.yield() in Python? Does that even make sense? - Stack Overflow - http://stackoverflow.com/questions/1908206/in-there-something-similar-to-javas-thread-yield-in-python-does-that-even-ma
* multithreading - How does a threading.Thread yield the rest of its quantum in Python? - Stack Overflow - http://stackoverflow.com/questions/787803/how-does-a-threading-thread-yield-the-rest-of-its-quantum-in-python

=== USB ===

Programming with PyUSB 1.0 - http://pyusb.sourceforge.net/docs/1.0/tutorial.html

PyUSB - http://pyusb.sourceforge.net/

git clone https://github.com/walac/pyusb

import usb.core
dev = usb.core.find()

# find our device
dev = usb.core.find(idVendor=0xfffe, idProduct=0x0001)

"'''Dealing with multiple identical devices''' - Sometimes you may have two identical devices connected to the computer. How can you differentiate them? Device objects come with two additional attributes which are not part of the USB Spec, but are very useful: bus and address attributes. First of all, it is worth to say that these attributes come from the backend and a backend is free to not support them, in which case they are set to None. That said, these attributes represent the bus number and bus address of the device and, as you might already have imagined, can be used to differentiate two devices with the same idVendor and idProduct attributes."

=== Tkinter ===

Python GUI Bitcoin: [http://bitcoin.stackexchange.com/questions/10594/simplest-way-of-obtaining-mtgox-exchange-rate-via-an-api]
<pre>
import json
import urllib2
from Tkinter import *

url='http://api.bitcoincharts.com/v1/markets.json'
req=urllib2.Request(url)
response=urllib2.urlopen(req).read()
output=json.loads(response)

mtusd_price = output[-10]['avg']
mteur_price = output[31]['avg']
mtgbp_price = output[44]['avg']

MTUSD= 'USD' + ' ' + str(mtusd_price)
MTEUR= 'EUR' + ' ' + str(mteur_price)
MTGBP = 'GBP' + ' ' + str(mtgbp_price)

root= Tk()
root.title("Bitcoin Price")
root.geometry("250x100")

USD = Label(root, text=MTUSD)
EUR = Label(root, text=MTEUR)
GBP = Label(root, text=MTGBP)
USD.pack()
EUR.pack()
GBP.pack()
mainloop()
</pre>

=== ctypes ===

"ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python."
:-- 15.17. ctypes — A foreign function library for Python — Python v2.7.7 documentation - https://docs.python.org/2/library/ctypes.html

Use "c scope" to build a database of symbols to see what options are available.

---

testlib.c
<pre>
#include <stdio.h>

void myprint(void);

void myprint()
{
printf("hello world\n");
}
</pre>

Compile:
gcc -shared -Wl,-soname,testlib -o testlib.so -fPIC testlib.c

testlibwrapper.py

Python:
<pre>
import ctypes

testlib = ctypes.CDLL('/full/path/to/testlib.so')
testlib.myprint()
</pre>

Source: python - ctypes - Beginner - Stack Overflow - http://stackoverflow.com/questions/5081875/ctypes-beginner

---

nzjrs's Gists - https://gist.github.com/nzjrs

---

<pre>
#include <stdio.h>

int sum(double *x,int n)
{
int i;
double counter;
counter = 0;
for(i=0;i<n;i++)
{
counter=counter+x[i];

}
return counter;
}
</pre>

<pre>
gcc -c sum.c
gcc -shared -o sum.so sum.o
</pre>

<pre>
from ctypes import *
my_sum=CDLL('sum.so')
a=numpy.array(range(10),dtype=float)
my_sum.sum(a.ctypes.data_as(c_void_p),int(10))
</pre>

<pre>
a=(c_double*10)()
for i in range(10):
a[i]=i
my_sum.sum(a,int(10))
</pre>

Source: Ctypes — Numerical Computing with Sage v6.2 - http://www.sagemath.org/doc/numerical_sage/ctypes.html

---

=== Warnings ===

if sys.version_info >= (1, 0, 0):
warnings.warn("Danger Will Robinson: trying Phi on untested Python 3.0!")

=== enum ===

Python 3: https://docs.python.org/3/library/enum.html

<pre>
from enum import Enum

# class syntax
class Color(Enum):
RED = 1
GREEN = 2
BLUE = 3

# functional syntax
Color = Enum('Color', ['RED', 'GREEN', 'BLUE'])
</pre>

---

Python 2: [http://stackoverflow.com/questions/36932/how-can-i-represent-an-enum-in-python]
<pre>
class Enum(set):
""" Basic enumerated type

Example
>>> Animals = phi.Enum(["DOG", "CAT", "HORSE"])
>>> Animals.DOG
'DOG'
>>> 'HORSE' in Animals
True
"""
def __getattr__(self, name):
if name in self:
return name
raise AttributeError("Missing Enum property: %s" % name)
def __setattr__(self, name, value): # this makes it read-only
raise AttributeError("Unable to modify Enum property: %s (%s)" % (name, value))

Animals = Enum(["DOG", "CAT", "HORSE"])

print(Animals.DOG) # 'DOG'
</pre>

----------

Alternative:
<pre>
class Stationery:
Pen, Pencil, Eraser = range(0, 3)
</pre>

One liner:
class Enum(tuple): __getattr__ = tuple.index

<pre>
>>> State = Enum(['Unclaimed', 'Claimed'])
>>> State.Claimed
1
>>> State[1]
'Claimed'
>>> State
('Unclaimed', 'Claimed')
>>> range(len(State))
[0, 1]
>>> [(k, State[k]) for k in range(len(State))]
[(0, 'Unclaimed'), (1, 'Claimed')]
>>> [(k, getattr(State, k)) for k in State]
[('Unclaimed', 0), ('Claimed', 1)]
</pre>

Python 3 supports enum natively with the enum.Enum library:

=== pySerial ===

https://pythonhosted.org/pyserial/

"This module encapsulates the access for the serial port. It provides backends for Python running on Windows, OSX, Linux, BSD (possibly any POSIX compliant system) and IronPython. The module named “serial” automatically selects the appropriate backend."

https://github.com/pyserial/

Intro - https://pythonhosted.org/pyserial/shortintro.html

---

<pre>
import serial
import time

ser = serial.Serial(
port='COM4',\
baudrate=9600,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)

print("connected to: " + ser.portstr)

#this will store the line
seq = []
count = 1

while True:
for c in ser.read():
seq.append(chr(c)) #convert from ANSII
joined_seq = ''.join(str(v) for v in seq) #Make a string from array

if chr(c) == '\n':
print("Line " + str(count) + ': ' + joined_seq)
seq = []
count += 1
break

ser.close()
</pre>

Ref: http://stackoverflow.com/questions/16077912/python-serial-how-to-use-the-read-or-readline-function-to-read-more-than-1-char

---

I modified the code to monitor a COM port: (Windows)

Monitor Serial:

<pre>
import serial, time, sys

logger = open("serial.txt", "w")
timer = time.time()

def log(msg, head=False):
msg = msg.strip()
if head:
msg = ">>> " + msg
logger.write(msg + "\n")
sys.stdout.write(msg + "\n")
newtime = time.time()
if newtime > timer + 1:
logger.flush()
sys.stdout.flush()

while True:

ser = None
try:
ser = serial.Serial(
port='COM1',\
baudrate=115200,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)
except serial.serialutil.SerialException as e:
log(str(e), True)
time.sleep(1)
continue

log("Connected to: " + ser.portstr, True)
count=1

try:
line = []
while True:
for c in ser.readline():
line.append(c)
if c == "\n":
log("".join(line))
line = []
except serial.serialutil.SerialException as e:
log(str(e), True)
time.sleep(1)
ser.close()
</pre>

=== MyBool ===

<pre>
>>> class NewBool(int):
... def __new__(cls, value):
... return int.__new__(cls, bool(value))

>>> y = NewBool(2)
>>> y == 1
True
</pre>

https://jfine-python-classes.readthedocs.io/en/latest/subclass-int.html

Also check out the __cmp__

== md5 ==

import md5
md5.new("Nobody inspects the spammish repetition").hexdigest()
# 'bb649c83dd1ea5c9d9dec9a18df0ffe9'

<pre>
$ echo -n "Nobody inspects the spammish repetition" | md5sum
bb649c83dd1ea5c9d9dec9a18df0ffe9 -
<pre>

== External Libraries ==

=== execnet ===

Python Package Index - execnet - https://pypi.python.org/pypi/execnet

Examples - http://codespeak.net/execnet/examples.html
* basic local and remote communication - http://codespeak.net/execnet/example/test_info.html

Execute source code in subprocess, communicate through a channel:
<pre>
>>> import execnet
>>> gw = execnet.makegateway()
>>> channel = gw.remote_exec("channel.send(channel.receive()+1)")
>>> channel.send(1)
>>> channel.receive()
2
</pre>

Compare current working directories
<pre>
>>> import execnet, os
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec("import os; channel.send(os.getcwd())")
>>> res = ch.receive()
>>> assert res == os.getcwd()
</pre>

Get information from remote ssh account
<pre>
>>> import execnet, os
>>> gw = execnet.makegateway("ssh=codespeak.net")
>>> channel = gw.remote_exec("""
... import sys, os
... channel.send((sys.platform, sys.version_info, os.getpid()))
... """)
>>> platform, version_info, remote_pid = channel.receive()
>>> platform
'linux2'
>>> version_info
(2, 4, 2, 'final', 0)
</pre>

remote-exec a function (avoiding inlined source part I)
<pre>
import execnet

def multiplier(channel, factor):
while not channel.isclosed():
param = channel.receive()
channel.send(param * factor)

gw = execnet.makegateway()
channel = gw.remote_exec(multiplier, factor=10)

for i in range(5):
channel.send(i)
result = channel.receive()
assert result == i * 10

gw.exit()
</pre>

remote-exec a module (avoiding inlined source part II)
:"You can pass a module object to remote_exec in which case its source code will be sent. No dependencies will be transferred so the module must be self-contained or only use modules that are installed on the “other” side. Module code can detect if it is running in a remote_exec situation by checking for the special __name__ attribute."

<pre>
# content of a module remote1.py

if __name__ == '__channelexec__':
channel.send('initialization complete')
</pre>

<pre>
>>> import execnet, remote1
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec(remote1)
>>> print (ch.receive())
initialization complete
</pre>

Use a callback instead of receive() and wait for completion
<pre>
>>> import execnet
>>> gw = execnet.makegateway()
>>> channel = gw.remote_exec("for i in range(10): channel.send(i)")
>>> l = []
>>> channel.setcallback(l.append, endmarker=None)
>>> channel.waitclose() # waits for closing, i.e. remote exec finish
>>> l
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, None]
</pre>

a simple command loop pattern
<pre>
# contents of: remotecmd.py
import os

def simple(arg):
return arg + 1

def listdir(path):
return os.listdir(path)

if __name__ == '__channelexec__':
for item in channel:
channel.send(eval(item))
</pre>

<pre>
>>> import execnet, remotecmd
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec(remotecmd)
>>> ch.send('simple(10)') # execute func-call remotely
>>> ch.receive()
11
</pre>

=== evdev ===

Install:
apt-get install python-setuptools python-dev
easy_install pip
pip install evdev

Simple read usage:
<pre>
from evdev import InputDevice
from evdev import ecodes

# look for a /dev/input/by-id/usb...kbd or similar
DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd"

dev = InputDevice(DEVICE)

while True:
r, w, x = select([dev], [], [], 5)
for event in dev.read():
if event.type == 1 and event.value == 1:
print event.code
</pre>

List LED states:
dev.leds()
dev.leds(verbose=True)

Inject usage: (does not appear to really work)
<pre>
from evdev import InputDevice
from evdev import ecodes
from evdev import UInput

DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd"
dev = InputDevice(DEVICE)

ui = UInput()

ui.write(ecodes.EV_KEY, ecodes.KEY_4, 1)
ui.write(ecodes.EV_KEY, ecodes.KEY_4, 0)
ui.syn()

ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 1)
ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 0)
ui.syn()

ui.close()
</pre>

Tutorial:
* evdev documentation — python-evdev 0.4.3 documentation - http://pythonhosted.org/evdev/index.html
References:
* python-evdev/doc/tutorial.rst at master · gvalkov/python-evdev · GitHub - https://github.com/gvalkov/python-evdev/blob/master/doc/tutorial.rst

=== ldap-python ===

python-ldap: LDAP client API for Python - http://www.python-ldap.org/

Index of Packages : Python Package Index - https://pypi.python.org/pypi/python-ldap/

yum install python-devel openldap-devel
pip install python-ldap

Articles:
* Python LDAP Applications: Part 1 - Installing and Configuring the Python-LDAP Library and Binding to an LDAP Directory | Packt Publishing - http://www.packtpub.com/article/installing-and-configuring-the-python-ldap-library-and-binding-to-an-ldap-directory

=== xmmp ===

PDF:
* XMPP: The Definitive Guide - Oreilly - http://oriolrius.cat/blog/wp-content/uploads/2009/10/Oreilly.XMPP.The.Definitive.Guide.May.2009.pdf

xmmp is the Jabber protocol

Best options:

MIT libraries.
* sleekxmpp (was: sleekxmpp) is pretty popular and is used for examples in Peter Saint-Andre's XMPP book from O'Reilly. It reportedly works well, and finally got an email list in April 2010, and has a chat room at sleek@conference.jabber.org.

GPL libraries.
* xmpppy is used by gajim.

LPGL libraries.
* pyxmpp is pretty good and uses libxml2 internally for xml parsing.
* pyxmpp2 is the next version of pyxmpp, runs on python 2.7 and 3.2, and removes the libxml2 requirement. Like many, it requires dnspython.
* jabberpy is the original and still works for a lot of tasks, but is thoroughly unmaintained.

Other libraries.
* Wokkel, mentioned in another post. That's a new one for me, based on twisted.

chat - Best python XMPP / Jabber client library? - Stack Overflow - http://stackoverflow.com/questions/1901828/best-python-xmpp-jabber-client-library

=== xmpppy ===

xmpppy: the jabber python project - http://xmpppy.sourceforge.net/

"xmpppy is a Python library that is targeted to provide easy scripting with Jabber. Similar projects are Twisted Words and jabber.py.

This library was not designed from scratch. It inherits some code from jabberpy and have very similar API in many places. Though it is separate project since it have almost completely different architecture and primarily aims to work with jabberd2 - the new Open Source Jabber Server.

xmpppy is distributed under the terms of GNU General Public License and can be freely redistributed without any charge."

wget http://sourceforge.net/projects/xmpppy/files/xmpppy/0.4.1/xmpppy-0.4.1.tar.gz/download -O xmpppy-0.4.1.tar.gz
tar -zvxf xmpppy-0.4.1.tar.gz
pip install pydns

--- basic connection ---

<pre>
#!/usr/bin/env python

import xmpp

user="username@gmail.com"
password="password"
server="gmail.com"

jid = xmpp.JID(user)
connection = xmpp.Client(server,debug=[])
connection.connect()
result = connection.auth(jid.getNode(), password,"LFY-client")

connection.sendInitPresence()

while connection.Process(1):
pass
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- basic receive ---

Print messages when received:
<pre>
#!/usr/bin/env python

import xmpp

user="username@gmail.com"
password="password"
server="gmail.com"

def message_handler(connect_object, message_node):
message = "Welcome to my first Gtalk Bot :)"
connect_object.send( xmpp.Message( message_node.getFrom() ,message))

jid = xmpp.JID(user)
connection = xmpp.Client(server)
connection.connect()
result = connection.auth(jid.getNode(), password, "LFY-client")
connection.RegisterHandler('message', message_handler)

connection.sendInitPresence()

while connection.Process(1):
pass
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- remote-control shell bot ---

Replace the simple bot’s message_handler() function with this new one:

<pre>
def message_handler(connect_object,message_node):
command = str(message_node.getBody())

process = subprocess.Popen(command,shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
message = process.stdout.read()
if message=="":
message=process.stderr.read()

connect_object.send( xmpp.Message( message_node.getFrom() ,message))
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- roster ---

<pre>
import xmpp
import sys

userID = 'myname@gmail.com'
password = 'mypassword'
ressource = 'Script'

jid = xmpp.protocol.JID(userID)
jabber = xmpp.Client(jid.getDomain(), debug=[])

connection = jabber.connect(('talk.google.com',5222))
auth = jabber.auth(jid.getNode(), password, ressource)

jabber.sendInitPresence(requestRoster=1)
myroster = jabber.getRoster()
</pre>

To get status, you have to wait for the other clients to send you their status:
<pre>
jabber.RegisterHandler('presence', myPresenceHandler)

def myPresenceHandler(con, event):
fromjid = event.getFrom().getStripped()
status = myroster.getStatus(fromjid)
</pre>

References:
python - XMPP chat: accessing contacts' status messages with xmppPy's Roster - Stack Overflow - http://stackoverflow.com/questions/2381597/xmpp-chat-accessing-contacts-status-messages-with-xmpppys-roster

--- full roster script ---

kparal/jabber-roster · GitHub - https://github.com/kparal/jabber-roster

A simple Python tool for listing your Jabber roster contacts. You can use it to easily backup list of your buddies.

This program is maintained, but not further developed. Bugs will be fixed, but no new features will be added. If you want to work on this program, don't hesitate to contact me, I will gladly assign you to the development team.

git clone https://github.com/kparal/jabber-roster
python jabber_roster.py

https://pypi.python.org/pypi/jabber-roster

--- vcard ---

This information is not in the roster. You will need to query the clients individually and get their vCard by sending this IQ :

<pre>
<iq from='stpeter@jabber.org/roundabout'
id='v1'
type='get'>
<vCard xmlns='vcard-temp'/>
</iq>
</pre>

My solution:
<pre>
def vcard(disp, jid):
msg = xmpp.protocol.Iq()
msg.setType('get')
msg.setTo(jid)
qc = msg.addChild('vCard')
qc.setAttr('xmlns', 'vcard-temp')
rep = disp.SendAndWaitForResponse(msg)
# to see what other fields are available in the XML output:
# print rep
userid=fname=lname=title=department=region=None
for i in rep.getChildren():
for j in i.getChildren():
if j.getName() == "TITLE":
title = j.getData().encode('utf-8')
for k in j.getChildren():
if k.getName() == "GIVEN":
fname = k.getData().encode('utf-8')
if k.getName() == "FAMILY":
lname = k.getData().encode('utf-8')
if k.getName() == "ORGUNIT":
department = k.getData().encode('utf-8')
if k.getName() == "REGION":
region = k.getData().encode('utf-8')
return fname, lname, title, department, region
</pre>

References:
* xmpppy - Retrieve gtalk nickname in python xmpp - Stack Overflow - http://stackoverflow.com/questions/4539477/retrieve-gtalk-nickname-in-python-xmpp
* XEP-0054: vcard-temp - http://xmpp.org/extensions/xep-0054.html

=== google-api-python-client ===

Google APIs Client Library for Python, which includes:
* google-api-python-client: The core Python library for accessing Google APIs.
* oauth2client: A Python client library for OAuth 2.0.
* Sample applications using google-api-python-client and oauth2client

-

https://developers.google.com/api-client-library/python/start/installation
* Calendar API - Python/Command Line

-
Download:
wget https://code.google.com/p/google-api-python-client/downloads/detail?name=google-api-python-client-1.2.zip
wget https://code.google.com/p/google-api-python-client/downloads/detail?name=oauth2client-1.2.zip

Git:
git clone https://github.com/google/google-api-python-client
git clone https://github.com/google/oauth2client

References:
* google-api-python-client - Google APIs Client Library for Python - Google Project Hosting - https://code.google.com/p/google-api-python-client/

== Documentation Tools ==

=== pydoc ===

pydoc - http://docs.python.org/library/pydoc.html
: Documentation generator and online help system

=== Doxygen and Doxypy ===

See [[Python/Doxygen]]

=== Sphinx ===

Sphinx - http://sphinx.pocoo.org/

"Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license.

It was originally created for the new Python documentation, and it has excellent facilities for the documentation of Python projects, but C/C++ is already supported as well, and it is planned to add special support for other languages as well. Of course, this site is also created from reStructuredText sources using Sphinx!"

=== epydoc ===

epydoc - http://epydoc.sourceforge.net/

"Epydoc is a tool for generating API documentation for Python modules, based on their docstrings. For an example of epydoc's output, see the API documentation for epydoc itself (html, pdf). A lightweight markup language called epytext can be used to format docstrings, and to add information about specific fields, such as parameters and instance variables. Epydoc also understands docstrings written in reStructuredText, Javadoc, and plaintext. For a more extensive example of epydoc's output, see the API documentation for Python 2.5. "

== Python 3 ==

=== Python 3 Wall of Superpowers ===

Python 3 Wall of Superpowers - https://python3wos.appspot.com/

"As listed on PyPI - packages in red don't support python 3, packages in green do. Hopefully one day everything will be greener. "

=== New in Python 3 ===

==== qualnames ====

New in 3.3
<pre>
>>> def f():
... def g():
... class H: pass
... return H
... return g
...

>>> H=f()()

>>> # 3.3:
>>> H
<class '__main__.f.<locals>.g.<locals>.H'>
>>> H.__qualname__
'f.<locals>.g.<locals>.H'

>>> # 2.6:
>>> H
<class __main__.H at 0x2b885044b890>
>>> H.__qualname__
AttributeError: class H has no attribute '__qualname__'
</pre>

== Python 2 or Python 3 ==

Python2orPython3 - PythonInfo Wiki - http://wiki.python.org/moin/Python2orPython3
:Short version: Python 2.x is the status quo, Python 3.x is the shiny new thing.
:At the time of writing (July 4, 2010), the final 2.7 release is out, with a statement of extended support for this end-of-life release. The 2.x branch will see no new major releases after that. 3.x is under active and continued development, with 3.1 already available and 3.2 due for release around the turn of the year.

Python 2.7.1 Release - http://www.python.org/download/releases/2.7.1/

What’s New In Python 3.0 — Python v3.0.1 documentation - http://docs.python.org/release/3.0.1/whatsnew/3.0.html
* print() requires parenthesis
* raw_input() is now input()

== Python for Windows ==

Python for Windows - http://www.python.org/download/windows/

=== Windows IDEs ===

==== PyCharm ====

http://www.pycharm.com/

=== Hide Console ===

fork() does not work on windows. So what else can we do?

How to hide console window in python? - http://stackoverflow.com/questions/764631/how-to-hide-console-window-in-python
* "Simply save it with a .pyw extension. This will prevent the console window from opening. Explanation at the bottom of section [http://docs.python.org/2/tutorial/interpreter.html#executable-python-scripts]"

=== 32bit vs 64bit ===

IDLE:

32bit:
<pre>
C:\Python27>python
Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
</pre>

64bit: (note: default path is C:\Python27)
<pre>
C:\Python27-64bit>python.exe
Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
</pre>

Task Manager:
python.exe *32

python.exe

Platform:
>>> import platform
>>> platform.python_compiler()
'MSC v.1500 32 bit (Intel)'

>>> import platform
>>> platform.python_compiler()
'MSC v.1500 64 bit (AMD64)'

Code: [http://stackoverflow.com/questions/2208828/detect-64bit-os-windows-in-python]
import struct
def is_python_64bit():
return (struct.calcsize("P") == 8)

=== Stand-Alone Executable ===

==== Python ZIP Application ====

Python can be run from a single bundled zip file - similar to a Java WAR! Python has been able to (Allow interpreter to execute a zip file http://bugs.python.org/issue1739468)

"Python quietly added a new feature in 2.6 that makes it possible to bundle up a directory full of Python code into a single executable file." [http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html]

When a python zip application is executed, it will execute '__main__.py' [http://docs.python.org/using/cmdline.html#interface-options]

mkdir app
touch app/__main__.py
cd app
zip -r ../app.zip *
cd ..
python app.zip

Load a resource - The easiest way to find and load a program bundle like this is to use the pkg_resources module. - http://packages.python.org/distribute/pkg_resources.html

<pre>
# file: app/__main__.py

import pkg_resources

def main():
print('The rain in Spain falls mainly in the plain.')
print(pkg_resources.resource_string('resources', 'inFrance.txt'))

if __name__ == '__main__':
main()
</pre>

Now use a bit of UNIX magic to turn app2.zip into an executable:
echo '#!/usr/bin/env python' | cat - app2.zip > app2
chmod +x app2

"The zip file format is designed to allow a small executable program to be inserted at the front (that's how self-extracting zip files are created), so this is kosher and doesn't corrupt the zip file." [http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html]

Now you can simply run app2 like any executable.
$ ./app2

References:
* Able Pear Software: Bundling Python files into a stand-alone executable - http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html
* PEP 441 -- Improving Python ZIP Application Support - http://www.python.org/dev/peps/pep-0441/
* Issue 1739468: Allow interpreter to execute a zip file - Python tracker - http://bugs.python.org/issue1739468
* Package Discovery and Resource Access using pkg_resources — Distribute documentation - http://packages.python.org/distribute/pkg_resources.html

==== Shell Python Loader ====

Came before the Python ZIP Application support.

Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe

"This takes your Python application and packages it into a single executable shell script. You application can be made of up multiple python scripts. The executable shell script is also compressed, so the resulting script is smaller than the original. This should work on any system that has Python and a Bourne shell (works on Cygwin too). Note that this does not package the Python interpreter into the executable. This packages only you python code into a zip file, but this still allows you to distribute your application as a single shell script that will run on any system with Python installed. In theory, you could zip the .pyc files, but the byte code is guaranteed to run only on exactly the same version Python interpreter (including minor version). It is safer to just package the .py files."

First you need to zip all your python files:
zip main.zip main.py spam.py eggs.py

zipheader.sh:
<pre>
#!/bin/sh
# This is a self-extracting executable.
# Execute this like any normal executable.
# You may need to "chmod a+x" this file.
# This is a binary ZIP file with a Python loader header.
#
# Bourne shell loader:
PYTHON=$(which python 2>/dev/null)
if [ ! -x "$PYTHON" ] ; then
echo "Python not found!"
exit 1
fi
exec $PYTHON -c "
# Python loader:
import sys, os
if int(sys.version[0])<2:
print 'Python version 2.3 final or greater is required.'
print 'Your version is', sys.version
os._exit(1)
major = sys.version_info[0]
minor = sys.version_info[1]
releaselevel = sys.version_info[3]
if (major==2 and minor<3) or (major==2 and minor==3 and releaselevel!='final'):
print 'Python version 2.3 final or greater is required.'
print 'Your version is', sys.version
os._exit(1)
sys.path.insert(0, sys.argv[1])
del sys.argv[0:1]
import main
main.main()
" $0 $@
# Zip file:
</pre>

Concatinate zipheader.sh with the zip file.
cat zipheader.sh main.zip > main

Set main as executable.
chmod +x main

"This is an improvement on Joerg Raedler's Python recipe, [http://code.activestate.com/recipes/215301-binding-main-skript-and-modules-to-one-executable-/ 215301]. The original closes stdin. The herefile in the shell script redirects stdin before python gets a chance to start. This disables raw_input() and anything else that reads sys.stdin. Unfortunately, Raedler's boot script closes stdin, which is a fairly big limitation. The herefile in the shell script (END_OF_PYTHON_CODE) redirects stdin before python starts; stdin is closed at the end of the herefile. This disables raw_input() and anything else that reads sys.stdin. "

References:
* Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe
* Build a compressed self-extracting executable script on UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/497000-build-a-compressed-self-extracting-executable-scri/
* binding main skript and modules to one executable with python-2.3 under UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/215301-binding-main-skript-and-modules-to-one-executable-/

==== py2exe ====

py2exe - http://www.py2exe.org/
: "py2exe is a Python Distutils extension which converts Python scripts into executable Windows programs, able to run without requiring a Python installation. "

Tutorial - http://www.py2exe.org/index.cgi/Tutorial

Note: Install the version that matches your version of python

Note: run installer "as administrator"

setup.py
<pre>
from distutils.core import setup
import py2exe

setup(console=['hello.py'])
</pre>

No console window version: [http://ubuntuforums.org/showthread.php?t=728170] [http://stackoverflow.com/questions/3275293/hiding-console-window-of-python-gui-app-with-py2exe]
<pre>
from distutils.core import setup
import py2exe

setup(windows=['hello.py'])
</pre>

Build:
python setup.py py2exe

pygame is more complex: http://www.pygame.org/wiki/Pygame2exe

==== py2app ====

==== pyinstaller ====

==== cx_Freeze ====

=== Windows Libraries ===

==== Python for Windows Extensions - PyWin32 ====

Python for Windows Extensions - http://starship.python.net/~skippy/win32/
* There is a build for each python version and architecture (32bit vs 64bit)

Follow event viewer
<pre>
import win32evtlog # install Python for Windows Extensions - http://starship.python.net/~skippy/win32/

# open event viewer
server = "localhost"
source = "System"
handle = win32evtlog.OpenEventLog(server, source)

# seek to end of log
flags = win32evtlog.EVENTLOG_BACKWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
events = win32evtlog.ReadEventLog(handle, flags, 0)

# return to normal reading direction
flags = win32evtlog.EVENTLOG_FORWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
win32evtlog.ReadEventLog(handle, flags, 0) # ignore initial logs

# loop through events
while True:
events = win32evtlog.ReadEventLog(handle, flags, 0)
if events:
for event in events:
data = event.StringInserts
if data:
# Example: 07/03/14 12:34:35 - MYEVENTSOURCE - 0 - 1 - 4 "HELLO"
print event.TimeGenerated, '-', event.SourceName, '-', event.EventCategory, '-', event.EventID, '-', event.EventType, '-', '"%s"' % " ".join(data)
time.sleep(.5) # yield cpu
</pre>

References:
* PyWin32: Getting Windows Event Logs « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/07/27/pywin32-getting-windows-event-logs/

==== PyCrypto for Windows ====

Download Install: http://www.voidspace.org.uk/python/modules.shtml#pycrypto

Installation: [http://stackoverflow.com/questions/11405549/how-do-i-install-pycrypto-on-windows]
easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win32-py2.7.exe
easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win-amd64-py2.7.exe

==== Paramiko for Windows ====

# Install PyCrypto (above)
# pip install paramiko

==== PyGame for Windows ====

Download - http://www.pygame.org/download.shtml
* http://pygame.org/ftp/pygame-1.9.1.win32-py2.7.msi

== Indentation ==

Python: Myths about Indentation - http://www.secnetix.de/olli/Python/block_indentation.hawk
:There are quite some prejudices and myths about Python's indentation rules among people who don't really know Python. I'll try to address a few of these concerns on this page.
*"Whitespace is significant in Python source code."
*"Python forces me to use a certain indentation style."
*"You cannot safely mix tabs and spaces in Python."
*"I just don't like it."
*"How does the compiler parse the indentation?"

== PEP - Python Enhancement Proposals ==

Python Enhancement Proposals (PEPs)

* Index of Python Enhancement Proposals (PEPs) - http://www.python.org/dev/peps/

== Convert Tabs to Spaces ==

Convert tabs to 4 spaces:
sed -i 's/\t/ /g' script.py

== Linting ==

* pep8 - python style guide checker
* pychecker - more verbose, imports code, slower
* pylint - most verbose, configurable, "rates" code
* pyflakes - least verbose (dead/redundant code)

rpmforge:
yum install pyflakes pychecker
# pylint - not sure which pacakge includes

=== pep8 ===

See [[Python/pep8]]

=== pychecker ===

See [[Python/pychecker]]

=== pylint ===

See [[Python/pylint]]

== PEP8 Style Guidlines ==

PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/

* Use spaces - Use 4 spaces per indentation level.
* Never mix tabs and spaces.
* Limit all lines to a maximum of 79 characters.
* Line wrapping - The preferred way of wrapping long lines is by using Python's implied line continuation inside parentheses, brackets and braces. Long lines can be broken over multiple lines by wrapping expressions in parentheses.
* Encoding - For Python 3.0 and beyond, UTF-8 is preferred over Latin-1, see PEP 3120.
* Imports should usually be on separate lines
* Avoid extraneous whitespace in most situations
* Always use 'self' for the first argument to instance methods.

=== PEP8 Style Checker ===

See [[#pep8]]

== Benefits of Python ==

What is the return on investment for this group by learning Python? The list of benefits below sums up what the language has to offer:
* Python is designed from the ground up to encourage maintainable, readable and reusable code. It has great support for particular paradigms such as object-oriented programming.
* Python code is usually around a quarter of the length of code generated by C++ or Java - less typing, less debugging and less maintenance.
* Python does not need to be compiled or linked, which allows for a more iterative programming style and a quicker development cycle.
* Python is extremely portable. Code can be easily exchanged between Windows, Mac OS and Linux with little to no modification.
* Python comes with a staggering array of standard libraries and mature third-party projects. Web development, scientific computing, operator system access and network scripting are all supported.
* Python can talk with many other languages easily. It has support for invoking C and C++ code and can itself be called from libraries written in these languages.

Source: http://quantstart.com/articles/Quant-Reading-List-Python-Programming/ - Quant Reading List Python Programming

Benefits of Python:
* Python is available for most operating systems, including Windows, UNIX, Linux, and Mac OS.
* Python's clean object-oriented design and extensive support libraries offer two to ten fold the programmer productivity seen with languages like C, C++, C#, Java, VB, and Perl.

Source: Python Development - Benefits of Python - Digital Mesh - http://www.digitalmesh.com/offshore-development-center/python-development/benefits-of-python.html

== Python Package Management ==

=== Install Module ===

Installing Python Modules — Python v2.7.1 documentation - http://docs.python.org/install/index.html

The new standard: Distutils
python setup.py install

Download and extract target package, then:
sudo python setup.py install

Note: Running setup.py install builds and installs all modules in one run

Help:
python setup.py --help

Incremental method:
python setup.py build
python setup.py install
python setup.py build --help # help

Force rebuild and install:
python setup.py build --force # if have to repeat
python setup.py install --force # if have to repeat

Default installation paths:
<pre>
Platform Standard installation location Default value
Unix (pure) prefix/lib/pythonX.Y/site-packages /usr/local/lib/pythonX.Y/site-packages
Unix (non-pure) exec-prefix/lib/pythonX.Y/site-packages /usr/local/lib/pythonX.Y/site-packages
Windows prefix C:\Python
</pre>

Alternate install path:
python setup.py install --home=<dir>

Alternate base folders:
<pre>
Type of file Installation Directory Override option
pure module distribution home/lib/python --install-purelib
non-pure module distribution home/lib/python --install-platlib
scripts home/bin --install-scripts
data home/share --install-data
</pre>

Modifying Python’s Search Path:
import sys
sys.path.append('/www/python/')

=== Python Package Index ===

user-submitted Python packages:
http://pypi.python.org/pypi

sudo python setup.py install

Package Site:
http://pypi.python.org/pypi/[PACKAGE]
http://pypi.python.org/pypi/test

If they are on http://pypi.python.org/:
pip install foo

[http://stackoverflow.com/questions/410163/for-python-programmers-is-there-anything-equivalent-to-perls-cpan For Python programmers, is there anything equivalent to Perl's CPAN? - Stack Overflow]

=== Easy Installation ===

EasyInstall - The PEAK Developers' Center - http://peak.telecommunity.com/DevCenter/EasyInstall
: Easy Install is a python module (easy_install) bundled with setuptools that lets you automatically download, build, install, and manage Python packages.

CheeseShopTutorial - PythonInfo Wiki - http://wiki.python.org/moin/CheeseShopTutorial
: EasyInstall (easy_install) gives you a quick and painless way to install packages remotely by connecting to the Package Index or even other websites via HTTP. It is somewhat analogous to the CPAN and PEAR tools for Perl and PHP, respectively.

=== pip ===

See [[Python/pip]]

==== Create PIP Package ====

See [[Python/pip#Create PIP Package]]

=== Distribute ===

distribute - Easily download, build, install, upgrade, and uninstall Python packages

distribute 0.6.24 : Python Package Index - http://pypi.python.org/pypi/distribute

Installation:
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/d/distribute/distribute-0.6.24.tar.gz#md5=17722b22141aba8235787f79800cc452
tar -zvxf distribute-0.6.24.tar.gz
cd distribute-0.6.24
sudo python setup.py install

Test:
echo "import distribute" | grep python

== Named Parameters ==

http://diveintopython.org/power_of_introspection/optional_arguments.html

Python allows function arguments to have default values; if the function is called without the argument, the argument gets its default value. Futhermore, arguments can be specified in any order by using named arguments. Stored procedures in SQL Server Transact/SQL can do this, so if you're a SQL Server scripting guru, you can skim this part.

Here is an example of info, a function with two optional arguments:

def info(object, spacing=10, collapse=1):

spacing and collapse are optional, because they have default values defined. object is required, because it has no default value. If info is called with only one argument, spacing defaults to 10 and collapse defaults to 1. If info is called with two arguments, collapse still defaults to 1.

Say you want to specify a value for collapse but want to accept the default value for spacing. In most languages, you would be out of luck, because you would need to call the function with three arguments. But in Python, arguments can be specified by name, in any order.

http://www.wellho.net/mouth/1871_Optional-and-named-parameters-in-Python.html

If you add a parameter with two asterixes (**) on the end of a function definition, that parameter is taken as being the name of a dictionary into which all otherwise unidentified parameters are stored.

<pre>
def getnet(gross, taxrate=17.5, **others):
net = gross / (1.0 + taxrate*0.01)
for more in others.keys():
print "GETNET:",more," - ",others[more]
return neto
</pre>

== Python vs other languages ==

[http://www.strombergers.com/python/ Python is Cool (and Perl is not), Especially for C/C++ Programmers]

==Experimenting with Memory Remanence==
[http://citp.princeton.edu/memory/exp/ Experimenting with Memory Remanence]

<pre>
#!/usr/bin/env python

# a pirate's favorite chemical element
a = ""
while 1: a += "ARGON"
</pre>

sudo strings /dev/mem | less

==UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)==
See [[UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)]]

== Program Layout Best Practices Presentation ==

Program layout best practices - Seth House <seth@eseth.com> - Utah Django User Group - 2011-06-23
* Slides - https://github.com/whiteinge/presentations/tree/master/upyug_2011-06-09_program-layout
* Video - http://blip.tv/utah-open-source/python-layout-best-practices-for-modules-django-cli-utah-django-user-group-5315808

Module skeleton:
<pre>
#!/usr/bin/env python
# coding: utf-8

"""Module docstring"""

__author__ = ’Mr. Me <me@example.net>’
__version__ = ’1.2.3’

if __name__==’__main__’:
main()
</pre>

Executing modules as scripts
# PEP 338:
python -m SimpleHTTPServer

== CLI package ==

cli — command line tools — pyCLI v1.1.1 documentation - http://packages.python.org/pyCLI/

== Python IDE ==

IntegratedDevelopmentEnvironments - PythonInfo Wiki - http://wiki.python.org/moin/IntegratedDevelopmentEnvironments
* IDEs with introspection-based code completion and integrated debugger
* IDEs with introspection-based code completion /or/ integrated debugger
* IDEs with integrated gui builder
* Editors
* Reviews

Is there a good, free Python IDE for Windows? - Stack Overflow - http://stackoverflow.com/questions/126753/is-there-a-good-free-python-ide-for-windows
* Eclipse with PyDev
* Aptana Studio 3
* Active State also offers Komodo Edit
* WingWare - http://www.wingware.com/
* Eric IDE - http://www.die-offenbachs.de/eric/index.html
* MMM-Experts - Products - http://mmm-experts.com/Products.aspx?ProductId=4
** PyScripter is a Python IDE built in Python for Delphi(P4D) components and created with the ambition to become competitive in functionality with commercial Windows-based IDEs available for other languages. Being built in a compiled language is rather snappier than some of the other Python IDEs and provides an extensive blend of features that make it a productive Python development environment.
* ActiveState Python comes with PythonWin which is reasonably good
* PyCharm from JetBrains is not free in general, but has free early access program.
* SPE - Stani's Python Editor, is pretty awesome. See http://pythonide.blogspot.com/.
* You may want to take a look at the pdb module - http://docs.python.org/lib/module-pdb.html
* Geany (http://www.geany.org/) is a fairly light text editor that does a lot of IDE-like stuff.
* SciTE: http://www.scintilla.org/SciTE.html
* Editra: http://editra.org/
* Netbeans has support for python development in their Python Early Access version. - http://www.netbeans.org/features/python/
* I use Notepad++ with the appropriate settings.
* The Eric Python IDE - http://www.die-offenbachs.de/eric/index.html
* BOa Constructor www.boa-constructor.sourceforge.net really nice! drag and drop stuff!
* DrPtyhon is also very nice, and has many plug-ins. http://drpython.sourceforge.net/
* Vim
* I will make a plug for my preference of Python development environments: SPE. SPE is an integrated development environment that provides code completion, integrated Python shell, calltips, and various helper tools like notes and todo lists. It also includes wxGlade so you can make wxPython applications easily. Plus, it includes the python debugger WinPdb.
* http://groups.google.com.tr/group/python_opengl this software is my sample visual python ide.its demo version.
* I use PyScripter, and find it pretty awesome.
* There is one Python IDE written in Java: jHepWork: http://jwork.org/jhepwork/ it can detect Python's indentation rules rather nicely (+ detects structure of a code and syntax)
* gvim built with your python version support and with this plugins:
** NERDTree (don't read he's blog ;)
** matchit (it's builtin in new versions just put a symlink or junction)
** pyflakes
** pysmell
* Try Python Tools for Visual Studio. Supports CPython and IronPython. - http://pytools.codeplex.com/

PythonEditors - PythonInfo Wiki - http://wiki.python.org/moin/PythonEditors
* List of editors

Python Ide - http://c2.com/cgi/wiki?PythonIde
: There are a lot of Ides for PythonLanguage springing up. Many of them written in Python

Is there a better Python IDE? « Lennart Regebro: Python, Plone, Web - http://regebro.wordpress.com/2010/10/09/is-there-a-better-python-ide/
: I’m currently a user of WingIDE, but sometimes I feel the need for something better, mainly something that is a bit more stable. WingIDE often hangs when switching projects, sometimes the file listing decides that you must double-click on the expand arrow instead of just click on it, etc. It still doesn’t have macros despite this being a much requested feature (how hard can it be to add, really?) and I never really get the hang of the SVN integration, so I just ignore it. And they keep recommending me to use WingIDE’s separate instance of GTK instead of the System GTK, which is daft. Their designs are ugly, and it should work with the system GTK so it looks the same. Anything else is just silly.

=== Open Komodo ===

Open Komodo - http://www.openkomodo.com/
: Open Komodo is the source code repository for Komodo Edit - a free multi-language editor for dynamic languages, based on the award-winning Komodo IDE. ActiveState created the Open Komodo code repository in August 2007 and it has been used since then for building all Komodo Edit releases.

=== Eclipse ===

Build an Eclipse development environment for Perl, Python, and PHP - http://www.eclipse.org/resources/resource.php?id=485
: This tutorial shows how Eclipse's DLTK makes it possible to build development tools for scripting languages. In particular, it explains how to implement syntax coloring, user preferences, and interpreter integration in a plug-in-based project.

See [[#PyDev]]

=== EasyEclipse ===

EasyEclipse - http://www.easyeclipse.org/
:EasyEclipse packages together Eclipse, the open-source software
development platform, and selected open source plugins.
:We select, assemble, test, patch, build installers and document
a full IDE, offered as reliable distributions and plugins.
:EasyEclipse is:
* Free and open-source,
* Easy to download and install, and
* Simple to maintain, without version and dependency issues.

EasyEclipse for Python - http://www.easyeclipse.org/site/distributions/python.html

EclipsePythonIntegration - PythonInfo Wiki - http://wiki.python.org/moin/EclipsePythonIntegration
: Using Eclipse as a Python editor

=== PyDev ===

PyDev - http://pydev.org/
: PyDev is a Python IDE for Eclipse, which may be used in Python, Jython and IronPython development.

---

Pydev - http://pydev.sourceforge.net/ (old link)

How to Configure Eclipse for Python - http://www.rose-hulman.edu/class/csse/resources/Eclipse/eclipse-python-configuration.htm

Python Development with PyDev and Eclipse - Tutorial - http://www.vogella.de/articles/Python/article.html
: This article describes how to write and debug Python programs with Eclipse

PyDev Getting Started - http://pydev.org/manual_101_root.html

Running a program - http://pydev.org/manual_101_run.html

---

Supported Task Tags:
* TODO:
* TASK:
* FIXME:

---

Installation:
# Platform Runtime Binary (download around 45-50 MB) - http://download.eclipse.org/eclipse/downloads/ or http://www.eclipse.org/downloads/
#* Example: Eclipse Classic 4.2.2
# Eclipse Software Add - PyDev: http://pydev.org/updates
# Eclipse Software Add - MercurialEclipse: http://cbes.javaforge.com/update [http://javaforge.com/project/HGE] [http://mercurial.selenic.com/wiki/MercurialEclipse]
# Switch perspectives to PyDev

=== The Eric Python IDE ===

The Eric Python IDE - http://eric-ide.python-projects.org/
:Eric is a full featured Python and Ruby editor and IDE, written in python. It is based on the cross platform Qt gui toolkit, integrating the highly flexible Scintilla editor control. It is designed to be usable as everdays' quick and dirty editor as well as being usable as a professional project management tool integrating many advanced features Python offers the professional coder. eric4 includes a plugin system, which allows easy extension of the IDE functionality with plugins downloadable from the net.

Eric Python IDE - Wikipedia - http://en.wikipedia.org/wiki/Eric_Python_IDE

=== Wingware Python IDE ===

The Intelligent Development Environment for Python Programmers - Wingware Python IDE - http://www.wingware.com/
:Wingware's Python IDE is an Integrated Development Environment designed specifically for the Python programming language. Since 1999, Wingware has focused on Python.

=== VIM ===

Python .vimrc:
<pre>
" Kenneth's Preferences

set nobackup " don't create backup files
"set nohlsearch " don't highlight search results
set hlsearch " do highlight search results
set incsearch " incremental search
set ignorecase " ignore case during search
set smartcase " do case sensitive search when upper case is typed
set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
set showmatch " briefly highlight matching bracket
set matchtime=1 " highlight match for 0.1 seconds
set background=dark " black background
set shiftwidth=4 " shift blocks by 4 spaces
set shortmess=at " use short messages so we don't have to hit enter
set expandtab " expand tabs into spaces
set scrolloff=5 " always show 5 lines before & after cursor
set shell=bash " use bash for the command shell
set shellcmdflag=-lc " make bash startup as a login shell for ~/.bashrc
"set nowrap " do not wrap long lines
set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set wildmenu " enable filename tab completion
set ruler " display the ruler: shows cursor position
set cpoptions+=$ " display '$' at end of cw sequence
set title " display filename and vim info in window title bar
set modelines=5 " search the first and last 5 lines for "vim:" modeline
set backspace=2 " backspace deletes

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

"if v:version > 700
" set cursorline " underline the current line
"endif

if &t_Co > 2 || has( "gui_running" )
syntax on
endif

if has( 'mouse' )
set mouse=a " enable mouse
endif

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

"folding settings
set foldmethod=indent "fold based on indent
set foldnestmax=10 "deepest fold is 10 levels
set nofoldenable "dont fold by default
set foldlevel=10 "this is just what i use

" Auto clear trailing white space in Python files
func! DeleteTrailingWS()
exec "normal mz"
%s/\s\+$//ge
exe "normal `z"
endfunc
autocmd BufWrite *.py :call DeleteTrailingWS()
</pre>

----

Older versions:
<pre>
" KENNETH SETTINGS
set nocp " turn off compatibility mode
set nomodeline " do not allow files to dictate vim behavior

set incsearch " incremental search
set ignorecase " ignore case during search

set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set background=dark " black background
syntax on " turn on syntax highlighting
set hlsearch " highligh last search pattern

set nowrap " do not wrap long lines
set scrolloff=5 " always show 5 lines before & after cursor
set ruler " show the cursor position all the time
set backspace=2 " allow backspace deletion
set bs=indent,eol,start " allow backspacing over everything in insert mode

set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set expandtab " expand tab into spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
filetype indent on " turn on auto indent for specific file types
</pre>

Other options:
<pre>
set number " show line numbers
set numberwidth=1 " remove space before largest line numbers

set smartindent " ?? - maybe included in 'filetype indent on'

set ai " always auto indent

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

</pre>

Kenneth's fio .vimrc:
<pre>

"source $VIM/mswin.vim

set nobackup " don't create backup files
"set nohlsearch " don't highlight search results
set incsearch " incremental search
set ignorecase " ignore case during search
set smartcase " do case sensitive search when upper case is typed
set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
set showmatch " briefly highlight matching bracket
set matchtime=1 " highlight match for 0.1 seconds
set background=dark " black background
set shiftwidth=4 " shift blocks by 4 spaces
set shortmess=at " use short messages so we don't have to hit enter
set expandtab " expand tabs into spaces
set scrolloff=5 " always show 5 lines before & after cursor
set shell=bash " use bash for the command shell
set shellcmdflag=-lc " make bash startup as a login shell for ~/.bashrc
"set nowrap " do not wrap long lines
set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set wildmenu " enable filename tab completion
set ruler " display the ruler: shows cursor position
set cpoptions+=$ " display '$' at end of cw sequence
set title " display filename and vim info in window title bar
set modelines=5 " search the first and last 5 lines for "vim:" modeline

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

if v:version > 700
set cursorline " underline the current line
endif

if &t_Co > 2 || has( "gui_running" )
syntax on
endif

if has( 'mouse' )
set mouse=a " enable mouse
endif

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

"folding settings
set foldmethod=indent "fold based on indent
set foldnestmax=10 "deepest fold is 10 levels
set nofoldenable "dont fold by default
set foldlevel=10 "this is just what i use
</pre>

== Interpreter vs Script ==

Interpreter:
>>> import os
>>> os.uname()
('Linux', 'demo.oeey.com', '2.6.18-194.8.1.el5', '#1 SMP Thu Jul 1 19:07:06 EDT 2010', 'i686')

Script:
#!/usr/bin/env python
import os
print os.uname()

== Web Development ==

WebProgramming - PythonInfo Wiki - http://wiki.python.org/moin/WebProgramming
: This topic guide attempts to cover every aspect of programming Web applications (both clients and servers) using Python.

HOWTO Use Python in the web — Python v2.7.2 documentation - http://docs.python.org/howto/webservers.html
: This document shows how Python fits into the web. It presents some ways to integrate Python with a web server, and general practices useful for developing web sites.

=== Plug Simple Python Web App ===

<pre>
-------- Original Message --------
Subject: Re: Writing a simple python web app
Date: Mon, 25 Oct 2010 19:03:31 -0400
From: Roberto Mello <roberto.mello@gmail.com>
To: Provo Linux Users Group <plug@plug.org>

On Sat, Sep 11, 2010 at 5:40 PM, Dave Smith <dave@thesmithfam.org> wrote:
> I'm creating a very simple web application, and I want to use python. My first guess was that mod_python
would provide the easiest entry point. Boy, was I wrong. All the mod_python tutorials spend 80% of their
time extolling the virtues of mod_python over CGI, but they are quite lean on specific examples. It's quite
disappointing. A natural progression then leads me to mod_wsgi, and since that has equally poor
documentation, to Django. However, Django is overkill for my app (I don't even have a database).

For something as simple as you have described, and to run on an
embedded platform, I would probably stay away from frameworks designed
to run on full hardware, and go with something more minimalistic.

Take a look at CherryPy [1] or web.py [2].

[1] http://www.cherrypy.org/
[2] http://webpy.org/

Roberto Mello
</pre>

---

See also [[web.py]]

See also [[CherryPy]]

See also [[Django]]

=== Apache CGI ===

See [[CGI]]

Great tutorial: https://www.tutorialspoint.com/python/python_cgi_programming.htm

My favorite option, make all .cgi files executable: (or .py)
<pre>
DirectoryIndex index.html index.php index.py
<Location />
# AddHandler cgi-script .cgi
AddHandler cgi-script .py
Options +ExecCGI
</Location>
</pre>

If you want to allow .htaccess:
* see [[Apache#.htaccess]]

To enable on Ubuntu:
sudo a2enmod cgi
service apache2 restart

Directory index:
DirectoryIndex index.py

Or if you really want to go crazy, this will try to run *every* file: [http://blog.kapsobor.de/archives/2006/09/05/apache_addhandler_multiple_extensions/]
<pre>
# this has problems with / directory being run as a script, instead of index too!
#<Location />
# SetHandler cgi-script
# Options +ExecCGI
#</Location>

<Files ~ ".">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>

or:
<pre>
<Files ~ "^.[a-zA-Z]*$">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>

Or just have index file:
<Files "index">

Note: Place within a <VirtualHost> section if needed.

Note: Make sure to 'chmod +x' the files.

Start all HTML output with:
print "Content-Type: text/html\n"

If you want just text output:
print "Content-Type: text/plain\n"

GET (from Environment):
<pre>
print "<pre>"
for env in os.environ:
print "%s = %s" % (env, os.environ[env])
print "</pre>"
</pre>

<pre>
# sample for: http://oeey.com/py/test.py/test?var1=ken&var2=barbie

REQUEST_URI = /py/index.py/test?var1=ken&var2=barbie
QUERY_STRING = var1=ken&var2=barbie
PATH_INFO = /test
REQUEST_METHOD = GET

SCRIPT_NAME = /py/test.py
SERVER_NAME = oeey.com
REMOTE_ADDR = 216.51.42.66
SERVER_ADDR = 192.168.108.4
DOCUMENT_ROOT = /www/py
SCRIPT_FILENAME = /www/py/test.py
HTTP_HOST = cgi.oeey.com
</pre>

POST (from stdin):
<pre>
post = sys.stdin.readline()
if post:
print post
</pre>

<pre>
CONTENT_TYPE = application/x-www-form-urlencoded 

yin=one+two+three&yang=four+five+six
</pre>

cgi helper class:
<pre>
import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage()

# field keys
print "fields keys: ", form.keys() # list of field keys

# fild values
if "name" in form: # if form.has_key('name'):
print "field 'name' is: ", form["name"].value
print "field 'name' is: ", form.getvalue('name')

# headers
print "dictionary of headers: ", form.headers
print form.type
print form.headers['content-type']
</pre>

To prevent XSS escape the characters:
print cgi.escape(form.getvalue('name'))

<pre>
import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage()
if (form.has_key("action") and form.has_key("name") \
and form.has_key("age")):
if (form["action"].value == "display"):
print "%s = %s" % (form["name"].value form["age"].value)
</pre>

Note: cgi helper class also does sessions and cookies.

References:
* CGI Tutorial | Web Python - http://webpython.codepoint.net/cgi_tutorial
* Python CGI: An Interactive Instruction - http://www.cs.virginia.edu/~lab2q/

See also:
* Web Python - http://webpython.codepoint.net/

---

REST Example:

<pre>
import os, re
def get_rest():
if not "PATH_INFO" in os.environ:
return ()
# filter for garbage
#print os.environ["PATH_INFO"]
path_info = re.findall('[/A-Za-z0-9_-]*', os.environ["PATH_INFO"])
if not path_info:
return ()
path_info = ''.join(path_info)
# collapse empty paths
paths = path_info.split('/')
final_paths = []
for path in paths:
if path:
final_paths.append(path)
return tuple(final_paths)
rest = get_rest();
</pre>

=== mod_python ===

AVOID! AVOID! AVOID! - if you want this simple, just use CGI!

See [[Apache mod_python]]

mod_python - http://docs.python.org/howto/webservers.html#mod-python

Install:
yum install mod_python

Config:
/etc/httpd/conf.d/python.conf

Simple:
<pre>

</pre>

:"People coming from PHP often find it hard to grasp how to use Python in the web. Their first thought is mostly mod_python, because they think that this is the equivalent to mod_php. Actually, there are many differences. What mod_python does is embed the interpreter into the Apache process, thus speeding up requests by not having to start a Python interpreter for each request. On the other hand, it is not “Python intermixed with HTML” in the way that PHP is often intermixed with HTML. The Python equivalent of that is a template engine. mod_python itself is much more powerful and provides more access to Apache internals. It can emulate CGI, work in a “Python Server Pages” mode (similar to JSP) which is “HTML intermingled with Python”, and it has a “Publisher” which designates one file to accept all requests and decide what to do with them.

:mod_python does have some problems. Unlike the PHP interpreter, the Python interpreter uses caching when executing files, so changes to a file will require the web server to be restarted. Another problem is the basic concept – Apache starts child processes to handle the requests, and unfortunately every child process needs to load the whole Python interpreter even if it does not use it. This makes the whole web server slower. Another problem is that, because mod_python is linked against a specific version of libpython, it is not possible to switch from an older version to a newer (e.g. 2.4 to 2.5) without recompiling mod_python. mod_python is also bound to the Apache web server, so programs written for mod_python cannot easily run on other web servers.

:These are the reasons why mod_python should be avoided when writing new programs. In some circumstances it still might be a good idea to use mod_python for deployment, but WSGI makes it possible to run WSGI programs under mod_python as well."

=== WSGI ===

WSGI - http://docs.python.org/howto/webservers.html#mod-wsgi

: "The Web Server Gateway Interface, or WSGI for short, is defined in PEP 333 and is currently the best way to do Python web programming. While it is great for programmers writing frameworks, a normal web developer does not need to get in direct contact with it. When choosing a framework for web development it is a good idea to choose one which supports WSGI."

See [[mod_wsgi]]

=== Django ===

See [[Django]]

=== CherryPy ===

See [[CherryPy]]

=== web.py ===

web.py: makes web apps

Think about the ideal way to write a web app. Write the code to make it happen.

web.py : Python Package Index - http://pypi.python.org/pypi/web.py

Home Page: http://webpy.org/

"Django lets you write web apps in Django. TurboGears lets you write web apps in TurboGears. Web.py lets you write web apps in Python."
:— Adam Atlas

NOTE: Can modify source code without having to restart app! :-)

---

PIP Install:
pip install web.py

---

Hello World: [http://webpy.org/]
<pre>
import web

urls = (
'/(.*)', 'hello'
)
app = web.application(urls, globals())

class hello:
def GET(self, name):
if not name:
name = 'World'
return 'Hello, ' + name + '!'

if __name__ == "__main__":
app.run()
</pre>

Note: app.py can be renamed.

Running:
python app.py

Note: You can add an IP address/port after the "app.py" bit to control where web.py launches the server. You can also tell it to run a fastcgi or scgi server.
python app.py 1234

---

Templates

<pre>
# templates/hello.html
$def with (name)

$if name:
I just wanted to say hello to $name.
$else:
Hello, world!
</pre>

app = ...
render = web.template.render('templates/')

index.GET:
name = 'Bob'
return render.hello(name) # render.[TEMPLATE_NAME](vars)

get variable: (/?name=Joe)
i = web.input(name=None)
return render.hello(i.name)

variable in url:
urls = ('/(.*)', 'hello')

class index:
def GET(self, name):
return render.hello(name)

# visit: /Joe

multiple variables in url:
urls = ('/(.*)/(.*)', 'hello')

class hello:
def GET(self, fname, lname):
return render.hello(fname, lname)

# visit: /Joe/Smith

=== lpthw.web ===

lpthw.web: a fork of web.py for Learn Python The Hard Way

Locks web.py at a specific version and removes some magic. Thanks to Aaron Swartz for making it originally.

lpthw.web: a fork of web.py for Learn Python The Hard Way - http://pypi.python.org/pypi/lpthw.web

Home Page: http://learnpythonthehardway.org/

This conflicts with web.py, so only install one.

=== web ===

web - Web modules for CGI and WSGI web programming

web : Python Package Index - http://pypi.python.org/pypi/web

Warning: This is an alpha release for those who are interested in
checking progress. No documentation and not for production use.

Auth, session, mail, form and templating tools for writing CGI scripts or WSGI web applications.

This version incompatible with previous versions becuase large portions of the code have been refactored into separate components.

== Issues ==

=== Hide DeprecationWarning ===

==== Import DeprecationWarning ====

Error:
<pre>
/opt/xmpppy-0.4.1/xmpp/auth.py:24: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import sha,base64,random,dispatcher,re
/opt/xmpppy-0.4.1/xmpp/auth.py:26: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5
</pre>

Hide deprecation warnings with code: [http://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python]
<pre>
import warnings

with warnings.catch_warnings():
warnings.filterwarnings("ignore",category=DeprecationWarning)
import md5, sha
</pre>

Hide with header:
#!/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line:
python -W ignore::DeprecationWarning script.py

References:
* How to ignore deprecation warnings in Python - Stack Overflow - http://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python

==== Funciton DeprecationWarning ====

Error:
<pre>
/opt/xmpppy-0.4.1/xmpp/transports.py:307: DeprecationWarning: socket.ssl() is deprecated. Use ssl.wrap_socket() instead.
tcpsock._sslObj = socket.ssl(tcpsock._sock, None, None)
</pre>

Hide with header:
#!/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line:
python -W ignore::DeprecationWarning script.py

==== Ubuntu 24 pip packages ====

<pre>
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
</pre>

== keywords ==

[[Category:Programming]]
[[Category:Python]]

Ubuntu/Python/3.14

2026-04-27T20:06:11Z

Kenneth: /* pip3.14 script */

== Install with DeadSnakes PPA ==

Dependency:
sudo apt install software-properties-common

Add Deadsnakes and install python 3.14
sudo add-apt-repository ppa:deadsnakes/ppa
# respond to prompts
sudo apt update
sudo apt install python3.14
# should also install:
sudo apt install python3.14-dev python3.14-venv

python3.14 --version

For development, virtuan env
sudo apt install python3.14-dev python3.14-venv

For a more complete installation including the Tkinter IDE and other utilities, you can use:
sudo apt install python3.14-full

Pip:
curl -sS https://bootstrap.pypa.io/get-pip.py | python3.14

python3.14 -m pip install pip --upgrade

python3.14 -m pip --version

python3.14 -m pip list

venv: (requires pip)
python3.14 -m venv -h

# check if pip is working
python3.14 -m ensurepip
# /usr/bin/python3.14: No module named ensurepip
# if this fails with the above, install pip and python3.14-venv ^

cd ~ ; rm -rf .test-venv ; mkdir .test-venv ; cd .test-venv ; python3.14 -m venv venv

== pip3.14 script ==

Dead-Snakes doesn't install the pip3.14 script for reasons (as you will see...) You can manually install, by copying the python 3.12 version, and changing the shebang header from python3 to python3.14. Only useful as a convenience for regular users, not for root.

cd /usr/bin
cp pip3.12 pip3.14
sed -i "s/python3/python3.14/" pip3.14

Note, calling pip directly as root will error out, but at least as users you can call it to install local packages
pip3.14 install pip --upgrade
ERROR: Cannot uninstall pip 24.0, RECORD file not found. Hint: The package was installed by debian.

/usr/bin/pip3.14
<pre>
#!/usr/bin/python3.14
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == "__main__":
sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])
sys.exit(main())
</pre>

Alternatively, you could also just create an alias:

alias pip3.14="python3.14 -m pip"

== keywords ==

Ubuntu/Python/3.14

2026-04-27T20:04:56Z

Kenneth: /* pip3.14 script */

== Install with DeadSnakes PPA ==

Dependency:
sudo apt install software-properties-common

Add Deadsnakes and install python 3.14
sudo add-apt-repository ppa:deadsnakes/ppa
# respond to prompts
sudo apt update
sudo apt install python3.14
# should also install:
sudo apt install python3.14-dev python3.14-venv

python3.14 --version

For development, virtuan env
sudo apt install python3.14-dev python3.14-venv

For a more complete installation including the Tkinter IDE and other utilities, you can use:
sudo apt install python3.14-full

Pip:
curl -sS https://bootstrap.pypa.io/get-pip.py | python3.14

python3.14 -m pip install pip --upgrade

python3.14 -m pip --version

python3.14 -m pip list

venv: (requires pip)
python3.14 -m venv -h

# check if pip is working
python3.14 -m ensurepip
# /usr/bin/python3.14: No module named ensurepip
# if this fails with the above, install pip and python3.14-venv ^

cd ~ ; rm -rf .test-venv ; mkdir .test-venv ; cd .test-venv ; python3.14 -m venv venv

== pip3.14 script ==

Dead-Snakes doesn't install the pip3.14 script for reasons... Can manually install, by copying the python 3.12 version, and changing the shebang header from python3 to python3.14

cd /usr/bin
cp pip3.12 pip3.14
sed -i "s/python3/python3.14/" pip3.14

Note, calling pip directly as root will error out, but at least as users you can call it to install local packages
pip3.14 install pip --upgrade
ERROR: Cannot uninstall pip 24.0, RECORD file not found. Hint: The package was installed by debian.

/usr/bin/pip3.14
<pre>
#!/usr/bin/python3.14
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == "__main__":
sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])
sys.exit(main())
</pre>

Alternatively, you could also just create an alias:

alias pip3.14="python3.14 -m pip"

== keywords ==

Ubuntu/Python/3.14

2026-04-27T20:04:17Z

Kenneth: /* pip3.14 script */

== Install with DeadSnakes PPA ==

Dependency:
sudo apt install software-properties-common

Add Deadsnakes and install python 3.14
sudo add-apt-repository ppa:deadsnakes/ppa
# respond to prompts
sudo apt update
sudo apt install python3.14
# should also install:
sudo apt install python3.14-dev python3.14-venv

python3.14 --version

For development, virtuan env
sudo apt install python3.14-dev python3.14-venv

For a more complete installation including the Tkinter IDE and other utilities, you can use:
sudo apt install python3.14-full

Pip:
curl -sS https://bootstrap.pypa.io/get-pip.py | python3.14

python3.14 -m pip install pip --upgrade

python3.14 -m pip --version

python3.14 -m pip list

venv: (requires pip)
python3.14 -m venv -h

# check if pip is working
python3.14 -m ensurepip
# /usr/bin/python3.14: No module named ensurepip
# if this fails with the above, install pip and python3.14-venv ^

cd ~ ; rm -rf .test-venv ; mkdir .test-venv ; cd .test-venv ; python3.14 -m venv venv

== pip3.14 script ==

Dead-Snakes doesn't install the pip3.14 script for reasons... Can manually install, by copying the python 3.12 version, and changing the shebang header from python3 to python3.14

cd /usr/bin
cp pip3.12 pip3.14
sed -i "s/python3/python3.14/" pip3.14

pip3.14 install pip --upgrade
ERROR: Cannot uninstall pip 24.0, RECORD file not found. Hint: The package was installed by debian.

/usr/bin/pip3.14
<pre>
#!/usr/bin/python3.14
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == "__main__":
sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])
sys.exit(main())
</pre>

Alternatively, you could also just create an alias:

alias pip3.14="python3.14 -m pip"

== keywords ==

Ubuntu/Python/3.14

2026-04-27T20:01:53Z

Kenneth: /* pip3.14 script */

== Install with DeadSnakes PPA ==

Dependency:
sudo apt install software-properties-common

Add Deadsnakes and install python 3.14
sudo add-apt-repository ppa:deadsnakes/ppa
# respond to prompts
sudo apt update
sudo apt install python3.14
# should also install:
sudo apt install python3.14-dev python3.14-venv

python3.14 --version

For development, virtuan env
sudo apt install python3.14-dev python3.14-venv

For a more complete installation including the Tkinter IDE and other utilities, you can use:
sudo apt install python3.14-full

Pip:
curl -sS https://bootstrap.pypa.io/get-pip.py | python3.14

python3.14 -m pip install pip --upgrade

python3.14 -m pip --version

python3.14 -m pip list

venv: (requires pip)
python3.14 -m venv -h

# check if pip is working
python3.14 -m ensurepip
# /usr/bin/python3.14: No module named ensurepip
# if this fails with the above, install pip and python3.14-venv ^

cd ~ ; rm -rf .test-venv ; mkdir .test-venv ; cd .test-venv ; python3.14 -m venv venv

== pip3.14 script ==

Dead-Snakes doesn't install the pip3.14 script for reasons... Can manually install, by copying the python 3.12 version, and changing the shebang header from python3 to python3.14

cd /usr/bin
cp pip3.12 pip3.14
sed -i "s/python3/python3.14/" pip3.14

/usr/bin/pip3.14
<pre>
#!/usr/bin/python3.14
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == "__main__":
sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])
sys.exit(main())
</pre>

Alternatively, you could also just create an alias:

alias pip3.14="python3.14 -m pip"

== keywords ==

Apache

2026-04-27T19:22:58Z

Kenneth: /* .htaccess */

{{subpages}}

[[Category:Apache]]

== Install Apache ==

# Debian / Ubuntu
sudo apt install apache2

# Redhat
yum install httpd

== Virtual Hosting ==
touch /etc/httpd/conf.d/virtual.conf

Virtual Host Configuration File:
/etc/httpd/conf.d/virtual.conf:

HTTP Virtual Host
<pre>
NameVirtualHost *:80

<VirtualHost *:80>
ServerName t0e.org
ServerAlias www.t0e.org

DocumentRoot /www

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>

<VirtualHost *:80>
ServerName www2.t0e.org
...
</VirtualHost>
</pre>

HTTPS Virtual Host:
<pre>
NameVirtualHost *:443

<VirtualHost *:443>
ServerName t0e.org
ServerAlias www.t0e.org

# SSL Enabled:
SSLEngine on
SSLCertificateFile /etc/t0e.org.pem

# If using a godaddy certificate chain:
#SSLCertificateFile /etc/contractpal.net.pem
#SSLCertificateKeyFile /etc/contractpal.net.pem
#SSLCertificateChainFile /etc/contractpal.net.pem

#Forwarding Traffic
# RewriteEngine on
# RewriteRule ^(/.*) http://localhost:8080/$1 [P]

DocumentRoot /www

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>
</pre>

References:
*[http://www.onlamp.com/pub/a/apache/2005/02/17/apacheckbk.html A Day in the Life of #Apache: Running Name-Based SSL Virtual Hosts in Apache]
*[http://httpd.apache.org/docs/1.3/vhosts/name-based.html Apache: Name-based Virtual Host Support]
*[http://www.networkworld.com/columnists/2006/011606internet.html Getting SSL to work across virtual hosts in Apache]

== CentOS common configuration changes ==

Keep alives:
# KeepAlive Off
KeepAlive On

Hide Apache version (in likely in /etc/apache2/conf-enabled/security.conf):
# ServerTokens Full
ServerTokens Prod

# ServerSignature On
ServerSignature Off

---

PHP config in /etc/php.ini (now likely in /etc/php/8.3/apache2/php.ini):
expose_php = Off

==Forwarding / Rewriting / Proxying Traffic==

<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName t0e.org
ServerAlias www.t0e.org

#Forwarding Traffic
RewriteEngine on
RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>
</pre>

Apache module mod_rewrite - RewriteRule - http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteRule
a2enmod rewrite

a2enmod proxy_http

[http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html Module mod_rewrite - URL Rewriting Engine]:
*RewriteEngine
*RewriteCond
*RewriteRule

[http://httpd.apache.org/docs/1.3/misc/rewriteguide.html Apache 1.3 - URL Rewriting Guide]

[http://httpd.apache.org/docs/1.3/mod/mod_proxy.html Apache module mod_proxy]:
*ProxyPass

---

<pre>
<VirtualHost *:80>
ServerName home.burgenerfamily.com
DocumentRoot /www/home
<Directory /www/home>
Options FollowSymLinks
RewriteEngine On
# allow existing files and directories to pass through
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
#RewriteRule ^static - [L,NC]
#RewriteCond %{REQUEST_FILENAME} !^favicon.ico$
#RewriteCond %{REQUEST_FILENAME} !^static/
#RewriteRule ^(.+)$ /index.php?title=$1 [PT,L,QSA]
RewriteRule ^(.*)$ /index.php/$1 [L,QSA]
</Directory>
ErrorLog logs/home.burgenerfamily.com-error_log
CustomLog logs/home.burgenerfamily.com-access_log common
</VirtualHost>
</pre>

== Proxy Pass ==

Enable Proxy: [https://www.digitalocean.com/community/tutorials/how-to-use-apache-as-a-reverse-proxy-with-mod_proxy-on-ubuntu-16-04]
sudo a2enmod proxy
sudo a2enmod proxy_http
systemctl restart apache2

Similar to forwarding, but takes care of redirects:

ProxyPass /mirror/foo/ http://foo.com/
ProxyPassReverse /mirror/foo/ http://foo.com/

Suppose the local server has address http://wibble.org/; then

ProxyPass /mirror/foo/ http://foo.com/
ProxyPassReverse /mirror/foo/ http://foo.com/

will not only cause a local request for the <http://wibble.org/mirror/foo/bar> to be internally converted into a proxy request to <http://foo.com/bar> (the functionality ProxyPass provides here). It also takes care of redirects the server foo.com sends: when http://foo.com/bar is redirected by him to http://foo.com/quux Apache adjusts this to http://wibble.org/mirror/foo/quux before forwarding the HTTP redirect response to the client.

<pre>
<VirtualHost *:80>
ServerName wiki.oeey.com

# Reverse Proxy
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/

ServerAdmin website@oeey.com
ErrorLog logs/wiki-error_log
CustomLog logs/wiki-access_log common
</VirtualHost>
</pre>

Proxy Pass to Jenkins: [https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache]
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName jenkins.oeey.com

#Forwarding Traffic
#RewriteEngine on
#RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ProxyPass / http://localhost:8080/ nocanon
ProxyPassReverse / http://localhost:8080/
ProxyRequests Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
<Proxy http://localhost:8080/*>
Order deny,allow
Allow from all
</Proxy>

ServerAdmin admin@oeey.com
ErrorLog logs/jenkins.oeey.com-error_log
CustomLog logs/jenkins.oeey.com-access_log common
</VirtualHost>
</pre>

=== Proxy Pass HTTPS ===

Proxy Pass to Jenkins:
<pre>
NameVirtualHost *:443
<VirtualHost *:443>
ServerName jenkins.oeey.com

# Enable SSL Support
SSLEngine on
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

#Forwarding Traffic
#RewriteEngine on
#RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ProxyPass / http://localhost:8080/ nocanon
ProxyPassReverse / http://localhost:8080/
ProxyPassReverse / http://jenkins.oeey.com/
ProxyRequests Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode
RequestHeader set X-Forwarded-Proto "http"

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
<Proxy http://localhost:8080/jenkins*>
Order deny,allow
Allow from all
</Proxy>

ServerAdmin admin@oeey.com
ErrorLog logs/jenkins.oeey.com-error_log
CustomLog logs/jenkins.oeey.com-access_log common
</VirtualHost>
</pre>

== Redirect ==

Redirect with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
Redirect permanent / http://www.oeey.com/
</VirtualHost>
</pre>

Redirect users to HTTPS with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
Redirect permanent / https://test.oeey.com/
</VirtualHost>
</pre>

Can be wrapped in Location (if you want)
<Location />
Redirect permanent / https://test.oeey.com/
</Location>

Redirect with RewriteEngine with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com

# Redirect HTTP to HTTPS
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>
</pre>

Redirect removing URI:
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
RedirectMatch "(.*)" "http://www.oeey.com/"
</VirtualHost>

[http://httpd.apache.org/docs/2.0/mod/mod_alias.html Apache Module mod_alias]:
*Alias
*Redirect
*RedirectPermanent

== Default Directory Index ==

to set the default pages to try and load (e.g. index.html, index.php):
DirectoryIndex index.html index.php

== Directory Index ==

To show full file names:
<pre>
<Directory />
Options Indexes
IndexOptions NameWidth=*
</Directory>
</pre>

Sample:
<pre>
<Directory "/www">
Options Indexes FollowSymLinks
IndexOptions FancyIndexing NameWidth=*
AllowOverride None
Require all granted
</Directory>
</pre>

== Hide Apache and PHP Banner ==

Hide PHP version (X-Powered-By) - Edit php.ini: [http://www.ducea.com/2006/06/16/apache-tips-tricks-hide-php-version-x-powered-by/]
expose_php = Off

Hide Apache version: [http://www.itsolutionskb.com/2008/10/hide-apache-information/]
# ServerSignature On
ServerSignature Off

# ServerTokens Full
ServerTokens Prod

References:
* Hide Apache ServerSignature/ServerTokens/PHP X-Powered-By - http://www.if-not-true-then-false.com/2009/howto-hide-and-modify-apache-server-information-serversignature-and-servertokens-and-hide-php-version-x-powered-by/

== .htaccess ==

Override configuration settings on a per directory (and sub directories) level with:
.htaccess

Allow override of all configuration options:
<pre>
<Directory /var/www/html>
AllowOverride All
</Directory>
</pre>

Example directory browsing in .htaccess:
Options +indexes

Redirect to index.php:
<pre>
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?path=$1 [NC,L,QSA]
</pre>
a2enmod rewrite
#a2enmod proxy_http

Seems to work for subdirectories:

<pre>
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^.*$ /subdir/index.php?path=$1 [L,QSA]
</pre>

== Allow from networks ==

<pre>
<Directory "...">
Order Deny,Allow
Deny from all
Allow from 127.0.0.0/255.255.255.0 10.10.0.0/16
</Directory>
</pre>

== Log Files ==

ErrorLog logs/test.com-error_log
CustomLog logs/test.com-access_log common

Make sure the soft link exists:
ln -s /var/log/httpd /etc/httpd/logs # RHEL based
ln -s /var/log/apache2 /etc/apache2/logs # Debian Based

*[http://httpd.apache.org/docs/2.0/logs.html Log Files v2.0]
*[http://httpd.apache.org/docs/1.3/logs.html#accesslog Log files v1.3]

===Log HTTPS X-Forwarded-For===
[http://www.webmasterworld.com/forum92/6134.htm Log the X-Forwarded-For] header in apache...
LogFormat "[X-Forwarded-For: %{X-Forwarded-For}i] %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "[X-Forwarded-For: %{X-Forwarded-For}i] %h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

== PHP ==

Install PHP with Apache: (Ubuntu)
#apt-get install apache2 php5 libapache2-mod-php5
apt-get install apache2 php libapache2-mod-php

== Apache SSL ==

Enable SSL with...

Apache can do virtual SSL hosting!

Install mod_ssl: [https://help.ubuntu.com/10.04/serverguide/httpd.html]
# redhat / rhel
sudo yum install mod_ssl

# debian / ubuntu
sudo a2enmod ssl
sudo a2ensite default-ssl
systemctl reload apache2

/etc/httpd/conf.d/ssl.conf:
<pre>
LoadModule ssl_module modules/mod_ssl.so
Listen 443
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
SSLPassPhraseDialog builtin
SSLSessionCache shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout 300
SSLMutex default
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>

Note: Delete the default VirtualHost section so we can add our own.

<pre>
# SSL Configuration - from ssl.conf virtual host example
SSLEngine On
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</pre>

<pre>
# SSL configuration
SSLEngine On
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile /etc/apache2/ssl/cert.crt
SSLCertificateKeyFile /etc/apache2/ssl/cert.key
SSLCertificateChainFile /etc/apache2/ssl/sf_bundle.crt
</pre>

<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName default.t0e.org

# Redirect HTTP to HTTPS
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

NameVirtualHost *:443
<VirtualHost *:443>
ServerName kb.t0e.org

# SSL Enabled: (only once in config file)
SSLEngine on
SSLCertificateFile /etc/t0e.org.pem
SSLCertificateKeyFile /etc/t0e.org.pem
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

DocumentRoot /www

ServerAdmin admin@kb.t0e.org.com
ErrorLog logs/kb.t0e.org-error_log
CustomLog logs/kb.t0e.org-access_log common
</VirtualHost>

<VirtualHost *:443>
ServerName wiki.t0e.org

DocumentRoot /www

ServerAdmin admin@wiki.t0e.org.com
ErrorLog logs/wiki.t0e.org-error_log
CustomLog logs/wiki.t0e.org-access_log common
</VirtualHost>
</pre>

Note: With SSL, Apache will always use the first default virtual hosts' certificate, so it only makes sense to define it there.

References:
*[http://www.apache-ssl.org/ Apache SSL]
*[http://httpd.apache.org/docs/2.0/ssl/ Apache SSL/TLS Encryption Documentation]
*[http://httpd.apache.org/docs/2.0/mod/mod_ssl.html mod_ssl]
*[http://www.securityfocus.com/infocus/1818 Article: Apache 2 with SSL/TLS: Step-by-Step, Part 1]
*[http://www.vanemery.com/Linux/Apache/apache-SSL.html Van's Apache SSL/TLS mini-HOWTO]
*[http://tomcat.apache.org/tomcat-3.3-doc/tomcat-ssl-howto.html Tomcat SSL Configuration]
*[http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html Apache: SSL/TLS Strong Encryption: FAQ]
*[http://www.networkworld.com/columnists/2006/011606internet.html Getting SSL to work across virtual hosts in Apache]
*[http://www.faqs.org/docs/Linux-HOWTO/SSL-RedHat-HOWTO.html Building a Secure RedHat Apache Server HOWTO]
* CommonMisconfigurations - Httpd Wiki - http://wiki.apache.org/httpd/CommonMisconfigurations

--- chain ---

If you have a chain, create the pem as "key, cert, chain" and set Apache config like so:

SSLCertificateFile /etc/ssl.pem
SSLCertificateKeyFile /etc/ssl.pem
SSLCertificateChainFile /etc/ssl.pem

or just

SSLCertificateFile /etc/ssl.pem
SSLCertificateChainFile /etc/ssl.pem

=== Ubuntu SnakeOil Cert ===

Install the snakoil fake cert <ref>https://superuser.com/questions/58149/host-based-snakeoil-certificates-in-ubuntu</ref>

sudo apt-get install ssl-cert

sudo make-ssl-cert generate-default-snakeoil --force-overwrite

/etc/ssl/certs/ssl-cert-snakeoil.pem
/etc/ssl/private/ssl-cert-snakeoil.key

== Linux System Password Protection ==

Using system accounts for HTTP authentication - http://wiki.zs64.net/Using_system_accounts_for_HTTP_authentication

httpd.conf

Edit httpd.conf to add these lines to the general configuration:

LoadModule auth_external_module libexec/apache2/mod_auth_external.so

(Should have been added by installing the port already.)

Add these lines to each virtual host where you'd like to use system accounts for authentication:

AddExternalAuth pwauth /usr/local/bin/pwauth
SetExternalAuthMethod pwauth pipe

Add these lines to each Directory or Location section that you want protected by authentication:

AuthType Basic
AuthExternal pwauth
require valid-user

== Password Protection with .htaccess and htpasswd ==

*[http://www.javascriptkit.com/howto/htaccess3.shtml Comprehensive guide to .htaccess: Password protection]
*[http://apache-server.com/tutorials/ATusing-htaccess.html Using .htaccess Files with Apache]
*[http://www.4webhelp.net/tutorials/misc/htaccess.php Password Protection by Rod]
*[http://www.spoono.com/php/tutorials/tutorial.php?id=30 .htaccess Password Protection in PHP]

Sample .htaccess file:

AuthUserFile /var/www/htpasswd
AuthGroupFile /dev/null
AuthName "Secret Place"
AuthType Basic
Require valid-user

#Require user <user>
#Require group <group>

To use with httpd.conf or virtual.conf, wrap in "<Directory>" tags:
<Directory />
AuthUserFile ...
...
</Directory>

Note: if you want in your httpd.conf (or virtual.conf) surround with "<Location />...</Location>"

To add password file:
htpasswd -c /var/www/htpasswd admin
htpasswd /var/www/htpasswd user1

# if you want to create a blank username with password:
htpasswd /path/htpasswd ''

To use .htaccess:
<Directory />
Options FollowSymLinks
AllowOverride '''AuthConfig'''
</Directory>

Or user the [http://www.webmaster-toolkit.com/htaccess-generator.shtml .htaccess generator]

[http://httpd.apache.org/docs/2.2/mod/core.html#require Apache HTTP Server - Require Directive]:

Require user userid [userid] ...
:Only the named users can access the resource.
Require group group-name [group-name] ...
:Only users in the named groups can access the resource.
Require valid-user
:All valid users can access the resource.

<pre>
AuthType Basic
AuthName "Restricted Resource"
AuthUserFile /web/users
AuthGroupFile /web/groups
Require group admin
</pre>

=== web based htpasswd manager ===

see [[change_htpasswd.pl]]

[http://home.xnet.com/~efflandt/pub/htpasswd.pl htpasswd.cgi by David Efflandt]

Perl CGI Script to Modify User Passwords:

This allows users to manage / change their own passwords.

Use the Perl CGI script [http://home.xnet.com/%7Eefflandt/pub/htpasswd.pl htpasswd.pl] [[http://www.yolinux.com/TUTORIALS/src/htpasswd_pl.txt cache]]

* Edit location of Perl .i.e.: /usr/bin/perl
** Not /usr/local/bin/perl
* Edit the script to specify location of the password file i.e. /var/www/PasswordDir/.htpasswd
* SELinux users must add the correct attribute i.e. chcon -R -h -t httpd_sys_content_t /var/www/PasswordDir
* The password file must be located in a directory where CGI is allowed to modify files.

File: httpd.conf (portion):

<pre>
..
...
<Directory "/var/www/PasswordDir">
Options -Indexes
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
...
..
</pre>

==Directory Browsing==
DocumentRoot /www/files
<Directory />
Options FollowSymLinks '''+Indexes'''
AllowOverride None
</Directory>

Note: The '+' may not be needed.

In a .htaccess:
Options +indexes

Long file names: [http://www.networkworld.com/article/2311687/software/showing-long-file-names-in-apache-directory-listings.html]
IndexOptions NameWidth=*

For root indexes, the welcome.conf may be overriding with the welcome screen. Make the following change: [http://www.linuxquestions.org/questions/linux-general-1/apache-2-root-directory-listing-201356/]
<pre>
<LocationMatch "^/$>
#Options -Indexes
Options Indexes
ErrorDocument 403 /error/noindex.html
</LocationMatch>
</pre>

== cgi ==

Script Alias folder:
ScriptAlias /cgi-bin/ "C:/Program Files/Apache Group/Apache/cgi-bin/"
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

My favorite option, make all .cgi files executable: (This will need to be surrounded with '<Location />' or '<Directory />' tags)
AddHandler cgi-script .cgi
Options +ExecCGI

Note: Make sure to 'chmod +x' the files.

Example:
<pre>
<VirtualHost *:80>
ServerName hg.t0e.org

DirectoryIndex hgweb.cgi
DocumentRoot /www/hg

<Location />
AddHandler cgi-script .cgi
Options +ExecCGI
</Location>

ServerAdmin admin@hg.t0e.org
ErrorLog logs/hg.t0e.org-error_log
CustomLog logs/hg.t0e.org-access_log common
</VirtualHost>
</pre>

Source: [http://www.thesitewizard.com/archive/addcgitoapache.shtml How to Add CGI Script Support (Perl, Python, etc) to Your Apache Server on Windows (thesitewizard.com)]

--

Only files without an extension:
<pre>
<VirtualHost *:80>
ServerName pi.t0e.org
DocumentRoot /www/pi
<Directory />
AllowOverride All
</Directory>
DirectoryIndex index index.php index.html index.htm
<Files ~ "^[a-zA-Z]*$">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>
=== CentOS Default ===

<pre>
#
# ScriptAliases are essentially the same as Aliases, except that
# documents in the realname directory are treated as applications and
# run by the server when requested rather than as documents sent to the client.
# The same rules about trailing "/" apply to ScriptAlias directives as to
# Alias.
#
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/var/www/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
</pre>

== Python CGI ==

See [[Python#Apache_CGI]]

==mod_proxy==
*http://www.apachetutor.org/admin/reverseproxies
*http://www.linuxfocus.org/English/March2000/article147.html
*http://www.linuxhelp.net/guides/apacheproxy/
*http://webauth.stanford.edu/manual/mod/mod_proxy.html
*http://www.apachetutor.org/admin/reverseproxies
*http://confluence.atlassian.com/display/DOC/Using+Apache+with+mod_proxy
*http://www.serverwatch.com/tutorials/article.php/3290851
*http://apache.webthing.com/mod_proxy_html/
*http://www.ibm.com/developerworks/linux/library/wa-lampsec/index.html

== prefork vs worker ==

Prefork performs better. Worker uses less memory. Prefork is most commonly used, and is the better choice for a web server.

To see which is compiled, and look for prefork.c or worker.c:
<pre>
# httpd -l

Compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c
</pre>

== Statistics ==

=== Connections ===

Count active connections:
netstat -ant | grep ESTABLISHED | grep :80 | wc -l

From PHP:
<?php
$number_of_users = shell_exec('netstat -ant | grep ESTABLISHED | grep :80 | wc -l');
echo $number_of_users;
?>

References:
* [http://www.experts-exchange.com/OS/Linux/Q_22825240.html Find out how many connections are on an apache server : apache, connections, many, find, server]

=== [http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status] ===

[http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status - Apache HTTP Server]

Summary

The Status module allows a server administrator to find out how well their server is performing. A HTML page is presented that gives the current server statistics in an easily readable form. If required this page can be made to automatically refresh (given a compatible browser). Another page gives a simple machine-readable list of the current server state.

The details given are:

* The number of worker serving requests
* The number of idle worker
* The status of each worker, the number of requests that worker has performed and the total number of bytes served by the worker (*)
* A total number of accesses and byte count served (*)
* The time the server was started/restarted and the time it has been running for
* Averages giving the number of requests per second, the number of bytes served per second and the average number of bytes per request (*)
* The current percentage CPU used by each worker and in total by Apache (*)
* The current hosts and requests being processed (*)

The lines marked "(*)" are only available if ExtendedStatus is On.

<pre>
LoadModule status_module modules/mod_status.so

#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
#ExtendedStatus On

#
# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Change the ".example.com" to match your domain to enable.
#
#<Location /server-status>
# SetHandler server-status
# Order deny,allow
# Deny from all
# Allow from .example.com
#</Location>
</pre>

References:
* [http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status - Apache HTTP Server]
* [http://www.experts-exchange.com/OS/Linux/Q_22825240.html Find out how many connections are on an apache server : apache, connections, many, find, server]

== TRACE and TRACK ==

ISO Audit may result in this "Medium" risk detected:

:"HTTP Methods Allowed. It is observed that HTTP methods GET HEAD TRACE OPTIONS are allowed on this host. Disable all unnecessary methods."

:"HTTP TRACE / TRACK Methods Allowed. The remote webserver supports the TRACE and/or TRACK methods. TRACE and TRACK are HTTP methods that are used to debug web server connections. Disable these methods."

=== Disabling TRACE ===

[http://rackerhacker.com/2007/08/28/apache-disable-trace-and-track-methods/ Apache: Disable TRACE and TRACK methods | Racker Hacker]:

Lots of PCI Compliance and vulnerability scan vendors will complain about TRACE and TRACK methods being enabled on your server. Since most providers run Nessus, you'll see this fairly often. Here's the rewrite rules to add:

RewriteEngine on
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]

These directives will need to be added to each VirtualHost.

For apache version 1.3.34 (or later 1.3.x versions), or apache 2.0.55 (or later), this has been made easy. Just add the line TraceEnable off

TraceEnable off

[http://publib.boulder.ibm.com/httpserv/ihsdiag/http_trace.html Disabling the HTTP TRACE method]:

The HTTP TRACE request method causes the data received by IBM HTTP Server from the client to be sent back to the client, as in the following example:

<pre>
$ telnet 127.0.0.1 8080
Trying...
Connected to 127.0.0.1.
Escape character is '^]'.
TRACE / HTTP/1.0
Host: foo
A: b
C: d

HTTP/1.1 200 OK
Date: Mon, 04 Oct 2004 14:07:59 GMT
Server: IBM_HTTP_SERVER
Connection: close
Content-Type: message/http

TRACE / HTTP/1.0
A: b
C: d
Host: foo

Connection closed.
</pre>

The TRACE capability could be used by vulnerable or malicious applications to trick a web browser into issuing a TRACE request against an arbitrary site and then send the response to the TRACE to a third party using web browser features.

Making the required configuration changes:

<pre>
<VirtualHost www.example.com>
...
# disable TRACE in the www.example.com virtual host
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]
</VirtualHost>
</pre>

Using SSL:
$ /usr/linux/bin/openssl s_client -connect 127.0.0.1:8444

For more information...

More background information on the concerns with TRACE is provided at http://www.apacheweek.com/issues/03-01-24#news.

Other:
* Apache Tips: Disable the HTTP TRACE method | MDLog:/sysadmin - http://www.ducea.com/2007/10/22/apache-tips-disable-the-http-trace-method/

=== TraceEnable ===

Apache Core Features - http://httpd.apache.org/docs/1.3/mod/core.html#traceenable

TraceEnable:
Syntax: TraceEnable [on|off|extended]
Default: TraceEnable on
Context: server config
Status: core (Windows, NetWare)
Compatibility: Available only in Apache 1.3.34, 2.0.55 and later

This directive overrides the behavior of TRACE for both the core server and mod_proxy. The default TraceEnable on permits TRACE requests per RFC 2616, which disallows any request body to accompany the request. TraceEnable off causes the core server and mod_proxy to return a 405 FORBIDDEN error to the client.

Finally, for testing and diagnostic purposes only, request bodies may be allowed using the non-compliant TraceEnable extended directive. The core (as an origin server) will restrict the request body to 64k (plus 8k for chunk headers if Transfer-Encoding: chunked is used). The core will reflect the full headers and all chunk headers with the request body. As a proxy server, the request body is not restricted to 64k. At this time the Apache 1.3 mod_proxy does not permit chunked request bodies for any request, including the extended TRACE request.

=== Disabling TRACK ===

The HTTP TRACK method

The TRACK method is a type of request supported by Microsoft web servers. It is not RFC compliant and is not supported directly by IBM HTTP Server. The method may be utilized as part of a cross-site scripting attack. See Vulnerability Note VU#288308 for more information.

Even though IBM HTTP Server does not support the TRACK method natively, it is possible for plug-in modules to provide support for it. To disable this capability for plug-in modules, in addition to disabling the TRACE method, add these two additional directives after the existing RewriteCond and RewriteRule directives which are used to disable TRACE:

RewriteCond %{REQUEST_METHOD} ^TRACK
RewriteRule .* - [F]

Here is a full example showing the directives to disable both TRACE and TRACK:

<pre>
<VirtualHost www.example.com>
...
# disable TRACE and TRACK in the www.example.com virtual host
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]
RewriteCond %{REQUEST_METHOD} ^TRACK
RewriteRule .* - [F]
</VirtualHost>
</pre>

=== Attack ===

Cross-Site Tracing issues

Earlier this week a paper was published, [http://www.cgisecurity.com/whitehat-mirror/WH-WhitePaper_XST_ebook.pdf "Cross-Site Tracing"] which gave details of how the TRACE HTTP request could be used in Cross-Site Scripting attacks. Unfortunately this issue has not been very well understood by the media and has received a unwarranted amount of attention.

When an HTTP TRACE request is sent to a web server that supports it, that server will respond echoing the data that is passed to it, including any HTTP headers. The paper explains that some browsers can be scripted to perform a TRACE request. A browser with this functionality could be made to issue a TRACE request against an arbitrary site and pass the results on elsewhere. Since browsers will only send authentication details and cookies to the sites that issue them this means a user having a browser with this functionality could be tricked into sending their cookies or authentication details for arbitrary sites to an attacker.

For example, if you visited a page that an attacker has carefully crafted, the page could cause your browser to bounce a TRACE request against some site for which you have authentication cookies. The result of the TRACE will be a copy of what was sent to the site, which will therefore include those cookies or authentication data. The carefully crafted page can then pass that information on to the attacker.

TRACE requests can be disabled by making a change to the Apache server configuration. Unfortunately it is not possible to do this using the Limit directive since the processing for the TRACE request skips this authorisation checking. Instead the following lines can be added which make use of the mod_rewrite module.

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]

'''Although the particular attack highlighted made use of the TRACE functionality to grab authentication details, this isn't a vulnerability in TRACE, or in the Apache web server.''' The same browser functionality that permits the published attack can be used for different attacks even if TRACE is disabled on the remote web server. For example an attacker could create a carefully crafted page that when visited submits a hidden request to some arbitrary site through your browser, grabs the result and passes it to the attacker.

== Performance Tuning ==

See [[Linux Performance Tuning]]

[http://www.linuxjournal.com/article/6041 Apache and Firewall Performance Tips from the Xenu.net Masters]

=== CentOS January 2009 Thread ===

<pre>
-------- Original Message --------
Subject: [CentOS] Apache Server Tuning for Performance
Date: Wed, 21 Jan 2009 00:09:38 +0530
From: linux-crazy <hicheerup@gmail.com>
Reply-To: CentOS mailing list <centos@centos.org>
To: CentOS mailing list <centos@centos.org>

Hi all,

I am facing facing performance issues with our web servers which is
working for concurrent 250 requests properly and then stops
responding when the requests are more than 250 .

The current configuration parameters are as follows :

apachectl -version
Server version: Apache/2.0.52
Server built: Jan 30 2007 09:56:16

Kernel : 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64
x86_64 x86_64 GNU/Linux

Server Hardware :
MAIN MEMORY
(i) Memory Size 4 GB Dual-Core Intel 5160 processors.

httpd.conf

Timeout 300
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 150

## Server-Pool Size Regulation (MPM specific)
# prefork MPM

# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule prefork.c>

StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 251
MaxClients 251
MaxRequestsPerChild 4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

I want to know about the difference between worker MPM and
Prefork MPM , how to find out which one will be used by my apache
server and the recommended one for highly loaded server.If some one
provide me the link that best explains above two comparison also be
very use full.

Can any one guide me tuning to be done for the maximum utilization
of the Resources and better performance of the Servers.

Regards,
lingu
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
</pre>

<pre>
Only use prefork MPM (worker is for Win32, AFAIK).

If you set ServerLimit to 251, that's the limit.
Set ServerLimit and MaxClients to the same value.
(Larger than 250)
OK?
And take a couple of minutes to study the apache documentation. It's
actually quite good and tranlated into many languages...

BUT: what your server is being able to handle also depends on what you
actually serve (PHP/JSP/Servelets/Perl/whatever!

This is usually not a problem that is easily described and solved in
two sentences.

Rainer
</pre>

<pre>
Increase your MaxClients configuration, also enable if you
haven't already the server-status module and monitor the
server via http://<your server name>/server-status it will
show how many workers are busy and what they are doing.

As for which to use, prefork is the old forked method of
doing things, the other uses threads. Depending on what kind
of application your running on top of apache, if it is not
thread safe(at some point many PHP modules were not thread
safe), you may want to use prefork. Otherwise you can use
the threading model.

If your not sure I'd say stick to prefork to be safe until
you can determine for sure that threading is safe.

nate
</pre>

<pre>
Most list members would likely advise sticking with the prefork
configuration. Without knowing what kind of applications you are
running on your webserver, I wouldn't suggest changing it.

Merely increasing the number of workers might make performance worse.

Use ps or top to figure out how much each apache worker is using. Then
decide how much ram you want to dedicate on your server to Apache,
without going into swap. (Over-allocating and then paging out memory
will only make performance much worse.) For example, if I have 2G or
ram, and I want 1.5 for apache workers, my average apache worker size
(resident memory) is 65MB, then I have room for 23 workers. (1024 * 1.5
) / 65. (There are more accurate ways to calculate this usage, like
taking shared memory into account.)

Upgrading the ram in your web server is a pretty fast interim solution.

Consider your application performance, too. The longer a request in your
application takes, the more workers are in use on your web server,
taking up more memory. If you have long-running queries in your
database, take care of those first.

Good luck

Jed
</pre>

<pre>
Jed Reynolds wrote:
>
> Merely increasing the number of workers might make performance worse.
>
> Use ps or top to figure out how much each apache worker is using. Then
> decide how much ram you want to dedicate on your server to Apache,
> without going into swap. (Over-allocating and then paging out memory
> will only make performance much worse.) For example, if I have 2G or
> ram, and I want 1.5 for apache workers, my average apache worker size
> (resident memory) is 65MB, then I have room for 23 workers. (1024 * 1.5
> ) / 65. (There are more accurate ways to calculate this usage, like
> taking shared memory into account.)

Pay attention to shared memory when doing this. A freshly-forked
process shares virtually all RAM with its parent. How much and how
quickly this changes varies wildly with the application type and
activity that causes the child's data to become unique. With some types
of applications (especially mod_perl) you may want to tune down the
number of hits each child services to increase memory sharing. Also, if
you are running external cgi programs you must take them into account.

> Upgrading the ram in your web server is a pretty fast interim solution.
>
> Consider your application performance, too. The longer a request in your
> application takes, the more workers are in use on your web server,
> taking up more memory. If you have long-running queries in your
> database, take care of those first.

You may also have to turn off or tune down the HTTP 1.1 connection
keepalives, trading the time it takes to establish a new connection for
the RAM it takes to keep the associated process waiting for another
request from the same client.

--
Les Mikesell
lesmikesell@gmail.com
</pre>

<pre>
Also take a look at apache alternatives like lighttpd and nginx. In certain
cases they do miracles.

Jure Pečar
</pre>

<pre>
I don't know about lighttpd (our results were mixed) - but NGINX is
really _very_ fast.
Last time I checked, it seems to be the fastest way to accelerate
webpage-delivery on generic hardware and with OSS-software.

But it's not a feature-monster like apache, so you still need that.

Rainer
</pre>

<pre>
Linux-crazy wrote on Wed, 21 Jan 2009 00:09:38 +0530:

> KeepAliveTimeout 150

My god, reduce this to 10 or 5.

> ServerLimit 251
> MaxClients 251

There's your limit. However, you should check with your hardware if upping
it is really desirable. With 4 GB I think you won't be able to handle much
more anyway.

Kai
</pre>

<pre>
Kai Schaetzl wrote:
> There's your limit. However, you should check with your hardware if upping
> it is really desirable. With 4 GB I think you won't be able to handle much
> more anyway.

That really depends. If you only shove out static pages and have one or
two or three odd cgis on the machine, you can flatten down the httpd
binary quite a bit by throwing out unneeded modules.

500 to 750 clients shouldn't be that much of a problem then ...

Cheers,

Ralph
</pre>

<pre>
Ralph Angenendt wrote on Wed, 21 Jan 2009 10:52:59 +0100:

> That really depends. If you only shove out static pages and have one or
> two or three odd cgis on the machine, you can flatten down the httpd
> binary quite a bit by throwing out unneeded modules.
>
> 500 to 750 clients shouldn't be that much of a problem then ...

Sure it depends ;-) I do serve dynamic pages but by removing the really
unnecessary modules I made my httpds much faster (especially on pipelined
image downloads) and they only have some 10 MB (RES) per worker.

But I would expect that they are running the default set of modules.

Kai
</pre>

<pre>
Kai Schaetzl schrieb:
> Ralph Angenendt wrote on Wed, 21 Jan 2009 10:52:59 +0100:
>
>
>> That really depends. If you only shove out static pages and have one or
>> two or three odd cgis on the machine, you can flatten down the httpd
>> binary quite a bit by throwing out unneeded modules.
>>
>> 500 to 750 clients shouldn't be that much of a problem then ...
>>
>
> Sure it depends ;-) I do serve dynamic pages but by removing the really
> unnecessary modules I made my httpds much faster (especially on pipelined
> image downloads) and they only have some 10 MB (RES) per worker.
>
> But I would expect that they are running the default set of modules.
>
> Kai
>
>

If you can separate-out the images to a specific URL (img.domain.com)
and put them all in the same directory, you can use NGINX to serve them.
Of course, the actual transfer-speed will not increase much - but
latency will go down to the absolute minimum.
And latency is what makes a page appear "fast" or "slow" to customers.
</pre>

<pre>
Ralph Angenendt wrote:
> Kai Schaetzl wrote:
>> There's your limit. However, you should check with your hardware if upping
>> it is really desirable. With 4 GB I think you won't be able to handle much
>> more anyway.
>
> That really depends. If you only shove out static pages and have one or
> two or three odd cgis on the machine, you can flatten down the httpd
> binary quite a bit by throwing out unneeded modules.
>
> 500 to 750 clients shouldn't be that much of a problem then ...

Yeah gotta monitor it.. just checked in on some servers I ran at
my last company and the front end proxies (99% mod_proxy) seem to
peak out traffic wise at about 230 workers. For no other reason other
than because I could each proxy has 8 apache instances running(4 for
HTTP 4 for HTTPS). CPU usage peaks at around 3%(dual proc single core),
memory usage around 800MB. About 100 idle workers. Keepalive was set
to 300 seconds due to poor application design.

One set of back end apache servers peaks traffic wise at around 100
active workers, though memory usage was higher, around 2GB because
of mod_fcgid running ruby on rails. CPU usage seems to be at around
25%(dual proc quad core) for that one application.

Each back end application had it's own dedicated apache instance(11
apps) for maximum stability/best performance monitoring. The bulk
of them ran on the same physical hardware though.

Traffic routing was handled by a combination of F5 load balancers
and the apache servers previously mentioned(F5 iRules were too slow
and F5's TMM was not scalable at the time).

nate
</pre>

<pre>
>
> > KeepAliveTimeout 150
>
>reduce this to 10 or 5.
>
>
> Kai
>
> --
> Kai Schätzl, Berlin, Germany

Kai,

what do you think about the "general Timeout

it is set to 300

ive never much thought about it, yet should we be consider and possible
reduce that one too?

- rh
</pre>

<pre>
RobertH wrote on Wed, 21 Jan 2009 11:26:41 -0800:

> what do you think about the "general Timeout
>
> it is set to 300
>
> ive never much thought about it, yet should we be consider and possible
> reduce that one too?

I'm using 120, but I don't think reducing this value has much impact.

Kai
</pre>

== Load Balancer ==

Apache Load Balancer

mod_proxy_balancer - Apache HTTP Server - mod_proxy_balancer - http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html

<pre>
<Proxy balancer://mycluster>
BalancerMember http://192.168.1.50:80
BalancerMember http://192.168.1.51:80
</Proxy>
ProxyPass /test balancer://mycluster
</pre>

== User Directories ==

Default:
UserDir public_html

Results in:
~/public_html/

To disable user directories:
UserDir disabled

Ubuntu user dir config:
<pre>
UserDir public_html
UserDir disabled root

<Directory /home/*/public_html>
AllowOverride FileInfo AuthConfig Limit Indexes
Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
<Limit GET POST OPTIONS>
Order allow,deny
Allow from all
</Limit>
<LimitExcept GET POST OPTIONS>
Order deny,allow
Deny from all
</LimitExcept>
</Directory>

</pre>

References:
* Per-user web directories - Apache HTTP Server - http://httpd.apache.org/docs/2.2/howto/public_html.html
* mod_userdir - Apache HTTP Server - http://httpd.apache.org/docs/2.2/mod/mod_userdir.html#userdir

== Minimal Memory Instance ==

Assuming running prefork module (check with 'apache -l'):

<pre>
KeepAlive off

<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 5
ServerLimit 20
MaxClients 20
MaxRequestsPerChild 10000
</IfModule>
</pre>

References:
* Running Apache On A Memory-Constrained VPS | Kalzumeus Software - http://www.kalzumeus.com/2010/06/19/running-apache-on-a-memory-constrained-vps/

---

TODO See:
* http://wiki.vpslink.com/Low_memory_MySQL_/_Apache_configurations
* http://www.narga.net/optimizing-apachephpmysql-low-memory-server/

== LDAP Authentication ==

# Red Hat / RHEL
sudo yum install mod_ldap
# allow ldap to connect through selinux:
sudo setsebool -P httpd_can_network_connect on

mod_ldap - Apache HTTP Server Version 2.4
https://httpd.apache.org/docs/2.4/mod/mod_ldap.html

== Issues ==

=== Could not reliably determine domain name ===

Warning:
Could not reliably determine the server's fully qualified domain name

Solution:
* Force ServerName in apache configs [http://askubuntu.com/questions/256013/could-not-reliably-determine-the-servers-fully-qualified-domain-name]
ServerName localhost

=== client denied by server configuration ===

browser:
403 Forbidden - You don't have permission to access / on this server.

Error:
<pre>
==> httpd/default-error_log <==
[Mon Sep 01 21:42:02.273169 2014] [authz_core:error] [pid 11221] [client 174.52.47.54:50675] AH01630: client denied by server configuration: /www/
</pre>

Add Directory allow permissions: [http://stackoverflow.com/questions/19445686/ubuntu-server-apache-2-4-6-client-denied-by-server-configuration-php-fpm]
<pre>
DocumentRoot /www
<Directory /www>
Order allow,deny
Allow from all
</Directory>
</pre>

or this worked better for me: [http://stackoverflow.com/questions/19445686/ubuntu-server-apache-2-4-6-client-denied-by-server-configuration-php-fpm] [https://mikegriffin.ie/blog/20140130-authz-core-error-client-denied-by-server-configuration/]
<pre>
DocumentRoot /www
<Directory /www>
Require all granted
</Directory>
</pre>

Actually this just worked on the latest Ubuntu system:
<pre>
<Directory /www/>
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>
</pre>

References:
* ClientDeniedByServerConfiguration - Httpd Wiki - https://wiki.apache.org/httpd/ClientDeniedByServerConfiguration

=== permissions missing component path ===

<pre>
==> httpd/default-error_log <==
[Mon Sep 01 21:50:09.256120 2014] [core:error] [pid 11511] (13)Permission denied: [client 174.52.47.54:50744] AH00035: access to \
/index.html denied (filesystem path '/www/index.html') because search permissions are missing on a component of the path
</pre>

== keywords ==

[[Category:Linux]]
[[Category:Web Development ‎]]
[[Category:Web]]
[[Category:HTML]]
[[Category:Internet]]

Apache

2026-04-27T19:21:55Z

Kenneth: /* .htaccess */

{{subpages}}

[[Category:Apache]]

== Install Apache ==

# Debian / Ubuntu
sudo apt install apache2

# Redhat
yum install httpd

== Virtual Hosting ==
touch /etc/httpd/conf.d/virtual.conf

Virtual Host Configuration File:
/etc/httpd/conf.d/virtual.conf:

HTTP Virtual Host
<pre>
NameVirtualHost *:80

<VirtualHost *:80>
ServerName t0e.org
ServerAlias www.t0e.org

DocumentRoot /www

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>

<VirtualHost *:80>
ServerName www2.t0e.org
...
</VirtualHost>
</pre>

HTTPS Virtual Host:
<pre>
NameVirtualHost *:443

<VirtualHost *:443>
ServerName t0e.org
ServerAlias www.t0e.org

# SSL Enabled:
SSLEngine on
SSLCertificateFile /etc/t0e.org.pem

# If using a godaddy certificate chain:
#SSLCertificateFile /etc/contractpal.net.pem
#SSLCertificateKeyFile /etc/contractpal.net.pem
#SSLCertificateChainFile /etc/contractpal.net.pem

#Forwarding Traffic
# RewriteEngine on
# RewriteRule ^(/.*) http://localhost:8080/$1 [P]

DocumentRoot /www

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>
</pre>

References:
*[http://www.onlamp.com/pub/a/apache/2005/02/17/apacheckbk.html A Day in the Life of #Apache: Running Name-Based SSL Virtual Hosts in Apache]
*[http://httpd.apache.org/docs/1.3/vhosts/name-based.html Apache: Name-based Virtual Host Support]
*[http://www.networkworld.com/columnists/2006/011606internet.html Getting SSL to work across virtual hosts in Apache]

== CentOS common configuration changes ==

Keep alives:
# KeepAlive Off
KeepAlive On

Hide Apache version (in likely in /etc/apache2/conf-enabled/security.conf):
# ServerTokens Full
ServerTokens Prod

# ServerSignature On
ServerSignature Off

---

PHP config in /etc/php.ini (now likely in /etc/php/8.3/apache2/php.ini):
expose_php = Off

==Forwarding / Rewriting / Proxying Traffic==

<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName t0e.org
ServerAlias www.t0e.org

#Forwarding Traffic
RewriteEngine on
RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>
</pre>

Apache module mod_rewrite - RewriteRule - http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteRule
a2enmod rewrite

a2enmod proxy_http

[http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html Module mod_rewrite - URL Rewriting Engine]:
*RewriteEngine
*RewriteCond
*RewriteRule

[http://httpd.apache.org/docs/1.3/misc/rewriteguide.html Apache 1.3 - URL Rewriting Guide]

[http://httpd.apache.org/docs/1.3/mod/mod_proxy.html Apache module mod_proxy]:
*ProxyPass

---

<pre>
<VirtualHost *:80>
ServerName home.burgenerfamily.com
DocumentRoot /www/home
<Directory /www/home>
Options FollowSymLinks
RewriteEngine On
# allow existing files and directories to pass through
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
#RewriteRule ^static - [L,NC]
#RewriteCond %{REQUEST_FILENAME} !^favicon.ico$
#RewriteCond %{REQUEST_FILENAME} !^static/
#RewriteRule ^(.+)$ /index.php?title=$1 [PT,L,QSA]
RewriteRule ^(.*)$ /index.php/$1 [L,QSA]
</Directory>
ErrorLog logs/home.burgenerfamily.com-error_log
CustomLog logs/home.burgenerfamily.com-access_log common
</VirtualHost>
</pre>

== Proxy Pass ==

Enable Proxy: [https://www.digitalocean.com/community/tutorials/how-to-use-apache-as-a-reverse-proxy-with-mod_proxy-on-ubuntu-16-04]
sudo a2enmod proxy
sudo a2enmod proxy_http
systemctl restart apache2

Similar to forwarding, but takes care of redirects:

ProxyPass /mirror/foo/ http://foo.com/
ProxyPassReverse /mirror/foo/ http://foo.com/

Suppose the local server has address http://wibble.org/; then

ProxyPass /mirror/foo/ http://foo.com/
ProxyPassReverse /mirror/foo/ http://foo.com/

will not only cause a local request for the <http://wibble.org/mirror/foo/bar> to be internally converted into a proxy request to <http://foo.com/bar> (the functionality ProxyPass provides here). It also takes care of redirects the server foo.com sends: when http://foo.com/bar is redirected by him to http://foo.com/quux Apache adjusts this to http://wibble.org/mirror/foo/quux before forwarding the HTTP redirect response to the client.

<pre>
<VirtualHost *:80>
ServerName wiki.oeey.com

# Reverse Proxy
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/

ServerAdmin website@oeey.com
ErrorLog logs/wiki-error_log
CustomLog logs/wiki-access_log common
</VirtualHost>
</pre>

Proxy Pass to Jenkins: [https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache]
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName jenkins.oeey.com

#Forwarding Traffic
#RewriteEngine on
#RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ProxyPass / http://localhost:8080/ nocanon
ProxyPassReverse / http://localhost:8080/
ProxyRequests Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
<Proxy http://localhost:8080/*>
Order deny,allow
Allow from all
</Proxy>

ServerAdmin admin@oeey.com
ErrorLog logs/jenkins.oeey.com-error_log
CustomLog logs/jenkins.oeey.com-access_log common
</VirtualHost>
</pre>

=== Proxy Pass HTTPS ===

Proxy Pass to Jenkins:
<pre>
NameVirtualHost *:443
<VirtualHost *:443>
ServerName jenkins.oeey.com

# Enable SSL Support
SSLEngine on
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

#Forwarding Traffic
#RewriteEngine on
#RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ProxyPass / http://localhost:8080/ nocanon
ProxyPassReverse / http://localhost:8080/
ProxyPassReverse / http://jenkins.oeey.com/
ProxyRequests Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode
RequestHeader set X-Forwarded-Proto "http"

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
<Proxy http://localhost:8080/jenkins*>
Order deny,allow
Allow from all
</Proxy>

ServerAdmin admin@oeey.com
ErrorLog logs/jenkins.oeey.com-error_log
CustomLog logs/jenkins.oeey.com-access_log common
</VirtualHost>
</pre>

== Redirect ==

Redirect with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
Redirect permanent / http://www.oeey.com/
</VirtualHost>
</pre>

Redirect users to HTTPS with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
Redirect permanent / https://test.oeey.com/
</VirtualHost>
</pre>

Can be wrapped in Location (if you want)
<Location />
Redirect permanent / https://test.oeey.com/
</Location>

Redirect with RewriteEngine with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com

# Redirect HTTP to HTTPS
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>
</pre>

Redirect removing URI:
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
RedirectMatch "(.*)" "http://www.oeey.com/"
</VirtualHost>

[http://httpd.apache.org/docs/2.0/mod/mod_alias.html Apache Module mod_alias]:
*Alias
*Redirect
*RedirectPermanent

== Default Directory Index ==

to set the default pages to try and load (e.g. index.html, index.php):
DirectoryIndex index.html index.php

== Directory Index ==

To show full file names:
<pre>
<Directory />
Options Indexes
IndexOptions NameWidth=*
</Directory>
</pre>

Sample:
<pre>
<Directory "/www">
Options Indexes FollowSymLinks
IndexOptions FancyIndexing NameWidth=*
AllowOverride None
Require all granted
</Directory>
</pre>

== Hide Apache and PHP Banner ==

Hide PHP version (X-Powered-By) - Edit php.ini: [http://www.ducea.com/2006/06/16/apache-tips-tricks-hide-php-version-x-powered-by/]
expose_php = Off

Hide Apache version: [http://www.itsolutionskb.com/2008/10/hide-apache-information/]
# ServerSignature On
ServerSignature Off

# ServerTokens Full
ServerTokens Prod

References:
* Hide Apache ServerSignature/ServerTokens/PHP X-Powered-By - http://www.if-not-true-then-false.com/2009/howto-hide-and-modify-apache-server-information-serversignature-and-servertokens-and-hide-php-version-x-powered-by/

== .htaccess ==

Override configuration settings on a per directory (and sub directories) level with:
.htaccess

Allow override of all configuration options:
<pre>
<Directory /var/www/html>
AllowOverride All
</Directory>
</pre>

Example directory browsing in .htaccess:
Options +indexes

Redirect to index.php:
<pre>
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?path=$1 [NC,L,QSA]
</pre>
a2enmod rewrite
#a2enmod proxy_http

== Allow from networks ==

<pre>
<Directory "...">
Order Deny,Allow
Deny from all
Allow from 127.0.0.0/255.255.255.0 10.10.0.0/16
</Directory>
</pre>

== Log Files ==

ErrorLog logs/test.com-error_log
CustomLog logs/test.com-access_log common

Make sure the soft link exists:
ln -s /var/log/httpd /etc/httpd/logs # RHEL based
ln -s /var/log/apache2 /etc/apache2/logs # Debian Based

*[http://httpd.apache.org/docs/2.0/logs.html Log Files v2.0]
*[http://httpd.apache.org/docs/1.3/logs.html#accesslog Log files v1.3]

===Log HTTPS X-Forwarded-For===
[http://www.webmasterworld.com/forum92/6134.htm Log the X-Forwarded-For] header in apache...
LogFormat "[X-Forwarded-For: %{X-Forwarded-For}i] %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "[X-Forwarded-For: %{X-Forwarded-For}i] %h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

== PHP ==

Install PHP with Apache: (Ubuntu)
#apt-get install apache2 php5 libapache2-mod-php5
apt-get install apache2 php libapache2-mod-php

== Apache SSL ==

Enable SSL with...

Apache can do virtual SSL hosting!

Install mod_ssl: [https://help.ubuntu.com/10.04/serverguide/httpd.html]
# redhat / rhel
sudo yum install mod_ssl

# debian / ubuntu
sudo a2enmod ssl
sudo a2ensite default-ssl
systemctl reload apache2

/etc/httpd/conf.d/ssl.conf:
<pre>
LoadModule ssl_module modules/mod_ssl.so
Listen 443
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
SSLPassPhraseDialog builtin
SSLSessionCache shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout 300
SSLMutex default
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>

Note: Delete the default VirtualHost section so we can add our own.

<pre>
# SSL Configuration - from ssl.conf virtual host example
SSLEngine On
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</pre>

<pre>
# SSL configuration
SSLEngine On
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile /etc/apache2/ssl/cert.crt
SSLCertificateKeyFile /etc/apache2/ssl/cert.key
SSLCertificateChainFile /etc/apache2/ssl/sf_bundle.crt
</pre>

<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName default.t0e.org

# Redirect HTTP to HTTPS
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

NameVirtualHost *:443
<VirtualHost *:443>
ServerName kb.t0e.org

# SSL Enabled: (only once in config file)
SSLEngine on
SSLCertificateFile /etc/t0e.org.pem
SSLCertificateKeyFile /etc/t0e.org.pem
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

DocumentRoot /www

ServerAdmin admin@kb.t0e.org.com
ErrorLog logs/kb.t0e.org-error_log
CustomLog logs/kb.t0e.org-access_log common
</VirtualHost>

<VirtualHost *:443>
ServerName wiki.t0e.org

DocumentRoot /www

ServerAdmin admin@wiki.t0e.org.com
ErrorLog logs/wiki.t0e.org-error_log
CustomLog logs/wiki.t0e.org-access_log common
</VirtualHost>
</pre>

Note: With SSL, Apache will always use the first default virtual hosts' certificate, so it only makes sense to define it there.

References:
*[http://www.apache-ssl.org/ Apache SSL]
*[http://httpd.apache.org/docs/2.0/ssl/ Apache SSL/TLS Encryption Documentation]
*[http://httpd.apache.org/docs/2.0/mod/mod_ssl.html mod_ssl]
*[http://www.securityfocus.com/infocus/1818 Article: Apache 2 with SSL/TLS: Step-by-Step, Part 1]
*[http://www.vanemery.com/Linux/Apache/apache-SSL.html Van's Apache SSL/TLS mini-HOWTO]
*[http://tomcat.apache.org/tomcat-3.3-doc/tomcat-ssl-howto.html Tomcat SSL Configuration]
*[http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html Apache: SSL/TLS Strong Encryption: FAQ]
*[http://www.networkworld.com/columnists/2006/011606internet.html Getting SSL to work across virtual hosts in Apache]
*[http://www.faqs.org/docs/Linux-HOWTO/SSL-RedHat-HOWTO.html Building a Secure RedHat Apache Server HOWTO]
* CommonMisconfigurations - Httpd Wiki - http://wiki.apache.org/httpd/CommonMisconfigurations

--- chain ---

If you have a chain, create the pem as "key, cert, chain" and set Apache config like so:

SSLCertificateFile /etc/ssl.pem
SSLCertificateKeyFile /etc/ssl.pem
SSLCertificateChainFile /etc/ssl.pem

or just

SSLCertificateFile /etc/ssl.pem
SSLCertificateChainFile /etc/ssl.pem

=== Ubuntu SnakeOil Cert ===

Install the snakoil fake cert <ref>https://superuser.com/questions/58149/host-based-snakeoil-certificates-in-ubuntu</ref>

sudo apt-get install ssl-cert

sudo make-ssl-cert generate-default-snakeoil --force-overwrite

/etc/ssl/certs/ssl-cert-snakeoil.pem
/etc/ssl/private/ssl-cert-snakeoil.key

== Linux System Password Protection ==

Using system accounts for HTTP authentication - http://wiki.zs64.net/Using_system_accounts_for_HTTP_authentication

httpd.conf

Edit httpd.conf to add these lines to the general configuration:

LoadModule auth_external_module libexec/apache2/mod_auth_external.so

(Should have been added by installing the port already.)

Add these lines to each virtual host where you'd like to use system accounts for authentication:

AddExternalAuth pwauth /usr/local/bin/pwauth
SetExternalAuthMethod pwauth pipe

Add these lines to each Directory or Location section that you want protected by authentication:

AuthType Basic
AuthExternal pwauth
require valid-user

== Password Protection with .htaccess and htpasswd ==

*[http://www.javascriptkit.com/howto/htaccess3.shtml Comprehensive guide to .htaccess: Password protection]
*[http://apache-server.com/tutorials/ATusing-htaccess.html Using .htaccess Files with Apache]
*[http://www.4webhelp.net/tutorials/misc/htaccess.php Password Protection by Rod]
*[http://www.spoono.com/php/tutorials/tutorial.php?id=30 .htaccess Password Protection in PHP]

Sample .htaccess file:

AuthUserFile /var/www/htpasswd
AuthGroupFile /dev/null
AuthName "Secret Place"
AuthType Basic
Require valid-user

#Require user <user>
#Require group <group>

To use with httpd.conf or virtual.conf, wrap in "<Directory>" tags:
<Directory />
AuthUserFile ...
...
</Directory>

Note: if you want in your httpd.conf (or virtual.conf) surround with "<Location />...</Location>"

To add password file:
htpasswd -c /var/www/htpasswd admin
htpasswd /var/www/htpasswd user1

# if you want to create a blank username with password:
htpasswd /path/htpasswd ''

To use .htaccess:
<Directory />
Options FollowSymLinks
AllowOverride '''AuthConfig'''
</Directory>

Or user the [http://www.webmaster-toolkit.com/htaccess-generator.shtml .htaccess generator]

[http://httpd.apache.org/docs/2.2/mod/core.html#require Apache HTTP Server - Require Directive]:

Require user userid [userid] ...
:Only the named users can access the resource.
Require group group-name [group-name] ...
:Only users in the named groups can access the resource.
Require valid-user
:All valid users can access the resource.

<pre>
AuthType Basic
AuthName "Restricted Resource"
AuthUserFile /web/users
AuthGroupFile /web/groups
Require group admin
</pre>

=== web based htpasswd manager ===

see [[change_htpasswd.pl]]

[http://home.xnet.com/~efflandt/pub/htpasswd.pl htpasswd.cgi by David Efflandt]

Perl CGI Script to Modify User Passwords:

This allows users to manage / change their own passwords.

Use the Perl CGI script [http://home.xnet.com/%7Eefflandt/pub/htpasswd.pl htpasswd.pl] [[http://www.yolinux.com/TUTORIALS/src/htpasswd_pl.txt cache]]

* Edit location of Perl .i.e.: /usr/bin/perl
** Not /usr/local/bin/perl
* Edit the script to specify location of the password file i.e. /var/www/PasswordDir/.htpasswd
* SELinux users must add the correct attribute i.e. chcon -R -h -t httpd_sys_content_t /var/www/PasswordDir
* The password file must be located in a directory where CGI is allowed to modify files.

File: httpd.conf (portion):

<pre>
..
...
<Directory "/var/www/PasswordDir">
Options -Indexes
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
...
..
</pre>

==Directory Browsing==
DocumentRoot /www/files
<Directory />
Options FollowSymLinks '''+Indexes'''
AllowOverride None
</Directory>

Note: The '+' may not be needed.

In a .htaccess:
Options +indexes

Long file names: [http://www.networkworld.com/article/2311687/software/showing-long-file-names-in-apache-directory-listings.html]
IndexOptions NameWidth=*

For root indexes, the welcome.conf may be overriding with the welcome screen. Make the following change: [http://www.linuxquestions.org/questions/linux-general-1/apache-2-root-directory-listing-201356/]
<pre>
<LocationMatch "^/$>
#Options -Indexes
Options Indexes
ErrorDocument 403 /error/noindex.html
</LocationMatch>
</pre>

== cgi ==

Script Alias folder:
ScriptAlias /cgi-bin/ "C:/Program Files/Apache Group/Apache/cgi-bin/"
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

My favorite option, make all .cgi files executable: (This will need to be surrounded with '<Location />' or '<Directory />' tags)
AddHandler cgi-script .cgi
Options +ExecCGI

Note: Make sure to 'chmod +x' the files.

Example:
<pre>
<VirtualHost *:80>
ServerName hg.t0e.org

DirectoryIndex hgweb.cgi
DocumentRoot /www/hg

<Location />
AddHandler cgi-script .cgi
Options +ExecCGI
</Location>

ServerAdmin admin@hg.t0e.org
ErrorLog logs/hg.t0e.org-error_log
CustomLog logs/hg.t0e.org-access_log common
</VirtualHost>
</pre>

Source: [http://www.thesitewizard.com/archive/addcgitoapache.shtml How to Add CGI Script Support (Perl, Python, etc) to Your Apache Server on Windows (thesitewizard.com)]

--

Only files without an extension:
<pre>
<VirtualHost *:80>
ServerName pi.t0e.org
DocumentRoot /www/pi
<Directory />
AllowOverride All
</Directory>
DirectoryIndex index index.php index.html index.htm
<Files ~ "^[a-zA-Z]*$">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>
=== CentOS Default ===

<pre>
#
# ScriptAliases are essentially the same as Aliases, except that
# documents in the realname directory are treated as applications and
# run by the server when requested rather than as documents sent to the client.
# The same rules about trailing "/" apply to ScriptAlias directives as to
# Alias.
#
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/var/www/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
</pre>

== Python CGI ==

See [[Python#Apache_CGI]]

==mod_proxy==
*http://www.apachetutor.org/admin/reverseproxies
*http://www.linuxfocus.org/English/March2000/article147.html
*http://www.linuxhelp.net/guides/apacheproxy/
*http://webauth.stanford.edu/manual/mod/mod_proxy.html
*http://www.apachetutor.org/admin/reverseproxies
*http://confluence.atlassian.com/display/DOC/Using+Apache+with+mod_proxy
*http://www.serverwatch.com/tutorials/article.php/3290851
*http://apache.webthing.com/mod_proxy_html/
*http://www.ibm.com/developerworks/linux/library/wa-lampsec/index.html

== prefork vs worker ==

Prefork performs better. Worker uses less memory. Prefork is most commonly used, and is the better choice for a web server.

To see which is compiled, and look for prefork.c or worker.c:
<pre>
# httpd -l

Compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c
</pre>

== Statistics ==

=== Connections ===

Count active connections:
netstat -ant | grep ESTABLISHED | grep :80 | wc -l

From PHP:
<?php
$number_of_users = shell_exec('netstat -ant | grep ESTABLISHED | grep :80 | wc -l');
echo $number_of_users;
?>

References:
* [http://www.experts-exchange.com/OS/Linux/Q_22825240.html Find out how many connections are on an apache server : apache, connections, many, find, server]

=== [http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status] ===

[http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status - Apache HTTP Server]

Summary

The Status module allows a server administrator to find out how well their server is performing. A HTML page is presented that gives the current server statistics in an easily readable form. If required this page can be made to automatically refresh (given a compatible browser). Another page gives a simple machine-readable list of the current server state.

The details given are:

* The number of worker serving requests
* The number of idle worker
* The status of each worker, the number of requests that worker has performed and the total number of bytes served by the worker (*)
* A total number of accesses and byte count served (*)
* The time the server was started/restarted and the time it has been running for
* Averages giving the number of requests per second, the number of bytes served per second and the average number of bytes per request (*)
* The current percentage CPU used by each worker and in total by Apache (*)
* The current hosts and requests being processed (*)

The lines marked "(*)" are only available if ExtendedStatus is On.

<pre>
LoadModule status_module modules/mod_status.so

#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
#ExtendedStatus On

#
# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Change the ".example.com" to match your domain to enable.
#
#<Location /server-status>
# SetHandler server-status
# Order deny,allow
# Deny from all
# Allow from .example.com
#</Location>
</pre>

References:
* [http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status - Apache HTTP Server]
* [http://www.experts-exchange.com/OS/Linux/Q_22825240.html Find out how many connections are on an apache server : apache, connections, many, find, server]

== TRACE and TRACK ==

ISO Audit may result in this "Medium" risk detected:

:"HTTP Methods Allowed. It is observed that HTTP methods GET HEAD TRACE OPTIONS are allowed on this host. Disable all unnecessary methods."

:"HTTP TRACE / TRACK Methods Allowed. The remote webserver supports the TRACE and/or TRACK methods. TRACE and TRACK are HTTP methods that are used to debug web server connections. Disable these methods."

=== Disabling TRACE ===

[http://rackerhacker.com/2007/08/28/apache-disable-trace-and-track-methods/ Apache: Disable TRACE and TRACK methods | Racker Hacker]:

Lots of PCI Compliance and vulnerability scan vendors will complain about TRACE and TRACK methods being enabled on your server. Since most providers run Nessus, you'll see this fairly often. Here's the rewrite rules to add:

RewriteEngine on
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]

These directives will need to be added to each VirtualHost.

For apache version 1.3.34 (or later 1.3.x versions), or apache 2.0.55 (or later), this has been made easy. Just add the line TraceEnable off

TraceEnable off

[http://publib.boulder.ibm.com/httpserv/ihsdiag/http_trace.html Disabling the HTTP TRACE method]:

The HTTP TRACE request method causes the data received by IBM HTTP Server from the client to be sent back to the client, as in the following example:

<pre>
$ telnet 127.0.0.1 8080
Trying...
Connected to 127.0.0.1.
Escape character is '^]'.
TRACE / HTTP/1.0
Host: foo
A: b
C: d

HTTP/1.1 200 OK
Date: Mon, 04 Oct 2004 14:07:59 GMT
Server: IBM_HTTP_SERVER
Connection: close
Content-Type: message/http

TRACE / HTTP/1.0
A: b
C: d
Host: foo

Connection closed.
</pre>

The TRACE capability could be used by vulnerable or malicious applications to trick a web browser into issuing a TRACE request against an arbitrary site and then send the response to the TRACE to a third party using web browser features.

Making the required configuration changes:

<pre>
<VirtualHost www.example.com>
...
# disable TRACE in the www.example.com virtual host
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]
</VirtualHost>
</pre>

Using SSL:
$ /usr/linux/bin/openssl s_client -connect 127.0.0.1:8444

For more information...

More background information on the concerns with TRACE is provided at http://www.apacheweek.com/issues/03-01-24#news.

Other:
* Apache Tips: Disable the HTTP TRACE method | MDLog:/sysadmin - http://www.ducea.com/2007/10/22/apache-tips-disable-the-http-trace-method/

=== TraceEnable ===

Apache Core Features - http://httpd.apache.org/docs/1.3/mod/core.html#traceenable

TraceEnable:
Syntax: TraceEnable [on|off|extended]
Default: TraceEnable on
Context: server config
Status: core (Windows, NetWare)
Compatibility: Available only in Apache 1.3.34, 2.0.55 and later

This directive overrides the behavior of TRACE for both the core server and mod_proxy. The default TraceEnable on permits TRACE requests per RFC 2616, which disallows any request body to accompany the request. TraceEnable off causes the core server and mod_proxy to return a 405 FORBIDDEN error to the client.

Finally, for testing and diagnostic purposes only, request bodies may be allowed using the non-compliant TraceEnable extended directive. The core (as an origin server) will restrict the request body to 64k (plus 8k for chunk headers if Transfer-Encoding: chunked is used). The core will reflect the full headers and all chunk headers with the request body. As a proxy server, the request body is not restricted to 64k. At this time the Apache 1.3 mod_proxy does not permit chunked request bodies for any request, including the extended TRACE request.

=== Disabling TRACK ===

The HTTP TRACK method

The TRACK method is a type of request supported by Microsoft web servers. It is not RFC compliant and is not supported directly by IBM HTTP Server. The method may be utilized as part of a cross-site scripting attack. See Vulnerability Note VU#288308 for more information.

Even though IBM HTTP Server does not support the TRACK method natively, it is possible for plug-in modules to provide support for it. To disable this capability for plug-in modules, in addition to disabling the TRACE method, add these two additional directives after the existing RewriteCond and RewriteRule directives which are used to disable TRACE:

RewriteCond %{REQUEST_METHOD} ^TRACK
RewriteRule .* - [F]

Here is a full example showing the directives to disable both TRACE and TRACK:

<pre>
<VirtualHost www.example.com>
...
# disable TRACE and TRACK in the www.example.com virtual host
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]
RewriteCond %{REQUEST_METHOD} ^TRACK
RewriteRule .* - [F]
</VirtualHost>
</pre>

=== Attack ===

Cross-Site Tracing issues

Earlier this week a paper was published, [http://www.cgisecurity.com/whitehat-mirror/WH-WhitePaper_XST_ebook.pdf "Cross-Site Tracing"] which gave details of how the TRACE HTTP request could be used in Cross-Site Scripting attacks. Unfortunately this issue has not been very well understood by the media and has received a unwarranted amount of attention.

When an HTTP TRACE request is sent to a web server that supports it, that server will respond echoing the data that is passed to it, including any HTTP headers. The paper explains that some browsers can be scripted to perform a TRACE request. A browser with this functionality could be made to issue a TRACE request against an arbitrary site and pass the results on elsewhere. Since browsers will only send authentication details and cookies to the sites that issue them this means a user having a browser with this functionality could be tricked into sending their cookies or authentication details for arbitrary sites to an attacker.

For example, if you visited a page that an attacker has carefully crafted, the page could cause your browser to bounce a TRACE request against some site for which you have authentication cookies. The result of the TRACE will be a copy of what was sent to the site, which will therefore include those cookies or authentication data. The carefully crafted page can then pass that information on to the attacker.

TRACE requests can be disabled by making a change to the Apache server configuration. Unfortunately it is not possible to do this using the Limit directive since the processing for the TRACE request skips this authorisation checking. Instead the following lines can be added which make use of the mod_rewrite module.

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]

'''Although the particular attack highlighted made use of the TRACE functionality to grab authentication details, this isn't a vulnerability in TRACE, or in the Apache web server.''' The same browser functionality that permits the published attack can be used for different attacks even if TRACE is disabled on the remote web server. For example an attacker could create a carefully crafted page that when visited submits a hidden request to some arbitrary site through your browser, grabs the result and passes it to the attacker.

== Performance Tuning ==

See [[Linux Performance Tuning]]

[http://www.linuxjournal.com/article/6041 Apache and Firewall Performance Tips from the Xenu.net Masters]

=== CentOS January 2009 Thread ===

<pre>
-------- Original Message --------
Subject: [CentOS] Apache Server Tuning for Performance
Date: Wed, 21 Jan 2009 00:09:38 +0530
From: linux-crazy <hicheerup@gmail.com>
Reply-To: CentOS mailing list <centos@centos.org>
To: CentOS mailing list <centos@centos.org>

Hi all,

I am facing facing performance issues with our web servers which is
working for concurrent 250 requests properly and then stops
responding when the requests are more than 250 .

The current configuration parameters are as follows :

apachectl -version
Server version: Apache/2.0.52
Server built: Jan 30 2007 09:56:16

Kernel : 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64
x86_64 x86_64 GNU/Linux

Server Hardware :
MAIN MEMORY
(i) Memory Size 4 GB Dual-Core Intel 5160 processors.

httpd.conf

Timeout 300
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 150

## Server-Pool Size Regulation (MPM specific)
# prefork MPM

# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule prefork.c>

StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 251
MaxClients 251
MaxRequestsPerChild 4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

I want to know about the difference between worker MPM and
Prefork MPM , how to find out which one will be used by my apache
server and the recommended one for highly loaded server.If some one
provide me the link that best explains above two comparison also be
very use full.

Can any one guide me tuning to be done for the maximum utilization
of the Resources and better performance of the Servers.

Regards,
lingu
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
</pre>

<pre>
Only use prefork MPM (worker is for Win32, AFAIK).

If you set ServerLimit to 251, that's the limit.
Set ServerLimit and MaxClients to the same value.
(Larger than 250)
OK?
And take a couple of minutes to study the apache documentation. It's
actually quite good and tranlated into many languages...

BUT: what your server is being able to handle also depends on what you
actually serve (PHP/JSP/Servelets/Perl/whatever!

This is usually not a problem that is easily described and solved in
two sentences.

Rainer
</pre>

<pre>
Increase your MaxClients configuration, also enable if you
haven't already the server-status module and monitor the
server via http://<your server name>/server-status it will
show how many workers are busy and what they are doing.

As for which to use, prefork is the old forked method of
doing things, the other uses threads. Depending on what kind
of application your running on top of apache, if it is not
thread safe(at some point many PHP modules were not thread
safe), you may want to use prefork. Otherwise you can use
the threading model.

If your not sure I'd say stick to prefork to be safe until
you can determine for sure that threading is safe.

nate
</pre>

<pre>
Most list members would likely advise sticking with the prefork
configuration. Without knowing what kind of applications you are
running on your webserver, I wouldn't suggest changing it.

Merely increasing the number of workers might make performance worse.

Use ps or top to figure out how much each apache worker is using. Then
decide how much ram you want to dedicate on your server to Apache,
without going into swap. (Over-allocating and then paging out memory
will only make performance much worse.) For example, if I have 2G or
ram, and I want 1.5 for apache workers, my average apache worker size
(resident memory) is 65MB, then I have room for 23 workers. (1024 * 1.5
) / 65. (There are more accurate ways to calculate this usage, like
taking shared memory into account.)

Upgrading the ram in your web server is a pretty fast interim solution.

Consider your application performance, too. The longer a request in your
application takes, the more workers are in use on your web server,
taking up more memory. If you have long-running queries in your
database, take care of those first.

Good luck

Jed
</pre>

<pre>
Jed Reynolds wrote:
>
> Merely increasing the number of workers might make performance worse.
>
> Use ps or top to figure out how much each apache worker is using. Then
> decide how much ram you want to dedicate on your server to Apache,
> without going into swap. (Over-allocating and then paging out memory
> will only make performance much worse.) For example, if I have 2G or
> ram, and I want 1.5 for apache workers, my average apache worker size
> (resident memory) is 65MB, then I have room for 23 workers. (1024 * 1.5
> ) / 65. (There are more accurate ways to calculate this usage, like
> taking shared memory into account.)

Pay attention to shared memory when doing this. A freshly-forked
process shares virtually all RAM with its parent. How much and how
quickly this changes varies wildly with the application type and
activity that causes the child's data to become unique. With some types
of applications (especially mod_perl) you may want to tune down the
number of hits each child services to increase memory sharing. Also, if
you are running external cgi programs you must take them into account.

> Upgrading the ram in your web server is a pretty fast interim solution.
>
> Consider your application performance, too. The longer a request in your
> application takes, the more workers are in use on your web server,
> taking up more memory. If you have long-running queries in your
> database, take care of those first.

You may also have to turn off or tune down the HTTP 1.1 connection
keepalives, trading the time it takes to establish a new connection for
the RAM it takes to keep the associated process waiting for another
request from the same client.

--
Les Mikesell
lesmikesell@gmail.com
</pre>

<pre>
Also take a look at apache alternatives like lighttpd and nginx. In certain
cases they do miracles.

Jure Pečar
</pre>

<pre>
I don't know about lighttpd (our results were mixed) - but NGINX is
really _very_ fast.
Last time I checked, it seems to be the fastest way to accelerate
webpage-delivery on generic hardware and with OSS-software.

But it's not a feature-monster like apache, so you still need that.

Rainer
</pre>

<pre>
Linux-crazy wrote on Wed, 21 Jan 2009 00:09:38 +0530:

> KeepAliveTimeout 150

My god, reduce this to 10 or 5.

> ServerLimit 251
> MaxClients 251

There's your limit. However, you should check with your hardware if upping
it is really desirable. With 4 GB I think you won't be able to handle much
more anyway.

Kai
</pre>

<pre>
Kai Schaetzl wrote:
> There's your limit. However, you should check with your hardware if upping
> it is really desirable. With 4 GB I think you won't be able to handle much
> more anyway.

That really depends. If you only shove out static pages and have one or
two or three odd cgis on the machine, you can flatten down the httpd
binary quite a bit by throwing out unneeded modules.

500 to 750 clients shouldn't be that much of a problem then ...

Cheers,

Ralph
</pre>

<pre>
Ralph Angenendt wrote on Wed, 21 Jan 2009 10:52:59 +0100:

> That really depends. If you only shove out static pages and have one or
> two or three odd cgis on the machine, you can flatten down the httpd
> binary quite a bit by throwing out unneeded modules.
>
> 500 to 750 clients shouldn't be that much of a problem then ...

Sure it depends ;-) I do serve dynamic pages but by removing the really
unnecessary modules I made my httpds much faster (especially on pipelined
image downloads) and they only have some 10 MB (RES) per worker.

But I would expect that they are running the default set of modules.

Kai
</pre>

<pre>
Kai Schaetzl schrieb:
> Ralph Angenendt wrote on Wed, 21 Jan 2009 10:52:59 +0100:
>
>
>> That really depends. If you only shove out static pages and have one or
>> two or three odd cgis on the machine, you can flatten down the httpd
>> binary quite a bit by throwing out unneeded modules.
>>
>> 500 to 750 clients shouldn't be that much of a problem then ...
>>
>
> Sure it depends ;-) I do serve dynamic pages but by removing the really
> unnecessary modules I made my httpds much faster (especially on pipelined
> image downloads) and they only have some 10 MB (RES) per worker.
>
> But I would expect that they are running the default set of modules.
>
> Kai
>
>

If you can separate-out the images to a specific URL (img.domain.com)
and put them all in the same directory, you can use NGINX to serve them.
Of course, the actual transfer-speed will not increase much - but
latency will go down to the absolute minimum.
And latency is what makes a page appear "fast" or "slow" to customers.
</pre>

<pre>
Ralph Angenendt wrote:
> Kai Schaetzl wrote:
>> There's your limit. However, you should check with your hardware if upping
>> it is really desirable. With 4 GB I think you won't be able to handle much
>> more anyway.
>
> That really depends. If you only shove out static pages and have one or
> two or three odd cgis on the machine, you can flatten down the httpd
> binary quite a bit by throwing out unneeded modules.
>
> 500 to 750 clients shouldn't be that much of a problem then ...

Yeah gotta monitor it.. just checked in on some servers I ran at
my last company and the front end proxies (99% mod_proxy) seem to
peak out traffic wise at about 230 workers. For no other reason other
than because I could each proxy has 8 apache instances running(4 for
HTTP 4 for HTTPS). CPU usage peaks at around 3%(dual proc single core),
memory usage around 800MB. About 100 idle workers. Keepalive was set
to 300 seconds due to poor application design.

One set of back end apache servers peaks traffic wise at around 100
active workers, though memory usage was higher, around 2GB because
of mod_fcgid running ruby on rails. CPU usage seems to be at around
25%(dual proc quad core) for that one application.

Each back end application had it's own dedicated apache instance(11
apps) for maximum stability/best performance monitoring. The bulk
of them ran on the same physical hardware though.

Traffic routing was handled by a combination of F5 load balancers
and the apache servers previously mentioned(F5 iRules were too slow
and F5's TMM was not scalable at the time).

nate
</pre>

<pre>
>
> > KeepAliveTimeout 150
>
>reduce this to 10 or 5.
>
>
> Kai
>
> --
> Kai Schätzl, Berlin, Germany

Kai,

what do you think about the "general Timeout

it is set to 300

ive never much thought about it, yet should we be consider and possible
reduce that one too?

- rh
</pre>

<pre>
RobertH wrote on Wed, 21 Jan 2009 11:26:41 -0800:

> what do you think about the "general Timeout
>
> it is set to 300
>
> ive never much thought about it, yet should we be consider and possible
> reduce that one too?

I'm using 120, but I don't think reducing this value has much impact.

Kai
</pre>

== Load Balancer ==

Apache Load Balancer

mod_proxy_balancer - Apache HTTP Server - mod_proxy_balancer - http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html

<pre>
<Proxy balancer://mycluster>
BalancerMember http://192.168.1.50:80
BalancerMember http://192.168.1.51:80
</Proxy>
ProxyPass /test balancer://mycluster
</pre>

== User Directories ==

Default:
UserDir public_html

Results in:
~/public_html/

To disable user directories:
UserDir disabled

Ubuntu user dir config:
<pre>
UserDir public_html
UserDir disabled root

<Directory /home/*/public_html>
AllowOverride FileInfo AuthConfig Limit Indexes
Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
<Limit GET POST OPTIONS>
Order allow,deny
Allow from all
</Limit>
<LimitExcept GET POST OPTIONS>
Order deny,allow
Deny from all
</LimitExcept>
</Directory>

</pre>

References:
* Per-user web directories - Apache HTTP Server - http://httpd.apache.org/docs/2.2/howto/public_html.html
* mod_userdir - Apache HTTP Server - http://httpd.apache.org/docs/2.2/mod/mod_userdir.html#userdir

== Minimal Memory Instance ==

Assuming running prefork module (check with 'apache -l'):

<pre>
KeepAlive off

<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 5
ServerLimit 20
MaxClients 20
MaxRequestsPerChild 10000
</IfModule>
</pre>

References:
* Running Apache On A Memory-Constrained VPS | Kalzumeus Software - http://www.kalzumeus.com/2010/06/19/running-apache-on-a-memory-constrained-vps/

---

TODO See:
* http://wiki.vpslink.com/Low_memory_MySQL_/_Apache_configurations
* http://www.narga.net/optimizing-apachephpmysql-low-memory-server/

== LDAP Authentication ==

# Red Hat / RHEL
sudo yum install mod_ldap
# allow ldap to connect through selinux:
sudo setsebool -P httpd_can_network_connect on

mod_ldap - Apache HTTP Server Version 2.4
https://httpd.apache.org/docs/2.4/mod/mod_ldap.html

== Issues ==

=== Could not reliably determine domain name ===

Warning:
Could not reliably determine the server's fully qualified domain name

Solution:
* Force ServerName in apache configs [http://askubuntu.com/questions/256013/could-not-reliably-determine-the-servers-fully-qualified-domain-name]
ServerName localhost

=== client denied by server configuration ===

browser:
403 Forbidden - You don't have permission to access / on this server.

Error:
<pre>
==> httpd/default-error_log <==
[Mon Sep 01 21:42:02.273169 2014] [authz_core:error] [pid 11221] [client 174.52.47.54:50675] AH01630: client denied by server configuration: /www/
</pre>

Add Directory allow permissions: [http://stackoverflow.com/questions/19445686/ubuntu-server-apache-2-4-6-client-denied-by-server-configuration-php-fpm]
<pre>
DocumentRoot /www
<Directory /www>
Order allow,deny
Allow from all
</Directory>
</pre>

or this worked better for me: [http://stackoverflow.com/questions/19445686/ubuntu-server-apache-2-4-6-client-denied-by-server-configuration-php-fpm] [https://mikegriffin.ie/blog/20140130-authz-core-error-client-denied-by-server-configuration/]
<pre>
DocumentRoot /www
<Directory /www>
Require all granted
</Directory>
</pre>

Actually this just worked on the latest Ubuntu system:
<pre>
<Directory /www/>
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>
</pre>

References:
* ClientDeniedByServerConfiguration - Httpd Wiki - https://wiki.apache.org/httpd/ClientDeniedByServerConfiguration

=== permissions missing component path ===

<pre>
==> httpd/default-error_log <==
[Mon Sep 01 21:50:09.256120 2014] [core:error] [pid 11511] (13)Permission denied: [client 174.52.47.54:50744] AH00035: access to \
/index.html denied (filesystem path '/www/index.html') because search permissions are missing on a component of the path
</pre>

== keywords ==

[[Category:Linux]]
[[Category:Web Development ‎]]
[[Category:Web]]
[[Category:HTML]]
[[Category:Internet]]

Apache

2026-04-27T19:15:12Z

Kenneth: /* .htaccess */

{{subpages}}

[[Category:Apache]]

== Install Apache ==

# Debian / Ubuntu
sudo apt install apache2

# Redhat
yum install httpd

== Virtual Hosting ==
touch /etc/httpd/conf.d/virtual.conf

Virtual Host Configuration File:
/etc/httpd/conf.d/virtual.conf:

HTTP Virtual Host
<pre>
NameVirtualHost *:80

<VirtualHost *:80>
ServerName t0e.org
ServerAlias www.t0e.org

DocumentRoot /www

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>

<VirtualHost *:80>
ServerName www2.t0e.org
...
</VirtualHost>
</pre>

HTTPS Virtual Host:
<pre>
NameVirtualHost *:443

<VirtualHost *:443>
ServerName t0e.org
ServerAlias www.t0e.org

# SSL Enabled:
SSLEngine on
SSLCertificateFile /etc/t0e.org.pem

# If using a godaddy certificate chain:
#SSLCertificateFile /etc/contractpal.net.pem
#SSLCertificateKeyFile /etc/contractpal.net.pem
#SSLCertificateChainFile /etc/contractpal.net.pem

#Forwarding Traffic
# RewriteEngine on
# RewriteRule ^(/.*) http://localhost:8080/$1 [P]

DocumentRoot /www

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>
</pre>

References:
*[http://www.onlamp.com/pub/a/apache/2005/02/17/apacheckbk.html A Day in the Life of #Apache: Running Name-Based SSL Virtual Hosts in Apache]
*[http://httpd.apache.org/docs/1.3/vhosts/name-based.html Apache: Name-based Virtual Host Support]
*[http://www.networkworld.com/columnists/2006/011606internet.html Getting SSL to work across virtual hosts in Apache]

== CentOS common configuration changes ==

Keep alives:
# KeepAlive Off
KeepAlive On

Hide Apache version (in likely in /etc/apache2/conf-enabled/security.conf):
# ServerTokens Full
ServerTokens Prod

# ServerSignature On
ServerSignature Off

---

PHP config in /etc/php.ini (now likely in /etc/php/8.3/apache2/php.ini):
expose_php = Off

==Forwarding / Rewriting / Proxying Traffic==

<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName t0e.org
ServerAlias www.t0e.org

#Forwarding Traffic
RewriteEngine on
RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>
</pre>

Apache module mod_rewrite - RewriteRule - http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteRule
a2enmod rewrite

a2enmod proxy_http

[http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html Module mod_rewrite - URL Rewriting Engine]:
*RewriteEngine
*RewriteCond
*RewriteRule

[http://httpd.apache.org/docs/1.3/misc/rewriteguide.html Apache 1.3 - URL Rewriting Guide]

[http://httpd.apache.org/docs/1.3/mod/mod_proxy.html Apache module mod_proxy]:
*ProxyPass

---

<pre>
<VirtualHost *:80>
ServerName home.burgenerfamily.com
DocumentRoot /www/home
<Directory /www/home>
Options FollowSymLinks
RewriteEngine On
# allow existing files and directories to pass through
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
#RewriteRule ^static - [L,NC]
#RewriteCond %{REQUEST_FILENAME} !^favicon.ico$
#RewriteCond %{REQUEST_FILENAME} !^static/
#RewriteRule ^(.+)$ /index.php?title=$1 [PT,L,QSA]
RewriteRule ^(.*)$ /index.php/$1 [L,QSA]
</Directory>
ErrorLog logs/home.burgenerfamily.com-error_log
CustomLog logs/home.burgenerfamily.com-access_log common
</VirtualHost>
</pre>

== Proxy Pass ==

Enable Proxy: [https://www.digitalocean.com/community/tutorials/how-to-use-apache-as-a-reverse-proxy-with-mod_proxy-on-ubuntu-16-04]
sudo a2enmod proxy
sudo a2enmod proxy_http
systemctl restart apache2

Similar to forwarding, but takes care of redirects:

ProxyPass /mirror/foo/ http://foo.com/
ProxyPassReverse /mirror/foo/ http://foo.com/

Suppose the local server has address http://wibble.org/; then

ProxyPass /mirror/foo/ http://foo.com/
ProxyPassReverse /mirror/foo/ http://foo.com/

will not only cause a local request for the <http://wibble.org/mirror/foo/bar> to be internally converted into a proxy request to <http://foo.com/bar> (the functionality ProxyPass provides here). It also takes care of redirects the server foo.com sends: when http://foo.com/bar is redirected by him to http://foo.com/quux Apache adjusts this to http://wibble.org/mirror/foo/quux before forwarding the HTTP redirect response to the client.

<pre>
<VirtualHost *:80>
ServerName wiki.oeey.com

# Reverse Proxy
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/

ServerAdmin website@oeey.com
ErrorLog logs/wiki-error_log
CustomLog logs/wiki-access_log common
</VirtualHost>
</pre>

Proxy Pass to Jenkins: [https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache]
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName jenkins.oeey.com

#Forwarding Traffic
#RewriteEngine on
#RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ProxyPass / http://localhost:8080/ nocanon
ProxyPassReverse / http://localhost:8080/
ProxyRequests Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
<Proxy http://localhost:8080/*>
Order deny,allow
Allow from all
</Proxy>

ServerAdmin admin@oeey.com
ErrorLog logs/jenkins.oeey.com-error_log
CustomLog logs/jenkins.oeey.com-access_log common
</VirtualHost>
</pre>

=== Proxy Pass HTTPS ===

Proxy Pass to Jenkins:
<pre>
NameVirtualHost *:443
<VirtualHost *:443>
ServerName jenkins.oeey.com

# Enable SSL Support
SSLEngine on
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

#Forwarding Traffic
#RewriteEngine on
#RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ProxyPass / http://localhost:8080/ nocanon
ProxyPassReverse / http://localhost:8080/
ProxyPassReverse / http://jenkins.oeey.com/
ProxyRequests Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode
RequestHeader set X-Forwarded-Proto "http"

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
<Proxy http://localhost:8080/jenkins*>
Order deny,allow
Allow from all
</Proxy>

ServerAdmin admin@oeey.com
ErrorLog logs/jenkins.oeey.com-error_log
CustomLog logs/jenkins.oeey.com-access_log common
</VirtualHost>
</pre>

== Redirect ==

Redirect with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
Redirect permanent / http://www.oeey.com/
</VirtualHost>
</pre>

Redirect users to HTTPS with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
Redirect permanent / https://test.oeey.com/
</VirtualHost>
</pre>

Can be wrapped in Location (if you want)
<Location />
Redirect permanent / https://test.oeey.com/
</Location>

Redirect with RewriteEngine with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com

# Redirect HTTP to HTTPS
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>
</pre>

Redirect removing URI:
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
RedirectMatch "(.*)" "http://www.oeey.com/"
</VirtualHost>

[http://httpd.apache.org/docs/2.0/mod/mod_alias.html Apache Module mod_alias]:
*Alias
*Redirect
*RedirectPermanent

== Default Directory Index ==

to set the default pages to try and load (e.g. index.html, index.php):
DirectoryIndex index.html index.php

== Directory Index ==

To show full file names:
<pre>
<Directory />
Options Indexes
IndexOptions NameWidth=*
</Directory>
</pre>

Sample:
<pre>
<Directory "/www">
Options Indexes FollowSymLinks
IndexOptions FancyIndexing NameWidth=*
AllowOverride None
Require all granted
</Directory>
</pre>

== Hide Apache and PHP Banner ==

Hide PHP version (X-Powered-By) - Edit php.ini: [http://www.ducea.com/2006/06/16/apache-tips-tricks-hide-php-version-x-powered-by/]
expose_php = Off

Hide Apache version: [http://www.itsolutionskb.com/2008/10/hide-apache-information/]
# ServerSignature On
ServerSignature Off

# ServerTokens Full
ServerTokens Prod

References:
* Hide Apache ServerSignature/ServerTokens/PHP X-Powered-By - http://www.if-not-true-then-false.com/2009/howto-hide-and-modify-apache-server-information-serversignature-and-servertokens-and-hide-php-version-x-powered-by/

== .htaccess ==

Override configuration settings on a per directory (and sub directories) level with:
.htaccess

Allow override of all configuration options:
<pre>
<Directory />
AllowOverride All
</Directory>
</pre>

Example directory browsing in .htaccess:
Options +indexes

Redirect to index.php:
<pre>
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?path=$1 [NC,L,QSA]
</pre>
a2enmod rewrite
#a2enmod proxy_http

== Allow from networks ==

<pre>
<Directory "...">
Order Deny,Allow
Deny from all
Allow from 127.0.0.0/255.255.255.0 10.10.0.0/16
</Directory>
</pre>

== Log Files ==

ErrorLog logs/test.com-error_log
CustomLog logs/test.com-access_log common

Make sure the soft link exists:
ln -s /var/log/httpd /etc/httpd/logs # RHEL based
ln -s /var/log/apache2 /etc/apache2/logs # Debian Based

*[http://httpd.apache.org/docs/2.0/logs.html Log Files v2.0]
*[http://httpd.apache.org/docs/1.3/logs.html#accesslog Log files v1.3]

===Log HTTPS X-Forwarded-For===
[http://www.webmasterworld.com/forum92/6134.htm Log the X-Forwarded-For] header in apache...
LogFormat "[X-Forwarded-For: %{X-Forwarded-For}i] %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "[X-Forwarded-For: %{X-Forwarded-For}i] %h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

== PHP ==

Install PHP with Apache: (Ubuntu)
#apt-get install apache2 php5 libapache2-mod-php5
apt-get install apache2 php libapache2-mod-php

== Apache SSL ==

Enable SSL with...

Apache can do virtual SSL hosting!

Install mod_ssl: [https://help.ubuntu.com/10.04/serverguide/httpd.html]
# redhat / rhel
sudo yum install mod_ssl

# debian / ubuntu
sudo a2enmod ssl
sudo a2ensite default-ssl
systemctl reload apache2

/etc/httpd/conf.d/ssl.conf:
<pre>
LoadModule ssl_module modules/mod_ssl.so
Listen 443
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
SSLPassPhraseDialog builtin
SSLSessionCache shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout 300
SSLMutex default
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>

Note: Delete the default VirtualHost section so we can add our own.

<pre>
# SSL Configuration - from ssl.conf virtual host example
SSLEngine On
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</pre>

<pre>
# SSL configuration
SSLEngine On
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile /etc/apache2/ssl/cert.crt
SSLCertificateKeyFile /etc/apache2/ssl/cert.key
SSLCertificateChainFile /etc/apache2/ssl/sf_bundle.crt
</pre>

<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName default.t0e.org

# Redirect HTTP to HTTPS
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

NameVirtualHost *:443
<VirtualHost *:443>
ServerName kb.t0e.org

# SSL Enabled: (only once in config file)
SSLEngine on
SSLCertificateFile /etc/t0e.org.pem
SSLCertificateKeyFile /etc/t0e.org.pem
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

DocumentRoot /www

ServerAdmin admin@kb.t0e.org.com
ErrorLog logs/kb.t0e.org-error_log
CustomLog logs/kb.t0e.org-access_log common
</VirtualHost>

<VirtualHost *:443>
ServerName wiki.t0e.org

DocumentRoot /www

ServerAdmin admin@wiki.t0e.org.com
ErrorLog logs/wiki.t0e.org-error_log
CustomLog logs/wiki.t0e.org-access_log common
</VirtualHost>
</pre>

Note: With SSL, Apache will always use the first default virtual hosts' certificate, so it only makes sense to define it there.

References:
*[http://www.apache-ssl.org/ Apache SSL]
*[http://httpd.apache.org/docs/2.0/ssl/ Apache SSL/TLS Encryption Documentation]
*[http://httpd.apache.org/docs/2.0/mod/mod_ssl.html mod_ssl]
*[http://www.securityfocus.com/infocus/1818 Article: Apache 2 with SSL/TLS: Step-by-Step, Part 1]
*[http://www.vanemery.com/Linux/Apache/apache-SSL.html Van's Apache SSL/TLS mini-HOWTO]
*[http://tomcat.apache.org/tomcat-3.3-doc/tomcat-ssl-howto.html Tomcat SSL Configuration]
*[http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html Apache: SSL/TLS Strong Encryption: FAQ]
*[http://www.networkworld.com/columnists/2006/011606internet.html Getting SSL to work across virtual hosts in Apache]
*[http://www.faqs.org/docs/Linux-HOWTO/SSL-RedHat-HOWTO.html Building a Secure RedHat Apache Server HOWTO]
* CommonMisconfigurations - Httpd Wiki - http://wiki.apache.org/httpd/CommonMisconfigurations

--- chain ---

If you have a chain, create the pem as "key, cert, chain" and set Apache config like so:

SSLCertificateFile /etc/ssl.pem
SSLCertificateKeyFile /etc/ssl.pem
SSLCertificateChainFile /etc/ssl.pem

or just

SSLCertificateFile /etc/ssl.pem
SSLCertificateChainFile /etc/ssl.pem

=== Ubuntu SnakeOil Cert ===

Install the snakoil fake cert <ref>https://superuser.com/questions/58149/host-based-snakeoil-certificates-in-ubuntu</ref>

sudo apt-get install ssl-cert

sudo make-ssl-cert generate-default-snakeoil --force-overwrite

/etc/ssl/certs/ssl-cert-snakeoil.pem
/etc/ssl/private/ssl-cert-snakeoil.key

== Linux System Password Protection ==

Using system accounts for HTTP authentication - http://wiki.zs64.net/Using_system_accounts_for_HTTP_authentication

httpd.conf

Edit httpd.conf to add these lines to the general configuration:

LoadModule auth_external_module libexec/apache2/mod_auth_external.so

(Should have been added by installing the port already.)

Add these lines to each virtual host where you'd like to use system accounts for authentication:

AddExternalAuth pwauth /usr/local/bin/pwauth
SetExternalAuthMethod pwauth pipe

Add these lines to each Directory or Location section that you want protected by authentication:

AuthType Basic
AuthExternal pwauth
require valid-user

== Password Protection with .htaccess and htpasswd ==

*[http://www.javascriptkit.com/howto/htaccess3.shtml Comprehensive guide to .htaccess: Password protection]
*[http://apache-server.com/tutorials/ATusing-htaccess.html Using .htaccess Files with Apache]
*[http://www.4webhelp.net/tutorials/misc/htaccess.php Password Protection by Rod]
*[http://www.spoono.com/php/tutorials/tutorial.php?id=30 .htaccess Password Protection in PHP]

Sample .htaccess file:

AuthUserFile /var/www/htpasswd
AuthGroupFile /dev/null
AuthName "Secret Place"
AuthType Basic
Require valid-user

#Require user <user>
#Require group <group>

To use with httpd.conf or virtual.conf, wrap in "<Directory>" tags:
<Directory />
AuthUserFile ...
...
</Directory>

Note: if you want in your httpd.conf (or virtual.conf) surround with "<Location />...</Location>"

To add password file:
htpasswd -c /var/www/htpasswd admin
htpasswd /var/www/htpasswd user1

# if you want to create a blank username with password:
htpasswd /path/htpasswd ''

To use .htaccess:
<Directory />
Options FollowSymLinks
AllowOverride '''AuthConfig'''
</Directory>

Or user the [http://www.webmaster-toolkit.com/htaccess-generator.shtml .htaccess generator]

[http://httpd.apache.org/docs/2.2/mod/core.html#require Apache HTTP Server - Require Directive]:

Require user userid [userid] ...
:Only the named users can access the resource.
Require group group-name [group-name] ...
:Only users in the named groups can access the resource.
Require valid-user
:All valid users can access the resource.

<pre>
AuthType Basic
AuthName "Restricted Resource"
AuthUserFile /web/users
AuthGroupFile /web/groups
Require group admin
</pre>

=== web based htpasswd manager ===

see [[change_htpasswd.pl]]

[http://home.xnet.com/~efflandt/pub/htpasswd.pl htpasswd.cgi by David Efflandt]

Perl CGI Script to Modify User Passwords:

This allows users to manage / change their own passwords.

Use the Perl CGI script [http://home.xnet.com/%7Eefflandt/pub/htpasswd.pl htpasswd.pl] [[http://www.yolinux.com/TUTORIALS/src/htpasswd_pl.txt cache]]

* Edit location of Perl .i.e.: /usr/bin/perl
** Not /usr/local/bin/perl
* Edit the script to specify location of the password file i.e. /var/www/PasswordDir/.htpasswd
* SELinux users must add the correct attribute i.e. chcon -R -h -t httpd_sys_content_t /var/www/PasswordDir
* The password file must be located in a directory where CGI is allowed to modify files.

File: httpd.conf (portion):

<pre>
..
...
<Directory "/var/www/PasswordDir">
Options -Indexes
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
...
..
</pre>

==Directory Browsing==
DocumentRoot /www/files
<Directory />
Options FollowSymLinks '''+Indexes'''
AllowOverride None
</Directory>

Note: The '+' may not be needed.

In a .htaccess:
Options +indexes

Long file names: [http://www.networkworld.com/article/2311687/software/showing-long-file-names-in-apache-directory-listings.html]
IndexOptions NameWidth=*

For root indexes, the welcome.conf may be overriding with the welcome screen. Make the following change: [http://www.linuxquestions.org/questions/linux-general-1/apache-2-root-directory-listing-201356/]
<pre>
<LocationMatch "^/$>
#Options -Indexes
Options Indexes
ErrorDocument 403 /error/noindex.html
</LocationMatch>
</pre>

== cgi ==

Script Alias folder:
ScriptAlias /cgi-bin/ "C:/Program Files/Apache Group/Apache/cgi-bin/"
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

My favorite option, make all .cgi files executable: (This will need to be surrounded with '<Location />' or '<Directory />' tags)
AddHandler cgi-script .cgi
Options +ExecCGI

Note: Make sure to 'chmod +x' the files.

Example:
<pre>
<VirtualHost *:80>
ServerName hg.t0e.org

DirectoryIndex hgweb.cgi
DocumentRoot /www/hg

<Location />
AddHandler cgi-script .cgi
Options +ExecCGI
</Location>

ServerAdmin admin@hg.t0e.org
ErrorLog logs/hg.t0e.org-error_log
CustomLog logs/hg.t0e.org-access_log common
</VirtualHost>
</pre>

Source: [http://www.thesitewizard.com/archive/addcgitoapache.shtml How to Add CGI Script Support (Perl, Python, etc) to Your Apache Server on Windows (thesitewizard.com)]

--

Only files without an extension:
<pre>
<VirtualHost *:80>
ServerName pi.t0e.org
DocumentRoot /www/pi
<Directory />
AllowOverride All
</Directory>
DirectoryIndex index index.php index.html index.htm
<Files ~ "^[a-zA-Z]*$">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>
=== CentOS Default ===

<pre>
#
# ScriptAliases are essentially the same as Aliases, except that
# documents in the realname directory are treated as applications and
# run by the server when requested rather than as documents sent to the client.
# The same rules about trailing "/" apply to ScriptAlias directives as to
# Alias.
#
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/var/www/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
</pre>

== Python CGI ==

See [[Python#Apache_CGI]]

==mod_proxy==
*http://www.apachetutor.org/admin/reverseproxies
*http://www.linuxfocus.org/English/March2000/article147.html
*http://www.linuxhelp.net/guides/apacheproxy/
*http://webauth.stanford.edu/manual/mod/mod_proxy.html
*http://www.apachetutor.org/admin/reverseproxies
*http://confluence.atlassian.com/display/DOC/Using+Apache+with+mod_proxy
*http://www.serverwatch.com/tutorials/article.php/3290851
*http://apache.webthing.com/mod_proxy_html/
*http://www.ibm.com/developerworks/linux/library/wa-lampsec/index.html

== prefork vs worker ==

Prefork performs better. Worker uses less memory. Prefork is most commonly used, and is the better choice for a web server.

To see which is compiled, and look for prefork.c or worker.c:
<pre>
# httpd -l

Compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c
</pre>

== Statistics ==

=== Connections ===

Count active connections:
netstat -ant | grep ESTABLISHED | grep :80 | wc -l

From PHP:
<?php
$number_of_users = shell_exec('netstat -ant | grep ESTABLISHED | grep :80 | wc -l');
echo $number_of_users;
?>

References:
* [http://www.experts-exchange.com/OS/Linux/Q_22825240.html Find out how many connections are on an apache server : apache, connections, many, find, server]

=== [http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status] ===

[http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status - Apache HTTP Server]

Summary

The Status module allows a server administrator to find out how well their server is performing. A HTML page is presented that gives the current server statistics in an easily readable form. If required this page can be made to automatically refresh (given a compatible browser). Another page gives a simple machine-readable list of the current server state.

The details given are:

* The number of worker serving requests
* The number of idle worker
* The status of each worker, the number of requests that worker has performed and the total number of bytes served by the worker (*)
* A total number of accesses and byte count served (*)
* The time the server was started/restarted and the time it has been running for
* Averages giving the number of requests per second, the number of bytes served per second and the average number of bytes per request (*)
* The current percentage CPU used by each worker and in total by Apache (*)
* The current hosts and requests being processed (*)

The lines marked "(*)" are only available if ExtendedStatus is On.

<pre>
LoadModule status_module modules/mod_status.so

#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
#ExtendedStatus On

#
# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Change the ".example.com" to match your domain to enable.
#
#<Location /server-status>
# SetHandler server-status
# Order deny,allow
# Deny from all
# Allow from .example.com
#</Location>
</pre>

References:
* [http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status - Apache HTTP Server]
* [http://www.experts-exchange.com/OS/Linux/Q_22825240.html Find out how many connections are on an apache server : apache, connections, many, find, server]

== TRACE and TRACK ==

ISO Audit may result in this "Medium" risk detected:

:"HTTP Methods Allowed. It is observed that HTTP methods GET HEAD TRACE OPTIONS are allowed on this host. Disable all unnecessary methods."

:"HTTP TRACE / TRACK Methods Allowed. The remote webserver supports the TRACE and/or TRACK methods. TRACE and TRACK are HTTP methods that are used to debug web server connections. Disable these methods."

=== Disabling TRACE ===

[http://rackerhacker.com/2007/08/28/apache-disable-trace-and-track-methods/ Apache: Disable TRACE and TRACK methods | Racker Hacker]:

Lots of PCI Compliance and vulnerability scan vendors will complain about TRACE and TRACK methods being enabled on your server. Since most providers run Nessus, you'll see this fairly often. Here's the rewrite rules to add:

RewriteEngine on
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]

These directives will need to be added to each VirtualHost.

For apache version 1.3.34 (or later 1.3.x versions), or apache 2.0.55 (or later), this has been made easy. Just add the line TraceEnable off

TraceEnable off

[http://publib.boulder.ibm.com/httpserv/ihsdiag/http_trace.html Disabling the HTTP TRACE method]:

The HTTP TRACE request method causes the data received by IBM HTTP Server from the client to be sent back to the client, as in the following example:

<pre>
$ telnet 127.0.0.1 8080
Trying...
Connected to 127.0.0.1.
Escape character is '^]'.
TRACE / HTTP/1.0
Host: foo
A: b
C: d

HTTP/1.1 200 OK
Date: Mon, 04 Oct 2004 14:07:59 GMT
Server: IBM_HTTP_SERVER
Connection: close
Content-Type: message/http

TRACE / HTTP/1.0
A: b
C: d
Host: foo

Connection closed.
</pre>

The TRACE capability could be used by vulnerable or malicious applications to trick a web browser into issuing a TRACE request against an arbitrary site and then send the response to the TRACE to a third party using web browser features.

Making the required configuration changes:

<pre>
<VirtualHost www.example.com>
...
# disable TRACE in the www.example.com virtual host
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]
</VirtualHost>
</pre>

Using SSL:
$ /usr/linux/bin/openssl s_client -connect 127.0.0.1:8444

For more information...

More background information on the concerns with TRACE is provided at http://www.apacheweek.com/issues/03-01-24#news.

Other:
* Apache Tips: Disable the HTTP TRACE method | MDLog:/sysadmin - http://www.ducea.com/2007/10/22/apache-tips-disable-the-http-trace-method/

=== TraceEnable ===

Apache Core Features - http://httpd.apache.org/docs/1.3/mod/core.html#traceenable

TraceEnable:
Syntax: TraceEnable [on|off|extended]
Default: TraceEnable on
Context: server config
Status: core (Windows, NetWare)
Compatibility: Available only in Apache 1.3.34, 2.0.55 and later

This directive overrides the behavior of TRACE for both the core server and mod_proxy. The default TraceEnable on permits TRACE requests per RFC 2616, which disallows any request body to accompany the request. TraceEnable off causes the core server and mod_proxy to return a 405 FORBIDDEN error to the client.

Finally, for testing and diagnostic purposes only, request bodies may be allowed using the non-compliant TraceEnable extended directive. The core (as an origin server) will restrict the request body to 64k (plus 8k for chunk headers if Transfer-Encoding: chunked is used). The core will reflect the full headers and all chunk headers with the request body. As a proxy server, the request body is not restricted to 64k. At this time the Apache 1.3 mod_proxy does not permit chunked request bodies for any request, including the extended TRACE request.

=== Disabling TRACK ===

The HTTP TRACK method

The TRACK method is a type of request supported by Microsoft web servers. It is not RFC compliant and is not supported directly by IBM HTTP Server. The method may be utilized as part of a cross-site scripting attack. See Vulnerability Note VU#288308 for more information.

Even though IBM HTTP Server does not support the TRACK method natively, it is possible for plug-in modules to provide support for it. To disable this capability for plug-in modules, in addition to disabling the TRACE method, add these two additional directives after the existing RewriteCond and RewriteRule directives which are used to disable TRACE:

RewriteCond %{REQUEST_METHOD} ^TRACK
RewriteRule .* - [F]

Here is a full example showing the directives to disable both TRACE and TRACK:

<pre>
<VirtualHost www.example.com>
...
# disable TRACE and TRACK in the www.example.com virtual host
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]
RewriteCond %{REQUEST_METHOD} ^TRACK
RewriteRule .* - [F]
</VirtualHost>
</pre>

=== Attack ===

Cross-Site Tracing issues

Earlier this week a paper was published, [http://www.cgisecurity.com/whitehat-mirror/WH-WhitePaper_XST_ebook.pdf "Cross-Site Tracing"] which gave details of how the TRACE HTTP request could be used in Cross-Site Scripting attacks. Unfortunately this issue has not been very well understood by the media and has received a unwarranted amount of attention.

When an HTTP TRACE request is sent to a web server that supports it, that server will respond echoing the data that is passed to it, including any HTTP headers. The paper explains that some browsers can be scripted to perform a TRACE request. A browser with this functionality could be made to issue a TRACE request against an arbitrary site and pass the results on elsewhere. Since browsers will only send authentication details and cookies to the sites that issue them this means a user having a browser with this functionality could be tricked into sending their cookies or authentication details for arbitrary sites to an attacker.

For example, if you visited a page that an attacker has carefully crafted, the page could cause your browser to bounce a TRACE request against some site for which you have authentication cookies. The result of the TRACE will be a copy of what was sent to the site, which will therefore include those cookies or authentication data. The carefully crafted page can then pass that information on to the attacker.

TRACE requests can be disabled by making a change to the Apache server configuration. Unfortunately it is not possible to do this using the Limit directive since the processing for the TRACE request skips this authorisation checking. Instead the following lines can be added which make use of the mod_rewrite module.

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]

'''Although the particular attack highlighted made use of the TRACE functionality to grab authentication details, this isn't a vulnerability in TRACE, or in the Apache web server.''' The same browser functionality that permits the published attack can be used for different attacks even if TRACE is disabled on the remote web server. For example an attacker could create a carefully crafted page that when visited submits a hidden request to some arbitrary site through your browser, grabs the result and passes it to the attacker.

== Performance Tuning ==

See [[Linux Performance Tuning]]

[http://www.linuxjournal.com/article/6041 Apache and Firewall Performance Tips from the Xenu.net Masters]

=== CentOS January 2009 Thread ===

<pre>
-------- Original Message --------
Subject: [CentOS] Apache Server Tuning for Performance
Date: Wed, 21 Jan 2009 00:09:38 +0530
From: linux-crazy <hicheerup@gmail.com>
Reply-To: CentOS mailing list <centos@centos.org>
To: CentOS mailing list <centos@centos.org>

Hi all,

I am facing facing performance issues with our web servers which is
working for concurrent 250 requests properly and then stops
responding when the requests are more than 250 .

The current configuration parameters are as follows :

apachectl -version
Server version: Apache/2.0.52
Server built: Jan 30 2007 09:56:16

Kernel : 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64
x86_64 x86_64 GNU/Linux

Server Hardware :
MAIN MEMORY
(i) Memory Size 4 GB Dual-Core Intel 5160 processors.

httpd.conf

Timeout 300
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 150

## Server-Pool Size Regulation (MPM specific)
# prefork MPM

# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule prefork.c>

StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 251
MaxClients 251
MaxRequestsPerChild 4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

I want to know about the difference between worker MPM and
Prefork MPM , how to find out which one will be used by my apache
server and the recommended one for highly loaded server.If some one
provide me the link that best explains above two comparison also be
very use full.

Can any one guide me tuning to be done for the maximum utilization
of the Resources and better performance of the Servers.

Regards,
lingu
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
</pre>

<pre>
Only use prefork MPM (worker is for Win32, AFAIK).

If you set ServerLimit to 251, that's the limit.
Set ServerLimit and MaxClients to the same value.
(Larger than 250)
OK?
And take a couple of minutes to study the apache documentation. It's
actually quite good and tranlated into many languages...

BUT: what your server is being able to handle also depends on what you
actually serve (PHP/JSP/Servelets/Perl/whatever!

This is usually not a problem that is easily described and solved in
two sentences.

Rainer
</pre>

<pre>
Increase your MaxClients configuration, also enable if you
haven't already the server-status module and monitor the
server via http://<your server name>/server-status it will
show how many workers are busy and what they are doing.

As for which to use, prefork is the old forked method of
doing things, the other uses threads. Depending on what kind
of application your running on top of apache, if it is not
thread safe(at some point many PHP modules were not thread
safe), you may want to use prefork. Otherwise you can use
the threading model.

If your not sure I'd say stick to prefork to be safe until
you can determine for sure that threading is safe.

nate
</pre>

<pre>
Most list members would likely advise sticking with the prefork
configuration. Without knowing what kind of applications you are
running on your webserver, I wouldn't suggest changing it.

Merely increasing the number of workers might make performance worse.

Use ps or top to figure out how much each apache worker is using. Then
decide how much ram you want to dedicate on your server to Apache,
without going into swap. (Over-allocating and then paging out memory
will only make performance much worse.) For example, if I have 2G or
ram, and I want 1.5 for apache workers, my average apache worker size
(resident memory) is 65MB, then I have room for 23 workers. (1024 * 1.5
) / 65. (There are more accurate ways to calculate this usage, like
taking shared memory into account.)

Upgrading the ram in your web server is a pretty fast interim solution.

Consider your application performance, too. The longer a request in your
application takes, the more workers are in use on your web server,
taking up more memory. If you have long-running queries in your
database, take care of those first.

Good luck

Jed
</pre>

<pre>
Jed Reynolds wrote:
>
> Merely increasing the number of workers might make performance worse.
>
> Use ps or top to figure out how much each apache worker is using. Then
> decide how much ram you want to dedicate on your server to Apache,
> without going into swap. (Over-allocating and then paging out memory
> will only make performance much worse.) For example, if I have 2G or
> ram, and I want 1.5 for apache workers, my average apache worker size
> (resident memory) is 65MB, then I have room for 23 workers. (1024 * 1.5
> ) / 65. (There are more accurate ways to calculate this usage, like
> taking shared memory into account.)

Pay attention to shared memory when doing this. A freshly-forked
process shares virtually all RAM with its parent. How much and how
quickly this changes varies wildly with the application type and
activity that causes the child's data to become unique. With some types
of applications (especially mod_perl) you may want to tune down the
number of hits each child services to increase memory sharing. Also, if
you are running external cgi programs you must take them into account.

> Upgrading the ram in your web server is a pretty fast interim solution.
>
> Consider your application performance, too. The longer a request in your
> application takes, the more workers are in use on your web server,
> taking up more memory. If you have long-running queries in your
> database, take care of those first.

You may also have to turn off or tune down the HTTP 1.1 connection
keepalives, trading the time it takes to establish a new connection for
the RAM it takes to keep the associated process waiting for another
request from the same client.

--
Les Mikesell
lesmikesell@gmail.com
</pre>

<pre>
Also take a look at apache alternatives like lighttpd and nginx. In certain
cases they do miracles.

Jure Pečar
</pre>

<pre>
I don't know about lighttpd (our results were mixed) - but NGINX is
really _very_ fast.
Last time I checked, it seems to be the fastest way to accelerate
webpage-delivery on generic hardware and with OSS-software.

But it's not a feature-monster like apache, so you still need that.

Rainer
</pre>

<pre>
Linux-crazy wrote on Wed, 21 Jan 2009 00:09:38 +0530:

> KeepAliveTimeout 150

My god, reduce this to 10 or 5.

> ServerLimit 251
> MaxClients 251

There's your limit. However, you should check with your hardware if upping
it is really desirable. With 4 GB I think you won't be able to handle much
more anyway.

Kai
</pre>

<pre>
Kai Schaetzl wrote:
> There's your limit. However, you should check with your hardware if upping
> it is really desirable. With 4 GB I think you won't be able to handle much
> more anyway.

That really depends. If you only shove out static pages and have one or
two or three odd cgis on the machine, you can flatten down the httpd
binary quite a bit by throwing out unneeded modules.

500 to 750 clients shouldn't be that much of a problem then ...

Cheers,

Ralph
</pre>

<pre>
Ralph Angenendt wrote on Wed, 21 Jan 2009 10:52:59 +0100:

> That really depends. If you only shove out static pages and have one or
> two or three odd cgis on the machine, you can flatten down the httpd
> binary quite a bit by throwing out unneeded modules.
>
> 500 to 750 clients shouldn't be that much of a problem then ...

Sure it depends ;-) I do serve dynamic pages but by removing the really
unnecessary modules I made my httpds much faster (especially on pipelined
image downloads) and they only have some 10 MB (RES) per worker.

But I would expect that they are running the default set of modules.

Kai
</pre>

<pre>
Kai Schaetzl schrieb:
> Ralph Angenendt wrote on Wed, 21 Jan 2009 10:52:59 +0100:
>
>
>> That really depends. If you only shove out static pages and have one or
>> two or three odd cgis on the machine, you can flatten down the httpd
>> binary quite a bit by throwing out unneeded modules.
>>
>> 500 to 750 clients shouldn't be that much of a problem then ...
>>
>
> Sure it depends ;-) I do serve dynamic pages but by removing the really
> unnecessary modules I made my httpds much faster (especially on pipelined
> image downloads) and they only have some 10 MB (RES) per worker.
>
> But I would expect that they are running the default set of modules.
>
> Kai
>
>

If you can separate-out the images to a specific URL (img.domain.com)
and put them all in the same directory, you can use NGINX to serve them.
Of course, the actual transfer-speed will not increase much - but
latency will go down to the absolute minimum.
And latency is what makes a page appear "fast" or "slow" to customers.
</pre>

<pre>
Ralph Angenendt wrote:
> Kai Schaetzl wrote:
>> There's your limit. However, you should check with your hardware if upping
>> it is really desirable. With 4 GB I think you won't be able to handle much
>> more anyway.
>
> That really depends. If you only shove out static pages and have one or
> two or three odd cgis on the machine, you can flatten down the httpd
> binary quite a bit by throwing out unneeded modules.
>
> 500 to 750 clients shouldn't be that much of a problem then ...

Yeah gotta monitor it.. just checked in on some servers I ran at
my last company and the front end proxies (99% mod_proxy) seem to
peak out traffic wise at about 230 workers. For no other reason other
than because I could each proxy has 8 apache instances running(4 for
HTTP 4 for HTTPS). CPU usage peaks at around 3%(dual proc single core),
memory usage around 800MB. About 100 idle workers. Keepalive was set
to 300 seconds due to poor application design.

One set of back end apache servers peaks traffic wise at around 100
active workers, though memory usage was higher, around 2GB because
of mod_fcgid running ruby on rails. CPU usage seems to be at around
25%(dual proc quad core) for that one application.

Each back end application had it's own dedicated apache instance(11
apps) for maximum stability/best performance monitoring. The bulk
of them ran on the same physical hardware though.

Traffic routing was handled by a combination of F5 load balancers
and the apache servers previously mentioned(F5 iRules were too slow
and F5's TMM was not scalable at the time).

nate
</pre>

<pre>
>
> > KeepAliveTimeout 150
>
>reduce this to 10 or 5.
>
>
> Kai
>
> --
> Kai Schätzl, Berlin, Germany

Kai,

what do you think about the "general Timeout

it is set to 300

ive never much thought about it, yet should we be consider and possible
reduce that one too?

- rh
</pre>

<pre>
RobertH wrote on Wed, 21 Jan 2009 11:26:41 -0800:

> what do you think about the "general Timeout
>
> it is set to 300
>
> ive never much thought about it, yet should we be consider and possible
> reduce that one too?

I'm using 120, but I don't think reducing this value has much impact.

Kai
</pre>

== Load Balancer ==

Apache Load Balancer

mod_proxy_balancer - Apache HTTP Server - mod_proxy_balancer - http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html

<pre>
<Proxy balancer://mycluster>
BalancerMember http://192.168.1.50:80
BalancerMember http://192.168.1.51:80
</Proxy>
ProxyPass /test balancer://mycluster
</pre>

== User Directories ==

Default:
UserDir public_html

Results in:
~/public_html/

To disable user directories:
UserDir disabled

Ubuntu user dir config:
<pre>
UserDir public_html
UserDir disabled root

<Directory /home/*/public_html>
AllowOverride FileInfo AuthConfig Limit Indexes
Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
<Limit GET POST OPTIONS>
Order allow,deny
Allow from all
</Limit>
<LimitExcept GET POST OPTIONS>
Order deny,allow
Deny from all
</LimitExcept>
</Directory>

</pre>

References:
* Per-user web directories - Apache HTTP Server - http://httpd.apache.org/docs/2.2/howto/public_html.html
* mod_userdir - Apache HTTP Server - http://httpd.apache.org/docs/2.2/mod/mod_userdir.html#userdir

== Minimal Memory Instance ==

Assuming running prefork module (check with 'apache -l'):

<pre>
KeepAlive off

<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 5
ServerLimit 20
MaxClients 20
MaxRequestsPerChild 10000
</IfModule>
</pre>

References:
* Running Apache On A Memory-Constrained VPS | Kalzumeus Software - http://www.kalzumeus.com/2010/06/19/running-apache-on-a-memory-constrained-vps/

---

TODO See:
* http://wiki.vpslink.com/Low_memory_MySQL_/_Apache_configurations
* http://www.narga.net/optimizing-apachephpmysql-low-memory-server/

== LDAP Authentication ==

# Red Hat / RHEL
sudo yum install mod_ldap
# allow ldap to connect through selinux:
sudo setsebool -P httpd_can_network_connect on

mod_ldap - Apache HTTP Server Version 2.4
https://httpd.apache.org/docs/2.4/mod/mod_ldap.html

== Issues ==

=== Could not reliably determine domain name ===

Warning:
Could not reliably determine the server's fully qualified domain name

Solution:
* Force ServerName in apache configs [http://askubuntu.com/questions/256013/could-not-reliably-determine-the-servers-fully-qualified-domain-name]
ServerName localhost

=== client denied by server configuration ===

browser:
403 Forbidden - You don't have permission to access / on this server.

Error:
<pre>
==> httpd/default-error_log <==
[Mon Sep 01 21:42:02.273169 2014] [authz_core:error] [pid 11221] [client 174.52.47.54:50675] AH01630: client denied by server configuration: /www/
</pre>

Add Directory allow permissions: [http://stackoverflow.com/questions/19445686/ubuntu-server-apache-2-4-6-client-denied-by-server-configuration-php-fpm]
<pre>
DocumentRoot /www
<Directory /www>
Order allow,deny
Allow from all
</Directory>
</pre>

or this worked better for me: [http://stackoverflow.com/questions/19445686/ubuntu-server-apache-2-4-6-client-denied-by-server-configuration-php-fpm] [https://mikegriffin.ie/blog/20140130-authz-core-error-client-denied-by-server-configuration/]
<pre>
DocumentRoot /www
<Directory /www>
Require all granted
</Directory>
</pre>

Actually this just worked on the latest Ubuntu system:
<pre>
<Directory /www/>
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>
</pre>

References:
* ClientDeniedByServerConfiguration - Httpd Wiki - https://wiki.apache.org/httpd/ClientDeniedByServerConfiguration

=== permissions missing component path ===

<pre>
==> httpd/default-error_log <==
[Mon Sep 01 21:50:09.256120 2014] [core:error] [pid 11511] (13)Permission denied: [client 174.52.47.54:50744] AH00035: access to \
/index.html denied (filesystem path '/www/index.html') because search permissions are missing on a component of the path
</pre>

== keywords ==

[[Category:Linux]]
[[Category:Web Development ‎]]
[[Category:Web]]
[[Category:HTML]]
[[Category:Internet]]

Apache

2026-04-27T19:13:44Z

Kenneth: /* .htaccess */

{{subpages}}

[[Category:Apache]]

== Install Apache ==

# Debian / Ubuntu
sudo apt install apache2

# Redhat
yum install httpd

== Virtual Hosting ==
touch /etc/httpd/conf.d/virtual.conf

Virtual Host Configuration File:
/etc/httpd/conf.d/virtual.conf:

HTTP Virtual Host
<pre>
NameVirtualHost *:80

<VirtualHost *:80>
ServerName t0e.org
ServerAlias www.t0e.org

DocumentRoot /www

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>

<VirtualHost *:80>
ServerName www2.t0e.org
...
</VirtualHost>
</pre>

HTTPS Virtual Host:
<pre>
NameVirtualHost *:443

<VirtualHost *:443>
ServerName t0e.org
ServerAlias www.t0e.org

# SSL Enabled:
SSLEngine on
SSLCertificateFile /etc/t0e.org.pem

# If using a godaddy certificate chain:
#SSLCertificateFile /etc/contractpal.net.pem
#SSLCertificateKeyFile /etc/contractpal.net.pem
#SSLCertificateChainFile /etc/contractpal.net.pem

#Forwarding Traffic
# RewriteEngine on
# RewriteRule ^(/.*) http://localhost:8080/$1 [P]

DocumentRoot /www

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>
</pre>

References:
*[http://www.onlamp.com/pub/a/apache/2005/02/17/apacheckbk.html A Day in the Life of #Apache: Running Name-Based SSL Virtual Hosts in Apache]
*[http://httpd.apache.org/docs/1.3/vhosts/name-based.html Apache: Name-based Virtual Host Support]
*[http://www.networkworld.com/columnists/2006/011606internet.html Getting SSL to work across virtual hosts in Apache]

== CentOS common configuration changes ==

Keep alives:
# KeepAlive Off
KeepAlive On

Hide Apache version (in likely in /etc/apache2/conf-enabled/security.conf):
# ServerTokens Full
ServerTokens Prod

# ServerSignature On
ServerSignature Off

---

PHP config in /etc/php.ini (now likely in /etc/php/8.3/apache2/php.ini):
expose_php = Off

==Forwarding / Rewriting / Proxying Traffic==

<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName t0e.org
ServerAlias www.t0e.org

#Forwarding Traffic
RewriteEngine on
RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ServerAdmin admin@t0e.org
ErrorLog logs/t0e.org-error_log
CustomLog logs/t0e.org-access_log common
</VirtualHost>
</pre>

Apache module mod_rewrite - RewriteRule - http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteRule
a2enmod rewrite

a2enmod proxy_http

[http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html Module mod_rewrite - URL Rewriting Engine]:
*RewriteEngine
*RewriteCond
*RewriteRule

[http://httpd.apache.org/docs/1.3/misc/rewriteguide.html Apache 1.3 - URL Rewriting Guide]

[http://httpd.apache.org/docs/1.3/mod/mod_proxy.html Apache module mod_proxy]:
*ProxyPass

---

<pre>
<VirtualHost *:80>
ServerName home.burgenerfamily.com
DocumentRoot /www/home
<Directory /www/home>
Options FollowSymLinks
RewriteEngine On
# allow existing files and directories to pass through
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
#RewriteRule ^static - [L,NC]
#RewriteCond %{REQUEST_FILENAME} !^favicon.ico$
#RewriteCond %{REQUEST_FILENAME} !^static/
#RewriteRule ^(.+)$ /index.php?title=$1 [PT,L,QSA]
RewriteRule ^(.*)$ /index.php/$1 [L,QSA]
</Directory>
ErrorLog logs/home.burgenerfamily.com-error_log
CustomLog logs/home.burgenerfamily.com-access_log common
</VirtualHost>
</pre>

== Proxy Pass ==

Enable Proxy: [https://www.digitalocean.com/community/tutorials/how-to-use-apache-as-a-reverse-proxy-with-mod_proxy-on-ubuntu-16-04]
sudo a2enmod proxy
sudo a2enmod proxy_http
systemctl restart apache2

Similar to forwarding, but takes care of redirects:

ProxyPass /mirror/foo/ http://foo.com/
ProxyPassReverse /mirror/foo/ http://foo.com/

Suppose the local server has address http://wibble.org/; then

ProxyPass /mirror/foo/ http://foo.com/
ProxyPassReverse /mirror/foo/ http://foo.com/

will not only cause a local request for the <http://wibble.org/mirror/foo/bar> to be internally converted into a proxy request to <http://foo.com/bar> (the functionality ProxyPass provides here). It also takes care of redirects the server foo.com sends: when http://foo.com/bar is redirected by him to http://foo.com/quux Apache adjusts this to http://wibble.org/mirror/foo/quux before forwarding the HTTP redirect response to the client.

<pre>
<VirtualHost *:80>
ServerName wiki.oeey.com

# Reverse Proxy
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/

ServerAdmin website@oeey.com
ErrorLog logs/wiki-error_log
CustomLog logs/wiki-access_log common
</VirtualHost>
</pre>

Proxy Pass to Jenkins: [https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache]
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName jenkins.oeey.com

#Forwarding Traffic
#RewriteEngine on
#RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ProxyPass / http://localhost:8080/ nocanon
ProxyPassReverse / http://localhost:8080/
ProxyRequests Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
<Proxy http://localhost:8080/*>
Order deny,allow
Allow from all
</Proxy>

ServerAdmin admin@oeey.com
ErrorLog logs/jenkins.oeey.com-error_log
CustomLog logs/jenkins.oeey.com-access_log common
</VirtualHost>
</pre>

=== Proxy Pass HTTPS ===

Proxy Pass to Jenkins:
<pre>
NameVirtualHost *:443
<VirtualHost *:443>
ServerName jenkins.oeey.com

# Enable SSL Support
SSLEngine on
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

#Forwarding Traffic
#RewriteEngine on
#RewriteRule ^(/.*) http://localhost:8080/$1 [P]

ProxyPass / http://localhost:8080/ nocanon
ProxyPassReverse / http://localhost:8080/
ProxyPassReverse / http://jenkins.oeey.com/
ProxyRequests Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode
RequestHeader set X-Forwarded-Proto "http"

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default (ie /etc/apache2/mods-enabled/proxy.conf in Ubuntu)
<Proxy http://localhost:8080/jenkins*>
Order deny,allow
Allow from all
</Proxy>

ServerAdmin admin@oeey.com
ErrorLog logs/jenkins.oeey.com-error_log
CustomLog logs/jenkins.oeey.com-access_log common
</VirtualHost>
</pre>

== Redirect ==

Redirect with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
Redirect permanent / http://www.oeey.com/
</VirtualHost>
</pre>

Redirect users to HTTPS with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
Redirect permanent / https://test.oeey.com/
</VirtualHost>
</pre>

Can be wrapped in Location (if you want)
<Location />
Redirect permanent / https://test.oeey.com/
</Location>

Redirect with RewriteEngine with URI intact:
<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com

# Redirect HTTP to HTTPS
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>
</pre>

Redirect removing URI:
NameVirtualHost *:80
<VirtualHost *:80>
ServerName test.oeey.com
RedirectMatch "(.*)" "http://www.oeey.com/"
</VirtualHost>

[http://httpd.apache.org/docs/2.0/mod/mod_alias.html Apache Module mod_alias]:
*Alias
*Redirect
*RedirectPermanent

== Default Directory Index ==

to set the default pages to try and load (e.g. index.html, index.php):
DirectoryIndex index.html index.php

== Directory Index ==

To show full file names:
<pre>
<Directory />
Options Indexes
IndexOptions NameWidth=*
</Directory>
</pre>

Sample:
<pre>
<Directory "/www">
Options Indexes FollowSymLinks
IndexOptions FancyIndexing NameWidth=*
AllowOverride None
Require all granted
</Directory>
</pre>

== Hide Apache and PHP Banner ==

Hide PHP version (X-Powered-By) - Edit php.ini: [http://www.ducea.com/2006/06/16/apache-tips-tricks-hide-php-version-x-powered-by/]
expose_php = Off

Hide Apache version: [http://www.itsolutionskb.com/2008/10/hide-apache-information/]
# ServerSignature On
ServerSignature Off

# ServerTokens Full
ServerTokens Prod

References:
* Hide Apache ServerSignature/ServerTokens/PHP X-Powered-By - http://www.if-not-true-then-false.com/2009/howto-hide-and-modify-apache-server-information-serversignature-and-servertokens-and-hide-php-version-x-powered-by/

== .htaccess ==

Override configuration settings on a per directory (and sub directories) level with:
.htaccess

Allow override of all configuration options:
<pre>
<Directory />
AllowOverride All
</Directory>
</pre>

Example directory browsing in .htaccess:
Options +indexes

Redirect to index.php:
<pre>
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?path=$1 [NC,L,QSA]
</pre>

== Allow from networks ==

<pre>
<Directory "...">
Order Deny,Allow
Deny from all
Allow from 127.0.0.0/255.255.255.0 10.10.0.0/16
</Directory>
</pre>

== Log Files ==

ErrorLog logs/test.com-error_log
CustomLog logs/test.com-access_log common

Make sure the soft link exists:
ln -s /var/log/httpd /etc/httpd/logs # RHEL based
ln -s /var/log/apache2 /etc/apache2/logs # Debian Based

*[http://httpd.apache.org/docs/2.0/logs.html Log Files v2.0]
*[http://httpd.apache.org/docs/1.3/logs.html#accesslog Log files v1.3]

===Log HTTPS X-Forwarded-For===
[http://www.webmasterworld.com/forum92/6134.htm Log the X-Forwarded-For] header in apache...
LogFormat "[X-Forwarded-For: %{X-Forwarded-For}i] %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "[X-Forwarded-For: %{X-Forwarded-For}i] %h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

== PHP ==

Install PHP with Apache: (Ubuntu)
#apt-get install apache2 php5 libapache2-mod-php5
apt-get install apache2 php libapache2-mod-php

== Apache SSL ==

Enable SSL with...

Apache can do virtual SSL hosting!

Install mod_ssl: [https://help.ubuntu.com/10.04/serverguide/httpd.html]
# redhat / rhel
sudo yum install mod_ssl

# debian / ubuntu
sudo a2enmod ssl
sudo a2ensite default-ssl
systemctl reload apache2

/etc/httpd/conf.d/ssl.conf:
<pre>
LoadModule ssl_module modules/mod_ssl.so
Listen 443
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
SSLPassPhraseDialog builtin
SSLSessionCache shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout 300
SSLMutex default
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>

Note: Delete the default VirtualHost section so we can add our own.

<pre>
# SSL Configuration - from ssl.conf virtual host example
SSLEngine On
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</pre>

<pre>
# SSL configuration
SSLEngine On
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile /etc/apache2/ssl/cert.crt
SSLCertificateKeyFile /etc/apache2/ssl/cert.key
SSLCertificateChainFile /etc/apache2/ssl/sf_bundle.crt
</pre>

<pre>
NameVirtualHost *:80
<VirtualHost *:80>
ServerName default.t0e.org

# Redirect HTTP to HTTPS
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

NameVirtualHost *:443
<VirtualHost *:443>
ServerName kb.t0e.org

# SSL Enabled: (only once in config file)
SSLEngine on
SSLCertificateFile /etc/t0e.org.pem
SSLCertificateKeyFile /etc/t0e.org.pem
SSLProtocol all -SSLv2
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

DocumentRoot /www

ServerAdmin admin@kb.t0e.org.com
ErrorLog logs/kb.t0e.org-error_log
CustomLog logs/kb.t0e.org-access_log common
</VirtualHost>

<VirtualHost *:443>
ServerName wiki.t0e.org

DocumentRoot /www

ServerAdmin admin@wiki.t0e.org.com
ErrorLog logs/wiki.t0e.org-error_log
CustomLog logs/wiki.t0e.org-access_log common
</VirtualHost>
</pre>

Note: With SSL, Apache will always use the first default virtual hosts' certificate, so it only makes sense to define it there.

References:
*[http://www.apache-ssl.org/ Apache SSL]
*[http://httpd.apache.org/docs/2.0/ssl/ Apache SSL/TLS Encryption Documentation]
*[http://httpd.apache.org/docs/2.0/mod/mod_ssl.html mod_ssl]
*[http://www.securityfocus.com/infocus/1818 Article: Apache 2 with SSL/TLS: Step-by-Step, Part 1]
*[http://www.vanemery.com/Linux/Apache/apache-SSL.html Van's Apache SSL/TLS mini-HOWTO]
*[http://tomcat.apache.org/tomcat-3.3-doc/tomcat-ssl-howto.html Tomcat SSL Configuration]
*[http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html Apache: SSL/TLS Strong Encryption: FAQ]
*[http://www.networkworld.com/columnists/2006/011606internet.html Getting SSL to work across virtual hosts in Apache]
*[http://www.faqs.org/docs/Linux-HOWTO/SSL-RedHat-HOWTO.html Building a Secure RedHat Apache Server HOWTO]
* CommonMisconfigurations - Httpd Wiki - http://wiki.apache.org/httpd/CommonMisconfigurations

--- chain ---

If you have a chain, create the pem as "key, cert, chain" and set Apache config like so:

SSLCertificateFile /etc/ssl.pem
SSLCertificateKeyFile /etc/ssl.pem
SSLCertificateChainFile /etc/ssl.pem

or just

SSLCertificateFile /etc/ssl.pem
SSLCertificateChainFile /etc/ssl.pem

=== Ubuntu SnakeOil Cert ===

Install the snakoil fake cert <ref>https://superuser.com/questions/58149/host-based-snakeoil-certificates-in-ubuntu</ref>

sudo apt-get install ssl-cert

sudo make-ssl-cert generate-default-snakeoil --force-overwrite

/etc/ssl/certs/ssl-cert-snakeoil.pem
/etc/ssl/private/ssl-cert-snakeoil.key

== Linux System Password Protection ==

Using system accounts for HTTP authentication - http://wiki.zs64.net/Using_system_accounts_for_HTTP_authentication

httpd.conf

Edit httpd.conf to add these lines to the general configuration:

LoadModule auth_external_module libexec/apache2/mod_auth_external.so

(Should have been added by installing the port already.)

Add these lines to each virtual host where you'd like to use system accounts for authentication:

AddExternalAuth pwauth /usr/local/bin/pwauth
SetExternalAuthMethod pwauth pipe

Add these lines to each Directory or Location section that you want protected by authentication:

AuthType Basic
AuthExternal pwauth
require valid-user

== Password Protection with .htaccess and htpasswd ==

*[http://www.javascriptkit.com/howto/htaccess3.shtml Comprehensive guide to .htaccess: Password protection]
*[http://apache-server.com/tutorials/ATusing-htaccess.html Using .htaccess Files with Apache]
*[http://www.4webhelp.net/tutorials/misc/htaccess.php Password Protection by Rod]
*[http://www.spoono.com/php/tutorials/tutorial.php?id=30 .htaccess Password Protection in PHP]

Sample .htaccess file:

AuthUserFile /var/www/htpasswd
AuthGroupFile /dev/null
AuthName "Secret Place"
AuthType Basic
Require valid-user

#Require user <user>
#Require group <group>

To use with httpd.conf or virtual.conf, wrap in "<Directory>" tags:
<Directory />
AuthUserFile ...
...
</Directory>

Note: if you want in your httpd.conf (or virtual.conf) surround with "<Location />...</Location>"

To add password file:
htpasswd -c /var/www/htpasswd admin
htpasswd /var/www/htpasswd user1

# if you want to create a blank username with password:
htpasswd /path/htpasswd ''

To use .htaccess:
<Directory />
Options FollowSymLinks
AllowOverride '''AuthConfig'''
</Directory>

Or user the [http://www.webmaster-toolkit.com/htaccess-generator.shtml .htaccess generator]

[http://httpd.apache.org/docs/2.2/mod/core.html#require Apache HTTP Server - Require Directive]:

Require user userid [userid] ...
:Only the named users can access the resource.
Require group group-name [group-name] ...
:Only users in the named groups can access the resource.
Require valid-user
:All valid users can access the resource.

<pre>
AuthType Basic
AuthName "Restricted Resource"
AuthUserFile /web/users
AuthGroupFile /web/groups
Require group admin
</pre>

=== web based htpasswd manager ===

see [[change_htpasswd.pl]]

[http://home.xnet.com/~efflandt/pub/htpasswd.pl htpasswd.cgi by David Efflandt]

Perl CGI Script to Modify User Passwords:

This allows users to manage / change their own passwords.

Use the Perl CGI script [http://home.xnet.com/%7Eefflandt/pub/htpasswd.pl htpasswd.pl] [[http://www.yolinux.com/TUTORIALS/src/htpasswd_pl.txt cache]]

* Edit location of Perl .i.e.: /usr/bin/perl
** Not /usr/local/bin/perl
* Edit the script to specify location of the password file i.e. /var/www/PasswordDir/.htpasswd
* SELinux users must add the correct attribute i.e. chcon -R -h -t httpd_sys_content_t /var/www/PasswordDir
* The password file must be located in a directory where CGI is allowed to modify files.

File: httpd.conf (portion):

<pre>
..
...
<Directory "/var/www/PasswordDir">
Options -Indexes
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
...
..
</pre>

==Directory Browsing==
DocumentRoot /www/files
<Directory />
Options FollowSymLinks '''+Indexes'''
AllowOverride None
</Directory>

Note: The '+' may not be needed.

In a .htaccess:
Options +indexes

Long file names: [http://www.networkworld.com/article/2311687/software/showing-long-file-names-in-apache-directory-listings.html]
IndexOptions NameWidth=*

For root indexes, the welcome.conf may be overriding with the welcome screen. Make the following change: [http://www.linuxquestions.org/questions/linux-general-1/apache-2-root-directory-listing-201356/]
<pre>
<LocationMatch "^/$>
#Options -Indexes
Options Indexes
ErrorDocument 403 /error/noindex.html
</LocationMatch>
</pre>

== cgi ==

Script Alias folder:
ScriptAlias /cgi-bin/ "C:/Program Files/Apache Group/Apache/cgi-bin/"
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

My favorite option, make all .cgi files executable: (This will need to be surrounded with '<Location />' or '<Directory />' tags)
AddHandler cgi-script .cgi
Options +ExecCGI

Note: Make sure to 'chmod +x' the files.

Example:
<pre>
<VirtualHost *:80>
ServerName hg.t0e.org

DirectoryIndex hgweb.cgi
DocumentRoot /www/hg

<Location />
AddHandler cgi-script .cgi
Options +ExecCGI
</Location>

ServerAdmin admin@hg.t0e.org
ErrorLog logs/hg.t0e.org-error_log
CustomLog logs/hg.t0e.org-access_log common
</VirtualHost>
</pre>

Source: [http://www.thesitewizard.com/archive/addcgitoapache.shtml How to Add CGI Script Support (Perl, Python, etc) to Your Apache Server on Windows (thesitewizard.com)]

--

Only files without an extension:
<pre>
<VirtualHost *:80>
ServerName pi.t0e.org
DocumentRoot /www/pi
<Directory />
AllowOverride All
</Directory>
DirectoryIndex index index.php index.html index.htm
<Files ~ "^[a-zA-Z]*$">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>
=== CentOS Default ===

<pre>
#
# ScriptAliases are essentially the same as Aliases, except that
# documents in the realname directory are treated as applications and
# run by the server when requested rather than as documents sent to the client.
# The same rules about trailing "/" apply to ScriptAlias directives as to
# Alias.
#
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/var/www/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
</pre>

== Python CGI ==

See [[Python#Apache_CGI]]

==mod_proxy==
*http://www.apachetutor.org/admin/reverseproxies
*http://www.linuxfocus.org/English/March2000/article147.html
*http://www.linuxhelp.net/guides/apacheproxy/
*http://webauth.stanford.edu/manual/mod/mod_proxy.html
*http://www.apachetutor.org/admin/reverseproxies
*http://confluence.atlassian.com/display/DOC/Using+Apache+with+mod_proxy
*http://www.serverwatch.com/tutorials/article.php/3290851
*http://apache.webthing.com/mod_proxy_html/
*http://www.ibm.com/developerworks/linux/library/wa-lampsec/index.html

== prefork vs worker ==

Prefork performs better. Worker uses less memory. Prefork is most commonly used, and is the better choice for a web server.

To see which is compiled, and look for prefork.c or worker.c:
<pre>
# httpd -l

Compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c
</pre>

== Statistics ==

=== Connections ===

Count active connections:
netstat -ant | grep ESTABLISHED | grep :80 | wc -l

From PHP:
<?php
$number_of_users = shell_exec('netstat -ant | grep ESTABLISHED | grep :80 | wc -l');
echo $number_of_users;
?>

References:
* [http://www.experts-exchange.com/OS/Linux/Q_22825240.html Find out how many connections are on an apache server : apache, connections, many, find, server]

=== [http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status] ===

[http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status - Apache HTTP Server]

Summary

The Status module allows a server administrator to find out how well their server is performing. A HTML page is presented that gives the current server statistics in an easily readable form. If required this page can be made to automatically refresh (given a compatible browser). Another page gives a simple machine-readable list of the current server state.

The details given are:

* The number of worker serving requests
* The number of idle worker
* The status of each worker, the number of requests that worker has performed and the total number of bytes served by the worker (*)
* A total number of accesses and byte count served (*)
* The time the server was started/restarted and the time it has been running for
* Averages giving the number of requests per second, the number of bytes served per second and the average number of bytes per request (*)
* The current percentage CPU used by each worker and in total by Apache (*)
* The current hosts and requests being processed (*)

The lines marked "(*)" are only available if ExtendedStatus is On.

<pre>
LoadModule status_module modules/mod_status.so

#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
#ExtendedStatus On

#
# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Change the ".example.com" to match your domain to enable.
#
#<Location /server-status>
# SetHandler server-status
# Order deny,allow
# Deny from all
# Allow from .example.com
#</Location>
</pre>

References:
* [http://httpd.apache.org/docs/2.0/mod/mod_status.html mod_status - Apache HTTP Server]
* [http://www.experts-exchange.com/OS/Linux/Q_22825240.html Find out how many connections are on an apache server : apache, connections, many, find, server]

== TRACE and TRACK ==

ISO Audit may result in this "Medium" risk detected:

:"HTTP Methods Allowed. It is observed that HTTP methods GET HEAD TRACE OPTIONS are allowed on this host. Disable all unnecessary methods."

:"HTTP TRACE / TRACK Methods Allowed. The remote webserver supports the TRACE and/or TRACK methods. TRACE and TRACK are HTTP methods that are used to debug web server connections. Disable these methods."

=== Disabling TRACE ===

[http://rackerhacker.com/2007/08/28/apache-disable-trace-and-track-methods/ Apache: Disable TRACE and TRACK methods | Racker Hacker]:

Lots of PCI Compliance and vulnerability scan vendors will complain about TRACE and TRACK methods being enabled on your server. Since most providers run Nessus, you'll see this fairly often. Here's the rewrite rules to add:

RewriteEngine on
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]

These directives will need to be added to each VirtualHost.

For apache version 1.3.34 (or later 1.3.x versions), or apache 2.0.55 (or later), this has been made easy. Just add the line TraceEnable off

TraceEnable off

[http://publib.boulder.ibm.com/httpserv/ihsdiag/http_trace.html Disabling the HTTP TRACE method]:

The HTTP TRACE request method causes the data received by IBM HTTP Server from the client to be sent back to the client, as in the following example:

<pre>
$ telnet 127.0.0.1 8080
Trying...
Connected to 127.0.0.1.
Escape character is '^]'.
TRACE / HTTP/1.0
Host: foo
A: b
C: d

HTTP/1.1 200 OK
Date: Mon, 04 Oct 2004 14:07:59 GMT
Server: IBM_HTTP_SERVER
Connection: close
Content-Type: message/http

TRACE / HTTP/1.0
A: b
C: d
Host: foo

Connection closed.
</pre>

The TRACE capability could be used by vulnerable or malicious applications to trick a web browser into issuing a TRACE request against an arbitrary site and then send the response to the TRACE to a third party using web browser features.

Making the required configuration changes:

<pre>
<VirtualHost www.example.com>
...
# disable TRACE in the www.example.com virtual host
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]
</VirtualHost>
</pre>

Using SSL:
$ /usr/linux/bin/openssl s_client -connect 127.0.0.1:8444

For more information...

More background information on the concerns with TRACE is provided at http://www.apacheweek.com/issues/03-01-24#news.

Other:
* Apache Tips: Disable the HTTP TRACE method | MDLog:/sysadmin - http://www.ducea.com/2007/10/22/apache-tips-disable-the-http-trace-method/

=== TraceEnable ===

Apache Core Features - http://httpd.apache.org/docs/1.3/mod/core.html#traceenable

TraceEnable:
Syntax: TraceEnable [on|off|extended]
Default: TraceEnable on
Context: server config
Status: core (Windows, NetWare)
Compatibility: Available only in Apache 1.3.34, 2.0.55 and later

This directive overrides the behavior of TRACE for both the core server and mod_proxy. The default TraceEnable on permits TRACE requests per RFC 2616, which disallows any request body to accompany the request. TraceEnable off causes the core server and mod_proxy to return a 405 FORBIDDEN error to the client.

Finally, for testing and diagnostic purposes only, request bodies may be allowed using the non-compliant TraceEnable extended directive. The core (as an origin server) will restrict the request body to 64k (plus 8k for chunk headers if Transfer-Encoding: chunked is used). The core will reflect the full headers and all chunk headers with the request body. As a proxy server, the request body is not restricted to 64k. At this time the Apache 1.3 mod_proxy does not permit chunked request bodies for any request, including the extended TRACE request.

=== Disabling TRACK ===

The HTTP TRACK method

The TRACK method is a type of request supported by Microsoft web servers. It is not RFC compliant and is not supported directly by IBM HTTP Server. The method may be utilized as part of a cross-site scripting attack. See Vulnerability Note VU#288308 for more information.

Even though IBM HTTP Server does not support the TRACK method natively, it is possible for plug-in modules to provide support for it. To disable this capability for plug-in modules, in addition to disabling the TRACE method, add these two additional directives after the existing RewriteCond and RewriteRule directives which are used to disable TRACE:

RewriteCond %{REQUEST_METHOD} ^TRACK
RewriteRule .* - [F]

Here is a full example showing the directives to disable both TRACE and TRACK:

<pre>
<VirtualHost www.example.com>
...
# disable TRACE and TRACK in the www.example.com virtual host
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]
RewriteCond %{REQUEST_METHOD} ^TRACK
RewriteRule .* - [F]
</VirtualHost>
</pre>

=== Attack ===

Cross-Site Tracing issues

Earlier this week a paper was published, [http://www.cgisecurity.com/whitehat-mirror/WH-WhitePaper_XST_ebook.pdf "Cross-Site Tracing"] which gave details of how the TRACE HTTP request could be used in Cross-Site Scripting attacks. Unfortunately this issue has not been very well understood by the media and has received a unwarranted amount of attention.

When an HTTP TRACE request is sent to a web server that supports it, that server will respond echoing the data that is passed to it, including any HTTP headers. The paper explains that some browsers can be scripted to perform a TRACE request. A browser with this functionality could be made to issue a TRACE request against an arbitrary site and pass the results on elsewhere. Since browsers will only send authentication details and cookies to the sites that issue them this means a user having a browser with this functionality could be tricked into sending their cookies or authentication details for arbitrary sites to an attacker.

For example, if you visited a page that an attacker has carefully crafted, the page could cause your browser to bounce a TRACE request against some site for which you have authentication cookies. The result of the TRACE will be a copy of what was sent to the site, which will therefore include those cookies or authentication data. The carefully crafted page can then pass that information on to the attacker.

TRACE requests can be disabled by making a change to the Apache server configuration. Unfortunately it is not possible to do this using the Limit directive since the processing for the TRACE request skips this authorisation checking. Instead the following lines can be added which make use of the mod_rewrite module.

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]

'''Although the particular attack highlighted made use of the TRACE functionality to grab authentication details, this isn't a vulnerability in TRACE, or in the Apache web server.''' The same browser functionality that permits the published attack can be used for different attacks even if TRACE is disabled on the remote web server. For example an attacker could create a carefully crafted page that when visited submits a hidden request to some arbitrary site through your browser, grabs the result and passes it to the attacker.

== Performance Tuning ==

See [[Linux Performance Tuning]]

[http://www.linuxjournal.com/article/6041 Apache and Firewall Performance Tips from the Xenu.net Masters]

=== CentOS January 2009 Thread ===

<pre>
-------- Original Message --------
Subject: [CentOS] Apache Server Tuning for Performance
Date: Wed, 21 Jan 2009 00:09:38 +0530
From: linux-crazy <hicheerup@gmail.com>
Reply-To: CentOS mailing list <centos@centos.org>
To: CentOS mailing list <centos@centos.org>

Hi all,

I am facing facing performance issues with our web servers which is
working for concurrent 250 requests properly and then stops
responding when the requests are more than 250 .

The current configuration parameters are as follows :

apachectl -version
Server version: Apache/2.0.52
Server built: Jan 30 2007 09:56:16

Kernel : 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64
x86_64 x86_64 GNU/Linux

Server Hardware :
MAIN MEMORY
(i) Memory Size 4 GB Dual-Core Intel 5160 processors.

httpd.conf

Timeout 300
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 150

## Server-Pool Size Regulation (MPM specific)
# prefork MPM

# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule prefork.c>

StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 251
MaxClients 251
MaxRequestsPerChild 4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

I want to know about the difference between worker MPM and
Prefork MPM , how to find out which one will be used by my apache
server and the recommended one for highly loaded server.If some one
provide me the link that best explains above two comparison also be
very use full.

Can any one guide me tuning to be done for the maximum utilization
of the Resources and better performance of the Servers.

Regards,
lingu
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
</pre>

<pre>
Only use prefork MPM (worker is for Win32, AFAIK).

If you set ServerLimit to 251, that's the limit.
Set ServerLimit and MaxClients to the same value.
(Larger than 250)
OK?
And take a couple of minutes to study the apache documentation. It's
actually quite good and tranlated into many languages...

BUT: what your server is being able to handle also depends on what you
actually serve (PHP/JSP/Servelets/Perl/whatever!

This is usually not a problem that is easily described and solved in
two sentences.

Rainer
</pre>

<pre>
Increase your MaxClients configuration, also enable if you
haven't already the server-status module and monitor the
server via http://<your server name>/server-status it will
show how many workers are busy and what they are doing.

As for which to use, prefork is the old forked method of
doing things, the other uses threads. Depending on what kind
of application your running on top of apache, if it is not
thread safe(at some point many PHP modules were not thread
safe), you may want to use prefork. Otherwise you can use
the threading model.

If your not sure I'd say stick to prefork to be safe until
you can determine for sure that threading is safe.

nate
</pre>

<pre>
Most list members would likely advise sticking with the prefork
configuration. Without knowing what kind of applications you are
running on your webserver, I wouldn't suggest changing it.

Merely increasing the number of workers might make performance worse.

Use ps or top to figure out how much each apache worker is using. Then
decide how much ram you want to dedicate on your server to Apache,
without going into swap. (Over-allocating and then paging out memory
will only make performance much worse.) For example, if I have 2G or
ram, and I want 1.5 for apache workers, my average apache worker size
(resident memory) is 65MB, then I have room for 23 workers. (1024 * 1.5
) / 65. (There are more accurate ways to calculate this usage, like
taking shared memory into account.)

Upgrading the ram in your web server is a pretty fast interim solution.

Consider your application performance, too. The longer a request in your
application takes, the more workers are in use on your web server,
taking up more memory. If you have long-running queries in your
database, take care of those first.

Good luck

Jed
</pre>

<pre>
Jed Reynolds wrote:
>
> Merely increasing the number of workers might make performance worse.
>
> Use ps or top to figure out how much each apache worker is using. Then
> decide how much ram you want to dedicate on your server to Apache,
> without going into swap. (Over-allocating and then paging out memory
> will only make performance much worse.) For example, if I have 2G or
> ram, and I want 1.5 for apache workers, my average apache worker size
> (resident memory) is 65MB, then I have room for 23 workers. (1024 * 1.5
> ) / 65. (There are more accurate ways to calculate this usage, like
> taking shared memory into account.)

Pay attention to shared memory when doing this. A freshly-forked
process shares virtually all RAM with its parent. How much and how
quickly this changes varies wildly with the application type and
activity that causes the child's data to become unique. With some types
of applications (especially mod_perl) you may want to tune down the
number of hits each child services to increase memory sharing. Also, if
you are running external cgi programs you must take them into account.

> Upgrading the ram in your web server is a pretty fast interim solution.
>
> Consider your application performance, too. The longer a request in your
> application takes, the more workers are in use on your web server,
> taking up more memory. If you have long-running queries in your
> database, take care of those first.

You may also have to turn off or tune down the HTTP 1.1 connection
keepalives, trading the time it takes to establish a new connection for
the RAM it takes to keep the associated process waiting for another
request from the same client.

--
Les Mikesell
lesmikesell@gmail.com
</pre>

<pre>
Also take a look at apache alternatives like lighttpd and nginx. In certain
cases they do miracles.

Jure Pečar
</pre>

<pre>
I don't know about lighttpd (our results were mixed) - but NGINX is
really _very_ fast.
Last time I checked, it seems to be the fastest way to accelerate
webpage-delivery on generic hardware and with OSS-software.

But it's not a feature-monster like apache, so you still need that.

Rainer
</pre>

<pre>
Linux-crazy wrote on Wed, 21 Jan 2009 00:09:38 +0530:

> KeepAliveTimeout 150

My god, reduce this to 10 or 5.

> ServerLimit 251
> MaxClients 251

There's your limit. However, you should check with your hardware if upping
it is really desirable. With 4 GB I think you won't be able to handle much
more anyway.

Kai
</pre>

<pre>
Kai Schaetzl wrote:
> There's your limit. However, you should check with your hardware if upping
> it is really desirable. With 4 GB I think you won't be able to handle much
> more anyway.

That really depends. If you only shove out static pages and have one or
two or three odd cgis on the machine, you can flatten down the httpd
binary quite a bit by throwing out unneeded modules.

500 to 750 clients shouldn't be that much of a problem then ...

Cheers,

Ralph
</pre>

<pre>
Ralph Angenendt wrote on Wed, 21 Jan 2009 10:52:59 +0100:

> That really depends. If you only shove out static pages and have one or
> two or three odd cgis on the machine, you can flatten down the httpd
> binary quite a bit by throwing out unneeded modules.
>
> 500 to 750 clients shouldn't be that much of a problem then ...

Sure it depends ;-) I do serve dynamic pages but by removing the really
unnecessary modules I made my httpds much faster (especially on pipelined
image downloads) and they only have some 10 MB (RES) per worker.

But I would expect that they are running the default set of modules.

Kai
</pre>

<pre>
Kai Schaetzl schrieb:
> Ralph Angenendt wrote on Wed, 21 Jan 2009 10:52:59 +0100:
>
>
>> That really depends. If you only shove out static pages and have one or
>> two or three odd cgis on the machine, you can flatten down the httpd
>> binary quite a bit by throwing out unneeded modules.
>>
>> 500 to 750 clients shouldn't be that much of a problem then ...
>>
>
> Sure it depends ;-) I do serve dynamic pages but by removing the really
> unnecessary modules I made my httpds much faster (especially on pipelined
> image downloads) and they only have some 10 MB (RES) per worker.
>
> But I would expect that they are running the default set of modules.
>
> Kai
>
>

If you can separate-out the images to a specific URL (img.domain.com)
and put them all in the same directory, you can use NGINX to serve them.
Of course, the actual transfer-speed will not increase much - but
latency will go down to the absolute minimum.
And latency is what makes a page appear "fast" or "slow" to customers.
</pre>

<pre>
Ralph Angenendt wrote:
> Kai Schaetzl wrote:
>> There's your limit. However, you should check with your hardware if upping
>> it is really desirable. With 4 GB I think you won't be able to handle much
>> more anyway.
>
> That really depends. If you only shove out static pages and have one or
> two or three odd cgis on the machine, you can flatten down the httpd
> binary quite a bit by throwing out unneeded modules.
>
> 500 to 750 clients shouldn't be that much of a problem then ...

Yeah gotta monitor it.. just checked in on some servers I ran at
my last company and the front end proxies (99% mod_proxy) seem to
peak out traffic wise at about 230 workers. For no other reason other
than because I could each proxy has 8 apache instances running(4 for
HTTP 4 for HTTPS). CPU usage peaks at around 3%(dual proc single core),
memory usage around 800MB. About 100 idle workers. Keepalive was set
to 300 seconds due to poor application design.

One set of back end apache servers peaks traffic wise at around 100
active workers, though memory usage was higher, around 2GB because
of mod_fcgid running ruby on rails. CPU usage seems to be at around
25%(dual proc quad core) for that one application.

Each back end application had it's own dedicated apache instance(11
apps) for maximum stability/best performance monitoring. The bulk
of them ran on the same physical hardware though.

Traffic routing was handled by a combination of F5 load balancers
and the apache servers previously mentioned(F5 iRules were too slow
and F5's TMM was not scalable at the time).

nate
</pre>

<pre>
>
> > KeepAliveTimeout 150
>
>reduce this to 10 or 5.
>
>
> Kai
>
> --
> Kai Schätzl, Berlin, Germany

Kai,

what do you think about the "general Timeout

it is set to 300

ive never much thought about it, yet should we be consider and possible
reduce that one too?

- rh
</pre>

<pre>
RobertH wrote on Wed, 21 Jan 2009 11:26:41 -0800:

> what do you think about the "general Timeout
>
> it is set to 300
>
> ive never much thought about it, yet should we be consider and possible
> reduce that one too?

I'm using 120, but I don't think reducing this value has much impact.

Kai
</pre>

== Load Balancer ==

Apache Load Balancer

mod_proxy_balancer - Apache HTTP Server - mod_proxy_balancer - http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html

<pre>
<Proxy balancer://mycluster>
BalancerMember http://192.168.1.50:80
BalancerMember http://192.168.1.51:80
</Proxy>
ProxyPass /test balancer://mycluster
</pre>

== User Directories ==

Default:
UserDir public_html

Results in:
~/public_html/

To disable user directories:
UserDir disabled

Ubuntu user dir config:
<pre>
UserDir public_html
UserDir disabled root

<Directory /home/*/public_html>
AllowOverride FileInfo AuthConfig Limit Indexes
Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
<Limit GET POST OPTIONS>
Order allow,deny
Allow from all
</Limit>
<LimitExcept GET POST OPTIONS>
Order deny,allow
Deny from all
</LimitExcept>
</Directory>

</pre>

References:
* Per-user web directories - Apache HTTP Server - http://httpd.apache.org/docs/2.2/howto/public_html.html
* mod_userdir - Apache HTTP Server - http://httpd.apache.org/docs/2.2/mod/mod_userdir.html#userdir

== Minimal Memory Instance ==

Assuming running prefork module (check with 'apache -l'):

<pre>
KeepAlive off

<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 5
ServerLimit 20
MaxClients 20
MaxRequestsPerChild 10000
</IfModule>
</pre>

References:
* Running Apache On A Memory-Constrained VPS | Kalzumeus Software - http://www.kalzumeus.com/2010/06/19/running-apache-on-a-memory-constrained-vps/

---

TODO See:
* http://wiki.vpslink.com/Low_memory_MySQL_/_Apache_configurations
* http://www.narga.net/optimizing-apachephpmysql-low-memory-server/

== LDAP Authentication ==

# Red Hat / RHEL
sudo yum install mod_ldap
# allow ldap to connect through selinux:
sudo setsebool -P httpd_can_network_connect on

mod_ldap - Apache HTTP Server Version 2.4
https://httpd.apache.org/docs/2.4/mod/mod_ldap.html

== Issues ==

=== Could not reliably determine domain name ===

Warning:
Could not reliably determine the server's fully qualified domain name

Solution:
* Force ServerName in apache configs [http://askubuntu.com/questions/256013/could-not-reliably-determine-the-servers-fully-qualified-domain-name]
ServerName localhost

=== client denied by server configuration ===

browser:
403 Forbidden - You don't have permission to access / on this server.

Error:
<pre>
==> httpd/default-error_log <==
[Mon Sep 01 21:42:02.273169 2014] [authz_core:error] [pid 11221] [client 174.52.47.54:50675] AH01630: client denied by server configuration: /www/
</pre>

Add Directory allow permissions: [http://stackoverflow.com/questions/19445686/ubuntu-server-apache-2-4-6-client-denied-by-server-configuration-php-fpm]
<pre>
DocumentRoot /www
<Directory /www>
Order allow,deny
Allow from all
</Directory>
</pre>

or this worked better for me: [http://stackoverflow.com/questions/19445686/ubuntu-server-apache-2-4-6-client-denied-by-server-configuration-php-fpm] [https://mikegriffin.ie/blog/20140130-authz-core-error-client-denied-by-server-configuration/]
<pre>
DocumentRoot /www
<Directory /www>
Require all granted
</Directory>
</pre>

Actually this just worked on the latest Ubuntu system:
<pre>
<Directory /www/>
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>
</pre>

References:
* ClientDeniedByServerConfiguration - Httpd Wiki - https://wiki.apache.org/httpd/ClientDeniedByServerConfiguration

=== permissions missing component path ===

<pre>
==> httpd/default-error_log <==
[Mon Sep 01 21:50:09.256120 2014] [core:error] [pid 11511] (13)Permission denied: [client 174.52.47.54:50744] AH00035: access to \
/index.html denied (filesystem path '/www/index.html') because search permissions are missing on a component of the path
</pre>

== keywords ==

[[Category:Linux]]
[[Category:Web Development ‎]]
[[Category:Web]]
[[Category:HTML]]
[[Category:Internet]]

Python

2026-04-27T19:11:56Z

Kenneth: /* Apache CGI */

{{subtop}}

== Python ==

Python (programming language) - Wikipedia - http://en.wikipedia.org/wiki/Python_(programming_language)
: Python is an interpreted, general-purpose high-level programming language whose design philosophy emphasizes code readability. Python aims to combine "remarkable power with very clear syntax", and its standard library is large and comprehensive. Its use of indentation for block delimiters is unique among popular programming languages.
: Python supports multiple programming paradigms, primarily but not limited to object-oriented, imperative and, to a lesser extent, functional programming styles. It features a fully dynamic type system and automatic memory management, similar to that of Scheme, Ruby, Perl, and Tcl. Like other dynamic languages, Python is often used as a scripting language, but is also used in a wide range of non-scripting contexts.
: The reference implementation of Python ([http://en.wikipedia.org/wiki/CPython CPython]) is free and open source software and has a community-based development model, as do all or nearly all of its alternative implementations. CPython is managed by the non-profit Python Software Foundation.
: Python interpreters are available for many operating systems, and Python programs can be packaged into stand-alone executable code for many systems using various tools.

Author:
:"Python was conceived in the late 1980s and its implementation was started in December 1989 by '''[http://www.python.org/~guido/ Guido van Rossum]''' at CWI in the Netherlands as a successor to the ABC programming language (itself inspired by SETL) capable of exception handling and interfacing with the Amoeba operating system. Van Rossum is Python's principal author, and his continuing central role in deciding the direction of Python is reflected in the title given to him by the Python community, Benevolent Dictator for Life (BDFL)." [http://en.wikipedia.org/wiki/Python_(programming_language)]

Name:
:"The language is named after the BBC show “Monty Python’s Flying Circus” and has nothing to do with reptiles. Making references to Monty Python skits in documentation is not only allowed, it is encouraged!" [http://docs.python.org/py3k/tutorial/appetite.html]

We as people are "Pythonites" [http://raspi.tv/2013/how-to-use-interrupts-with-python-on-the-raspberry-pi-and-rpi-gpio-part-2]

== [http://www.python.org/dev/peps/pep-0020/ The Zen of Python] ==

"The core philosophy of the language is summarized by the document 'PEP 20 (The Zen of Python)'" [http://en.wikipedia.org/wiki/Python_%28programming_language%29]
* PEP 20 -- The Zen of Python - http://www.python.org/dev/peps/pep-0020/

<pre>
Abstract

Long time Pythoneer Tim Peters succinctly channels the BDFL's
guiding principles for Python's design into 20 aphorisms, only 19
of which have been written down.

The Zen of Python

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Easter Egg

>>> import this
</pre>

Other Sources:
* http://www.python.org/doc/humor/
* The Zen of Python – as a Poster « CODE POETRY - http://codepoetry.wordpress.com/2010/03/23/the-zen-of-python-as-a-poster/
** Zen of Python (PDF) http://codepoetry.files.wordpress.com/2010/03/zen-of-python.pdf

== Documentation ==

* Overview — Python v2.7.1 documentation - http://docs.python.org/index.html
** Download — Python v2.7.1 documentation - http://docs.python.org/download.html

* Overview — Python v3.2 documentation - http://docs.python.org/py3k/
** Download — Python v3.2 documentation - http://docs.python.org/py3k/download.html

* PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/

== Tutorials ==

* The Python Tutorial — Python v2.7.1 documentation - http://docs.python.org/tutorial/

* Python 101 -- Introduction to Python - http://www.rexx.com/~dkuhlman/python_101/python_101.html

* 4. More Control Flow Tools — Python v2.7.1 documentation - http://docs.python.org/tutorial/controlflow.html

== Linux Installation ==

RHEL Distribution installation:
yum -y install python python-setuptools

Ubuntu Distribution installation: [https://help.ubuntu.com/community/Mercurial]
# enable 'universe' in the sources.list
# "Mercurial is included in the Universe repository - activate this repo first"
sudo apt-get install mercurial

CentOS 5 Option Recommended Dependencies:
yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel

# this leaves: 2.6.6: _bsddb, bsddb185, dl, imageop, sunaudiodev
# this leaves: 2.7.1: _bsddb, bsddb185, dl, imageop, sunaudiodev
# this leaves: 3.2: none

CentOS 9 Option Recommended Dependencies:
# gdbm-devel not found in centos9?
yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel sqlite-devel bzip2-devel tk-devel tcl-devel

Ubuntu 12.04 LTS 64bit dependencies:
apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev

# maybe: lib64readline-dev libsqlite3-dev tk-dev

# this leaves: 2.7: _bsddb, bsddb185, dl, imageop, sunaudiodev

=== 2.x ===

CentOS dependencies:
sudo yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel

Ubuntu 12.04 LTS 64bit dependencies:
sudo apt-get update
sudo apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev

Ubuntu 24.04 LTS 64bit dependencies:
sudo apt-get update
sudo apt-get install gcc make sudo libbz2-dev tcl-dev libncurses-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev libffi-dev
# optional install
sudo apt-get install checkinstall

2.7.15:
<pre>
# PYVER=2.6.6
# PYVER=2.7.5
# PYVER=2.7.6
# PYVER=2.7.9
# PYVER=2.7.10
# PYVER=2.7.15
PYVER=2.7.18
mkdir -p ~/.src ; cd ~/.src
wget --no-check-certificate http://www.python.org/ftp/python/$PYVER/Python-$PYVER.tgz
tar -zvxf Python-$PYVER.tgz
cd Python-$PYVER
# ./configure --prefix=/opt/python-$PYVER
# make clean && make
./configure --prefix=/opt/python-$PYVER && make clean && make
sudo make install

sudo /opt/python-$PYVER/bin/python -m ensurepip --upgrade
sudo /opt/python-$PYVER/bin/pip install --upgrade pip
sudo /opt/python-$PYVER/bin/pip install requests

export PATH=/opt/python-$PYVER/bin:$PATH
echo -e "\n\nexport PATH=/opt/python-$PYVER/bin:\$PATH\n" >> ~/.bash_profile
# echo -e "\n\nexport PATH=/opt/python27/bin:\$PATH\n" >> ~/.bashrc

# NOTE: remove any old versions from your .bash_profile
</pre>

Bootstrapping the pip installer - https://docs.python.org/2/library/ensurepip.html

=== 3.x ===

3.8.16:
<pre>
#VER=3.7.0
#VER=3.8.15
#VER=3.10.9
VER=3.8.16
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/${VER}/Python-${VER}.tgz
tar -zvxf Python-${VER}.tgz
cd Python-${VER}
./configure --prefix=/opt/python${VER}
make

sudo make install

## OPTIONAL alternative install path:
sudo checkinstall
## note: 'checkinstall' makes a .deb, which you can uninstall with easier, but if you are using --prefix, you do need to
## premake the folders
sudo mkdir -p /opt/python${VER}/bin
sudo mkdir -p /opt/python${VER}/lib
shortVER=`echo $VER | cut -d '.' -f 1-2` # get short version eg. 3.10 from 3.10.9
sudo mkdir -p /opt/python${VER}/lib/python${shortVER}
sudo mkdir -p /opt/python${VER}/lib/python${shortVER}/test
# sudo checkinstall # hit Y, and enter twice
## note: older versions have problems with the virtualization of the file system, so do FS Translation = no
checkinstall --fstrans=no

export PATH=/opt/python${VER}/bin:$PATH
echo -e "\n\nexport PATH=/opt/python${VER}/bin:\$PATH\n" >> ~/.bash_profile

# have to manually update version
rm /opt/python${VER}/bin/python
ln -sfn python3.8 /opt/python${VER}/bin/python
#cp /opt/python${VER}/bin/python3.8 /opt/python${VER}/bin/python

# DEPENDENCIES
sudo apt install ...
yum install libffi-devel openssl-devel
</pre>

Note, if you want the libraries to be shared (like python-dev):
./configure --enable-shared

---

VER=3.6.9

VER=3.6.8

---

3.2.0:
<pre>
mkdir -p ~/src ; cd ~/src
wget http://www.python.org/ftp/python/3.2/Python-3.2.tgz
tar -zvxf Python-3.2.tgz
cd Python-3.2
./configure --prefix=/opt/python32
make
sudo make install

export PATH=/opt/python32/bin:$PATH
echo -e "\n\nexport PATH=/opt/python32/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.3.0:
<pre>
mkdir -p ~/src ; cd ~/src
wget http://www.python.org/ftp/python/3.3.0/Python-3.3.0.tgz
tar -zvxf Python-3.3.0.tgz
cd Python-3.3.0
./configure --prefix=/opt/python3.3.0
make
sudo make install

export PATH=/opt/python3.3.0/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.3.0/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.3.4:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/3.3.4/Python-3.3.4.tgz
tar -zvxf Python-3.3.4.tgz
cd Python-3.3.4
./configure --prefix=/opt/python3.3.4
make clean
make
sudo make install

export PATH=/opt/python3.3.4/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.3.4/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.4.0:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz
tar -zvxf Python-3.4.0.tgz
cd Python-3.4.0
./configure --prefix=/opt/python3.4.0
make
sudo make install

export PATH=/opt/python3.4.0/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.4.0/bin:\$PATH\n" >> ~/.bash_profile
</pre>

=== setuptools ===

Purpose: to install pip

Note: required for several 'python setup.py install' installations.

Python Package Index : setuptools 0.6c11 - http://pypi.python.org/pypi/setuptools

By Package Manager:
yum install python-setuptools

By Source:
#VER=2.0.1
#VER=2.2
VER=3.4.4
mkdir -p ~/.src ; cd ~/.src
wget --no-check-certificate http://pypi.python.org/packages/source/s/setuptools/setuptools-$VER.tar.gz
tar -zvxf setuptools-$VER.tar.gz
cd setuptools-$VER
# sudo python setup.py install
sudo /opt/python-2.7.9/bin/python setup.py install

Fix header:
# Only needed on older versions
sed -i 's#!/usr/bin/python#!/usr/bin/env python#g' /opt/python279/bin/easy_install

Install pip:
sudo /opt/python-2.7.9/bin/easy_install pip

Sample pip usage:
sudo /opt/python-2.7.9/bin/pip install paramiko

== IDLE and Interactive Mode ==

Interactive Mode - "When commands are read from a tty, the interpreter is said to be in interactive mode. In this mode it prompts for the next command with the primary prompt, usually three greater-than signs (>>>); for continuation lines it prompts with the secondary prompt, by default three dots (...)." [http://docs.python.org/2/tutorial/interpreter.html#invoking-the-interpreter]

Can use '_' to reference last output:
>>> 5 + 6
11
>>> print _
11

References:
* 2. Using the Python Interpreter — Python v2.7.3 documentation - http://docs.python.org/2/tutorial/interpreter.html

== Python 2.7 EOL ==

<blockquote>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
</blockquote>

== Show All Python in Path ==

compgen -c | grep -i "python"

== Code ==

=== Hello World ===

[http://en.wikipedia.org/wiki/Shebang_%28Unix%29 shebang] (also called a hashbang) of Python:
#!/usr/bin/env python
print "Hello World"

NOTE: can also just use "#!/usr/bin/python"

=== Standard Script ===

Standard Script:
<pre>
def main(args):
# do logic

if __name__ == "__main__":
sys.exit(main(sys.argv))
</pre>

Needed for things like pylint

=== White Space ===

Python uses white space to indicate code blocks. Instead of ''{'' use a '':'' and indent consistently (4 spaces).

if a == b:
print 'equal'

Convert tabs to spaces:
sed -i 's/\t/ /g' *.py

Remove trailing whitespace:
sed -i 's/\s*$//' *.py
sed --in-place 's/[[:space:]]\+$//' fileName

Clean project:
# find . -name "*py" -exec dos2unix {} \;
# find . -name "*py" -exec sed -i 's/\t/ /g' {} \;
# find . -name "*py" -exec sed -i 's/\s*$//' {} \;
find . -name "*py" -exec dos2unix {} \; -exec sed -i 's/\t/ /g' {} \; -exec sed -i 's/\s*$//' {} \;

==== Trim Tailing White Space ====

<pre>
sed -e 's/[[:space:]]*$//'
</pre>

<pre>
find . -name "*.py" -exec sed -i -e 's/[[:space:]]*$//' {} \;
</pre>

ref: [https://stackoverflow.com/questions/369758/how-to-trim-whitespace-from-a-bash-variable]

=== Print ===

print - http://docs.python.org/tutorial/inputoutput.html

Print: # include new line
print("Hello World")
print "Hello World" # python 2 only

Suppress new line: [http://stackoverflow.com/questions/493386/how-to-print-in-python-without-newline-or-space]
<pre>
print('#', end='') # python 3
print "Hello World", # python 2 only (does produce a single separating white space though)

import sys
sys.stdout.write('.') # no separating white space
</pre>

# flush output
# optional, if new line won't come for some time
import sys
sys.stdout.flush()

Multiple with separator:
print("hello", "world") # separated by spaces
print("hello", "world", sep = '.') # separated by period

Formatting:
<pre>
mystr = "abc"
myint = 5
myflt = 3.141592
print( "%s %d %.3f" % (mystr, myint, myflt) )
print( "{0} {1} {2}".format(mystr, myint, myflt) ) # string's format function (python > 2.7)
print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name
print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding
</pre>

Padded floating point decimal number: (rounds)
print "%06.2f" % 2.319 # "002.32"
print "%6.2f" % 2.311 # " 2.31"
print "{:06.2f}".format(2.319) # "002.32"
print "{:6.2f}".format(2.319) # " 2.32"

# Python 3 version where var=2.321234
print(f"{var:.2f}") # "2.32"
print(f"{var:6.2f}") # " 2.32"
print(f"{var:06.2f}") # "002.32"
print(f"{var:08.2f}") # "00002.32"
print(f"{var:08.4f}") # "002.3212"
# Python 3 version where var=2.329 - rounding
print(f"{var:06.2f}") # "002.33"

=== Comments ===

# this is a comment
# this is a comment
x = y # this is a comment

Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended)
"""
multi line comment
"""

'''
multi line comment can comment out """
'''

This can be used like a C "#if 0" concept to comment out whole sections of code.

Using false block to comment out a block code: (does not ignore bad syntax though)
if False:
# code in this block is ignored

=== doc strings ===

Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended)
"""
multi line comment
"""

=== Help ===

dir - list functions and properties of objects:
import math
dir(math) # module
dir(dir) # function
dir("") # object - "", [], {}, etc...
dir(__builtins__) # built in functions
dir() # list imported objects, and dunder methods

Note: dunder (double under) methods determine special functionality (eg. __add__ for +)

help - manual printout: (class doc string, function doc string, data, etc)
import math
help(math) # full module info
help(len) # full function info
help(["foo"]) # full object info

Pydoc: (command line, same as 'import math, help(math)'
$ pydoc math

Documentation String: (aka docstring, doc string)
print(math.pow.__doc__) # module doc string
print(len.__doc__) # function doc string

Object or function type:
type(x)

mylist = []
isinstance(mylist, list) # True
type(s) is list # True

Note: isinstance() and type() not equivalent when objects and subclass get involved

IDLE interactive help:
help()

=== Variables ===

Basic Data Types
str = "string" # string
num = 5 # integer
flt = 5.0 # float
nothing = None # None (not the same as undefined)

Undefined. Variables start off in an 'undefined' state. It is always better to initially set the variable to None to avoid this state. To check if undefined: [http://code.activestate.com/recipes/59892-testing-if-a-variable-is-defined/] [http://code.activestate.com/recipes/59892/]
<pre>
try:
x
except NameError:
x = None
</pre>

Rules: [http://www.python.org/dev/peps/pep-0008/ PEP8 - Method Names and Instance Variables]
* any length
* letters, numbers, underscore
* first character can't be number
* case sensitive
* cannot use keywords

Multiple assignment:
x = y = z = 0 # Zero x, y and z
x, y, z = 1, "two", 3.0 # x = 1, y = "two", z = 3.0

Swap variables: (done in parallel)
a, b = b, a

Unpack variables: (requires equal items on each side)
a, b, c = [1, 2, 3]

Other Data Types:
* tuple
* list
* dictionary
* set
* objects

Everything in Python is an object that has:
* identity: id(obj)
* type: type(obj)
* value: obj
** mutable - id() stays the same on modification (dictionary, lists, objects)
** immutable - id() changes on modification as new object is created (string, integer, tuple)

==== Check if Variable Exists ====

<pre>
try:
a # does a exist in the current namespace
except NameError:
a = 10 # nope
</pre>

'a' in vars() or 'a' in globals()
'a' in vars(__builtins__)

<pre>
if hasattr(a, 'property'):
doStuff(a.property)
else:
otherStuff()
</pre>

Ref: [https://stackoverflow.com/questions/1592565/determine-if-variable-is-defined-in-python] [https://stackoverflow.com/questions/610883/how-to-know-if-an-object-has-an-attribute-in-python]

=== Numbers ===

num = 5
print "num: %d" % num

>>> print "Today's stock price: %f" % 50.4625 1
50.462500
>>> print "Today's stock price: %.2f" % 50.4625 2
50.46
>>> print "Change since yesterday: %+.2f" % 1.5 3
+1.50

Arithmetic:
+ - * /
// # integer division
% # modulus
** # exponent

Compact Arithmetic:
i+=1 # Increment - Note, there is no 'i++'
i*=2 # Multiply
i%=2 # modulus
...

Math Functions:
import math
math.ceil(x)
math.floor(x)
math.trunc(x)
math.pow(x,y)

abs(x) # built in

String interpolation:
%% % character
%d integer
%x hex
%X hex (upper)
%f float
%s string

x = 1/81
print('value: %.2f' % x) # 0.01
print('value: %.5f' % x) # 0.012345
print('%d %d %d' % (1, 2, 3))

Type check:
if not isinstance(myvar, (int, float)): ...

=== Strings ===

STRINGS ARE IMMUTABLE! (to modify convert to a list)

Strings: (String Constants)
str = "Hello"
str = 'Hello'
str = """
Hello
World
""" # can also use '''...'''

String literals can span multiple lines: (no wrapping when \ used)
hello = "hello\
world"

Length:
len("str")

Raw string: (back slashes are not escaped)
myrawstr = r"hello\nworld\
how are you"
myrawstr = r"C:\" # ERROR - limitation of what won't be escaped

Concatenation:
str = 'hot' + 'dog'
print "Hello" "World"

Can't concat numbers
str = 5 + 'test' # error
num = 5
str = num + 'test' # error
str = str(5) + str(num) + 'test' # ok

Repeated Concatenation:
str = 10 * 'ha'
str = 'ha' * 10 # same
print '-'*60 # --------------------

String Indexing:
s = 'hello'
print( s[0] ) # is 'h'
print( s[-1] ) # is 'o'
print( s[len(s) - 1 ] ) # is also 'o'

Modify Strings: [http://stackoverflow.com/questions/1228299/change-one-character-in-a-string-in-python]
s[1] = 'x' # ERROR!
ERROR: can't modify string by index

# solution:
s = list("Hello World")
s[0] = 'C'
print( "".join(s) )

String Slicing:
s = 'hello'
print( s[0:3] ) # 'hel'
i = 1
print( s[i:i+1] ) # 'e'
print( s[:3] ) # 0:3 'hel'
print( s[4:] ) # 4:end 'o'
print( s[:] ) # start:end 'hello'
print( s[-2:] ) # -2:end 'lo'

For loop:
s = 'hello'
for c in s:
print c # c = 'h' first time

Escape characters:
\\ \' \" \n \r \t

Alternate loop:
s = 'hello'
for i in range( len(s) ):
print s[i]

Ord and Char (ASCII):
ord('a') # 97
chr(97) # 'a'

0-9 : 48-57
A-Z : 65-90
a-z : 97-122

List Functions: (dir on any string)
dir('')
dir(str)

Help:
help('')
help(str)

Padding:
"a".rjust(2, '0') # '0a'
"1".ljust(4, '0') # '1000'

String Test Functions:
"HELLO".isupper() # True
"hello".islower() # True
"hello".isalpha() # True
"12.22".isdigit() # False (because of '.')
"12.22".isnumeric() # False (because of '.')
"hel2".isalnum() # True
"e" in "hello" # True
"hello".startswith('he') # True
"hello".endswith('lo') # True

Note: To check for floats: [http://stackoverflow.com/questions/354038/how-do-i-check-if-a-string-is-a-number-in-python]
<pre>
def is_number(s):
try:
float(s)
return True
except ValueError:
return False
</pre>

String Modify Functions: (sort of... Strings are immutable)
"HELLO".lower() # 'hello'
"hello".upper() # 'HELLO'
" hello ".strip() # 'hello' remove leading/tailing whitespace
" hello ".lstrip() # 'hello ' remove left whitespace
" hello ".rstrip() # ' hello' remove right whitespace
"www.com".split('.') # ['www', 'com'] <- list
"www.com".partition('.') # ('www', '.', 'com') <- tuple
"".join( ('a','b','c') ) # "abc" join strings, list or tuple
"-".join( ('a','b','c') ) # "a-b-c" join with separator
"23".zfill(4) # "0023"

String Search Functions:
"hello".find('lo') # 3 (-1 on fail)
"hello".index('lo') # 3 (exception on fail)
"hello".rfind('lo') # 3 right to left (-1 on fail)
"hello".rindex('lo') # 3 right to left (exception on fail)

Formatting: (form of concatenation)
<pre>
mystr = "abc"
myint = 5
myflt = 3.141592
print( "%s %d %.3f" % (mystr, myint, myflt) )
print( "{0} {1} {2}".format(mystr, myint, myflt) ) # pep 3101 style - string's format function (python > 2.7)
print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name
print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding
</pre>

Split multi line string:
for line in str.split('\n'): print(line)
lines = str.splitlines() # returns list
lines = str.splitlines(True) # returns list (each includes new line characters)

Input a string:
name = input('What is your name: ').strip() # python 3
name = raw_input('What is your name: ').strip() # python 2

Leet Speak:
leet_table = ''.maketrans('EIOBT', '31087')
'BE COOL. SPEAK LEET!'.translate(leet_table)

String Functions:
newstr = mystr.strip() # remove white space on both sides
newstr = mystr.rstrip() # remove white space on right side
newstr = mystr.lstrip() # remove white space on left side
newstr = mystr.replace('\n', ' ') # replace string with another
newstr = mystr.replace('jello', 'hello') # replace string with another

==== String Formatting ====

PyFormat: Using % and .format() for great good! - https://pyformat.info/

===== Zero Padding =====

3 zero padding:
"{:03}".format(1) # 003

3 zero padding, specify position:
"{1:03}".format(1, 2) # 002

Old method:
"%03d" % 2 # 002

=== chr and ord ===

Ord and Char (ASCII):
ord('a') # 97
chr(97) # 'a'

=== Range ===

Creates a list of numbers. Remember: range always ends one less than you want!

Count:
ten = 10
one = 1
range(ten) # [0,1,...,9] starts with zero by default
range(one, ten + 1) # [1,...,10] actually get 1 to 10
range(ten, one - 1, -1) # [10,9,...,1] count down
range(2, 10 + 1, 2) # [2,4,...,10] even numbers

=== drange ===

Solution for decimal range() step value:
<pre>
def drange(start, stop, step):
r = start
while r < stop:
yield r
r += step

sub_ten = drange(0.0, 1.0, 0.1)
</pre>

References:
* floating point - Python decimal range() step value - Stack Overflow - http://stackoverflow.com/questions/477486/python-decimal-range-step-value

=== Casting ===

Convert:
float(3)
str(85)
int(8.6)
round(5.5)

Check Type:
type(5) # <class 'int'>
type(5.0) # <class 'float'>
type('5') # <class 'str'>
type(None) # <class 'NoneType'>
type(print) # <class 'builtin_function_or_method'>
type(()) # <class 'tuple'>
type([]) # <class 'list'>

mytype = int
if type(1) is mytype: ...

=== Boolean Logic ===

True and False:
bol = True
bol = False

Boolean Operators: (in order precedence)
a == b
a != b
not a
a and b
a or b

Note: Short-circuit evaluated - 'and' and 'or'

Return Boolean:
return myint >= 16

Get Boolean from string: [http://stackoverflow.com/questions/715417/converting-from-a-string-to-boolean-in-python]
# Note: no built in way to do this

myString = "false"
val = (myString == "true")

True if myString=="True" else False

s in ['true', '1', 't', 'y', 'yes', 'yeah', 'yup', 'certainly', 'uh-huh']

def str2bool(v):
return v.lower() in ("yes", "true", "t", "1")

=== None ===

None is Python's version of NULL or NIL.

None is a singleton (Python only has one copy of None in the interpreter).

if myvar is None:
...

=== If Conditionals ===

Operators:
==, !=, not, and, or, <, >, <>, <=, >=, in, not in, is, is not

If statement:
<pre>
if a == b:
...
elif c < age < d:
...
else:
...
</pre>

Shorthand conditional: (ternary operator)
val = True if a == b else False

is - checks for identical objects - same id()
# None is a singleton, so this works great for None
if myvar is None:
...

Other shorthand:
<pre>
def mycmp(x, y):
if x > y: return True
else: return False
</pre>

Switch/Case statement:
* there is no built in switch/case statement, use "if ... elif ... elif ..." sequence instead

=== For Loop ===

''for'' is used for iterating over an iterable sequence (eg. lists, dictionaries, tuples). (for each)

# range(start, end + 1, skip)

for i in range(10):
print(i) # 0 .. 9

for i in range(5, 10):
print(i) # 5 .. 9

for i in range(10, 0, -1):
print(i) # 10 .. 1

for i in [1, 2, 3]: # iterate over list
print(i) # 1 2 3

Discard variable: [http://bytes.com/topic/python/answers/757444-loop-without-variable]
for _ in range(10): # repeat and discard variable (good for pep8 check)
print "hello world"

for key in my_dict.keys(): ...
for value in my_dict.values(): ...
for key, value in my_dict.items(): ...

Compact version:
for i in range(10): print(i) # 0 .. 9

Access index: [http://stackoverflow.com/questions/522563/accessing-the-index-in-python-for-loops]
# using enumeration
for idx, val in enumerate(mylist):
print idx, val

# using range
for ix in range(len(mylist)):
print mylist[ix]

# list comprehension
[ (ix, mylist[ix]) for ix in range(len(mylist))]

=== While Loop ===

i = 0
while i < 10:
i = i + 1

while True:
... # infinite loop

=== Breaking Loops ===

break statement:
while True:
if s == 'done':
break # jump out of the loop
...

continue statement: (works with while too)
for i in range(10):
if i == 2:
continue # skip 2
...

=== Functions ===

No return value: (None)
x = print('')
# x is 'None'

Note: specifying no return value is the same as this:
return None

No Operation - 'pass':
def noop():
pass

Function:
<pre>
def area(radius):
""" This is a doc string """ # can be " ... " also...
import math
return math.pi * radius ** 2
</pre>

Function Doc string:
print(area.__doc__)
foo.func_doc

Function name:
foo.func_name

Local variable:
name = "old"
def cname():
name = "new"
print(name) # is still 'old'

Global variable:
name = "old"
def cname():
global name
name = "new"
cname()
print(name) # is now 'new'

By convention starting point is: (not required)
def main():

Parameters: (with default)
def foo(x, y, z = "default"):

Note: don't use mutable for defaults, as the object is remembered across runs! (unless that is what you want)
<pre>
# each call to t() will increase list
def t(foo=[]):
foo.append('1')
print(foo)
</pre>

<pre>
# fix mutable issue
def named_param(a, foo=None):
foo = foo or []
if not foo:
foo.append(a)
</pre>

Parameters by keyword:
def foo(x = '1', y = '2', z = '3'):
...
foo(y = 'a', x = 'b') # order is not important with keywords

Trick for optional parameter:
def do(x = None):
if x == None:

Function tests - functions are callable
callable(myfunc)

=== Type Hints ===

: "Note The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers, IDEs, linters, etc."

def surface_area_of_cube(edge_length: float) -> str:
return f"The surface area of the cube is {6 * edge_length ** 2}."

Without the hints:
def surface_area_of_cube(edge_length):
...

ref: https://docs.python.org/3/library/typing.html

=== Pass ===

Pass is a null operation (noop):

Do nothing, noop, take no action:
pass

Great for creating minimal classes:
class MyEmptyClass:
pass

Greate for making stub functions/methods:
def initlog(*args):
pass # remember to implement this!

=== Slice ===

Take take a slice of strings, lists, tuples, etc...
a = [0,1,2,3,4,5]
a[0] # 0 - not a slice
a[-1] # 5 - not a slice
a[0:1] # [0]
a[1:3] # [1, 2]
a[:2] # [0, 1]
a[:-1] # [0, 1, 2, 3, 4] - all but last
a[2:] # [2, 3, 4, 5]
a[::2] # [0, 2, 4] - stride
a[::-1] # [5, 4, 3, 2, 1, 0] - reverse
range(0,10)[::2] # [0, 2, 4, 6, 8] - stride

a = '012345'
a[1:3] # '12'
a[::2] # '024'

=== [http://docs.python.org/reference/compound_stmts.html Compound Statements] ===

Compound statements: (generally discouraged)
fun1(); fun2(); fun3()

Compound with statements: [http://docs.python.org/reference/compound_stmts.html#the-with-statement]
with A() as a, B() as b: ...

#is equivalent to
with A() as a:
with B() as b:
...

[http://www.python.org/dev/peps/pep-0343 PEP 0343] - The “with” statement

f = open('file')
with f: ...

Try: (quick ignore all exceptions, very dangerous)
try: my_broken_function()
except: pass

=== Passing Command Line Arguments ===

Parameters are stored in the 'sys.argv' variable:
import sys
print( sys.argv[0] ) # name of script
print( sys.argv[1] ) # first argument
for arg in sys.argv: print( arg ) # list all arguments

Print script name:
os.path.basename(sys.argv[0])

Print path to script:
print sys.path[0]

Paths: [http://code.activestate.com/recipes/474083-get-the-path-of-the-currently-executing-python-scr/]
<pre>
import os,sys
print "CWD: ",os.getcwd()
print "Script: ",sys.argv[0]
print ".EXE: ",os.path.dirname(sys.executable)
print "Script dir: ", os.path.realpath(os.path.dirname(sys.argv[0]))
pathname, scriptname = os.path.split(sys.argv[0])
print "Relative script dir: ",pathname
print "Script dir: ", os.path.abspath(pathname)
</pre>

os.path.realpath(__file__)

print 'sys.argv[0] =', sys.argv[0]
pathname = os.path.dirname(sys.argv[0])
print 'path =', pathname
print 'full path =', os.path.abspath(pathname)

=== Environment Variables ===

import os
a = os.environ.get('MYVAR') # get environment var
a = os.environ.get('MYVAR','default') # get environment var
a = os.environ['MYVAR'] # get environment var
os.environ['MYVAR'] = 'something' # set environment var
os.environ.update({'MYVAR':'something'}) # set environment var

for env in os.environ:
print "%s = %s" % (env, os.environ[env])

with open("environment.txt", "w") as f:
for env in os.environ:
f.write("%s = %s\n" % (env, os.environ[env]))

=== PYTHONPATH ===

Environment Search path:
PYTHONPATH=...

From Code:
sys.path.append('...')

Get site-package directory from shell: [http://stackoverflow.com/questions/122327/how-do-i-find-the-location-of-my-python-site-packages-directory]
python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

=== Import and Modules ===

Import and Modules - http://docs.python.org/tutorial/modules.html

Import module:
import package # standard import
import package.code # nested name space
from package import code # import specifics from module
import module # standard import
from math import sin # import specifics from module
from module import myfunc as fun1 # alias function/object
import math as other_math # alias module
import longname as ln # alias module
from module import * # BAD BAD BAD!

import math
math.pow(x,y)

Group and organize by:
* stdlib libraries
* local libraries
* 3rd party libraries

Remove import:
del math

from math import *
pow(x,y)

from math import pow
pow(x,y)

import math as math_lib # rename
math_lib.pow(x,y)

List functions in module:
import math
dir(math)

List all built in functions:
dir(__builtins__)

Note: Modules have the extension .py

Simply module Example:

# mymod.py:
def hello():
""" this is a doc string """
print "Hello World"

# Use the module:
import mymod
mymod.hello():

# Use everything in the the module, except for names beginning with '_': (bad practice!)
from mymod import *
hello()

# Use specific functions in the module:
from mymod import hello, hi
hello()

# Function assignment:
h = mymod.hello
h()

# module name
mymod.__name__ # mymod
__name__ # used inside of module will print current module or "__main__" if directly called

Note: careful when building own modules, as the import literally imports and executes the code, so any code not in functions is executed!

Check if being called directly or indirectly:
if __name__ == "__main__":
# execute tests...

Standard Script Setup:
<pre>
def main(args):
# do logic
if __name__ == "__main__":
sys.exit(main(sys.argv))
</pre>

Easter egg:
# The Zen of Python
import this

Search path:
* environment variable 'PYTHONPATH' (on Unix, this is usually .:/usr/local/lib/python)
* sys.path (list of paths, initialized to PYTHONPATH)
** sys.path.append('path')

Module folder: [http://stackoverflow.com/questions/391879/organising-my-python-project] [http://stackoverflow.com/questions/279237/python-import-a-module-from-a-folder] [http://stackoverflow.com/questions/72852/how-to-do-relative-imports-in-python]
/kenlib/kenmod.py
# so you can do "import kenlib.kenmod"
# create file '__init__.py' in kenlib/

Check if package exists:
<pre>
try:
import argparse
except ImportError:
print >> sys.stderr, "Please install argparse from http://code.google.com/p/
argparse."
exit(1)
</pre>

=== ifmain ===

if __name__ == '__main__':
sys.exit(main(sys.argv))

if __name__ == '__main__':
sys.exit(main(sys.argv[1:]) or 0)

=== Lists ===

* Lists can be modified (mutable)
* Tuples are constant (immutable)

List: (mutable or modifiable)
x = [] # empty list
x = [5] # singleton list

s = list("Hello World")
print(s) # s = ['H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd']
s[0] = 'X'

s.append(x) # append x
s.insert(i, x) # insert x at i
del(s[i]) # delete item at i
s.sort() # sort (modify in place!)
s.reverse() # reverse
s.index(x) # index of x in s
new = old[:] # create copy of list
new = list(old) # create copy of list

Check if item in list:
if x in s: ...

Filter list of strings based on contents: [https://stackoverflow.com/questions/2152898/filtering-a-list-of-strings-based-on-contents]
res = [k for k in lst if 'ab' in k]

List comprehension
[n * n for n in range(1, 11)] # squares of numbers 1 to 10
[c for c in 'pizza'] # list of string 'pizza'
[c.upper() for c in 'pizza'] # list of string 'pizza' uppercase
[n for n in nums if n > 0] # filters out only positive numbers in nums list
[fname for fname in os.listdir(path) # on multiple lines
if os.path.isfile(fname)
if fname.endswith('.py')]

Enumeration
for index, value in enumerate(mylist): print index, value

Length:
len(x)

Concatenation:
x + s
x * 2

Convert list to string:
"".join(mylist) # no separator
" ".join(mylist) # space separator
",".join(mylist) # comma separator

Test Membership
x in s

Multi dimensional list:
mylist = [ [0,"hi"], [1,"bye"] ]
print( mylist[0][1] )

Sorting: (see lambda)
data.sort() # alpha numerica sort
data.reverse() # alpha numerica sort

<pre>
# custom sort
data = [1, 5, 3, 9]
def mylistcmp(x, y):
if x == y: return 0
if x < y:
return -1
else: # swap 1 and -1 here to reverse
return 1

data.sort(cmp=mylistcmp)
</pre>

Slices:
lst = ['1a', '2b', '3c', '4d', '5e', '6f']
lst[0] # '1a'
lst[-1] # '6f' last item
lst[0:2] # ['1a', '2b']
lst[0:2] = ['11', '22'] # lst = ['11', '22', '3c', '4d', '5e', '6f'] - assignment to slice
lst[0:2] = [] # remove items
lst[0:0] = ['1a', '2b'] # insert items (at beginning)
lst[-1:-1] = ['7g', '8h'] # insert items (at item BEFORE last), not really what you want, use append()
lst[-1:] = ['7g', '8h'] # replace last item
lst[:] = [] # clear entire list
del(lst[0:2]) # delete slice items

Note: it is not safe to modify a sequence while being iterated over. Create a copy first:
for x in mylst[:]: # make a slice copy of the entire list
if len(x) > 6: a.insert(0,x)

Make list from tuple:
list( range(10) )

Merge multiple lists: [http://www.testingreflections.com/node/view/4930]
<pre>
def merge(seq):
merged = []
for s in seq:
for x in s:
merged.append(x)
return merged

# sample usage:
foo = [['a', 'b'],['c'],['d', 'e', 'f']]
print merge(foo)
>>>['a', 'b', 'c', 'd', 'e', 'f']
</pre>

=== Tuple ===

Tuple: (immutable or constant)
x = () # empty tuple
x = (5,) # singleton tuple
x = (5) # NOTE: integer with parenthesis, not tuple

s = tuple("Hello World")
print(s) # s = ('H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd')
# ERROR: s[0] = 'X'

Make list from tuple:
list( range(10) )

Use in conditionals:
if answer in ('y', 'ye', 'yes'): ...

Convert tuple string to tuple: [http://stackoverflow.com/questions/3945856/converting-string-to-tuple-and-adding-to-tuple]
x = "(1,2,3)"
t = tuple(int(v) for v in re.findall("[0-9]+", x))

Another Convert tuple string to tuple: [http://stackoverflow.com/questions/3945856/converting-string-to-tuple-and-adding-to-tuple]
import ast
ast.literal_eval("(1,2,3,4)") # (1,2,3,4)

=== Named Tuple ===

What are "named tuples" in Python? - Stack Overflow - http://stackoverflow.com/questions/2970608/what-are-named-tuples-in-python

<pre>
from collections import namedtuple
Point = namedtuple('Point', 'x y')
pt1 = Point(1.0, 5.0)
pt2 = Point(2.5, 1.5)
print p1 # point(x=1, y=5)
print p1.x # 1.0
print p1[0] # 1.0
</pre>

=== Dictionary ===

NOTE: Dictionaries are also known as associative arrays, maps or hash tables.

Dictionary: (mutable or modifiable)

color = {} # empty dictionary
color = {'red' : 1, 'blue', 2}
print( color['red'] ) # 1
print( color.get('red') ) # 1
print( color.get('red', 'default') ) # red if found, or return 'default'
color['green'] = 3 # able to add new item on the fly!
'red' in color # True - check if index exists
del(color["red"]) # delete item
len(color) # 2 - count of items

List in dictionary
<pre>
d.setdefault(key,[]).append(member) # tricky!

# more tricky: set default type as list
from from collections import defaultdict
d = defaultdict(list)
d[key].append(member)
</pre>

Enumeration:
for index, key in enumerate(mylist): print index, key

functions:
d.clear()
d.copy()
d.get(key)
d.get(key, default_val)
d.update(e) # update with (key, value) pairs in e
d.setdefault(key, value) # only update if not exist, and return set value

Views: (adjust on the fly as dictionary changes)
d.items() (key, values)
for k, v in d.items(): print(k, v)

d.keys()
for k in d.keys(): print(k)

d.values()
for v in d.values(): print(v)

Sorting: (see also lambda)
<pre>
# custom sort of dictionaries in list
data = [dict(number=x) for x in '036149'] # create list of dictionaries
def mydictcmp(x, y):
if x['number'] == y['number']: return 0
if x['number'] < y['number']:
return -1
else: # swap 1 and -1 here to reverse
return 1

# data.sort(cmp=mydictcmp) # modifies data

new_data = sorted(data, cmp=mydictcmp) # better solution
</pre>

Sorting:
<pre>
my_dict = {'a':1, 'b':4, 'c':3, 'd':9}
my_dict_keys = my_dict.keys()

def my_dict_sort(x, y):
if my_dict[x] == my_dict[y]: return 0
if my_dict[x] < my_dict[y]: return -1
if my_dict[x] > my_dict[y]: return 1

#sorted_my_dict = sorted(my_dict_keys, cmp=my_dict_sort, reverse=True)
sorted_tuple = sorted(my_dict_keys, cmp=my_dict_sort)
print "Sorted Dictionary:"
for key in sorted_tuple:
print "%s: %d" % (key, my_dict[key])

# a: 1, c: 3, b: 4, d: 9,
# reverse: d: 9, b: 4, c: 3, a: 1,
</pre>

Sort and keys:
keys = sorted(mydict.keys())

Sort by value: [http://stackoverflow.com/questions/613183/python-sort-a-dictionary-by-value]
import operator
x = {1: 2, 3: 4, 4:3, 2:1, 0:0}
sorted_x = sorted(x.iteritems(), key=operator.itemgetter(1))

Sort options:
# LIST.sort(cmp=None, key=None, reverse=False) -- stable sort *IN PLACE*;
# sorted(iterable, cmp=None, key=None, reverse=False)

Another: (works well!)
sorted(my_dict.items(), key=lambda x: x[1])

See: http://wiki.python.org/moin/HowTo/Sorting/

=== Sets ===

Sets are special lists with unique values (good for removing duplicates)
lst = [1, 1, 6, 8, 1, 5]
s = set(lst)
print(s) # set([8, 1, 5, 6])
s.add(item)
item in s # true
s.remove(item)

Union (return combination of all):
s.union(t)

Intersection (return matching):
s.intersection(t)

Symmetric Difference (return only items not matching either, inverse of intersection):
s.symmetric_difference(t)
list( set(mylist).symmetric_difference(set(mylist2)) )

<pre>
Operation Equivalent Result
len(s) cardinality of set s
x in s test x for membership in s
x not in s test x for non-membership in s
s.issubset(t) s <= t test whether every element in s is in t
s.issuperset(t) s >= t test whether every element in t is in s
s.union(t) s | t new set with elements from both s and t
s.intersection(t) s & t new set with elements common to s and t
s.difference(t) s - t new set with elements in s but not in t
s.symmetric_difference(t) s ^ t new set with elements in either s or t but not both
s.copy() new set with a shallow copy of s
</pre>

References:
* sets — Unordered collections of unique elements - http://docs.python.org/library/sets.html

=== Files ===

Open file, get file descriptor:
fd = open('/dev/null')

File Modes:
'r' read (default)
'w' write (truncate and create if needed)
'a' append (create if needed)

'b' binary
't' text (default)

'+' read and write

Common Functions: (fd)

f.read() # read whole file to string
f.read(n) # read n bytes from file to string
f.readline() # read single line as string (while loop)
f.readlines() # read all lines to list

f.write(str) # write string to file (does not auto include new line)
f.writelines(list) # write list to file (does not auto include new lines)

f.seek(0) # seek to start of file
f.seek(pos) # seek to position
f.truncate() # truncate file

f.close() # close file (fd)
f.closed # True/False flag that reports if file is closed

Help:
help(open('/dev/null'))
pydoc file

Read Text File Line by Line
f = open(fname, 'r') # ('r' optional)
for line in f: # or f.readlines()
print(line)
f.close() # optional

Write list to file (with new lines):
f.writelines('\n'.join(mylist))

One liner read whole file:
print( open( fname, 'r' ).read() )

Write text file:
# f = open(fname, 'a') # append text file
f = open(fname, 'w')
f.write("Hello\n")
f.write( str(10) ) # convert numbers to strings first
f.writelines(["line one\n", "line two\n"])
f.close

Read binary file:
def is_gif(fname):
f = open(fname, 'br') # not sure 'b' is correct
first4 = tuple(f.read(4))
return first4 == (0x47, 0x49, 0x46, 0x38)

Read all characters of file:
c = f.read(1)
while c:
# if c == ... logic
c = f.read(1)

Reading file in while loop: [http://ubuntuforums.org/showthread.php?t=1092300]
<pre>
f=open("file")
while True:
line=f.readline()
if not line: break
print line
f.close()
</pre>

with - implicit close - [http://www.python.org/dev/peps/pep-0343/ PEP 343 -- The "with" Statement]: [http://docs.python.org/reference/compound_stmts.html#the-with-statement] (python >= 2.5)
with open("/etc/passwd") as f:
# ..., f.read(), etc

With statement: (promptly close open handle at end of block)
with open(fname, 'r') as f:
for ...

<pre>
# eqivalent to:
f = open("/etc/passwd")
f.__enter__()
try:
...
finally:
f.__exit__()
</pre>

cat: (for grabbing a configuration line)
<pre>
def cat(filename):
with open(filename) as f:
return f.readline().strip()
</pre>

=== With ===

PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/

'with' structure:
<pre>
with EXPR as VAR:
BLOCK
</pre>

translates to:
<pre>
VAR = EXPR
VAR.__enter__()
try:
BLOCK
finally:
VAR.__exit__()
</pre>

Note: The "as VAR" part is optional. Good for locking:
<pre>
with locking(myLock):
BLOCK
</pre>

with - implicit close - [http://www.python.org/dev/peps/pep-0343/ PEP 343 -- The "with" Statement]: [http://docs.python.org/reference/compound_stmts.html#the-with-statement] (python >= 2.5)
with open("/etc/passwd") as f:
...

With statement: (promptly close open handle at end of block)
with open(fname, 'r') as f:
for ...

See also [[#Files]]

References:
* PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/
* PEP 340 -- Anonymous Block Statements - http://www.python.org/dev/peps/pep-0340/

=== Try Exception Handling ===

Exception Handling - http://docs.python.org/release/2.6.6/tutorial/errors.html

throw execption:
raise [EXCEPTION]
raise IOError("This is a test!")

Catch exceptions:
try:
except ( [EXCEPTION], [EXCEPTION] ):
except [EXCEPTION]:
except [EXCEPTION] as err: # use "[EXCEPTION], err" in python < 2.6
print("Error: {}".format(err))
except:
raise # reraise caught error that is not handled
else: # if no exception caught
finally: # always executed

Note: Usually a good idea to re-raise exception if you don't handle it (just 'raise')

General Exception: (all exceptions inherit from Exception)
try:
raise Exception('spam', 'eggs')
except Exception as e:
pass

Custom Exception Class:
<pre>
class MyError(Exception):
"""My Error Exception"""

class MyException(Exception):
pass

rasie MyError("My Error")
</pre>

<pre>
class ValidationError(Exception):
def __init__(self, message, Errors):

# Call the base class constructor with the parameters it needs
Exception.__init__(self, message)

# Now for your custom code...
self.Errors = Errors
def __str__(self):
return "MyErrors - %s" % self.Errors
</pre>

Rename standard exception:
<pre>
class Failure(StandardError):
"""Exception for unexpected failures."""
</pre>

Note: a good place to hide the exception classes in the __init__.py file.

==== exception line number ====

Python When I catch an exception, how do I get the type, file, and line number? - Stack Overflow [http://stackoverflow.com/questions/1278705/python-when-i-catch-an-exception-how-do-i-get-the-type-file-and-line-number]
<pre>
import sys, os

try:
raise NotImplementedError("No error")
except Exception as e:
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print(exc_type, fname, exc_tb.tb_lineno)
</pre>

==== traceback ====

<pre>
import traceback
try:
...
except Exception as E:
...
print 'Traceback: %s', traceback.format_exc()
</pre>

Catching multiple exceptions and getting properties from the exception:
<pre>
import traceback
import sys

class MyError(Exception):
"""My Error Exception"""

try:
#raise ValueError("a value error") # <type 'exceptions.ValueError'>
raise MyError("a my error") # <class '__main__.MyError'>
raise Exception("won't be caught") # <type 'exceptions.Exception'>
except (ValueError, MyError) as e:
print type(e) # <class '__main__.MyError'>
print e # __str__() = a my error
msg = str(e) # msg = 'a my error'
print e.__doc__ # My Error Exception
print e.__str__() # a my error
print e.__repr__() # MyError('a my error',)
print e.args # ('a my error',)
x = e # __getitem__() allows args to be unpacked directly
# x, y = e # if two args were passed, use len(e.args) to see count

# manually print stack trace (import traceback, sys)
print "Exception in user code:"
print '-'*60
traceback.print_exc(file=sys.stdout)
# stack_trace = traceback.format_exc()
print '-'*60
</pre>

=== Object Oriented Programming ===

Terms:
* encapsulates
* inheritance
* polymorphism
* method overriding

==== class ====

class animal.py:
<pre>
class Animal(object): # default "(object)" not required
" Animal class "
def __init__(self, name):
self.name = name # name is private variable
self.__iam = "animal" # __iam is private variable
def __str__(self): # string representation
return "name = '%s'" % (self.name)
def __repr__(self): # object representation (just typing instance in IDLE, or repr())
return 'Animal(%s)' % str(self) # returns __str__
def speak(self, msg='Nothing to say'):
print(msg)
</pre>

Note: if no __str__ defined, but __repr__ is, then: __str__ = __repr__

Usage:
<pre>
from animal import Animal
a = Animal('good')
print(a.__doc__) # Animal class
print(a) # name = 'good'
a # Animal(name = 'good')
print(a.name) # good
a.speak('hi') # hi
print(a._Animal__iam) # animal - access to private variable (shouldn't do)
</pre>

subclass dog.py:
<pre>
from animal import Animal
class Dog(Animal):
" Dog class "
def __init__(self, name):
self.name = name # name is private variable
self.__iam = "dog" # __iam is private variable
# NOTICE: no __str__, inherited from super class
def __repr__(self): # override to reflect "dog"
return 'Dog(%s)' % str(self) # returns __str__
def speak(self, msg='Nothing to say'): # override method
super(Dog, self).speak('Dog says: ' + msg) # call super class method
</pre>

Subclass usage:
<pre>
from dog import Dog
d = Dog('bad')
print(d.__doc__) # Dog class
print(d) # name = 'bad'
d # Dog(name = 'bad')
print(d.name) # bad
d.speak('hi') # Dog says: hi
print(d._Dog__iam) # dog - access to private variable (shouldn't do)
</pre>

Super: [http://stackoverflow.com/questions/576169/understanding-python-super-with-init-methods]
# python 3.0
super().__init__()

# python 2.7
super(self.__class__, self).__init__()
super([CLASSNAME], self).__init__()

<pre>
class Rectangle(Polygon):
def __init__(self, id, width, height):
# super(Rectangle, self).__init__(id)
super(self.__class__, self).__init__(id)
self.shape = (width, height)
</pre>

Get class name:
self.__class__.__name__
obj.__class__.__name__

Full name:
obj.__module__ + "." + obj.__class__.__name__

Private variables: (uses name mangling)
self.__age
# direct access: _classname__varname (eg. p._Person__age = 44)

Tests:
isinstance(myobj, myclass) # check if myobj is a class or subclass of myclass
issubclass(A, B) # check if A is subclass of B
issubclass(Exception, BaseException) # True

Multiple Inheritance: (methods searched in order)
class DerviedClass(Base1, Base2, Base3):
pass

Standard startup script:
class MyClass():
...
@classmethod
def main(cls, argv=None):
...
# return cls() # creator
if __name__ == "__main__";
sys.exit(MyClass.main(sys.argv))

Raise an error if a method is not overridden (not implemented)
def _get_dist(self):
"""Find linux disto. Implemented in subclasses."""
raise NotImplementedError

==== Multiple Constructors ====

<pre>
class Animal(object):

def __init__(self, name):
self.name = name

@classmethod
def create_fido(cls):
return cls("fido")
</pre>

==== Descriptor Classes ====

"In general, a descriptor is an object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol: __get__(), __set__(), and __delete__(). If any of those methods are defined for an object, it is said to be a descriptor." [http://docs.python.org/2/reference/datamodel.html#implementing-descriptors]

<pre>
object.__get__(self, instance, owner)

Called to get the attribute of the owner class (class attribute access) or of an instance of that
class (instance attribute access). owner is always the owner class, while instance is the instance
that the attribute was accessed through, or None when the attribute is accessed through the owner.
This method should return the (computed) attribute value or raise an AttributeError exception.

object.__set__(self, instance, value)

Called to set the attribute on an instance instance of the owner class to a new value, value.
</pre>

Example: [http://stackoverflow.com/questions/3798835/understanding-get-and-set-and-python-descriptors]
<pre>
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)

class Temperature(object):
celsius = Celsius()

t = Temperature()
t.celsius = 5 # calls Celsius.__set__
print t.celsius # calls Celsius.__get__
</pre>

Note: Also appears in 'pydoc' under "Data descriptors defined here" with object's docstring. Trick: the __doc__ can be overridden in the __init__ method, which makes for dynamic docstrings!

References:
* Descriptor HowTo Guide — Python v2.7.5 documentation - http://docs.python.org/2/howto/descriptor.html
* 3.4.2.2. Implementing Descriptors¶ - 3. Data model — Python v2.7.5 documentation - http://docs.python.org/2/reference/datamodel.html#implementing-descriptors

=== [http://docs.python.org/reference/simple_stmts.html#the-exec-statement exec code] ===

DANGEROUS!

Execute code in string:
exec( 'print("hi")' )

Example of executing a startup script for IDLE:
if os.path.isfile('.pythonrc.py'): exec(open('.pythonrc.py').read())

import os
filename = os.environ.get('PYTHONSTARTUP')
if filename and os.path.isfile(filename):
exec(open(filename).read())

=== eval ===

VERY DANGEROUS

Will execute a string as though it were pure python code!

eval("print 'hello'")

=== Common Header Format ===

<pre>
__author__ = "software team"
__date__ ="$Feb 15, 2011 1:42:17 PM$"
__version__ = "1.1.0.441 bandelier@66d9fe1edf3e"
</pre>

<pre>
Next should be authorship information. This information should follow this format:

__author__ = "Rob Knight, Gavin Huttley, and Peter Maxwell"
__copyright__ = "Copyright 2007, The Cogent Project"
__credits__ = ["Rob Knight", "Peter Maxwell", "Gavin Huttley",
"Matthew Wakefield"]
__license__ = "GPL"
__version__ = "1.0.1"
__maintainer__ = "Rob Knight"
__email__ = "rob@spot.colorado.edu"
__status__ = "Production"
</pre>
Source: Python Coding Guidelines - http://bayes.colorado.edu/PythonGuidelines.html

These will show up in the help() like such:
<pre>
Help on module test:

NAME
test

FILE
/home/kenneth/test.py

FUNCTIONS
test()

DATA
__author__ = 'test author'
__version__ = '1.0'

VERSION
1.0

AUTHOR
test author
</pre>

References:
* Python: What is the common header format? - Stack Overflow - http://stackoverflow.com/questions/1523427/python-what-is-the-common-header-format

=== Common Environment Variables ===

Command line and environment — Python v2.7.2 documentation - http://docs.python.org/using/cmdline.htm

PYTHONUNBUFFERED
: If this is set to a non-empty string it is equivalent to specifying the -u option. [http://docs.python.org/using/cmdline.html#envvar-PYTHONUNBUFFERED]
: -u - Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
* See also stdout - Python output buffering - Stack Overflow - http://stackoverflow.com/questions/107705/python-output-buffering
export PYTHONUNBUFFERED=1
set PYTHONUNBUFFERED=1

PYTHONVERBOSE
:If this is set to a non-empty string it is equivalent to specifying the -v option. If set to an integer, it is equivalent to specifying -v multiple times.
: -v - Print a message each time a module is initialized, showing the place (filename or built-in module) from which it is loaded.
export PYTHONVERBOSE=1
set PYTHONVERBOSE=1

PYTHONPATH
: Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored. [http://docs.python.org/using/cmdline.html#envvar-PYTHONPATH]
export PYTHONPATH=/path1/:path2/
set PYTHONPATH=C:\path1\;C:\path2\

=== Queue ===

Good for threading.

"The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics." [http://docs.python.org/2/library/queue.html]

<pre>
import Queue

bucket = Queue.Queue()
bucket.put("something")

while True:
try:
item = bucket.get(block=False)
except Queue.Empty:
pass
else:
# do stuff with queue
</pre>

Important methods:
Queue.qsize()
Queue.empty()
Queue.put(item[, block[, timeout]])
Queue.get([block[, timeout]])

-

Example: http://docs.python.org/2/library/queue.html
<pre>
def worker():
while True:
item = q.get()
do_work(item)
q.task_done()

q = Queue()
for i in range(num_worker_threads):
t = Thread(target=worker)
t.daemon = True
t.start()

for item in source():
q.put(item)

q.join() # block until all tasks are done
</pre>

-

"The module implements three types of queue, which differ only in the order in which the entries are retrieved. In a FIFO queue, the first tasks added are the first retrieved. In a LIFO queue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the heapq module) and the lowest valued entry is retrieved first."

Queue Types:
class Queue.Queue(maxsize=0) # FIFO Queue
class Queue.LifoQueue(maxsize=0) # LIFO Queue
class Queue.PriorityQueue(maxsize=0) # Priority Queue

References:
* 8.10. Queue — A synchronized queue class — Python v2.7.5 documentation - http://docs.python.org/2/library/queue.html

== Recipes ==

=== Remove Numbers from String ===

Using list comprehension: [http://stackoverflow.com/questions/12851791/removing-numbers-from-string]
<pre>
result = ''.join([i for i in s if not i.isdigit()])
</pre>

Using pattern matching:
<pre>
>>> re.compile(r'\D+').findall("test 1234 test 1234")
['test ', ' test ']

>>> re.compile(r"\d").sub("", "test 1234 test 1234")
'test test '
</pre>

=== Retry Wrapper ===

<pre>
import functool
class SomethingRetry:

def retry_with_backoff(retries: int, base_delay: float, exceptions: tuple[type[Exception], ...] = (Exception,)) -> callable:
"""
Retries a function with exponential backoff, doubling on each attempt.
Arguments:
retries: Number of retry attempts.
base_delay: Initial delay between retries in seconds.
exceptions: Tuple of exceptions to catch and retry.
"""

def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(1, retries + 1):
try:
return func(*args, **kwargs)
except exceptions as e:
print(f"Warning: Attempt {attempt} failed with error: {e}")
if attempt == retries:
print("Error: All retry attempts failed.")
raise
sleep_time = base_delay * (2 ** (attempt - 1))
print(f"Notice: Retrying in {sleep_time} seconds...")
time.sleep(sleep_time)
return wrapper
return decorator

@retry_with_backoff(retries=6, base_delay=10, exceptions=(Exception,))
def do_something(self, sample_param: str):
</pre>

== Standard Libraries ==

=== user input ===

<pre>
line = raw_input() # Python 2
# line = input() # Python 2
line = input('What is your name: ')
print(": " + line.strip() + " :")
</pre>

=== sys - stdout stderr stdin ===

STDOUT:
import sys
sys.stdout.write("Hello ") # does not write new line
sys.stdout.write("World\n")

STDERR:
import sys
sys.stderr.write("ERROR!\n")

STDIN:
import sys
line = sys.stdin.readline()

line = input() # use raw_input() in Python 2
line = input('What is your name: ')
print(": " + line.strip() + " :")

Read piped stdin line by line: [http://stackoverflow.com/questions/1450393/how-do-you-read-from-stdin-in-python]
# cat /etc/passwd | ./readlines.py # linux
# type file.txt | python readlines.py # windows

# fileinput.input() - returns fileinput object which is iterative
import fileinput
for line in fileinput.input():
print(": " + line + " :")

# sys.stdin.readlines() - returns list of strings
import sys
for line in sys.stdin.readlines():
print(": " + line + " :")

# sys.stdin - returns object which is iterative
import sys
for line in sys.stdin:
print(": " + line + " :")

# ugly while loop: [http://ubuntuforums.org/showthread.php?t=1092300]
while True:
try:
s = raw_input("Say something ")
except EOFError:
break
print ": %s" % s

Check if stdin has any waiting data: [http://stackoverflow.com/questions/3762881/how-do-i-check-if-stdin-has-some-data]
<pre>
import sys
import select

# if select.select([sys.stdin,],[],[],0.0)[0]: # alternative old method

if not sys.stdin.isatty():
print "Have data!"
print sys.stdin.readlines()
else:
print "No data"
</pre>

<pre>
if len(sys.argv) > 1:
message = ' '.join(sys.argv[1:])
elif not sys.stdin.isatty():
# read from pipe in
message = sys.stdin.readline()
else:
# read from user
sys.stdout.write("Message: ")
message = raw_input().strip()
</pre>

=== [http://docs.python.org/library/re.html Regular Expressions] ===

Import:
import re # user regular expressions

Match vs Search:
re.match("cde", "abcdef") # no match (match searches beginning of string, first match)
re.match("abc", "abcdef") # match (match searches beginning of string, first match)
re.search("cd", "abcdef") # match (matches any location, first match)

Both re.match() and re.search() return objects that are not intuitive to use. You can check the object's group() method for what matched, but I prefer re.findall().

Match:
s = 'done'
if re.match('done|quit', s) != None # True

Search:
s = 'done'
if re.search('do', s) != None # True

Replace:
msg = 'jello world'
msg = re.sub('jello', 'hello', msg)

For case intensive searches append 3rd parameter "re.I".

Find all:
re.findall("[a-z]+", "aaa111bbb222") # ['aaa', 'bbb']
re.findall("111([a-z]+)", "aaa111bbb222") # ['bbb']
re.findall("hello", "HELLO", re.I) # ['HELLO'] - Case Insensitive

Split:
>>> re.split('[a-z]+','111aaa222bbb')
['111', '222', '']

>>> re.split('([a-z]+)','111aaa222bbb') # parenthesis cause return of all
['111', 'aaa', '222', 'bbb', '']

Replacement Substitution:
# re.sub(pattern, repl, string[, count, flags])
re.sub(r'\\', r'/', r'\\data\files')

Compiled regular express for repeated use: (performance)
valid = re.compile(r"[0-9]+")
valid.match("aaa111bbb") # not valid
match = valid.search("aaa111bbb222") # valid
print(match.group()) # what matched: '111'

=== Date and Time ===

Sleep in seconds:
import time
time.sleep(seconds)

Use the following functions to convert between time representations:
<table border="1">
<tr><th>From</th><th>To</th><th>Use</th></tr>
<tr><td>seconds since the epoch</td><td>struct_time in UTC</td><td>gmtime()</td></tr>
<tr><td>seconds since the epoch</td><td>struct_time in local time</td><td>localtime()</td></tr>
<tr><td>struct_time in UTC</td><td>seconds since the epoch</td><td>calendar.timegm()</td></tr>
<tr><td>struct_time in local time</td><td>seconds since the epoch</td><td>mktime()</td></tr>
</table>

Date formats:
<pre>
>>> time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
'2014-04-04 12:22:43'

>>> time.time()
1396635763.0581281

>>> datetime.datetime.now()
datetime.datetime(2014, 4, 4, 12, 22, 43, 58456)
# datetime(year, month, day, hour, minute, second, microsecond)

>>> time.localtime()
time.struct_time(tm_year=2014, tm_mon=4, tm_mday=4, tm_hour=12, tm_min=22, tm_sec=43, tm_wday=4, tm_yday=94, tm_isdst=1)
</pre>

Current epoch time:
int(time.time())

Time zone offset:
standard_tz_offset = - time.timezone / 3600 # 7
daylight_tz_offset = - time.altzone / 3600 # 6

Is daylight saving time: [http://stackoverflow.com/questions/2881025/python-daylight-savings-time]
import time
time.localtime()
_.tm_isdst # 0 or 1

mst_epoch_time = time.time() # server in MST/MDT
if time.localtime().tm_isdst: # are we in daylight saving time?
utc_epoch_time = mst_epoch_time + time.altzone
else:
utc_epoch_time = mst_epoch_time + time.timezone

Format current time: [http://docs.python.org/2/library/time.html#time.strftime]
time.strftime('%d%b%y') # '12May11'
time.strftime('%Y.%m.%d %H:%M:%S') # '2013.08.09 14:01:28'

Time conversions:
<pre>
# epoch to str
epoch_time = time.time() # 1375992088.7867229
dt = datetime.datetime.fromtimestamp( epoch_time ) # datetime.datetime(2013, 8, 8, 14, 1, 28, 786723)
date_str = dt.strftime("%Y-%m-%d %H:%M:%S") # '2013-08-08 14:01:28'
tz_offset = - time.altzone / 3600
if tz_offset < 0:
tz_offset_str = "-%02d00" % abs(tz_offset)
else:
tz_offset_str = "+%02d00" % abs(tz_offset)
print date_str + " " + tz_offset_str # '2013-08-08 14:01:28 -0600'
</pre>

<pre>
# UTC epoch to str
gm_epoch_time = calendar.timegm(time.gmtime()) # 1375992114
dt = datetime.utcfromtimestamp( gm_epoch_time ) # datetime.datetime(2013, 8, 8, 20, 1, 54)
dt.strftime("%Y-%m-%d %H:%M:%S +0000") # '2013-08-08 20:01:54 +0000'
dt.strftime("%Y-%m-%dT%H:%M:%SZ") # '2013-08-08 20:01:54 +0000'
</pre>

<pre>
# Local time str to epoch
iso_string='2013-08-08T14:01:28'
epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S' ) ) # 1375992088.0

iso_string='2013-08-08T14:01:28 MDT'
epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S %Z' ) ) # 1375992088.0

# not sure how to specify numerical time zone and pull it in!

# UTC (Zulu) str to epoch, and back again
iso_string = '2013-08-08T20:01:54Z'
epoch_time = calendar.timegm( time.strptime( iso_string.replace('Z', 'GMT'), '%Y-%m-%dT%H:%M:%S%Z' ) )
print epoch_time # 1375992114
print "%sZ" % datetime.utcfromtimestamp(timestamp).isoformat() # 2013-08-08T20:01:54Z
</pre>

<pre>
# Convert epoch from Local to UTC
epoch_time = 1375992114
tz_offset = - time.altzone / 3600
gm_epoch_time = epoch_time - tz_offset * 3600 # 1376013714
</pre>

<pre>
# Convert epoch from UTC to Local
gm_epoch_time = 1376013714
tz_offset = - time.altzone / 3600
epoch_time = gm_epoch_time + tz_offset * 3600 # 1375992114
</pre>

struct_time:
time.localtime() # local zime zone
time.gmtime() # UTC time

today = time.localtime()
today_year = today.tm_year
today_month = today.tm_mon
today_day = today.tm_mday

time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime())

Zulu time to epoch time: [http://stackoverflow.com/questions/127803/how-to-parse-iso-formatted-date-in-python]
s = '2008-09-03T20:56:35.450686Z'
event_time_struct = time.strptime(s.split('.')[0]+ "UTC", "%Y-%m-%dT%H:%M:%S%Z")
epoch_time = calendar.timegm(event_time_struct)

Parse string to time.struct_time (like gmtime()): [http://docs.python.org/2/library/time.html#time.strptime]
time.strptime(time.ctime()) # current time - default format is ctime format: "%a %b %d %H:%M:%S %Y"
time.strptime(time.ctime(time.time())) # default format is ctime format: "%a %b %d %H:%M:%S %Y"
time.strptime("30 Nov 00", "%d %b %y")

Build from ticks: [http://www.tutorialspoint.com/python/python_date_time.htm]
localtime = time.localtime( time.time() ) # build time.struct_time from ticks
print time.asctime( localtime ) # Tue Jan 13 10:17:09 2009

dt = datetime.datetime.fromtimestamp( time.time() ) # build datetime.datetime struct from ticks
dt.strftime("%Y-%m-%d %H:%M:%S") # '2013-07-31 23:07:48'
time.strptime( time.time(), ")

<pre>
from datetime import date
now = date.today() # now = datetime.date(2003, 12,2)
now.strftime("%m-%d-%y") # '12-02-03'
# dates support calendar arthimetic
birthday = date(1964, 7, 31)
age = now - birthday
age.days # 14368
</pre>

ts = (datetime.datetime.now().strftime("%Y%m%d%H%M")) # 201105191118
tstxt = (datetime.datetime.now().strftime("%m-%d-%Y %H:%M")) # 05-19-2011 11:18

Seconds since epoch:
int(time.time()) # 1305842304

Zulu and back again: ([http://aboutsimon.com/2013/06/05/datetime-hell-time-zone-aware-to-unix-timestamp/ Datetime hell])
<pre>
import time
from calendar import timegm
from datetime import datetime

if __name__ == '__main__':
iso_string = '2013-06-05T15:19:10Z'

timestamp = timegm(
time.strptime(
iso_string.replace('Z', 'GMT'),
'%Y-%m-%dT%H:%M:%S%Z'
)
)

print iso_string
print timestamp
print '-> %sZ' % datetime.utcfromtimestamp(timestamp).isoformat()
</pre>

http://wiki.python.org/moin/TimeTransitionsImage?action=AttachFile&do=get&target=v1.png [http://wiki.python.org/moin/WorkingWithTime]

References:
* WorkingWithTime - PythonInfo Wiki - http://wiki.python.org/moin/WorkingWithTime
* Datetime hell. Time zone aware to UNIX timestamp. | About me and the things I do - http://aboutsimon.com/2013/06/05/datetime-hell-time-zone-aware-to-unix-timestamp/
* WorkingWithTime - Python Wiki - http://wiki.python.org/moin/WorkingWithTime

---

Something I cooked up...

<pre>
def utc_time(self, years_offset=60):
mst_epoch_time = time.time() # server is in MST
if time.localtime().tm_isdst: # are we daylight saving time?
utc_epoch_time = mst_epoch_time + time.altzone
else:
utc_epoch_time = mst_epoch_time + time.timezone
return utc_epoch_time

def tz_time(self, tz_offset, daylight_adjust=True):
""" return the future time based on timezone offset
@tz_offset is timezone offset, eg -7 for MST
@daylight_adjust is flag to auto adjust for daylight time"""
if daylight_adjust:
if time.localtime().tm_isdst: # are we daylight saving time?
_tz_offset = tz_offset - 1
else:
_tz_offset = tz_offset
_utc_epoch_time = self.utc_time()
_adj_time = _utc_epoch_time + _tz_offset * 60 * 60
return _adj_time

def future_time(self, tz_offset=-7, daylight_adjust=True, years_offset=60):
""" Calculate some future year """
_tz_time = self.tz_time(tz_offset)
future_utc_epoch_time = _tz_time + years_offset * 365 * 24 * 60 * 60
# account for leap years
cur_year = datetime.datetime.now().year
cur_year_leap = cur_year / 4
future_year = cur_year + years_offset
future_year_leap = future_year / 4
leap_days = future_year_leap - cur_year_leap
future_utc_epoch_time += leap_days * 24 * 60 * 60
return future_utc_epoch_time
</pre>

=== Math ===

Math Functions:
import math
math.ceil(x)
math.floor(x)
math.trunc(x)
math.pow(x,y)

abs(x) # built in

=== Random ===

import random
random.choice(['a', 'b', 'c'])
random.sample(range(100), 10)
random.random() # random float 0 < x < 1
random.randrange(3) # integer in [0, 1, 2]
random.randrange(1, 3) # integer in [1, 2]
random.randint(1, 3) # integer in [1, 2, 3]

=== Operating System Operations ===

Import OS Library:
import os

Common Functions:
# directory functions
os.getcwd() # get current directory
os.listdir(p) # get list of directory contents
os.chdir(p) # change directory
os.mkdir(p) # make directory
os.makedirs(p) # make directory and all intermediate
os.rmdir(p) # remove empty directory, see shutil.rmtree()
os.removedirs(p) # remove empty directories, see shutil.rmtree()
os.path.exists(p) # does file/directory exist?
os.path.isdir(p) # does directory exist
os.path.join(folder1, folder2, file, etc) # portable path join

# file functions
os.remove(p) # remove file
os.unlink(p) # same as os.remove()
os.path.exists(p) # does file/directory exist?
os.path.isfile(p) # does file exist
os.stat(fname) # get file stats
os.stat(fname).st_size # file size
os.link(src, dst) # create hard link
os.readlink(p) # read symbolic link
os.rename(old, new) # rename/move file or directory
os.tempnam(p) # temporary unique name for target path
os.tmpfile() # create temporary file
os.tmpnam() # temporary file name
os.path.getsize(filename) # get file size

# system functions
os.fork() # fork process
os.getpid() # get process id
os.getppid() # get parent's process id
os.kill(pid, sig) # kill a process
os.getlogin() # get user's login name

# execution
os.system(cmd) # output to stdout, return exit_status
os.popen(cmd) # returns a pipe to command

Note: to get path to script:
sys.path[0]

Get directory and file names:
os.path.dirname(path) # '/root/test.txt' -> '/root'
os.path.basename(path) # '/root/test.txt' -> 'test.txt'
os.path.splitext(path) # '/root/test.test.txt' -> ('/root/test.test', '.txt') - extract extention
filename.split(".")[-1] # '/root/test.test.txt' -> 'txt' - extract extention

Normalize path, eliminating double slashes, etc.
os.path.normpath(path) # '/root/..' -> '/' or 'test/..' -> '.'
os.path.abspath(path) # '/root/..' -> '/' or 'test/..' -> '/root/test'

List dir: (returns list)
os.listdir( os.getcwd() )

List of files:
[p for p in os.listdir(os.getcwd())
if os.path.isfile(p)]

List of folders:
[p for p in os.listdir(os.getcwd())
if os.path.isdir(p)]

Build path for local OS: (using appropriate slash)
pathstr = os.path.join("folder", "folder", "file_or_folder")

File and directory management with shutil
import shutil
shutil.copyfile('file1', 'file2')
shutil.move('file1', 'folder2/file1')

File Wildcards:
import glob
glob.glob('*.py') # return list of files

Walk directory tree: [http://stackoverflow.com/questions/120656/directory-listing-in-python]
<pre>
import os
for dirname, dirnames, filenames in os.walk('.'):
for subdirname in dirnames:
print os.path.join(dirname, subdirname)
for filename in filenames:
print os.path.join(dirname, filename)
</pre>

ugly 'deltree' (by kenneth)
<pre>
import os
def deltree(path):
files = os.listdir(path)
for file in files:
file_path = os.path.join(path, file)
if os.path.isdir(file_path):
deltree(file_path)
os.rmdir(file_path)
else:
os.remove(file_path)
</pre>

Cleaner 'deltree' (by Kep)
<pre>
import shutil
# Blow away (if needed), and reclone to proper version
if os.path.isdir("xxx"):
shutil.rmtree("xxx")
</pre>

== System Platform ==

<pre>
import platform

if platform.system() == "Linux":
# ...
if platform.system() == "Windows":
# ...
else:
# ...
</pre>

=== Temp Files ===

import tempfile
filename = tempfile.mktemp()
f = open(filename, 'w')
...
f.close()

import os
os.remove(filename) # delete temp file

=== Pickling ===

Pickling (serializing, making objects persistent):
import pickle
grades = {'bob' : [4, 5, 6],
'sue' : [3, None, 7, 7]}
outfile = open(fname, 'wb')
pickle.dump(grades, outfile)

infile = open(fname, 'rb')
grades = pickle.load(infile)

=== [http://docs.python.org/library/logging.html Logging] ===

* logging — Logging facility for Python — Python 3 documentation - https://docs.python.org/3/library/logging.html
* Logging HOWTO — Python Dev Version documentation - http://docs.python.org/dev/howto/logging.html
* Logging Cookbook — Python Dev Version documentation - http://docs.python.org/dev/howto/logging-cookbook.html

Setup logger:
import logging
logging.basicConfig(level=logging.DEBUG,format='[%(asctime)s.%(msecs)03d] %(levelname)s %(filename)s(line:%(lineno)d): %(message)s')

Log Levels:
import logging
msg = "i broke"
logging.debug(msg)
logging.info(msg)
logging.warning(msg) # WARNING:root:i broke
logging.error(msg) # ERROR:root:i broke
logging.critical(msg) # CRITICAL:root:i broke

log.setLevel(level) # debug, info, warn/warning, error, critical
log.setLevel(logging.DEBUG)

A good convention to use when naming loggers is to use a module-level logger, in each module which uses logging, named as follows: [http://docs.python.org/dev/howto/logging.html]
logger = logging.getLogger(__name__)

Simple example: [http://stackoverflow.com/questions/345991/python-no-handlers-could-be-found-for-logger-opengl-error]
<pre>
log_format='%(asctime)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
logging.basicConfig(format=log_format, datefmt=log_date_format)
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)

log.info('hi')
</pre>

Formatter function:
<pre>
def makeFormatter(format_str="", date_format="%Y-%m-%d %H:%M:%S"):
if not format_str:
format_str = "%(asctime)s | %(levelname)s | %(filename)s:%(lineno)d:%(funcName)s | %(message)s"
return logging.Formatter(fmt=format_str, datefmt=date_format)
</pre>

Medium example:
<pre>
# Create logger
log_format='%(asctime)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
log_formatter = logging.Formatter(fmt=log_format, datefmt=log_date_format)
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)

# Log to console
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(log_formatter)
log.addHandler(ch)

# Log to file
fh = logging.FileHandler('my_logger.log')
fh.setLevel(logging.DEBUG)
fh.setFormatter(log_formatter)
log.addHandler(fh)
</pre>

Log all loggers to file - attach to root logger
<pre>
import logging

logger = logging.getLogger(__name__)

# Configure Logger
log_format='%(asctime)s | %(name)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
log_formatter = logging.Formatter(fmt=log_format, datefmt=log_date_format)
logging.basicConfig(level=logging.DEBUG, format=log_format, datefmt=log_date_format)

# Setup Log to File handler, and attach to root logger
root_logger = logging.getLogger() # Calling getLogger() without a name returns the root logger
fh = logging.FileHandler('logger.txt')
fh.setLevel(logging.DEBUG)
fh.setFormatter(log_formatter)
root_logger.addHandler(fh)
</pre>

Application Setup (not basic log configuration carries across imported modules):
<pre>
import logging
def main():
logging.basicConfig(filename='myapp.log', level=logging.INFO)
logging.info('Started')
...
logging.info('Finished')

if __name__ == '__main__':
main()
</pre>

Log multiple lines:
<pre>
for line in output.splitlines():
log.debug(line)
</pre>

Set log level: (warning is default)
logging.basicConfig(level=logging.DEBUG)

Set log level from string value: [http://docs.python.org/dev/howto/logging.html]
<pre>
# assuming loglevel is bound to the string value obtained from the
# command line argument. Convert to upper case to allow the user to
# specify --log=DEBUG or --log=debug
numeric_level = getattr(logging, loglevel.upper(), None)
if not isinstance(numeric_level, int):
raise ValueError('Invalid log level: %s' % loglevel)
logging.basicConfig(level=numeric_level, ...)
</pre>

Log to file:
logging.basicConfig(filename='example.log',level=logging.DEBUG)
logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG) # do not append

Format output: (once set, can't be changed?)
# WARNING:MESSAGE
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)

# 05/23/2011 12:50:02 PM MESSAGE
logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')

# 05/23/2011 12:55:28 PM : WARNING : MESSAGE
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')

# 2011-05-23 13:05:30 : WARNING : MESSAGE
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

# 2011-05-23 13:05:30 | WARNING | FILE:FUNC | MESSAGE
logging.basicConfig(format='%(asctime)s | %(levelname)s | %(filename)s:%(funcName)s | %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

Log record attributes - http://docs.python.org/library/logging.html#logrecord-attributes

Format Example:
<pre>
FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' # doesn't work!
logging.basicConfig(format=FORMAT)
d = { 'clientip' : '192.168.0.1', 'user' : 'fbloggs' }
logger = logging.getLogger('tcpserver')
logger.warning('Protocol problem: %s', 'connection reset', extra=d)
</pre>

<pre>
import logging

# create logger with 'spam_application'
logger = logging.getLogger('spam_application')
logger.setLevel(logging.DEBUG)

# create file handler which logs even debug messages
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)

# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)

# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(fh)
logger.addHandler(ch)

logger.info('Message to loggers and handlers')
</pre>

=== Templating ===

from string import Template
t = Template('${name}'s fav dish is $food')
t.substitute(name = 'ken', food = 'beans') # throws exception if missing one, but can have more
t.safe_substitute(name = 'ken') # will fill in what it can

Change delimiter:
class MYT(Template):
delimiter = '%'
minput = input("File name format (%d - date, %s - size): ")
myt = MYT(minput)
myt.substitute(%d=date, %s=size)

=== [http://docs.python.org/library/threading.html Threading] ===

Basic thread: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import thread

def someFunc():
print "someFunc was called"

thread.start_new_thread(someFunc, ())
</pre>

More complete basic thread:
<pre>
import thread
import threading
import time

def run_thread (threadname, sleeptime):
print "Thread start: " + threadname
time.sleep(sleeptime)
print "Thread end: " + threadname
thread.interrupt_main() # throw keyboard interrupt to kill main program

if __name__ == "__main__":
thread.start_new_thread(run_thread, ("Thread1", 3))
try:
while 1:
pass
except:
print "Thread1 exited...."
</pre>

Can use a global counter instead of exception to track number of threads:
<pre>
import thread
import threading
import time

threadcount = 3

def run_thread (threadname, sleeptime):
global threadcount
print "Thread start: " + threadname
time.sleep(sleeptime)
print "Thread end: " + threadname
threadcount -= 1

if __name__ == "__main__":
thread.start_new_thread(run_thread, ("Thread1", 3))
thread.start_new_thread(run_thread, ("Thread2", 4))
thread.start_new_thread(run_thread, ("Thread3", 2))
while 1:
if threadcount == 0:
break
else:
pass
print "completed"
</pre>

---

Thread object: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import threading

t1 = threading.Thread(target=someFunc)
# with parameters: Thread(target=myfunction, args=('MyStringHere',1)).start()
t1.start()
t1.join()
</pre>

Basic threading with parameters:
<pre>
import threading
t = threading.Thread(None, someFunc, None, (param1, param2))
t.start()
t.join()
</pre>

<pre>
t = threading.Thread(target=client_recv, args=(s, name))
while True:
t.join(timeout=2)
if t.isAlive():
print "Thread is dead..."
break
</pre>

Threading as a class: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import threading

class FuncThread(threading.Thread):
def __init__(self, target, *args):
self._target = target
self._args = args
threading.Thread.__init__(self)

def run(self):
self._target(*self._args)

# Example usage
def someOtherFunc(data, key):
print "someOtherFunc was called : data=%s; key=%s" % (str(data), str(key))

t1 = FuncThread(someOtherFunc, [1,2], 6)
t1.start()
t1.join()
</pre>

Threading as a class #2:
<pre>
import threading
import time
import sys

class MyT(threading.Thread):
def __init__(self, name, time):
threading.Thread.__init__(self)
self.name = name
self.time = time
def run(self):
print("Thread Start: " + self.name)
for i in range(self.time):
print("[" + self.name + "]", end='')
sys.stdout.flush()
time.sleep(1)
print("Thread Stop: " + self.name)

myt1 = MyT('1', 10)
myt1.start()

myt2 = MyT('2', 15)
myt2.start()

myt3 = MyT('3', 5)
myt3.start()

myt4 = MyT('4', 10)
myt4.start()

myt1.join()
myt2.join()
myt3.join()
myt4.join()
</pre>

Locking:
<pre>
lock=thread.allocate_lock()

# pass lock to thread, and use in thread as such:
lock.acquire()
#some operation
lock.release()
</pre>

Returning data from a thread (using a [http://docs.python.org/library/queue.html queue]): [http://stackoverflow.com/questions/1886090/return-value-from-thread] [http://stackoverflow.com/questions/2829329/catch-a-threads-exception-in-the-caller-thread-in-python]
<pre>
# Pass exception back to parent
import sys
import threading
import Queue

class ExcThread(threading.Thread):
def __init__(self, bucket):
threading.Thread.__init__(self)
self.bucket = bucket
def run(self):
try:
raise Exception('An error occured here.')
except Exception:
self.bucket.put(sys.exc_info())

def main():
bucket = Queue.Queue()
thread_obj = ExcThread(bucket)
thread_obj.start()
while True:
try:
exc = bucket.get(block=False)
except Queue.Empty:
pass
else:
exc_type, exc_obj, exc_trace = exc
# deal with the exception
print "Exception Caught:", exc_type, exc_obj
print "Exception Trace:", exc_trace
thread_obj.join(0.1)
if thread_obj.isAlive():
continue
else:
break

if __name__ == '__main__':
main()
</pre>

<pre>
Exception Caught: <type 'exceptions.Exception'> An error occured here.
Exception Trace: <traceback object at 0x2b2230a285f0>
</pre>

Another Exception catching queue example: (kenneth's example)
<pre>
# thread function
def run_thread(queue, item):
...
queue.put( Exception("Failure on item:" + item) )

threads = []
queue = Queue.Queue()

# start threads
for item in items:
t = Thread(None, run_thread, None, (queue, item))
t.start()
threads.append(t)

# wait for threads
for t in threads:
t.join()

# catch and reraise exceptions
while not queue.empty():
raise(queue.get())
</pre>

Returning data from a thread (using a callback function): [http://stackoverflow.com/questions/1886090/return-value-from-thread]
:"Another approach is to pass a callback function to the thread. This gives a simple, safe and flexible way to return a value to the parent, anytime from the new thread."
<pre>
import threading
import time
import sys

class MyThread(threading.Thread):
def __init__(self, cb):
threading.Thread.__init__(self)
self.callback = cb
def run(self):
for i in range(10):
self.callback(i)
time.sleep(1)

def count(x):
print x
sys.stdout.flush()

t = MyThread(count)
t.start()
</pre>

References:
* 16.2. threading — Higher-level threading interface — Python v2.7.1 documentation - http://docs.python.org/library/threading.html
* An Introduction to Thread Programming - http://www.prasannatech.net/2008/08/introduction-to-thread-programming.html
* Python - Multithreaded Programming - http://www.tutorialspoint.com/python/python_multithreading.htm
* Running Functions as Threads in Python « Software Ramblings - http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html
* Understanding Threading in Python LG #107 - http://linuxgazette.net/107/pai.html
* multithreading - Simple threading in Python 2.6 using thread.start_new_thread() - Stack Overflow - http://stackoverflow.com/questions/849674/simple-threading-in-python-2-6-using-thread-start-new-thread
* python - Return value from thread - Stack Overflow - http://stackoverflow.com/questions/1886090/return-value-from-thread

---

Interrupt main thread:
<pre>
# source: http://stackoverflow.com/questions/4591917/python-how-to-quit-cli-when-stuck-in-blocking-raw-input

import thread
import time

def main():
try:
m = thread.start_new_thread(killable_input, tuple())
while 1:
time.sleep(0.1)
except KeyboardInterrupt:
print "exception"

def killable_input():
w = thread.start_new_thread(normal_input, tuple())
i = thread.start_new_thread(wait_sometime, tuple())

def normal_input():
s = raw_input("input:")

def wait_sometime():
time.sleep(4) # or any other condition to kill the thread
print "too slow, killing imput"
thread.interrupt_main()

if __name__ == '__main__':
main()
</pre>

=== urllib ===

See [[Python/urllib]]

=== Command line arguments ===

<pre>
import sys
for x in sys.argv:
print x
</pre>

=== Execute other python script ===

import sys
sys.argv = ['', 'my', 'args', 'here']
execfile('myscript.py') # does not support arguments directly

=== Exit Code ===

sys.exit(n) http://docs.python.org/library/sys.html:
import sys
sys.exit(1) # exit code of 1 (default is 0)
sys.exit("some error") # exit code of 1, and message displayed

=== [http://docs.python.org/library/syslog.html syslog] ===

<pre>
import syslog
syslog.openlog('[WATER]')
syslog.syslog('Processing started')
if error:
syslog.syslog(syslog.LOG_ERR, 'Processing started')
</pre>

<pre>
import syslog
syslog.openlog('[WATER]')
def log(msg):
syslog.syslog(str(msg))
log("Starting...")
</pre>

=== Subprocess ===

Subprocess - http://docs.python.org/library/subprocess.html

Methods of subprocess (or external command execution):
* subprocess (only method to rule/replace all others)
* os.system
* os.spawn*
* os.popen*
* popen2.*
* commands.*

My New Favorite:
<pre>
import subprocess
class Run(object):
class RC(int):
pass
def __new__(cls, cmd, *args, **kwargs):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
rc = cls.RC(p.wait())
rc.output = "\n".join(p.stdout.readlines())
rc.error = "\n".join(p.stderr.readlines())
return rc

if Run("some command"): print "do something" # on rc of not 0 (negative or positive)
</pre>

My Favorite:
<pre>
def exe(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
#output = " | ".join(p.stdout.readlines())
output = "\n".join(p.stdout.readlines())
rc = p.wait()
return (rc, output)

def exe_checked(cmd, checked=True):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
#output = " | ".join(p.stdout.readlines())
output = "\n".join(p.stdout.readlines())
rc = p.wait()
if rc and checked:
raise Exception("Command returned non zero [{0}]: {1}".format(rc, cmd)
return (rc, output)

def exe_with_stderr(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subproc
ess.PIPE)
output = " | ".join(p.stdout.readlines())
outputerr = " | ".join(p.stderr.readlines())
rc = p.wait()
return (rc, output, outputerr)
</pre>

<pre>
def exec_program(cmd):
s = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = s.communicate()
result = s.wait()
return stdout, stderr, result
</pre>

Pass along output and return true error code:
import subprocess
retcode = subprocess.call(["ls", "-l"]) # true return code, output sent to STDOUT

retcode = subprocess.call(["foo.sh","args"],shell=True) # Execute in shell, with shell processing
retcode = subprocess.call(['C:\\Temp\\a b c\\Notepad.exe'])
retcode = subprocess.call(["D:/Python24/Python.exe", "Skripta.py", "arg1", "arg2"])

Note: shell=True is equivalent to: subprocess.Popen(['/bin/sh', '-c', args[0], args[1], ...])

Process STDOUT and STDERR and get return code:
<pre>
#!/usr/bin/env python
import subprocess
import string

cmd = "cat /etc/passwd | grep ken"
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) # stderr=subprocess.STDOUT to combine

out = string.join(p.stdout.readlines())
outerr = string.join(p.stderr.readlines())
retcode = str(p.wait())

print("STDOUT: " + out)
print("STDERR: " + outerr)
print("RETCODE: " + retcode)
</pre>

Above with stderr redirected to stdout (combined):
<pre>
import subprocess
p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
print line,
retval = p.wait()
</pre>

Replacing shell backtick: [http://docs.python.org/library/subprocess.html]
# shell code:
# OUTPUT=`mycmd myarg`
from subprocess import Popen
output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]

Replacing shell pipeline [http://docs.python.org/library/subprocess.html]
# shell code:
# OUTPUT=`dmesg | grep hda`
from subprocess import Popen
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

Replacing os.system(): [http://docs.python.org/library/subprocess.html]
# old:
sts = os.system("mycmd" + " myarg")

# new:
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)[1]

Control an executed process:
<pre>
cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

# cycle while process continues alive
while p.poll() == None:

# check if input
if select.select([p.stdout.fileno()], [], [], 1)[0]:
while select.select([p.stdout.fileno()], [], [], .1)[0]:
line = p.stdout.readline()
# ... process

# write something to process
p.stdin.write("tp %s %s\n" % (user1, user2))

...

# time to force a quit: (choose which you need)
p.kill() # SIGKILL
p.terminate() # SIGTERM
os.kill(p.pid, signal.SIGTERM) # same as first, but not as clean
</pre>

==== Other Command Execution ====

Old method with os.system(): [http://docs.python.org/library/os.html]
import os
encoded_wait_code = os.system("some command") # output sent to STDOUT
# Note: returns code of 0 if command returns 0, but wait() encoded value for all other values

Quick and dirty old method (prints stderr, captures stdout)
<pre>
import os
out = string.join( os.popen(cmd).readlines() )
</pre>

Quick and dirty old method (prints stderr, captures stdout, captures ugly wait() encoded return code)
<pre>
import os
p = os.popen(cmd)
stdout = "".join( p.readlines() )
encoded_wait_code = p.close() # None if 0
rcode = os.WEXITSTATUS(ugly_rcode) # fix ugly wait() encoded rcode
print(stdout)
print(rcode)

if os.popen(cmd).close():
print "error with cmd: %s" % cmd
sys.exit(1)
</pre>

???:
<pre>
os.spawnvp(os.P_WAIT,"/usr/bin/mplayer",["","-ao","pcm",file])
# or
os.spawnvp(os.P_NOWAIT,"/usr/bin/mplayer",["","-ao","pcm",file])
</pre>

Replacing os.system(): [http://docs.python.org/library/subprocess.html]
# old:
sts = os.system("mycmd" + " myarg")

# new:
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)[1]

More realistic example replacing os.system(): [http://docs.python.org/library/subprocess.html]
<pre>
try:
retcode = call("mycmd" + " myarg", shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e
</pre>

==== select ====

Select:

select.select(rlist, wlist, xlist[, timeout])
: This is a straightforward interface to the Unix select() system call. The first three arguments are sequences of ‘waitable objects’: either integers representing file descriptors or objects with a parameterless method named fileno() returning such an integer
: The optional timeout argument specifies a time-out as a floating point number in seconds. When the timeout argument is omitted the function blocks until at least one file descriptor is ready. A time-out value of zero specifies a poll and never blocks.

if select.select([sys.stdin.fileno()], [], [], .1)[0]:
line = p.stdout.readline()

==== full wrapped execution ====

Process timeout (by Kenneth):
<pre>
import subprocess
import time
import os
import signal

def cmd(cmd, timeout=300, retry=0):
# Execute command, time progress, and kill on timeout

# (status, output) = commands.getstatusoutput(cmd)
start_time = None
status = None
output = None
for retry_count in range(retry+1):
if retry_count > 0:
print("RETRY: " + str(retry_count) + " " + cmd)
start_time = time.time()
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
timeout_fail = False
while p.poll() == None:
if time.time() - start_time > timeout:
# kill process
print("TIMEOUT: " + str(timeout) + "sec - killing (" + str(p.pid) + ") - " + cmd)
try:
os.kill(p.pid, signal.SIGTERM)
except OSError as e:
print("Kill exception: " + str(e))
timeout_fail = True
break
#print(".")
time.sleep(1)
status = p.poll()
output = p.stdout.read()
if timeout_fail == False:
break
seconds = int(time.time() - start_time)
return (status, output, seconds)
</pre>

---

Process input and output wrapper (from minecraft):
<pre>
import subprocess
import select
import re
import sys

cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)

# cycle while process continues alive
while p.poll() == None:

if select.select([p.stdout.fileno()], [], [], 1)[0]:
while select.select([p.stdout.fileno()], [], [], .1)[0]:
line = p.stdout.readline()
# <kiloforce> -tp joe
match = re.findall("<(.*)> -tp (.*)", line)
if match:
user1 = match[0][0]
user2 = match[0][1]
print ">> ACTION: Teleporting %s to %s" % (user1, user2)
p.stdin.write("tp %s %s\n" % (user1, user2))
if line:
print ">> SERVER: %s" % line.strip()
else:
print "The server died!"
sys.exit(1)

if select.select([sys.stdin.fileno()], [], [], .1)[0]:
line = sys.stdin.readline()
print ">> CONSOLE: %s" % line.strip()
p.stdin.write("%s" % line)
</pre>

==== Command With Timeout ====

python - Using module 'subprocess' with timeout - Stack Overflow - http://stackoverflow.com/questions/1191374/using-module-subprocess-with-timeout

<pre>
import subprocess, threading

class Command(object):
def __init__(self, cmd):
self.cmd = cmd
self.process = None

def run(self, timeout):
def target():
print 'Thread started'
self.process = subprocess.Popen(self.cmd, shell=True)
self.process.communicate()
print 'Thread finished'

thread = threading.Thread(target=target)
thread.start()

thread.join(timeout)
if thread.is_alive():
print 'Terminating process'
self.process.terminate()
thread.join()
print self.process.returncode

command = Command("echo 'Process started'; sleep 2; echo 'Process finished'")
command.run(timeout=3)
command.run(timeout=1)
</pre>

My version:
<pre>
import subprocess
import threading

class Command(object):
def __init__(self, cmd, capture=False, shell=True):
self.cmd = cmd
self.process = None
self.thread = None
self.shell = shell
if type(cmd) == list:
self.shell = False
self.rc = None
self.pid = None
self.stdout = None
self.stderr = None
self.capture = capture # capture stdout?

def run(self, timeout=None, wait=True):
def target():
print '>> Thread started (%s)' % self.cmd
if self.capture:
self.process = subprocess.Popen(self.cmd, shell=self.shell, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
else:
self.process = subprocess.Popen(self.cmd, shell=self.shell)
print '>> Thread PID: %d' % self.process.pid
self.pid = self.process.pid
self.stdout, self.stderr = self.process.communicate()
self.rc = self.process.returncode
print '>> Thread finished: %d' % self.rc

self.thread = threading.Thread(target=target)
self.thread.start()

if wait:
self.wait(timeout)
return self # so you can do: c = Command().run()

def wait(self, timeout=None):
print '>> Waiting for thread...'
if timeout:
self.thread.join(timeout)
else:
self.thread.join()
if self.thread.is_alive():
print '>> Terminating process'
self.process.terminate()
self.rc = self.process.returncode
self.thread.join()
if self.thread.is_alive():
print '>> Failed to terminate'
else:
print '>> Thread completed'
#print self.process.returncode

def is_alive(self ):
return self.thread.is_alive()

command = Command('do something').run()

command2 = Command(['do', 'something', 'long']).run(wait=False)
command2.wait(timeout=10)

command3 = Command('want output', capture=True).run()
print command3.stdout
</pre>

=== System call ===

os.system("some_command with args")
os.system("some_command < input_file | another_command > output_file")
stream = os.popen("some_command with args")

Here's a summary of the ways to call external programs and the advantages and disadvantages of each:

1.
: os.system("some_command with args") passes the command and arguments to your system's shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example,
: os.system("some_command < input_file | another_command > output_file")
: However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs.
: http://docs.python.org/lib/os-process.html

2.
: stream = os.popen("some_command with args") will do the same thing as os.system except that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything.
: http://docs.python.org/lib/os-newstreams.html

3.
: The Popen class of the subprocess module. This is intended as a replacement for os.popen but has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you'd say
: print Popen("echo Hello World", stdout=PIPE, shell=True).stdout.read()
: instead of
: print os.popen("echo Hello World").read()
: but it is nice to have all of the options there in one unified class instead of 4 different popen functions.
: http://docs.python.org/lib/node528.html

4.
: The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply wait until the command completes and gives you the return code. For example:
: return_code = call("echo Hello World", shell=True)
: http://docs.python.org/lib/node529.html

5.
: The os module also has all of the fork/exec/spawn functions that you'd have in a C program, but I don't recommend using them directly.

: The subprocess module should probably be what you use.

Source: [http://stackoverflow.com/questions/89228/how-to-call-external-command-in-python]

== Advanced Code ==

=== Executing Code from Command Line ===

python -c [cmd]
# -c cmd : program passed in as string (terminates option list)

<pre>
python -c "import sys;exec(sys.stdin.read())" << "EOF"
def test1():
print "test1"
test1()
print "test"
EOF
</pre>

=== Integer Return Code Class ===

Python3 version:
<pre>
class RC(int):
def __new__(cls, rc, stdout='', stderr=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, stdout='', stderr=''):
super().__init__()
self.rc = rc # redundant
self.stdout = stdout
self.stderr = stderr
def __repr__(self):
"""Optional string output"""
return "(%s, '%s', '%s')" % (self.rc, self.stdout, self.stderr)
</pre>

Python2 version:
<pre>
class RC(int):
def __new__(cls, rc, stdout='', stderr=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, stdout='', stderr=''):
super(self.__class__, self).__init__(rc)
self.rc = rc # redundant
self.stdout = stdout
self.stderr = stderr
def __repr__(self):
"""Optional string output"""
return "(%s, '%s', '%s')" % (self, self.stdout, self.stderr)
</pre>

sample:
<pre>
>>> rc1 = RC(5)
>>> rc2 = RC(0, 'out', 'err')
>>> rc1
(5, '', '')
>>> rc2
(5, 'out', 'err')
>>> rc2.stdout
'out'
>>> int(rc1)
5
>>> 0 + rc1
5
>>> if rc1 <= 5: print "yes"
...
yes
>>> if rc2: print "non-zero"
...
</pre>

=== Smart Execute ===

<pre>
class RC(int):
def __new__(cls, rc, output=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, output=''):
super().__init__()
self.rc = rc # redundant
self.output = output
def __repr__(self):
"""Optional string output"""
return "(%s, '%s')" % (self.rc, self.output)

def smart_execute(cmd, display=True, checked=False):
output = ""
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, shell=True, encoding='utf-8')
for c in iter(lambda: process.stdout.read(1), ""):
if display:
sys.stdout.buffer.write(c.encode('utf-8'))
output += c
process.stdout.close()
rc = process.wait()
if checked and rc:
raise subprocess.CalledProcessError(ret, cmd)
return RC(rc, output.strip())
</pre>

Not smart version:
<pre>
def execute(cmd, return_output=False):
out = ""
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, shell=True, encoding='utf-8')
for c in iter(lambda: process.stdout.read(1), ""):
sys.stdout.buffer.write(c.encode('utf-8'))
if return_output:
out += c
process.stdout.close()
ret = process.wait()
if ret:
raise subprocess.CalledProcessError(ret, cmd)
if return_output:
return out.strip()
</pre>
</pre>

=== select ===

usage: select.select(rlist, wlist, xlist[, timeout]) [http://docs.python.org/2/library/select.html]
* rlist: wait until ready for reading
* wlist: wait until ready for writing
* xlist: wait for an “exceptional condition” (see the manual page for what your system considers such a condition)
* The optional timeout argument specifies a time-out as a floating point number in seconds.

if select.select([p.stdout.fileno()], [], [], 1)[0]:

=== ANSI Color ===

* ANSI escape code - http://en.wikipedia.org/wiki/ANSI_escape_code

---

unicode - Print in terminal with colors using Python? - Stack Overflow - https://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python

<pre>
class bcolors:
HEADER = '\033[95m'
OKBLUE = '\033[94m'
OKGREEN = '\033[92m'
WARNING = '\033[93m'
FAIL = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'

print bcolors.WARNING + "Warning: No active frommets remain. Continue?" + bcolors.ENDC
</pre>

<pre>
class colors:
black = 0
red = 1
green = 2
yellow = 3
blue = 4
magenta = 5
cyan = 6
white = 7
fg = 30 # foreground add
bg = 40 # background add
bfg = 90 # bold foreground add
bbg = 100 # bold background add

print '\033[%dm red foreground \033[0m' % (colors.red + colors.fg)
print '\033[%dm bold red foreground \033[0m' % (colors.red + colors.bfg)
print '\033[%dm blue background \033[0m' % (colors.blue + colors.bg)
print '\033[%dm bold blue background \033[0m' % (colors.blue + colors.bbg)
</pre>

---

Before you start spouting escape sequences, you should check that stdout is a tty. You can do this with sys.stdout.isatty()

<pre>
sys.stdout.isatty()
</pre>

<pre>
def hilite(string, status, bold):
attr = []
if status:
# green
attr.append('32')
else:
# red
attr.append('31')
if bold:
attr.append('1')
return '\x1b[%sm%s\x1b[0m' % (';'.join(attr), string)
</pre>

Python | change text color in shell - Stack Overflow - https://stackoverflow.com/questions/2330245/python-change-text-color-in-shell

---

For reference, a small bash script which prints a table of escape sequences: colortable16.sh: ([http://www.frexx.de/xterm-256-notes/ The 256 color mode of xterm], [http://plug.org/pipermail/plug/2008-March/016821.html The 90s called; they want their 256 colors back])
<pre>
#!/bin/bash

# prints a color table of 8bg * 8fg * 2 states (regular/bold)
echo
echo Table for 16-color terminal escape sequences.
echo Replace ESC with \\033 in bash.
echo
echo "Background | Foreground colors"
echo "---------------------------------------------------------------------"
for((bg=40;bg<=47;bg++)); do
for((bold=0;bold<=1;bold++)) do
echo -en "\033[0m"" ESC[${bg}m | "
for((fg=30;fg<=37;fg++)); do
if [ $bold == "0" ]; then
echo -en "\033[${bg}m\033[${fg}m [${fg}m "
else
echo -en "\033[${bg}m\033[1;${fg}m [1;${fg}m"
fi
done
echo -e "\033[0m"
done
echo "--------------------------------------------------------------------- "
done

echo
echo
</pre>

Python version: (modified by me)
<pre>
#!/usr/bin/env python

import sys

# prints a color table of 8bg * 8fg * 2 states (regular/bold)
print
print "Table for 16-color terminal escape sequences."
print "Replace ESC with \\033 in bash."
print
print "Background | Foreground colors"
print "---------------------------------------------------------------------"
for bg in range(40, 47+1):
for bold in range(0, 1+1):
sys.stdout.write("\033[0m"" ESC[{bg}m | ".format(bg=bg))
for fg in range(30, 37+1):
if bold == 0:
sys.stdout.write("\033[{bg}m\033[{fg}m [{fg}m ".format(bg=bg, fg=fg))
else:
sys.stdout.write("\033[{bg}m\033[1;{fg}m [1;{fg}m".format(bg=bg, fg=fg))
sys.stdout.write("\033[0m\n")
print "--------------------------------------------------------------------- "

print
print
</pre>

=== Execute External Application ===

Execute program: (and send output to STDOUT)
encoded_wait_code = os.system('mkdir mydir') # output sent to STDOUT

Get output and ugly return code: (STDERR goes to STDOUT)
p = os.popen(cmd)
print p.readlines()
encoded_wait_code = p.close()

Execute other python script
import sys
sys.argv = [, 'my', 'args', 'here']
execfile('myscript.py') # does not support arguments directly

Subprocess:
* See [[#Subprocess]]

=== Program Layout ===

<pre>
PackageName/
README
setup.py
bin/
script
docs/
test/ # some include in package_name
pacakgename/
__init__.py
code1.py
subpackage/
__init__.py
</pre>

=== __init__.py ===

Purpose: Makes the folder into a package library

NOTE: This file is most commonly blank

Good place to put your package functions.
<pre>
# ken/__init__.py
def load():
...
</pre>

<pre>
import ken
ken.load()
</pre>

Can import other modules to then use as package modules:
<pre>
# ken/__init__.py
from ken.client import Server
</pre>

<pre>
import ken
ken.Server()
</pre>

Good place to put your Exceptions.
<pre>
class MyException(Exception):
""" My Error Exception """
</pre>

Use the __all__ to specify what gets imported on 'from package import *'
__all__ = ['func1', 'func2']

=== Metadata ===

VERSION = (0, 2, 1)
or
__version__ = '0.2.1'

Also See [[#pydoc]]

---

__version__ = '1.10'
__author__ = 'Joshua Roesslein'
__license__ = 'MIT'

=== setup.py ===

See [[#Distributing Python Modules with Distutils]]

=== Distributing Python Modules with Distutils ===

Distributing Python Modules — http://docs.python.org/distutils/index.html

Uses setup.py for package installation.
python setup.py install

==== Simple example ====

To install foo.py:
<pre>
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
</pre>

Source: http://docs.python.org/distutils/introduction.html

==== klib example ====

klib setup.py:
<pre>
from distutils.core import setup

setup(name = "klib",
version = "101",
description = "klib library",
url = "http://www.oeey.com/klib",
author = "Kenneth",
author_email = "kenneth@oeey.com",
license = "Kenneth License",
long_description = """Really long text here.""",
platforms = "Any Platform",

packages = ['klib','klib.messenger'],
)
</pre>

Installed to:
/opt/python26/lib/python2.6/site-packages/klib/
/opt/python26/lib/python2.6/site-packages/klib-100-py2.6.egg-info

Metadata: (/opt/python26/lib/python2.6/site-packages/klib-101-py2.6.egg-info)
<pre>
Metadata-Version: 1.0
Name: klib
Version: 101
Summary: klib library
Home-page: http://www.oeey.com/klib
Author: Kenneth
Author-email: kenneth@oeey.com
License: Kenneth License
Description: Really long text here.
Platform: Any platform
</pre>

Any fields not included will be marked:
Platform: UNKNOWN

References:
* 2. Writing the Setup Script — Python v2.7.2 documentation - http://docs.python.org/distutils/setupscript.html
* http://wiki.python.org/moin/Distutils/Tutorial

==== Uninstall Package ====

Collect files during install:
python setup.py install --record install-record.txt
sudo rm $(cat install-record.txt)

References:
* installation - python setup.py uninstall - Stack Overflow - http://stackoverflow.com/questions/1550226/python-setup-py-uninstall

=== Distributing Python Modules with Setuptools ===

<pre>
#!/usr/bin/env python
import setuptools

setuptools.setup(name="tweepy",
version="1.10",
description="Twitter library for python",
license="MIT",
author="Joshua Roesslein",
author_email="tweepy@googlegroups.com",
url="http://github.com/tweepy/tweepy",
packages = setuptools.find_packages(),
keywords= "twitter library",
zip_safe = True)

</pre>

=== Decorators ===

Decorators wrap functions

Function Counter: [http://stackoverflow.com/questions/739654/understanding-python-decorators]
<pre>
def fcount(func):
def wrapper(*args, **kw):
wrapper.count = wrapper.count + 1
return func(*args, **kw)
wrapper.count = 0
return wrapper

@fcount
def f(n):
return n*n

print "f.count = ", f.count
print "f(10) = ", f(10)
print "f.count = ", f.count
print "f(3) = ", f(3)
print "f.count = ", f.count
print "f(12) = ", f(12)
print "f.count = ", f.count
print "f(15) = ", f(15)
print "f.count = ", f.count

# Output:
# f.count = 0
# f(10) = 100
# f.count = 1
# f(3) = 9
# f.count = 2
# f(12) = 144
# f.count = 3
# f(15) = 225
# f.count = 4
</pre>

Great reference: Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

Example 1:
@dec1
def func(arg1, arg2, ...): pass

# equivalent to:
def func(arg1, arg2, ...): pass
func = dec1(func)

Example 2:
<pre>
@dec2
@dec1
def func(arg1, arg2, ...):
pass
</pre>

<pre>
# equivalent to:
def func(arg1, arg2, ...):
pass
func = dec2(dec1(func))
</pre>

Example 3:
@decomaker(argA, argB, ...)
def func(arg1, arg2, ...):
pass

# equivalent to:
func = decomaker(argA, argB, ...)(func)

# example:
@accepts(int,int)
@returns(float)
def bar(low,high):
pass

Example 4: (overrides and ignores passed in function - bad, unless desired)
<pre>
def spamrun(fn):
def sayspam(*args):
print "spam, spam, spam"
return sayspam

@spamrun
def useful(a, b):
print a**2 + b**2
</pre>

Example 5: (Decorator that modifies behavior of undecorated func)
<pre>
def addspam(fn):
def new(*args):
print "spam, spam, spam"
return fn(*args)
return new

@addspam
def useful(a, b):
print a**2 + b**2
</pre>

My Example 1: (print args and continue to function)
<pre>
def showargs(fn):
def doshowargs(*args, **kw):
print "Args for:", fn.__name__, args, kw
return fn(*args, **kw)
return doshowargs

@showargs
def useful(a, b):
print a**2 + b**2

</pre>

Another Example: [http://stackoverflow.com/questions/739654/understanding-python-decorators]
<pre>
def makebold(fn):
def wrapped(*args, **kw):
return "" + fn(*args, **kw) + ""
return wrapped

def makeitalic(fn):
def wrapped(*args, **kw):
return "" + fn(*args, **kw) + ""
return wrapped

@makebold
@makeitalic
def hello(msg):
return msg

print hello("hello world")
# output: hello world
</pre>

References:
* PEP 318 -- Decorators for Functions and Methods - http://www.python.org/dev/peps/pep-0318/
* Charming Python: Decorators make magic easy - http://www.ibm.com/developerworks/linux/library/l-cpdecor/index.html
* What are some common uses for Python decorators? - Stack Overflow - http://stackoverflow.com/questions/489720/what-are-some-common-uses-for-python-decorators
* Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

---

Utah Python July 2013 Meeting - Decorator Examples: [https://www.youtube.com/watch?v=uh9uhzitdvw]

Example of uses:
* Count function calls
* Add additional output (debugging)
* Timing code
* Thread synchronization (locks)
* Type checking

<pre>
## Simple wrapping call
def shout(f):
def wrapper(*args):
print('This is before {name}'.format(name=f.__name__))
ret = f(*args)
print('This is after {name}'.format(name=f.__name__))
return ret
wrapper.__name__ = f.__name__
return wrapper

@shout
def echo(text):
print(text)

echo('this is a test')
</pre>

<pre>
## Counting function calls
def count(f):
def wrapper(*args):
wrapper.counter += 1
print("Function called {count} times".format(count=wrapper.counter))
return f(*args)
wrapper.counter = 0
# change wrapper name, or it will just be 'wrapper' (for inspection)
wrapper.__name__ = f.__name__
return wrapper

@count
def donothing():
pass

donothing()
donothing()
print("Function called a total of {count} times".format(count=donothing.counter))
</pre>

<pre>
## Timing your code
import time

def timer(f):
def wrapper(*arg):
t = time.time()
ret = f(*arg)
print("Took {elapsedtime} seconds".format(elapsedtime=time.time()-t))
return ret
return wrapper

@timer
def fact(n):
prod = 1
while n > 0:
prod *= n
n -= 1
return prod

total = fact(10000)
</pre>

<pre>
## Synchronization
import threading

def synchronized(lock):
def wrap(f):
def wrapper(*args, **kwargs):
lock.acquire()
try:
return f(*args, **kwargs)
finally:
lock.release()
return wrapper
return wrap

mylock = threading.Lock()

@synchronized(mylock)
def myfunc():
pass
</pre>

<pre>
## Type checking
def myMethod(ID, name):
''' Old way '''
if not (isinstance(ID, 'uint') and isinstance(name, 'utf8string')):
raise Exception()

def accepts(*typeargs):
''' Decorator way '''
def wrap(f):
def wrapper(*args):
for a, t in zip(args, typeargs):
if not isinstance(a, t):
raise Exception('{a} not of type {t}'.format(a=a, t=t))
return f(*args)
return wrapper
return wrap

@accepts(int)
def add1(n):
return n + 1
</pre>

<pre>
## Memoize
def memoize(f):
def wrapper(*args):
if not wrapper.ret:
wrapper.ret = f(*args)
return wrapper.ret
wrapper.ret = False
wrapper.__name__ = f.__name__
return wrapper

import time

@memoize
def sumto10():
count = 0
sum = 0
while count < 10:
sum += count
time.sleep(1)
count += 1
return sum

# first time will count slow
print sumto10()
# second time will return cached value instantly
print sumto10()
</pre>

Memoize: "Caches a function's return value each time it is called. If called later with the same arguments, the cached value is returned (not reevaluated)" [http://wiki.python.org/moin/PythonDecoratorLibrary#Memoize]

<pre>
## Nested decorators
import time
@shout
@memoize
def test1():
print "calculating"
time.sleep(3)
return 100
</pre>

Order is important! For tested decorators the execution order is from the top to the bottom. If you reverse @shout and @memoize, the @shout will not run because of the way @memoize was written.

<pre>
## registry
registry = []
# can also decorate decorators:
@shout
def registry(f):
registry.append(f)
return f

@register
def test2():
pass
</pre>

Decorator is just a function that returns a function.

Can use functools to automatically update decorator wrappers with __name__, etc [http://docs.python.org/2/library/functools.html]
<pre>
## Auto syntax sygar the wrapper for you
import functools
def my_decorator(f):
@functools.wraps(f)
def wrapper(*args, **kwds):
print 'Calling decorated function'
return f(*args, **kwds)
return wrapper
</pre>

=== Class Method Decorators ===

Static methods: [http://docs.python.org/library/functions.html#staticmethod]
: Similar to C++ and Java static methods
<pre>
class C:
@staticmethod
def say(msg):
print(msg)

C.say('hi')
</pre>

Class method: [http://docs.python.org/library/functions.html#classmethod]
: A class method receives the class as implicit first argument, just like an instance method receives the instance.
: Class methods are different than C++ or Java static methods. If you want those, see staticmethod() in this section.
<pre>
@classmethod
def setup_parser(cls, parser):
</pre>

See also [[#Class Properties]]

=== Class Properties ===

Properties: (not using decorators)
<pre>
class C(object):
def __init__(self):
self._x = None
def getx(self):
print("get x")
return self._x
def setx(self, value):
print("set x")
self._x = value
def delx(self):
print("del x")
del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")
</pre>

<pre>
#usage
c = C()
c.x = 5
print(c.x)
del(c.x)
</pre>

Getter and Setter Property Decorators: [http://docs.python.org/library/functions.html#property]
<pre>
class C(object): # make sure to inherit from object for these properties to work!
def __init__(self):
self._x = None

@property
def x(self):
"""I'm the 'x' property."""
print("get x")
return self._x

@x.setter # python >= 2.6
def x(self, value):
print("set x")
self._x = value

@x.deleter # python >= 2.6
def x(self):
print("del x")
del self._x
</pre>

<pre>
#usage
c = C()
c.x = 5
print(c.x)
del(c.x)
</pre>

Note: If you don't add the @x.setter, the property will be read only, and will throw a "AttributeError: can't set attribute" if attempted to be set.

Using x.getter example: [http://wiki.python.org/moin/AlternativeDescriptionOfProperty]
<pre>
class Cls(object):
def __init__(self):
self._x = 100

x = property()

@x.getter
def x(self):
"""I'm the 'x' property."""
return self._x

@x.setter
def x(self, value):
self._x = value

@x.deleter
def x(self):
del self._x
</pre>

=== Custom Decorators ===

decorator
@dec1
def func(arg1, arg2, ...): pass

equivalent to:
def func(arg1, arg2, ...): pass
func = dec1(func)

Basic layout (names are arbitrary):
<pre>
def my_decorator(function_name):
# update wrapper.__doc__ and .func_name
# or @functools.wraps(my_wrapper)
def mywrapper(*args, **kw):
# do something before
result = function_name(*args, **kw)
# do something after
return result
return my_wrapper
</pre>

Example:
<pre>
def limit(length):
def decorator(function):
def wrapper(*args, **kw):
result = function(*args, **kw)
result = result[:length]
return result
return wrapper
return decorator
</pre>

<pre>
@limit(3)
def echo(foo): return foo

echo('123456') # '123'
</pre>

=== assert ===

assert True
assert False # exception
assert 1 == 1
assert 1 == 2, "some message" # AssertionError: some message

=== Snake Case ===

Convert Camel Case to Snake Case: [http://stackoverflow.com/questions/1175208/elegant-python-function-to-convert-camelcase-to-snake-case]

<pre>
def snake_case(name):
s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name)
return re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1).lower()
</pre>

<pre>
>>> convert('CamelCase')
'camel_case'
</pre>

=== Camel Case ===

Camel Case: [http://stackoverflow.com/questions/8347048/camelcase-every-string-any-standard-library]

>>> ''.join(x for x in 'make IT camel CaSe'.title() if not x.isspace())
'MakeItCamelCase'

To titleize a sentence:
>>> "this is a test".title()
"This Is A Test"

=== Lambda ===

Simple functions
mul = lambda a, b: a * b

# equivalent to:
def mul(a, b):
return a * b

Sort usage:
data = [dict(number=x) for x in '036149']
data.sort(key=lambda x: float(x['number']))

Compare usage:
data = [dict(number=x) for x in '036149']
data.sort(cmp=lambda x,y: cmp(x['number'], y['number']))

Boolean:
lambda x: x % 3 == 0

Multiple parameter:
lambda x, y: x + y

Call another function:
lambda word: len(word)

Strings:
lambda line: line.split()[2]

Embedded:
def transform(n):
return lambda x: x + n

No parameters:
lambda: 1

Default parameters:
lambda x=1: x+1

References:
* Python: Lambda Functions - http://www.secnetix.de/olli/Python/lambda_functions.hawk

=== Map ===

Apply a function to items of a sequence:
map(str, range(3)) # ['0', '1', '2']

Note: list comprehensions can replace map and filter

=== Reduce ===

Apply a function to pairs of the sequence:
import operator
reduce(operator.mul, [1,2,3,4]) # 24 - ((1*2)*3)*4)

Note: sum or for loop can replace reduce

=== Filter ===

Return a sequence of items for which function(item) is True
filter(lambda x: x >= 0, [0, -1, 3]) # [0, 3]

Note: list comprehensions can replace map and filter

Find matching items in two lists:
a = ['a','b','c']
b = ['c','d']
filter(lambda x: x in a, b) # returns only 'c' as it is in both lists

=== List comprehension ===

Nothing more than Syntactic Sugar

5. Data Structures — Python v2.7.5 documentation - http://docs.python.org/2/tutorial/datastructures.html#list-comprehensions
: "List comprehensions provide a concise way to create lists."
: "A list comprehension consists of '''brackets''' containing an '''expression''' followed by a '''for clause''', then '''zero or more for or if clauses'''. "

Remove newlines from each item in a list:
mylist = [x.strip() for x in mylist]

Simple:
doubled = [x*2 for x in mylist]
squares = [x**2 for x in range(10)]

If:
[ 2*x for x in range(1,10) if x >= 5 ]

Complex:
[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]

Expanding PI:
[str(round(pi, i)) for i in range(1, 6)]

Return list of words
my_list = [current_word.lower() for current_word in words]

Utah Python July 2013 Meeting: (http://basepi.net)
<pre>
# Iterative
squares = []
for x in range(10):
squares.append(x**2)
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

## Functional
squares = map(lambda x: x**2, range(10))

## List comprehension
squares = [x**2 for x in range(10)]
even_squares = [x**2 for x in range(10) if x%2 == 0]

# get hash of strings in list:
import sha
strings = ['Python is cool!', 'I love python']
hashes = [sha.sha(x).digest() for x in strings]

# flatten matrix:
matrix = [[1, 2, 3], [4, 5, 6]]
[x for row in matrix for x in row]
# [1, 2, 3, 4, 5, 6]

# build matrix:
[(x,y) for x in range(10) for y in range(10)]
# matrix of not both odd and not both even
[(x,y) for x in range(10) for y in range(10) if x%2 != y%2]

# suck lines of files into list, ignoring commented lines
[line.strip().split("\t") for line in open("my_file.tab")
if not line.startswith('#')]
</pre>

References:
* 5.1.4. List Comprehensions - Data Structures — Python v2.7.2 documentation - http://docs.python.org/tutorial/datastructures.html#list-comprehensions
* Utah Python July 2013 Meeting - YouTube - https://www.youtube.com/watch?v=uh9uhzitdvw [http://basepi.net]

=== [http://docs.python.org/reference/expressions#calls Mass Parameter Passing] ===

Using: *args and **kw
* *args a tuple of remaining unnamed parameters
* **kw a dictionary of remaining named parameters

def param_func(a, b='b', *args, **kw):
print( [a, b, args, kw] )

WARNING: If not careful, this can result in this error: "TypeError: param_func() got multiple values for keyword argument 'myvar'". Might be better to just use:

def param_func(*args, **kw):
print( [args, kw] )

param_func(1, 2, 3, 4, 5) # [1, 2, (3, 4, 5), {}]
param_func(1, 2, 3, 4, 5, joe='test') # [1, 2, (3, 4, 5), {'joe': 'test'}]
mylist = ("a", "c", "g", "h", 5)
param_func(mylist) # [('a', 'c', 'g', 'h', 5), 'b', (), {}]
param_func(*mylist) # ['a', 'c', ('g', 'h', 5), {}] *tricky*
Note: *mydict will strip out just keys and treat like list
Note: **mydict will treat like named parameters

def mywrapper(*args, **kw):
# do something before
result = function_name(*args, **kw)

Passing arbitrary parameters:
<pre>
def fun2(**kw):
print kw # {'a': '1', 'b': '2', 'c': '3'}

def fun1(c='3', **kw):
print kw # {'a': '1', 'b': '2'}
fun2(c='3', **kw)

fun1(a='1', b='2')
</pre>

=== [http://www.python.org/dev/peps/pep-0234/ Iteration Protocol] ===

[http://www.python.org/dev/peps/pep-0234/ PEP 234 -- Iterators]

Making a class iterable:
<pre>
class mycount(object):
def __init__(self, max):
self.x = 0
self.max = max
def __iter__(self): # return object that implements __next__() or next()
return self
def next(self):
if self.x < self.max:
self.x += 1
return self.x
else:
self.x = 0
raise StopIteration()
</pre>

for i in mycount(10): print i # 1 ... 10
print [x for x in mycount(10) ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10)
for i in c: print i # 1 ... 10
[ x for x in c ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

=== Generators ===

Generators are functions that return to their previous state (using 'yield') when iterated over, which can be used in for loops.

==== Yield ====

Improve on Iterators:
* Infinite data (infinite iterators)
* Memory-intensive iteration
* Cooperative Multi-threading

Generators are functions that return to their previous state when iterated over, which can be used in for loops.

Note: Generators throw StopIteration when completed

Note: the 'yield' keyword will stash away the state of the function and return the value. (Continuations)

<pre>
def gen_range(end):
cur = 0
while cur < end:
yield cur
# return here next time
cur += 1

# for loop iteration
for i in gen_range(10): print i

# manual iteration
my_gen = gen_range(10)
try:
while True:
print my_gen.next()
except StopIteration:
pass

# list comprehension
numbers = [x for x in gen_range(10)]
</pre>

Note: one can place an empty 'return' in a generator if needed. A 'return' with argument is invalid.

Making a class generate:
<pre>
class mycount(object):
def __init__(self, max):
self.max = max
def __iter__(self):
return self.next() # change here
def next(self): # change here
x = 0
while x < self.max:
x += 1
yield x

for i in mycount(10): print i # 1 ... 10
print [x for x in mycount(10) ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10)
for i in c: print i # 1 ... 10
[ x for x in c ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

itr = mycount(10).next()
itr.next() # 1
itr.next() # 2 - eventually throws StopIteration exception
</pre>

Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
<pre>
* Iterators are used to traverse a list
* Generators are more powerful Iterators

# simple 'n' number in functional form
def firstn(n):
num = 0
nums = []
while num < n:
nums.append(num)
num += 1
return nums

# simple 'n' number in generator form
def firstn(n):
num = 0
while num < n:
yield num
num += 1

-

# Infinite list generator
def counter():
num = 0
while True:
yield num
num += 1

# Fibbonacci generator
def fib():
num1 = 0
num2 = 1
yield num1
while True:
num1, num2 = num2, num1 + num2
yield num1

# Fibbonacci generator 2
def fib(n):
num1 = 0
num2 = 1
count = 0
yield num1
while count < n:
count += 1
num1, num2 = num2, num1 + num2
yield num1
</pre>

Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
<pre>
## Cooperative Multi-Threading:
# Cooperative multithreading relies on the threads themselves to relinquish control once they are at a stopping point.

def mythread(name):
print('Starting Thread: {name}...'.format(name=name))
counter = 10
while counter > 0:
print('Thread: {name} doing some work!'.format(name=name))
counter -= 1
yield

thread1 = mythread('Alice')
thread2 = mythread('Bob')
thread3 = mythread('Trudy')

try:
while True:
thread1.next()
thread2.next()
thread3.next()
except StopIteration:
pass
</pre>

References:
* Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
** http://files.basepi.net/talks/pyarmory.zip

=== dunder methods ===

Some dunder (double under) methods determine what will happen when + (__add__) or other similar numerical symbols used.

<pre>
class str_add(object):
def __init__(self, name):
self.name = name
def __add__(self, other):
self.name += other.name
return self
# note: only need to return self if you want to go beyond 'a + b'

a = str_add('a_')
b = str_add('b_')
c = str_add('c_')
d = a + b + c
print(a.name) # a_b_c_
print(b.name) # b_
print(c.name) # c_
print(d.name) # a_b_c_
</pre>

Other common methods to override:
* __add__ for +
* __sub__ for -
* __mul__ for *
* __div__ for /
* __mod__ for %

References:
* Emulating numeric types - http://docs.python.org/2/reference/datamodel.html#emulating-numeric-types

=== [http://docs.python.org/library/doctest.html doctest] ===

test.py:
<pre>
#!/usr/bin/env python
"""
This is the "example" module.

>>> double(5)
10
"""

def double(n):
"""
This is double function

>>> double(3)
6
"""
return n * 2

if __name__ == "__main__":
import doctest
doctest.testmod()
</pre>

=== unittest ===

unittest.TestCase

Here is a short script to test three functions from the random module:
<pre>
import random
import unittest

class TestSequenceFunctions(unittest.TestCase):

def setUp(self):
self.seq = range(10)

def test_shuffle(self):
# make sure the shuffled sequence does not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, range(10))

# should raise an exception for an immutable sequence
self.assertRaises(TypeError, random.shuffle, (1,2,3))

def test_choice(self):
element = random.choice(self.seq)
self.assertTrue(element in self.seq)

def test_sample(self):
with self.assertRaises(ValueError):
random.sample(self.seq, 20)
for element in random.sample(self.seq, 5):
self.assertTrue(element in self.seq)

if __name__ == '__main__':
unittest.main()
</pre>

The unittest module can be used from the command line to run tests from modules, classes or even individual test methods:
python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method
python -m unittest -v test_module # verbose
python -m unittest -h # help

References:
* 25.3. unittest — Unit testing framework — Python v2.7.4 documentation - http://docs.python.org/2/library/unittest.html
* Python Debugging Tips - Google Drive - https://docs.google.com/presentation/d/130Wl--IzbDh9ymF6KHMu0WrsIwz6k32pav1gFbsw7NI/edit?usp=sharing

=== Internet Access ===

<pre>
# Python 3
from urllib.request import urlopen
for line in urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'):
line = line.decode('utf-8')
if 'EST' in line or 'EDT' in line: # look for eastern time
print(line)
</pre>

=== Simple SMTP ===

Requires local STMP server.

<pre>
import smtplib
server = smtplib.SMTP("localhost")
server.sendmail('mrken@demo.oeey.com', 'kenneth@demo.oeey.com',
"""To: kenneth@demo.oeey.com
From: mrken@demo.oeey.com
Subject: This is fun

Beware of the dog.
""")
server.quit()
</pre>

<pre>
s = smtplib.SMTP(smtp_server)
</pre>

with TLS:
<pre>
s = smtplib.SMTP("smtp.gmail.com",587)
s.ehlo()
s.starttls()
s.ehlo()
</pre>

SSL:
<pre>
s = smtplib.SMTP_SSL('smtp.mail.yahoo.com', 465)
</pre>

=== Data Compress ===

zlib, gzip, bz2, zipfile, tarfile

<pre>
import zlib
s = b'hello world' # only in python >3
len(s)
t = zlib.compress(s)
zlib.decompress(t)
zlib.crc32(s)
</pre>

=== gzip ===

<pre>
import gzip
content = "Lots of content here"
f = gzip.open('file.txt.gz', 'wb')
f.write(content)
f.close()
</pre>

<pre>
import gzip
f_in = open('file.txt', 'rb')
f_out = gzip.open('file.txt.gz', 'wb')
f_out.writelines(f_in)
f_out.close()
f_in.close()
</pre>

=== Performance Timer ===

Time a task repeated 10 times:
<pre>
from timeit import Timer
Timer('t=a; a=b; b=t', 'a=1; b=2').timeit()
Timer('print("hi")').timeit(10) # repeat 10 times and time
</pre>

Simple method:
<pre>
start_time = int(time.time())
... task ...
compute_time = int(time.time()) - start_time
</pre>

=== Python Version Check ===

Solution:

__init__.py:
<pre>
# Check Python Version:
import sys
assert sys.version_info >= (2, 6, 0), 'requires Python 2.6 or higher'
del sys
</pre>

resulting error if not > 2.6:
<pre>
>>> import nucleon
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "nucleon/__init__.py", line 9, in ?
assert sys.version_info >= (2, 6, 0), 'nucleon requires Python 2.6 or higher'
AssertionError: nucleon requires Python 2.6 or higher
</pre>

Other solutions:

Note: this will throw syntax error on 2.4 due to ternary
<pre>
import sys
if sys.version_info < (2, 4):
raise "must use python 2.5 or greater"
else:
# syntax error in 2.4, ok in 2.5
x = 1 if True else 2
print x
</pre>

maybe solution:
<pre>
try:
eval("1 if True else 2")
except SyntaxError:
# doesn't have ternary
</pre>

<pre>
>>> try:
... from my_module import twoPointSixCode
... except Exception:
... print "can't import, probably because your python is too old!"
>>>
</pre>

<pre>
#!/usr/bin/env/python
import sys
if sys.version_info<(2,6,0):
sys.stderr.write("You need python 2.6 or later to run this script\n")
exit(1)
</pre>

<pre>
>>> import sys
>>> sys.hexversion # hex printed as integer
33817584
>>> '%x' % sys.hexversion # output as hex, not zero fill 2.4.3
'20403f0'
>>> sys.hexversion < 0x02060000 # test if version less than 2.6
True
>>> print "0x%08x" % sys.hexversion # print as hex and zero fill and prepend '0x'
0x020403f0
</pre>

<pre>
import sys
if sys.hexversion < 0x02060500:
print >> sys.stderr, "This script requires python 2.6.5 or higher."
exit(1)
</pre>

<pre>
>>> import sys
>>> sys.version_info
(2, 6, 4, 'final', 0)
>>> if not sys.version_info[:2] == (2, 6):
... print "Error, I need python 2.6"
... else:
... from my_module import twoPointSixCode
>>>
</pre>

References:
* http://stackoverflow.com/questions/446052/python-best-way-to-check-for-python-version-in-program-that-uses-new-language-fe
* http://stackoverflow.com/questions/1093322/how-do-i-check-what-version-of-python-is-running-my-script

=== [http://docs.python.org/library/struct.html Struct - Binary Data Record Layout] ===

Format Characters:
<pre>
Format C Type Python type Standard size
x pad byte no value
c char str of len 1 1
b signed char integer 1
B unsigned char integer 1
? _Bool bool 1
h short integer 2
H unsigned short integer 2
i int integer 4
I unsigned int integer 4
l long integer 4
L unsigned long integer 4
q long long integer 8
Q unsigned long long integer 8
f float float 4
d double float 8
s char[] string
p char[] string
P void * integer
</pre>

H # two byte unassigned number
I # four byte unassigned number
< # standard size and in little-endian byte order

import struct
data = struct.pack('<IIIHH', 1, 2, 3, 4, 5)
# data = '\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x05\x00'

fields = struct.unpack('<IIIHH', data)
# fields = (1L, 2L, 3L, 4, 5)
# using splice: fields = struct.unpack('<IIIHH', data[start:start+16])

f1, f2, f3, f4, f5 = fields

hex(f1) # hex a field:
# '0x1L'

>>> struct.pack("<5sh", "helloo", 11)
'hello\x0b\x00'
>>> struct.unpack("<5sh", 'hello\x0b\x00')
('hello', 11)

=== pdb interactive debugger ===

# uncomment the following to use the interactive debugger:
# import pdb; pdb.set_trace()

=== [http://docs.python.org/library/telnetlib.html Telnet] ===

create:
class telnetlib.Telnet([host[, port[, timeout]]])

Note: The instance is initially not connected by default; the open() method must be used to establish a connection. Alternatively, the host name and optional port number can be passed to the constructor, to, in which case the connection to the server will be established before the constructor returns.

open if not initially done so:
Telnet.open(host[, port[, timeout]])

Functions:
Telnet.read_until(expected[, timeout]) # Read until a given string, expected, is encountered or until timeout seconds have passed.
# on timeout return whatever is in queue, possibly empty string
Telnet.close() # close connection
Telnet.read_all() # Read all data until EOF; block until connection closed.
Telnet.write(buffer) # Write a string to the socket, doubling any IAC characters. This can block if the connection is blocked.
Telnet.expect(list[, timeout]) # Read until one from a list of a regular expressions matches.
# Return a tuple of three items: the index in the list of matches; the match object; and the text read to match.

Read until end of line:
line = tn.read_until("\n")

<pre>
import getpass
import sys
import telnetlib

HOST = "localhost"
user = raw_input("Enter your remote account: ")
password = getpass.getpass()

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ")
tn.write(user + "\n")
if password:
tn.read_until("Password: ")
tn.write(password + "\n")

tn.write("ls\n")
tn.write("exit\n")

print tn.read_all()
</pre>

<pre>
#!/usr/bin/env python

import sys
import telnetlib

HOST = "216.119.193.252"
USER = "admin"
PASS = "@lt12345"

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ")
tn.write(USER + "\n")
tn.read_until("Password: ")
tn.write(PASS + "\n")
tn.read_until("# ")

tn.write("terminal length 0\n")
tn.read_until("# ")

tn.write("show running-config\n")
configlines = tn.read_until("# ")

tn.write("exit\n")
tn.read_all()

print(configlines)
</pre>

=== SSH ===

Cheap method:
os.system('ssh user@server command')

SecureShell - PythonInfo Wiki - http://wiki.python.org/moin/SecureShell
: There are several ways to use Secure Shell (SSH) in Python
* pyssh - http://pyssh.sourceforge.net/
* paramiko - Paramiko is a native Python implementation of SSH. - http://www.lag.net/paramiko/
* conch - conch is another native implementation of SSH and part of the Twisted Matrix project - http://twistedmatrix.com/projects/conch
* SSH wrapper - Some tools just wrap around existing ssh/sftp implementations
** keyphrene - http://www.keyphrene.com/products/org.keyphrene/
** keyphrene SSH wrapper - http://membres.lycos.fr/fredp/python/pyscp.html
* Fabric - http://docs.fabfile.org/en/1.4.0/index.html
* pexpect - Noah.org - http://www.noah.org/wiki/Pexpect

SFTP:
* SFTP in Python: Really Simple SSH | The best of Zeth - http://zeth.net/post/332/

pexpect:
wget http://pexpect.sourceforge.net/pexpect-2.3.tar.gz
tar xzf pexpect-2.3.tar.gz
cd pexpect-2.3
sudo python ./setup.py install

=== SFTP ===

SFTP in Python? (platform independent) - Stack Overflow - http://stackoverflow.com/questions/432385/sftp-in-python-platform-independent

paramiko: ssh2 protocol for python - http://www.lag.net/paramiko/

pip install paramiko

mkdir -p ~/.src ; cd ~/.src
wget http://www.lag.net/paramiko/download/paramiko-1.7.7.1.zip
unzip paramiko-1.7.7.1.zip
cd paramiko-1.7.7.1
python setup.py install

SFTP Methods:
* chdir
* chmod
* chown
* close
* file
* from_transport
* get
* get_channel
* getcwd
* listdir
* listdir_attr
* logger
* lstat
* mkdir
* normalize
* open
* put
* readlink
* remove
* rename
* request_number
* rmdir
* sock
* stat
* symlink
* truncate
* ultra_debug
* unlink
* utime

<pre>
import paramiko

host = "THEHOST.com" #hard-coded
port = 22
transport = paramiko.Transport((host, port))

password = "THEPASSWORD" #hard-coded
username = "THEUSERNAME" #hard-coded
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

import sys
path = './THETARGETDIRECTORY/' + sys.argv[1] #hard-coded
localpath = sys.argv[1]
sftp.put(localpath, path)

sftp.close()
transport.close()
print 'Upload done.'
</pre>

VMware Example:
<pre>
#!/usr/bin/env python

import paramiko
import sys

host = "sftp2.engx.vmware.com"
port = 443
transport = paramiko.Transport((host, port))

username = "xxxx"
password = "xxx"
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

print dir(sftp)

print sftp.chdir("build-uploader")
print sftp.listdir()

sftp.close()
transport.close()
</pre>

=== paramiko ===

See [[Python/paramiko]]

=== [http://docs.python.org/library/getpass.html Get Password] ===

Prompt for password:
getpass.getpass(prompt='Password: ')

import getpass
user = raw_input("Enter your remote account: ")
password = getpass.getpass()

=== [http://docs.python.org/library/configparser.html ini files] ===

<pre>
ini_filename = 'my.ini'

import ConfigParser
parser = ConfigParser.SafeConfigParser()
parser.read(ini_filename)

sections = parser.sections() # get sections
for section in sections:
print "[" + section + "]"
for name, value in parser.items(section):
print name + " = " + value
</pre>

<pre>
if parser.has_option(section, option):
print( parser.get(section, option) )
</pre>

To have the option names not convert to lower case:
parser.optionxform = str

Ignore section: [http://stackoverflow.com/questions/2885190/using-pythons-configparser-to-read-a-file-without-section-name]
ini_str = '[root]\n' + open(ini_path, 'r').read()
ini_fp = StringIO.StringIO(ini_str)
config = ConfigParser.RawConfigParser()
config.readfp(ini_fp)

Get option:
config.has_option('root', 'some_key')
config.get('root', 'some_key')

config.items('root') # list of Key Value tuples

=== optparse - Option Parser ===

[http://docs.python.org/library/optparse.html - Option Parser

Note: Deprecated since version 2.7: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

<pre>
import optparse
parser = optparse.OptionParser()

parser.add_option('--name', default=None)

parser.add_option(
'-l', '--loglevel',
help='Log Level (default: info)',
default="info", action='store', type='string')

(options, args) = parser.parse_args()

# if no parameters passed show help
if len(sys.argv) == 1:
parser.print_help()
sys.exit(1)

# if a stand alone parameter passed, override option
if args:
options.target = args[0]

# show option:
print(options.loglevel)
</pre>

Note: Order is important when listing add_options!

Pretty Description:
parser = optparse.OptionParser(description="Purpose: to do something")
# or
parser.description = "Purpose: to do something"

Parse arguments:
(options, args) = parser.parse_args()
Note: If you like, you can pass a custom argument list to parse_args(), but that’s rarely necessary: by default it uses sys.argv[1:].

Change Usage Message:
<pre>
usage = "%prog [options]" # default message
# Usage: <prog> [options]" # where <prog> is replaced with the script name

parser = OptionParser(usage=usage)
# or
parser.usage = usage # default message
# or
parser.set_usage(usage)
</pre>

Remove option: (does not remove default value from 'options' variable!!! - stupid!)
parser.remove_option('--loglevel')

Modify an option: (changes are saved to parser automatically, unless in an option group)
option = parser.get_option('--loglevel')
option.help = 'something else'
option.default = 'debug'

# or
parser.remove_option("--loglevel")
parser.add_option("--loglevel", help="something else", default="info")

Modify an option in an option group:
opt_group = parser.get_option_group("--loglevel")
opt_group.remove_option("--loglevel")
opt_group.add_option("--loglevel", help="something else", default="info")

Types: "string", "int", "long", "choice", "float" and "complex" [http://docs.python.org/library/optparse.html#optparse-standard-option-types]
Defaults:
* Default actions is 'store'.
* Default type is 'string'.
* Default dest is the long name of the parameter (or short name if no long name provided)
* Default value is None

<pre>
from optparse import OptionParser
[...]
parser = OptionParser()

# string
parser.add_option("-f", "--file", dest="filename",
help="write report to FILE", metavar="FILE")

# boolean - action is what to do when flag found
parser.add_option("-q", "--quiet",
action="store_true", dest="verbose", default=False,
help="don't print status messages to stdout")

# integer
parser.add_option(
"-m", "--minutes",
help="Minutes (default: 60)",
default=60, type='int')

(options, args) = parser.parse_args()

print "Filename: " + options.filename
</pre>

fake handling of required option:
<pre>
if not option.filename:
parser.print_help()
parser.error("filename is required")
</pre>

change default usage:
parser = OptionParser(usage="Usage: %prog [args] arg1 arg2")

Really ugly way to print help usage :-(
<pre>
def Usage():
pname = sys.argv[0]
cmd='%s -h' % (pname)
os.system(cmd)

if (len(sys.argv) < 2):
Usage()
sys.exit(1)
</pre>

Groups:
<pre>
# OptionGroup(parser, title, description=None)
group = OptionGroup(parser, "Dangerous Options",
"Caution: use these options at your own risk. "
"It is believed that some of them bite.")
group.add_option("-g", action="store_true", help="Group option.")
parser.add_option_group(group)
</pre>

Hide an option:
* Option.help
: "Help text to print for this option when listing all available options after the user supplies a help option (such as --help). If no help text is supplied, the option will be listed without help text. To hide this option, use the special value optparse.SUPPRESS_HELP."
<pre>
parser.add_option("--secret", help=SUPPRESS_HELP)
</pre>

Issues with Option Parser:
* No handling for positional arguments
* No adding help for positional arguments
* No handling for required field

=== argparse ===

argparse - http://docs.python.org/library/argparse.html

argparse — Parser for command-line options, arguments and sub-commands

NOTE: MUCH MORE POWERFUL THAN OPTION PARSER!

Note: New in version 2.7.

Installation on <= 2.6
<pre>
# Install argparse: - http://code.google.com/p/argparse/
# REQUIRES SETUPTOOLS
mkdir -p ~/.src ; cd ~/.src
wget http://argparse.googlecode.com/files/argparse-1.2.1.tar.gz
tar -zvxf argparse-1.2.1.tar.gz
cd argparse-1.2.1
sudo python setup.py install
</pre>

The argparse module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse will figure out how to parse those out of sys.argv. The argparse module also automatically generates help and usage messages and issues errors when users give the program invalid arguments.

<pre>
import argparse

parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
const=sum, default=max,
help='sum the integers (default: find the max)')

args = parser.parse_args()
print args.accumulate(args.integers)
</pre>

<pre>
import argparse

argparser = demoproject.util.argparse.ArgumentParser(
usage='Kick off demoproject benchmarks.')
argparser.add_argument(
'--card',
help='What cards to run on. Depends on Jenkins NODE_LABEL environment'
'variable for correct mappings.')
argparser.add_argument(
'--suite', metavar='SUITE',
help='Specify the appropriate benchmark suite as defined in '
'the benchmark ini file. Defaults to standard_perf',
default='standard_perf')
args = argparser.parse_args()

print args.card
</pre>

Remove an argument: [http://stackoverflow.com/questions/32807319/disable-remove-argument-in-argparse]
<pre>
def remove_options(parser, options):
""" remove_options(parser, ['--some-option', '--some-option2']) """
for option in options:
for action in parser._actions:
if vars(action)['option_strings'][0] == option:
parser._handle_conflict_resolve(None,[(option,action)])
break
</pre>

=== [http://docs.python.org/library/commands.html commands] ===

35.16. commands — Utilities for running commands — Python v2.7.1 documentation - http://docs.python.org/library/commands.html

Note: Deprecated since version 2.6: The commands module has been removed in Python 3.0. Use the subprocess module instead.

The commands module contains wrapper functions for os.popen() which take a system command as a string and return any output generated by the command and, optionally, the exit status.

Get status and output:
status, output = commands.getstatusoutput(cmd)

Get output ignoring status:
output = commands.getoutput(cmd)

NOTE: DO NOT USE getstatus() AS IT DOES NOT DO WHAT YOU THINK IT DOES!
status = commands.getstatus(file)

<pre>
>>> import commands

>>> commands.getstatusoutput('ls /bin/ls')
(0, '/bin/ls')

>>> commands.getstatusoutput('cat /bin/junk')
(256, 'cat: /bin/junk: No such file or directory')

>>> commands.getstatusoutput('/bin/junk')
(256, 'sh: /bin/junk: not found')

>>> commands.getoutput('ls /bin/ls')
'/bin/ls'

>>> commands.getstatus('/bin/ls')
'-rwxr-xr-x 1 root 13352 Oct 14 1994 /bin/ls'
</pre>

=== [http://docs.python.org/library/_winreg.html Windows Registry] ===

Note: The _winreg module has been renamed to winreg in Python 3.0.

* 32.3. winreg – Windows registry access — Python v3.2 documentation - http://docs.python.org/py3k/library/winreg.html
* 34.3. _winreg – Windows registry access — Python v2.7.1 documentation - http://docs.python.org/library/_winreg.html

Read from registry:
<pre>
from _winreg import *
key = OpenKey(HKEY_LOCAL_MACHINE, r'Software\Microsoft\Outlook Express', 0, KEY_ALL_ACCESS)
QueryValueEx(key, "InstallRoot")
</pre>

Write to registry:
<pre>
keyVal = r'Software\Microsoft\Internet Explorer\Main'
try:
key = OpenKey(HKEY_CURRENT_USER, keyVal, 0, KEY_ALL_ACCESS)
except:
key = CreateKey(HKEY_CURRENT_USER, keyVal)
SetValueEx(key, "Start Page", 0, REG_SZ, "http://www.blog.pythonlibrary.org/")
CloseKey(key)
</pre>

Enumerate registry keys:
<pre>
from _winreg import EnumKey, HKEY_USERS

try:
i = 0
while True:
subkey = EnumKey(HKEY_USERS, i)
print subkey
i += 1
except WindowsError:
# WindowsError: [Errno 259] No more data is available
pass
</pre>

Another enumeration example:
<pre>
# File: winreg-example-1.py

import _winreg

explorer = _winreg.OpenKey(
_winreg.HKEY_CURRENT_USER,
"Software\\Microsoft\\Windows\\CurrentVersion\\Explorer"
)

# list values owned by this registry key
try:
i = 0
while 1:
name, value, type = _winreg.EnumValue(explorer, i)
print repr(name),
i += 1
except WindowsError:
print

value, type = _winreg.QueryValueEx(explorer, "Logon User Name")

print
print "user is", repr(value)
</pre>

Enable Remote Desktop Example: (by Kenneth)
<pre>
from _winreg import *
key = OpenKey(HKEY_LOCAL_MACHINE, r'SYSTEM\CurrentControlSet\Control\Terminal Server', 0, KEY_ALL_ACCESS)
SetValueEx(key, "fDenyTSConnections", 0, REG_DWORD, 0)
ValueTypeTuple = QueryValueEx(key, "fDenyTSConnections") # (0, 4)
print("Please reboot the system to complete the changes")
</pre>

References:
* Python’s _winreg: Editing the Windows Registry « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/03/20/pythons-_winreg-editing-the-windows-registry/

=== JSON ===

JSON - http://docs.python.org/library/json.html

ENCODING TO JSON STRING (type str)

import json
json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
# '["foo", {"bar": ["baz", null, 1.0, 2]}]'

compact encoding:
import json
json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',',':'))
# '[1,2,3,{"4":5,"6":7}]'

Pretty printing:
import json
print json.dumps({'4': 5, '6': 7}, sort_keys=True, indent=4)
# {
# "4": 5,
# "6": 7
# }

DECODING JSON STRING TO PYTHON LIST OR DICTIONARY OBJECT

import json
json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]')
# [u'foo', {u'bar': [u'baz', None, 1.0, 2]}]

Load from file:
f = open(filename)
j = json.load(f)
print j["kind"] # "dual-controller-adapter"
#{
# "kind": "dual-controller-adapter",
#}

From Command Line:
echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool

# get value
echo '{"test":1,"test2":2}' | python -c 'import sys,json;data=json.loads(sys.stdin.read()); print data["test"]'

# read large file
python -c "import json;f=open('file.json');data=json.load(f);print json.dumps(data,indent=2)"
python -c "import json;print json.dumps(json.load(open('file.json')),indent=2)"

Load from web:
<pre>
import httplib
import json

def get_http_json(server, path):
""" This performs an HTTP GET against a server and path and returns the
resulting page data.

server The server we're hitting, e.g. demo.com
path The path we're hitting, e.g. api/part/MA12345/
"""
connection = httplib.HTTPConnection(server)
try:
url = 'http://%s/%s' % (server, path)
print 'Querying url %s' % url
connection.request('GET', url)
response = connection.getresponse()
if response.status == httplib.OK:
return json.loads(response.read())
else:
print 'Unable to get %s. %s %s' % (url, response.reason,
response.status)
return None
finally:
connection.close()

j = get_http_json(server, path)
</pre>

Pretty printing from the command line:
echo '{"json":"obj"}' | python -mjson.tool
#{
# "json": "obj"
#}

References:
* 18.2. json — JSON encoder and decoder — Python v2.7.1 documentation - http://docs.python.org/library/json.html

=== Sockets ===

See [[Python/Sockets]]

=== [http://docs.python.org/library/base64.html base64] ===

import base64

Encode:
encoded = base64.b64encode('data to be encoded')

Decode:
base64.b64decode(encoded)

=== daemonize ===

Creating a daemon the Python way « Python recipes « ActiveState Code - http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/

<pre>
import os
import sys

try:
pid = os.fork()
except OSError, e:
raise Exception, "%s [%d]" % (e.strerror, e.errno)

if (pid != 0): # parent
os._exit(0)

# create new process session
os.setsid()

# redirect stdin, stdout, stderr
os.open('/dev/null', os.O_RDWR)
os.dup2(0, 1)
os.dup2(0, 2)

#... tasks ...
</pre>

My Solution:
<pre>
import os
import sys
import signal
import syslog

HUP = False

def log(msg):
syslog.syslog(str(msg))
syslog.openlog('[WATER]')

def kill_signal_handler(signal, frame):
log("Being killed off: {0} {1}".format(signal, frame))
log("Shutting Down!")
sys.exit(0)
signal.signal(signal.SIGTERM, kill_signal_handler)
signal.signal(signal.SIGALRM, kill_signal_handler)

def hup_signal_handler(signal, frame):
global HUP
log("Signal reload received: {0} {1}".format(signal, frame))
HUP=True
signal.signal(signal.SIGHUP, hup_signal_handler)

def main_loop():
global HUP

while True:
# ... handle HUP=True ...

# ... main code ...

# wait 60 seconds before we start the next loop
time.sleep(60) # signals will interrupt this

if __name__ == "__main__":
log("Starting Daemon...")
child_id = os.fork()
if child_id:
log("Daemonized!")
sys.exit(0)
os.setsid()
# redirect stdin, stdout, stderr
os.open('/dev/null', os.O_RDWR)
os.dup2(0, 1)
os.dup2(0, 2)

# write pid
with open("water.pid", "w") as f:
f.write(str(os.getpid()))

while True:
try:
main_loop()
except Exception as e:
log("Critical Failure Caught: %s" % str(e))
log("Critical Failure - Restarting!")
time.sleep(30)
#raise e
</pre>

ln -s /opt/pyton27/bin/python pyton-water
./python-water water.py
killall python-water

==== fork ====

<pre>
import os
import sys
import time

child_id = os.fork()
if child_id:
print "parent exiting"
sys.exit(0)

while True:
time.sleep(3)
print "child"
</pre>

For windows, to hide the console, see [[#Hide Console]]

=== csv ===

13.1. csv — CSV File Reading and Writing — Python v2.7.2 documentation - http://docs.python.org/library/csv.html

Read:
<pre>
import csv
spamReader = csv.reader(open('eggs.csv', 'rb'), delimiter=' ', quotechar='|')
for row in spamReader:
print ', '.join(row)
</pre>

Write:
<pre>
import csv
spamWriter = csv.writer(open('eggs.csv', 'wb'), delimiter=' ',
quotechar='|', quoting=csv.QUOTE_MINIMAL)
spamWriter.writerow(['Spam'] * 5 + ['Baked Beans'])
spamWriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam'])
</pre>

=== SQLite ===

11.13. sqlite3 — DB-API 2.0 interface for SQLite databases — Python v2.7.2 documentation - http://docs.python.org/library/sqlite3.html

sqlite3 - New in version 2.5.

<pre>
import sqlite3

conn = sqlite3.connect('/tmp/example')

c = conn.cursor()

# Create table
c.execute('''create table stocks
(date text, trans text, symbol text,
qty real, price real)''')

# Insert a row of data
c.execute("""insert into stocks
values ('2006-01-05','BUY','RHAT',100,35.14)""")

# Show affected row count
print c.rowcount

# Save (commit) the changes
conn.commit()

# We can also close the cursor if we are done with it
c.close()
</pre>

parameter substitution:
<pre>
# Never do this -- insecure!
symbol = 'IBM'
c.execute("... where symbol = '%s'" % symbol)

# Do this instead
t = (symbol,)
c.execute('select * from stocks where symbol=?', t)
c.execute('select * from stocks where symbol=?', (symbol,))

# Larger example
for t in [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00),
('2006-04-06', 'SELL', 'IBM', 500, 53.00),
]:
c.execute('insert into stocks values (?,?,?,?,?)', t)
</pre>

select results:
<pre>
c = conn.cursor()
c.execute('select * from stocks order by price')

for row in c:
print row # row is a tuple

# OR (but not both)

rows = c.fetchall() # rows is list of tuples
print len(rows)
for row in rows:
print row
</pre>

Show which version of sqlite3: [http://zetcode.com/db/sqlitepythontutorial/]
<pre>
#!/usr/bin/python
# -*- coding: utf-8 -*-

import sqlite3 as lite
import sys

con = None

try:
con = lite.connect('test.db')

cur = con.cursor()
cur.execute('SELECT SQLITE_VERSION()')

data = cur.fetchone()

print "SQLite version: %s" % data

except lite.Error, e:

print "Error %s:" % e.args[0]
sys.exit(1)

finally:

if con:
con.close()
</pre>

Or simply: [http://stackoverflow.com/questions/1545479/force-python-to-forego-native-sqlite3-and-use-the-installed-latest-sqlite3-ver]
# echo -e "import sqlite3\nprint sqlite3.sqlite_version_info" | python
(3, 3, 6)

=== MySQL Database ===

Ubuntu packages:
* python-mysqldb - Python interface to MySQL
* python-mysql.connector - pure Python implementation of MySQL Client/Server protocol

apt-get install python-mysqldb
>>> import _mysql

apt-get install python-mysql.connector

PIP:
* https://pypi.python.org/pypi/MySQL-python
* https://pypi.python.org/pypi/mysqlclient

pip install MySQL-python

pip install mysqlclient

---

_mysql - http://mysql-python.sourceforge.net/MySQLdb.html

apt-get install python-mysqldb

<pre>
import _mysql

db=_mysql.connect(host="localhost",user="user",
passwd="pass",db="db")

db.query("""SELECT spam, eggs, sausage FROM breakfast
WHERE price < 5""")

r=db.store_result()
# ...or...
r=db.use_result()

row = r.fetch_row()
</pre>

=== couchdb ===

Getting started with couchdb-python — couchdb-python v0.8 documentation - http://packages.python.org/CouchDB/getting-started.html

<pre>
couch = couchdb.Server('http://127.0.0.1:5984/')

# database
db = couch.create('test') # create new
db = couch['mydb'] # or get existing

# print databases:
for db in couch:
print db

doc = {'foo': 'bar'}
doc_id, doc_rev = db.save(doc)
# (u'9278f7691c3421227a9f8436540008f9', u'1-4c6114c65e295552ab1019e2b046b10e')
# note: this also updates doc with _rev and _id
doc = db['9278f7691c3421227a9f8436540008f9']

# better to create a document using an id you want, instead of the randomly generated one
db[doc.id] = doc # this will write immediately to the database

# print all docs:
for id in db:
print id

# delete doc
db.delete(doc)

# delete database
couch.delete('test')
</pre>

Search database:
<pre>
# show all records
map_fun = '''function(doc) { emit(doc.name, null); }'''

# row = <Row id=u'12341234123412341234', key=None, value=None>
# emit(key, value) - returns the values for key and value

map_fun = '''
function(doc) {
if (doc.username == 'virt-01')
emit(doc.username, null);
}
'''

# row = <Row id=u'12341234123412341234', key=None, value=None>
for row in db.query(map_fun):
print row.id

for row in db.query(map_fun, descending=True):
print row.id
</pre>

couchdb-python - CouchDB Python Library - Google Project Hosting - http://code.google.com/p/couchdb-python/

Installation:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/C/CouchDB/CouchDB-0.8.tar.gz#md5=e32b39e459e1fa2586ea252712a11a59
tar -zvxf CouchDB-0.8.tar.gz
cd CouchDB-0.8
python setup.py install
</pre>

Test:
echo "import couchdb" | python

=== nose ===

nose: nose extends unittest to make testing easier

nose : Python Package Index - http://pypi.python.org/pypi/nose

Install:
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/n/nose/nose-1.1.2.tar.gz#md5=144f237b615e23f21f6a50b2183aa817
tar -zvxf nose-1.1.2.tar.gz
cd nose-1.1.2
sudo python setup.py install

skeleton:
mkdir skeleton
cd skeleton
mkdir bin NAME tests docs
touch NAME/__init__.py
touch tests/__init__.py
touch tests/NAME_tests.py

nose test skeleton:
<pre>
# tests/NAME_tests.py
from nose.tools import *
import klib

def setup():
print "SETUP!"

def teardown():
print "TEAR DOWN!"

def test_basic():
print "I RAN!"

# def test_something_else():
</pre>

Usage:
nosetests

See Also:
* [[#doctest]]
* [[#unittest]]

=== pydoc ===

25.1. pydoc — Documentation generator and online help system — Python v2.7.2 documentation - http://docs.python.org/library/pydoc.html

pydoc - The pydoc module automatically generates documentation from Python modules.

pydoc - same as 'import [library], help([library])'
$ pydoc [library]

Note: lists some special 'module metadata variables' separately (e.g. version, author).

special pydoc recognized metadata sections:
__version__
__author__
__credits__
__date__

==== pydoc function module ====

<pre>
Help on package klib:

NAME
klib - This is the name

FILE
/root/klib/klib/__init__.py

DESCRIPTION
These are module doc strings
that give helpful details

PACKAGE CONTENTS
markone

FUNCTIONS
func1()
This is a doc string
for func1

DATA
PI = 3.141
__all__ = ['func1', 'PI']
__author__ = 'Kenneth <kenneth@oeey.com>'
__copyright__ = 'Copyright 2007, The Project'
__credits__ = 'Credits go to me'
__date__ = '26 February 2001'
__email__ = 'ken@oeey.com'
__license__ = 'GPL'
__maintainer__ = 'Ken'
__status__ = 'Production'
__version__ = '$Revision: 83492 $'

VERSION
83492

DATE
26 February 2001

AUTHOR
Kenneth <kenneth@oeey.com>

CREDITS
Credits go to me
</pre>

Source:
<pre>
"""
This is the name

These are module doc strings
that give helpful details
"""

#__version__ = "1.0"
__version__ = "$Revision: 83492 $"
__author__ = "Kenneth <kenneth@oeey.com>"

__date__ = "26 February 2001"
__credits__ = "Credits go to me"

__all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license'
__license__ = "GPL"
__copyright__ = "Copyright 2007, The Project"
__maintainer__ = "Ken"
__email__ = "ken@oeey.com"
__status__ = "Production"

def func1():
"""
This is a doc string
for func1
"""
</pre>

==== pydoc class module ====

Sample:
<pre>
Help on module class_pydoc:

NAME
class_pydoc - This is the name

FILE
/root/class_pydoc.py

DESCRIPTION
These are module doc strings
that give helpful details

CLASSES
Toad

class Toad
| doc string for object
|
| Methods defined here:
|
| __init___(self, name)
| Constructor
|
| instance_function(self)
| Instance Function
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| class_function(cls) from __builtin__.classobj
| Class Function
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| my_class_attribute = 'sample class attribute'

DATA
PI = 3.141
__author__ = 'Kenneth <kenneth@oeey.com>'
__copyright__ = 'Copyright 2007, The Project'
__credits__ = 'Credits go to me'
__date__ = '26 February 2001'
__email__ = 'ken@oeey.com'
__license__ = 'GPL'
__maintainer__ = 'Ken'
__status__ = 'Production'
__version__ = '1.0'

VERSION
1.0

DATE
26 February 2001

AUTHOR
Kenneth <kenneth@oeey.com>

CREDITS
Credits go to me
</pre>

Code:
<pre>
"""
This is the name

These are module doc strings
that give helpful details
"""

__version__ = "1.0"
#__version__ = "$Revision: 83492 $"
__author__ = "Kenneth <kenneth@oeey.com>"

__date__ = "26 February 2001"
__credits__ = "Credits go to me"

#__all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license'
__license__ = "GPL"
__copyright__ = "Copyright 2007, The Project"
__maintainer__ = "Ken"
__email__ = "ken@oeey.com"
__status__ = "Production"

class Toad:
""" doc string for object """

my_class_attribute = 'sample class attribute'

def __init___(self, name):
""" Constructor """

def instance_function(self):
""" Instance Function """

@classmethod
def class_function(cls):
""" Class Function """
</pre>

=== username ===

User ID:
os.getuid()

User Name:
# fails it stdin piped to program
os.getlogin()

User Name:
import sys
import pwd
def get_username():
return pwd.getpwuid( os.getuid() )[ 0 ]

Environment:
os.environ['USER']

=== hostname ===

import socket
print(socket.gethostname())

os.uname()[1]

hostname = os.system('hostname')

=== Networking ===

==== Get Hostname ====

import socket
print(socket.gethostname())

==== Convert hostname to ip address ====

>>> socket.gethostbyaddr('demo')
('demo.oeey.com', [], ['10.10.10.10'])

/usr/local/bin/whatip: (or nametoip)
<pre>
#!/usr/bin/env python

#
# whatip - convert hostname to ip address
# author: Kenneth Burgener <kenneth@oeey.com> (FEB 2012)
#

import socket
import sys

if len(sys.argv) != 2:
print "Convert Hostname to IP Address"
print "Usage: %s HOSTNAME" % sys.argv[0].split('/')[-1]
sys.exit(1)

hostname = sys.argv[1]
try:
result = socket.gethostbyaddr(hostname)
print result[2][0]
except (socket.gaierror, socket.herror) as e:
print "Error: Unable to resolve '%s'" % hostname
sys.exit(1)
</pre>

==== Get Public IP Address ====

<pre>
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("gmail.com",80))
print(s.getsockname()[0])
s.close()
</pre>

networking - Finding local IP addresses using Python's stdlib - Stack Overflow - http://stackoverflow.com/questions/166506/finding-local-ip-addresses-using-pythons-stdlib

==== Python Network Server ====

<pre>
#!/usr/bin/python # This is server.py file

import socket # Import socket module

s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.
s.bind((host, port)) # Bind to the port

s.listen(5) # Now wait for client connection.
while True:
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
c.send('Thank you for connecting')
c.close() # Close the connection
</pre>

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
* 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

==== Python Network Client ====

<pre>
#!/usr/bin/python # This is client.py file

import socket # Import socket module

s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.

s.connect((host, port))
print s.recv(1024)
s.close # Close the socket when done
</pre>

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
* 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

=== Run Library Module as Script ===

Run python with -m option:
-m mod : run library module as a script (terminates option list)

Example:
cat [json_file] | python -m json.tool
# same as:
cat [json_file] | python /opt/python26/lib/python2.6/json/tool.py

Good for when library is buried deep in the library directories

=== Hex ===

<pre>
>>> hex(10) # print integer as hex string
'0xa'
>>> sys.hexversion # python version as integer
33817584
>>> hex(sys.hexversion) # python version as hex
'0x20403f0'
>>> print "0x%08x" % sys.hexversion # python version as hex
'0x20403f0'
>>> int('0xa', 16) # hex string to integer
10
>>> int('0a', 16) # hex string to integer
10
>>> int('0xa', 0) # hex string to integer auto detected du to '0x'
10
</pre>

=== Binary ===

Integer to Binary String with Negative Number Support: (2's complement)
<pre>
# source: http://www.gossamer-threads.com/lists/python/python/645216
def bin(x):
if x < -32768 or x > 65535:
raise Exception("Out of Range Error: %d" % x)
if x < 0:
x = 65535 + x + 1
bin_str = ''
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(15, 11, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(11, 7, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(7, 3, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(3, -1, -1))
return bin_str
</pre>

Integer to Binary String:
<pre>
# source: http://www.gossamer-threads.com/lists/python/python/645216
def bin(x):
if x < 0 or x > 65535:
raise Exception("Out of Range Error: %d" % x)
bin_str = ''
bin_str = ''.join(x & (1 << i) and '1' or '0' for i in range(15, 7, -1))
bin_str = bin_str + ' '
bin_str = bin_str + ''.join(x & (1 << i) and '1' or '0' for i in range(7, -1, -1))
return bin_str

print bin(0) # '00000000 00000000'
print bin(1) # '00000000 00000001'
print bin(255) # '00000000 11111111'
print bin(256) # '00000001 00000000'
print bin(43690) # '10101010 10101010'
print bin(65535) # '11111111 11111111'
print bin(65536) # '00000000 00000000' - rolled over to next set!
</pre>

Integer to Binary:
bin(170) # '0b10101010'

Binary to Integer:
int('10101010', 2) # 170

=== bitwise ===

See [[#Binary]]

BitwiseOperators - Python Wiki - http://wiki.python.org/moin/BitwiseOperators

<pre>
The Operators:

x << y
Returns x with the bits shifted to the left by y places (and new bits on the right-hand-side are zeros). This is the same as multiplying x by 2**y.
x >> y
Returns x with the bits shifted to the right by y places. This is the same as //'ing x by 2**y.
x & y
AND - Does a "bitwise and". Each bit of the output is 1 if the corresponding bit of x AND of y is 1, otherwise it's 0.
x | y
OR - Does a "bitwise or". Each bit of the output is 0 if the corresponding bit of x AND of y is 0, otherwise it's 1.
~ x
Returns the complement of x - the number you get by switching each 1 for a 0 and each 0 for a 1. This is the same as -x - 1.
x ^ y
XOR - Does a "bitwise exclusive or". Each bit of the output is the same as the corresponding bit in x if that bit in y is 0, and it's the complement of the bit in x if that bit in y is 1.
</pre>

=== Script Path ===

<pre>
if __name__ == "__main__":
# Change directory to where this script lives so we can find relative
# paths from here regardless of where the working directory was when we
# ran the script.
# APP_DIR = os.path.dirname(sys.argv[0])
APP_DIR = os.path.dirname(os.path.abspath(sys.argv[0))
if APP_DIR:
os.chdir(APP_DIR)
</pre>

normpath(path) # Normalize path, eliminating double slashes, etc.
abspath(path) # Return an absolute path

Path to a module: [http://stackoverflow.com/questions/247770/retrieving-python-module-path] (couldn't fine a_module)
import a_module
import os
path = os.path.dirname(a_module.__file__)

Path to module: [http://stackoverflow.com/questions/50499/in-python-how-do-i-get-the-path-and-name-of-the-file-that-is-currently-executin]
import inspect, os
print inspect.getfile(inspect.currentframe()) # script filename (usually with path)
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))) # script directory

Even better:
print os.path.realpath(__file__)
print os.path.dirname(__file__)
print os.path.basename(__file__)

references:
* http://stackoverflow.com/questions/595305/python-path-of-script

keywords: scriptpath script path program path pathname

=== Singleton ===

GLOBAL OBJECT!

None is a singleton (Python only has one copy of None in the interpreter).

Make your own singleton:

override the new method like this:
<pre>
class Singleton(object):
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Singleton, cls).__new__(
cls, *args, **kwargs)
return cls._instance

if __name__ == '__main__':
s1=Singleton()
s2=Singleton()
if(id(s1)==id(s2)):
print "Same"
else:
print "Different"
</pre>

Using decorator -implementation from PEP318? Implementing the singleton pattern with a decorator:
<pre>
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance

@singleton
class MyClass:
...
</pre>

* Python and the Singleton Pattern - Stack Overflow - http://stackoverflow.com/questions/42558/python-and-the-singleton-pattern

=== XML ===

<pre>
from xml.dom.minidom import parse, parseString

dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name

datasource = open('c:\\temp\\mydata.xml')
dom2 = parse(datasource) # parse an open file

dom3 = parseString('<myxml>Some data<empty/> some more data</myxml>')
</pre>

Pretty Print: [http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python]
<pre>
import xml.dom.minidom

xml = xml.dom.minidom.parse(xml_fname) # or xml.dom.minidom.parseString(xml_string)
pretty_xml_as_string = xml.toprettyxml()
</pre>

Pretty Print with etree [http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python]
<pre>
import lxml.etree as etree

x = etree.parse("filename")
print etree.tostring(x, pretty_print = True)
</pre>

References:
* 19.7. xml.dom.minidom — Lightweight DOM implementation — Python v2.7.3 documentation - http://docs.python.org/library/xml.dom.minidom.html
* Pretty printing XML in python - Stack Overflow - http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python

---

Simply don't include self.output.startElement and self.output.endElement for elements you don't want:

simple_filter_parser.py: (my version of [http://freegeographytools.com/2010/removin-time-data-out-of-a-gpx-or-kml-file-to-make-it-work-right-in-google-earth])
<pre>
# this parses out the time element of a gpx for Google Earth
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class my_filter_xml(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self.indent = 0
self.ignore = False
self.output.startDocument()

def startElement(self, name, attrs):
print " " * self.indent + "start: " + name
if attrs.keys():
keys = " " * (self.indent + 2) + "attrs:"
for key in attrs.keys():
keys = keys + " " + key + "=" + attrs[key]
print keys
self.indent = self.indent + 2
#self.output.startElement(name, {})
if name == "time":
self.ignore = True
#if not self.ignore:
self.output.startElement(name, attrs)

def characters(self, ch):
if ch.strip():
print " " * self.indent + "ch: " + ch.strip()
if not self.ignore:
self.output.characters(ch)
else:
if not self.ignore:
self.output.characters("\n")

def endElement(self, name):
self.indent = self.indent - 2
print " " * self.indent + "/end: " + name
#if not self.ignore:
self.output.endElement(name)
if name == "time":
self.ignore = False

if __name__ == '__main__':
source = open("tracks.gpx")
dest = open("tracks_out.gpx", 'w+')
xml.sax.parse(source, my_filter_xml(dest, True))
</pre>

---

fixdescriptor.py:
<pre>
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self._outfile = outfile
self._verbose = verbose
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self._filter = [ 'payload',
'payloads',
'file-list',
'file',
'checksum' ]
self._keep = [ 'vib',
'maintenance-mode' ]
self._inside = None

def startElement(self, name, attrs):
# Certain elements are kept unchanged
if name in self._keep:
if self._verbose:
print '%s element start found' % (name)
self.output.startElement(name, {})
if name == 'maintenance-mode':
if self._verbose:
print 'inside maintenance-mode'
self._inside = name
return
# Other elements are filtered (removed)
elif name in self._filter:
if self._verbose:
print '%s element start found' % (name)
self._inside = name
return
self.output.startElement(name, attrs)

def characters(self, ch):
if self._inside == 'maintenance-mode':
if self._verbose:
print 'new content for maintenance-mode'
self.output.characters('false')
elif self._inside not in self._filter:
self.output.characters(ch)

def endElement(self, name):
# If it's not a checksum element, only print it
if name in self._filter + self._keep:
self._inside = None
if self._verbose:
print '%s element end found' % (name)
if name in self._filter:
return
self.output.endElement(name)

if __name__ == '__main__':
#
parser = optparse.OptionParser(usage='%prog ...')
parser.add_option('-v', '--verbose',
action='store_true', default=False,
help='Print more information.')
parser.add_option('--input-xml',
type='str', default='descriptor.xml',
help='Name of input XML [default: %default]')
parser.add_option('--output-xml',
type='str', default='fixed_descriptor.xml',
help='Name of output XML [default: %default]')
opts, args = parser.parse_args(argv[1:])
source = open(opts.input_xml)
dest = open(opts.output_xml, 'w+')
xml.sax.parse(source, fix_descriptor(dest, opts.verbose))
</pre>

--

Simple read and output parsing:
<pre>
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self._outfile = outfile
self._verbose = verbose
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self._inside = None

def startElement(self, name, attrs):
# attrs.items() is a list of tuples (name, value)
print "start:", name, "/", attrs.items()
self.output.startElement(name, attrs)

def characters(self, ch):
print "char:", self._inside, "/", ch
self.output.characters(ch)

def endElement(self, name):
print "end:", name
self.output.endElement(name)

if __name__ == '__main__':
parser = optparse.OptionParser(usage='%prog ...')
parser.add_option('-v', '--verbose',
action='store_true', default=False,
help='Print more information.')
parser.add_option('--input-xml',
type='str', default='descriptor.xml',
help='Name of input XML [default: %default]')
parser.add_option('--output-xml',
type='str', default='fixed_descriptor.xml',
help='Name of output XML [default: %default]')
opts, args = parser.parse_args(argv[1:])
source = open(opts.input_xml)
dest = open(opts.output_xml, 'w+')
xml.sax.parse(source, fix_descriptor(dest, opts.verbose))
</pre>

=== Unicode ===

Unicode string:
u'Hello World !'

Raw Unicode string:
>>> ur'Hello\u0020World !'
u'Hello World !'

Convert:
u = u'hello\xe4'
str(u) # UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 5: ordinal not in range(128)
s = u.encode('utf8') # 's' is now type str

References:
* Introduction to Unicode Strings - http://docs.python.org/tutorial/introduction.html#unicode-strings

=== Zip ===

See [[Python/Zip]]

=== signals ===

signal.SIGTERM : kill [pid]
signal.SIGALRM : kill -ALRM [pid]
signal.SIGHUP : kill -HUP [pid]

<pre>
import signal, os

def handler(signum, frame):
print 'Signal handler called with signal', signum
raise IOError("Couldn't open device!")

# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)

# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)

signal.alarm(0) # Disable the alarm
</pre>

Note: use signal.SIGTERM to handle "kill" command

References:
* 17.4. signal — Set handlers for asynchronous events — Python v2.7.5 documentation - http://docs.python.org/2/library/signal.html

==== Ctrl+C ====

<pre>
#!/usr/bin/env python
import signal
import sys
def signal_handler(signal, frame):
print 'You pressed Ctrl+C!'
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
print 'Press Ctrl+C'

# linux only:
signal.pause()

# windows alternative:
#import time
#time.sleep(60)
</pre>

References:
* control - How do I capture SIGINT in Python? - Stack Overflow - http://stackoverflow.com/questions/1112343/how-do-i-capture-sigint-in-python

==== Non-blocking raw_input for Python ====

Non-blocking raw_input for Python:
<pre>
# source: http://www.garyrobinson.net/2009/10/non-blocking-raw_input-for-python.html

import signal

class AlarmException(Exception):
pass

def alarmHandler(signum, frame):
raise AlarmException

def nonBlockingRawInput(prompt='', timeout=20):
signal.signal(signal.SIGALRM, alarmHandler)
signal.alarm(timeout)
try:
text = raw_input(prompt)
signal.alarm(0)
return text
except AlarmException:
print '\nPrompt timeout. Continuing...'
signal.signal(signal.SIGALRM, signal.SIG_IGN)
return ''
</pre>

=== email ===

<pre>
import smtplib
import email.mime.text

msg_text = "Hello World"
msg = email.mime.text.MIMEText(msg_text)

# me == the sender's email address
# you == the recipient's email address
msg['Subject'] = 'The contents of %s' % textfile
msg['From'] = me
msg['To'] = you

# Send the message via our own SMTP server, but don't include the
# envelope header.
s = smtplib.SMTP('localhost')
s.sendmail(me, [you], msg.as_string())
s.quit()
</pre>

my nagios fake /bin/mail:
<pre>
#!/usr/bin/env python

import smtplib
import email.mime.text
import optparse
import os
import sys
import pwd

os.system('logger mail called')

parser = optparse.OptionParser()
parser.add_option(
'-s', '--subject',
help='Subject',
default="")
parser.add_option(
'-t', '--to',
help='To',
dest="send_to",
default=None)
parser.add_option(
'-f', '--from',
help='From',
dest="send_from",
default=None)
parser.add_option(
'-S', '--server',
help='server',
default="prime")
(options, args) = parser.parse_args()

def get_username():
return pwd.getpwuid( os.getuid() )[ 0 ]

if options.send_to:
send_to = options.send_to
else:
if len(args) == 1:
send_to = args[0]
else:
parser.print_help()
sys.exit(1)

if options.send_from:
send_from = options.send_from
else:
username = get_username()
hostname = os.uname()[1]
send_from = username + "@" + hostname

msg_text = sys.stdin.read()
msg = email.mime.text.MIMEText(msg_text)

msg['Subject'] = options.subject
msg['From'] = send_from
msg['To'] = send_to

s = smtplib.SMTP(options.server)
s.sendmail(send_from, [send_to], msg.as_string())
s.quit()
</pre>

Authenticated (with gmail):
<pre>
import smtplib
import email.mime.text

# from_email = "kenneth@demo.oeey.com"
# to_email = "kenneth@demo.oeey.com"
from_email = "Kenneth Burgener <kenneth@demo.oeey.com>"
to_emails = ["Kenneth Burgener <kenneth@demo.oeey.com>", "Kenneth Cell <xxx@xxx.com>"]

msg_text = "hello world"
msg = email.mime.text.MIMEText(msg_text)
msg['Subject'] = 'Test text'
msg['From'] = from_email
# msg['To'] = to_email
msg['To'] = ", ".join(to_emails)
# s = smtplib.SMTP('localhost')
print "sending..."
s = smtplib.SMTP_SSL('smtp.gmail.com', '465')
s.login('xxx', 'xxx')
# s.sendmail(from_email, [to_email], msg.as_string())
s.sendmail(from_email, to_emails, msg.as_string())
s.quit()
print "sent!"
</pre>

Source:
* 20.12. smtplib — SMTP protocol client — Python v2.7.5 documentation - http://docs.python.org/2/library/smtplib.html
* 18.1.11. email: Examples — Python v2.7.5 documentation - http://docs.python.org/2/library/email-examples.html

=== pprint ===

8.18. pprint — Data pretty printer — Python v2.7.5 documentation - http://docs.python.org/2/library/pprint.html

import pprint
pprint.pprint(myVar)

import pprint
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(myVar)

Note: Useless on objects!

Dump object:
<pre>
def dumpobject(obj):
print "<<<---"
print type(obj), obj.__class__.__name__
for attr in dir(obj):
if attr[0] == "_": continue # ignore under/dunder properties
if str(type(getattr(obj, attr))) == "<type 'instancemethod'>":
print attr, ": METHOD"
else:
print attr, ":", type(getattr(obj, attr)), ":", getattr(obj, attr)
print "END --->>>"
</pre>

=== Curses ===

Curses Programming with Python — Python v3.3.2 documentation - http://docs.python.org/3/howto/curses.html

"The curses library supplies a terminal-independent screen-painting and keyboard-handling facility for text-based terminals; such terminals include VT100s, the Linux console, and the simulated terminal provided by various programs. Display terminals support various control codes to perform common operations such as moving the cursor, scrolling the screen, and erasing areas. Different terminals use widely differing codes, and often have their own minor quirks.

In a world of graphical displays, one might ask “why bother”? It’s true that character-cell display terminals are an obsolete technology, but there are niches in which being able to do fancy things with them are still valuable. One niche is on small-footprint or embedded Unixes that don’t run an X server. Another is tools such as OS installers and kernel configurators that may have to run before any graphical support is available."

---

Thread from Utah Python User's Group mailing list:

-

Hello, I am trying to learn more about curses in python for a desktop application. Does anybody have any recommendations on tutorials or documentation?

-Corban

-

We've use urwid at work. Seems to be a good approach if you aren't already tied to something. Learning curve can be a bit steep (well, compared to what you'd think it should be). Good luck!

http://excess.org/urwid/

Gabriel Gunderson

-

This video series looks pretty good:

http://www.youtube.com/user/pythoncursestutorial?feature=watch

Official HOWTOs from the python.org site:

http://docs.python.org/2/howto/curses.html
http://docs.python.org/3/howto/curses.html

As Gabe mentioned, there are other libraries that provide similar or
greater functionality, so if you aren't doing this specifically to
learn curses, it's worth at least looking at a few of them to compare.

There's a brief comparison of three possibilities here:

http://stackoverflow.com/questions/8349085/python-ncurses-cdk-urwid-difference

Daniel Fackrell

-

Here's another option that looks promising:

https://pypi.python.org/pypi/blessings/

Daniel Fackrell

=== dialog ===

pythondialog - http://pythondialog.sourceforge.net/
: "pythondialog is a Python wrapper for the dialog utility originally written by Savio Lam, and later rewritten by Thomas E. Dickey. Its purpose is to provide an easy to use, pythonic and comprehensive Python interface to dialog. This allows one to make simple text-mode user interfaces on Unix-like systems (including Linux). "

Note: The latest versions (2.12 and later) only support Python 3; users who really want to stick to Python 2 should use version 2.11 for now.

Install:
# latest:
# wget http://sourceforge.net/projects/pythondialog/files/latest/download?source=dlp

wget http://sourceforge.net/projects/pythondialog/files/pythondialog/2.11/python2-pythondialog-2.11.tar.gz/download
tar -zvxf python2-pythondialog-2.11.tar.gz
cd pythondialog-2.11

import dialog
d = dialog.Dialog()
if d.yesno("Continue?") == d.DIALOG_OK:
print "Continuing..."

---

My solution

kiloforce / dialog / source / — Bitbucket - https://bitbucket.org/kiloforce/dialog

<pre>
#!/usr/bin/env python

""" Wrapper for Linux 'dialog' command """

__author__ = "Kenneth Burgener <kenneth@k.ttak.org>"
__copyright__ = "(c) 2013"
__date__ = "$Sep 27, 2013$"
__version__ = "1.0"
__credits__ = r"""
XXXXXXXXXXXXXXXXXX
X Dialog Wrapper #
##################
"""

import subprocess
import time
import sys

class Dialog():
""" Python wrapper for Linux 'dialog' command

dialog - display dialog boxes from shell scripts

>>> import dialog
>>> dlg = dialog.Dialog()
>>> rc = dlg.msgbox('hello world')

Return codes:
YES = 0
OK = 0
EXIT = 0
NO = 1
CANCEL = 1
ESC = 255
"""

YES = 0
OK = 0
EXIT = 0
NO = 1
CANCEL = 1
ESC = 255

def __init__(self):
rc = self._exe('which dialog > /dev/null')[0]
if rc != 0:
print "Error: failed to find Linux dialog command"
sys.exit(1)

def _exe(self, cmd, strip=True):
p = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE)
output = "\n".join(p.stderr.readlines())
if strip:
output = output.strip()
rc = p.wait()
return (rc, output)

def rc_to_str(self, rc):
if rc == 0:
return "YES"
elif rc == 1:
return "NO"
elif rc == 255:
return "ESX"
else:
raise Exception("Unknown return code: %s" % rc)

def clear(self):
"""
Clear screen

Example:
>>> dialog.clear()"""
rc, output = self._exe('clear')
assert rc in (0,), "clear rc"
return rc

def msgbox(self, msg):
"""
Simple message box

@param msg (str) = display message
@return: rc (int)

Example:
>>> rc = dialog.msgbox('hello world')"""
rc, output = self._exe('dialog --msgbox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.ESC), "msgbox rc"
return rc

def yesno(self, msg):
"""
Yes/No dialog

@param msg (str) = display message
@return: rc (int)

Example:
>>> rc = dialog.yesno('continue')"""
rc, output = self._exe('dialog --yesno "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.NO, self.ESC), "yesno rc"
return rc

def infobox(self, msg):
"""
Dialog that does not wait for user's response
(do your own pause)

@param msg (str) = display message
@return: rc (int) always zero

Example:
>>> for i in range(10):
>>> dialog.infobox('counting: ' + str(i))
>>> time.sleep(3)"""
rc, output = self._exe('dialog --infobox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES,), "infobox rc"
return rc

def inputbox(self, msg):
"""
Get user input

@param msg (str) = display message
@return: rc (int), output (str/None)

Example:
>>> rc, output = dialog.inputbox('name:')"""
rc, output = self._exe('dialog --inputbox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.NO, self.ESC), "inputbox rc"
if not output:
output = None
return rc, output

def textbox(self, filename):
"""
Display a file

@param filename (str) = file to display
@return: rc (int)

Example:
>>> dialog.textbox('/etc/passwd')"""
rc, output = self._exe('dialog --textbox {filename} 0 0'.format(filename=filename))
assert rc in (self.YES, self.ESC), "textbox rc"
return rc

def menu(self, msg, item_list):
"""
Single selection checkbox menu

@param msg (str) = display message
@param item_list (str list) = list of items to display
@return: rc (int), selection (int/None)

Example:
>>> rc, selection = dialog.menu('Select multiple options:',
... ('option 1', 'option 2'))"""
item_str = ""
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
rc, output = self._exe('dialog --menu "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
if output:
selection = int(output)
else:
selection = None
assert rc in (self.YES, self.NO, self.ESC), "menu rc"
return rc, selection

def checklist(self, msg, item_list, checked_list=None):
"""
Multiple selection checkbox menu

@param msg (str) = display message
@param item_list (str list) = list of items
@param checked_list (int list) = default items to check
@return: rc (int), selections (int list/None)

Example:
>>> rc, selections = dialog.checklist('Select multiple options:',
... ('option 1', 'option 2')
"""
item_str = ""
assert type(item_list) in (type(None), type(()), type([])), "checklist: item_list must be a list"
assert type(checked_list) in (type(None), type(()), type([])), "checklist: checked_list must be a list"
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
if checked_list and i+1 in checked_list:
item_str += " on"
else:
item_str += " off"
rc, output = self._exe('dialog --checklist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
print item_str
selections = []
for choice in output.split('"'):
if choice.strip():
selections.append(int(choice))
if selections:
selections = tuple(selections)
else:
selections = None
assert rc in (self.YES, self.NO, self.ESC), "checklist rc"
return rc, selections

def radiolist(self, msg, item_list, checked_item=None):
"""
Single selection radio menu, with default

@param msg (str) = display message
@param item_list (str list) = list of items
@param checked_item (int) = default item to select
@return: rc (int), selection (int)

Example:
>>> rc, selection = dialog.radiolist('Select one option:',
>>> ('option 1', 'option 2'), 2)"""
item_str = ""
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
if checked_item and i+1 == int(checked_item):
item_str += " on"
else:
item_str += " off"
rc, selection = self._exe('dialog --radiolist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
if not selection:
selection = None
assert rc in (self.YES, self.NO, self.ESC), "radiolist rc"
return rc, selection

def gauge(self, msg, percent):
"""
Progress bar (does not wait for user's input)

@param msg (str) = display message
@param percent (int) = progress
@return: rc (int) always zero

Example:
>>> for i in range(0, 100+1, 2):
>>> dialog.gauge('Progress:', i)
>>> time.sleep(.1)"""
rc, output = self._exe('echo {percent} | dialog --gauge "{msg}" 0 0'.format(percent=percent, msg=msg))
assert rc in (self.YES,), "guage rc"
return rc

def progress(self, msg, percent):
"""
Alias for self.guage()
"""
return self.gauge(msg=msg, percent=percent)

def form(self, msg, item_list):
"""
Multiple item entry form

@param msg (str) = display message
@param item_list (str list) = list of items
@return: rc (int), answers (str list)

Example:
>>> rc, answers = dialog.form('Enter details',
... ('first name', 'last name'))"""
item_str = ""
name_len = 1
for i in range(len(item_list)):
if len(item_list[i])+3 > name_len:
name_len = len(item_list[i])+3
for i in range(len(item_list)):
item_str += " \"" + item_list[i] + ":\" " + str(i+1) + " 1 \"\" " + str(i+1) +\
" " + str(name_len) + " 40 0"
rc, answers = self._exe('dialog --form "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str),
strip=False)
answers = answers.split('\n\n')
if not answers:
answers = None
assert rc in (self.YES, self.NO, self.ESC), "form rc"
return rc, answers

def test(self):
"""
Test features

>>> import dialog
>>> dlg = dialog.Dialog()
>>> dlg.test()
"""

def wait_user(msg):
self.msgbox("%s" % msg)

# rc_to_str
wait_user("Testing return code...")
print self.rc_to_str(self.YES) # YES
print self.rc_to_str(self.NO) # NO
print self.rc_to_str(self.ESC) # ESX
try:
print self.rc_to_str(111) # exception
raise Exception("Did not throw exception")
except Exception as rc:
pass

# msgbox
wait_user("Testing msgbox...")
rc = self.msgbox("msgbox")
if rc not in (self.YES, self.ESC):
raise Exception("Unknown return code for msgbox: %s" % rc)

# yesno
wait_user("Testing yesno...")
rc = self.yesno("yesno")
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for yesno: %s" % rc)

# infobox
wait_user("Testing infobox...")
rc = self.infobox("infobox sleep 1 sec")
if rc not in (self.YES,):
raise Exception("Unknown return code for infobox %s" % rc)
time.sleep(1)

# inputbox
wait_user("Testing inputbox...")
rc, output = self.inputbox("inputbox")
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for inputbox: %s" % rc)
wait_user("Result: %s" % output)

# textbox
wait_user("Testing textbox...")
rc = self.textbox("/etc/passwd")
if rc not in (self.YES, self.ESC):
raise Exception("Unknown return code for textbox: %s" % rc)

# menu
wait_user("Testing menu...")
rc, selection = self.menu("menu", ("option 1", "option 2"))
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for menu: %s" % rc)
wait_user("Result: %s" % selection)

# checklist
wait_user("Testing checklist...")
rc, selection = self.checklist("menu", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for checklist: %s" % rc)
wait_user("Result: %s" % str(selection))
wait_user("Testing checklist with default selection...")
rc, selection = self.checklist("menu", ["option 1", "option 2"], [1, 2])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for checklist: %s" % rc)
wait_user("Result: %s" % str(selection))

# radiolist
wait_user("Testing radiolist...")
rc, selection = self.radiolist("menu", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for radiolist: %s" % rc)
wait_user("Result: %s" % str(selection))
wait_user("Testing radiolist with default selection...")
rc, selection = self.radiolist("menu", ["option 1", "option 2"], 2)
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for radiolist: %s" % rc)
wait_user("Result: %s" % str(selection))

# guage
wait_user("Testing guage...")
for i in range(4+1):
rc = self.gauge("test guage", i*25)
# rc = self.progress("test progress", i*25)
if rc not in (self.YES,):
raise Exception("Unknown return code for guage: %s" % rc)
time.sleep(1)

# form
wait_user("Testing form...")
rc, answers = self.form("form", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for form: %s" % rc)
wait_user("Result: %s" % str(answers))

# done with test, cleanup
wait_user("Done with tests...")
self.clear()

if __name__ == "__main__":
# dialog = Dialog()
# dialog.test()
pass
</pre>

=== Busy Loop and Thread.yield() ===

"I want to tell my Python threads to yield, and so avoid hogging the CPU unnecessarily. In Java, you could do that using the Thread.yield() function"

Solution:
time.sleep(0)
time.sleep(0.00001)
time.sleep(epsilon)

References:
* multithreading - In there something similar to Java's Thread.yield() in Python? Does that even make sense? - Stack Overflow - http://stackoverflow.com/questions/1908206/in-there-something-similar-to-javas-thread-yield-in-python-does-that-even-ma
* multithreading - How does a threading.Thread yield the rest of its quantum in Python? - Stack Overflow - http://stackoverflow.com/questions/787803/how-does-a-threading-thread-yield-the-rest-of-its-quantum-in-python

=== USB ===

Programming with PyUSB 1.0 - http://pyusb.sourceforge.net/docs/1.0/tutorial.html

PyUSB - http://pyusb.sourceforge.net/

git clone https://github.com/walac/pyusb

import usb.core
dev = usb.core.find()

# find our device
dev = usb.core.find(idVendor=0xfffe, idProduct=0x0001)

"'''Dealing with multiple identical devices''' - Sometimes you may have two identical devices connected to the computer. How can you differentiate them? Device objects come with two additional attributes which are not part of the USB Spec, but are very useful: bus and address attributes. First of all, it is worth to say that these attributes come from the backend and a backend is free to not support them, in which case they are set to None. That said, these attributes represent the bus number and bus address of the device and, as you might already have imagined, can be used to differentiate two devices with the same idVendor and idProduct attributes."

=== Tkinter ===

Python GUI Bitcoin: [http://bitcoin.stackexchange.com/questions/10594/simplest-way-of-obtaining-mtgox-exchange-rate-via-an-api]
<pre>
import json
import urllib2
from Tkinter import *

url='http://api.bitcoincharts.com/v1/markets.json'
req=urllib2.Request(url)
response=urllib2.urlopen(req).read()
output=json.loads(response)

mtusd_price = output[-10]['avg']
mteur_price = output[31]['avg']
mtgbp_price = output[44]['avg']

MTUSD= 'USD' + ' ' + str(mtusd_price)
MTEUR= 'EUR' + ' ' + str(mteur_price)
MTGBP = 'GBP' + ' ' + str(mtgbp_price)

root= Tk()
root.title("Bitcoin Price")
root.geometry("250x100")

USD = Label(root, text=MTUSD)
EUR = Label(root, text=MTEUR)
GBP = Label(root, text=MTGBP)
USD.pack()
EUR.pack()
GBP.pack()
mainloop()
</pre>

=== ctypes ===

"ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python."
:-- 15.17. ctypes — A foreign function library for Python — Python v2.7.7 documentation - https://docs.python.org/2/library/ctypes.html

Use "c scope" to build a database of symbols to see what options are available.

---

testlib.c
<pre>
#include <stdio.h>

void myprint(void);

void myprint()
{
printf("hello world\n");
}
</pre>

Compile:
gcc -shared -Wl,-soname,testlib -o testlib.so -fPIC testlib.c

testlibwrapper.py

Python:
<pre>
import ctypes

testlib = ctypes.CDLL('/full/path/to/testlib.so')
testlib.myprint()
</pre>

Source: python - ctypes - Beginner - Stack Overflow - http://stackoverflow.com/questions/5081875/ctypes-beginner

---

nzjrs's Gists - https://gist.github.com/nzjrs

---

<pre>
#include <stdio.h>

int sum(double *x,int n)
{
int i;
double counter;
counter = 0;
for(i=0;i<n;i++)
{
counter=counter+x[i];

}
return counter;
}
</pre>

<pre>
gcc -c sum.c
gcc -shared -o sum.so sum.o
</pre>

<pre>
from ctypes import *
my_sum=CDLL('sum.so')
a=numpy.array(range(10),dtype=float)
my_sum.sum(a.ctypes.data_as(c_void_p),int(10))
</pre>

<pre>
a=(c_double*10)()
for i in range(10):
a[i]=i
my_sum.sum(a,int(10))
</pre>

Source: Ctypes — Numerical Computing with Sage v6.2 - http://www.sagemath.org/doc/numerical_sage/ctypes.html

---

=== Warnings ===

if sys.version_info >= (1, 0, 0):
warnings.warn("Danger Will Robinson: trying Phi on untested Python 3.0!")

=== enum ===

Python 3: https://docs.python.org/3/library/enum.html

<pre>
from enum import Enum

# class syntax
class Color(Enum):
RED = 1
GREEN = 2
BLUE = 3

# functional syntax
Color = Enum('Color', ['RED', 'GREEN', 'BLUE'])
</pre>

---

Python 2: [http://stackoverflow.com/questions/36932/how-can-i-represent-an-enum-in-python]
<pre>
class Enum(set):
""" Basic enumerated type

Example
>>> Animals = phi.Enum(["DOG", "CAT", "HORSE"])
>>> Animals.DOG
'DOG'
>>> 'HORSE' in Animals
True
"""
def __getattr__(self, name):
if name in self:
return name
raise AttributeError("Missing Enum property: %s" % name)
def __setattr__(self, name, value): # this makes it read-only
raise AttributeError("Unable to modify Enum property: %s (%s)" % (name, value))

Animals = Enum(["DOG", "CAT", "HORSE"])

print(Animals.DOG) # 'DOG'
</pre>

----------

Alternative:
<pre>
class Stationery:
Pen, Pencil, Eraser = range(0, 3)
</pre>

One liner:
class Enum(tuple): __getattr__ = tuple.index

<pre>
>>> State = Enum(['Unclaimed', 'Claimed'])
>>> State.Claimed
1
>>> State[1]
'Claimed'
>>> State
('Unclaimed', 'Claimed')
>>> range(len(State))
[0, 1]
>>> [(k, State[k]) for k in range(len(State))]
[(0, 'Unclaimed'), (1, 'Claimed')]
>>> [(k, getattr(State, k)) for k in State]
[('Unclaimed', 0), ('Claimed', 1)]
</pre>

Python 3 supports enum natively with the enum.Enum library:

=== pySerial ===

https://pythonhosted.org/pyserial/

"This module encapsulates the access for the serial port. It provides backends for Python running on Windows, OSX, Linux, BSD (possibly any POSIX compliant system) and IronPython. The module named “serial” automatically selects the appropriate backend."

https://github.com/pyserial/

Intro - https://pythonhosted.org/pyserial/shortintro.html

---

<pre>
import serial
import time

ser = serial.Serial(
port='COM4',\
baudrate=9600,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)

print("connected to: " + ser.portstr)

#this will store the line
seq = []
count = 1

while True:
for c in ser.read():
seq.append(chr(c)) #convert from ANSII
joined_seq = ''.join(str(v) for v in seq) #Make a string from array

if chr(c) == '\n':
print("Line " + str(count) + ': ' + joined_seq)
seq = []
count += 1
break

ser.close()
</pre>

Ref: http://stackoverflow.com/questions/16077912/python-serial-how-to-use-the-read-or-readline-function-to-read-more-than-1-char

---

I modified the code to monitor a COM port: (Windows)

Monitor Serial:

<pre>
import serial, time, sys

logger = open("serial.txt", "w")
timer = time.time()

def log(msg, head=False):
msg = msg.strip()
if head:
msg = ">>> " + msg
logger.write(msg + "\n")
sys.stdout.write(msg + "\n")
newtime = time.time()
if newtime > timer + 1:
logger.flush()
sys.stdout.flush()

while True:

ser = None
try:
ser = serial.Serial(
port='COM1',\
baudrate=115200,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)
except serial.serialutil.SerialException as e:
log(str(e), True)
time.sleep(1)
continue

log("Connected to: " + ser.portstr, True)
count=1

try:
line = []
while True:
for c in ser.readline():
line.append(c)
if c == "\n":
log("".join(line))
line = []
except serial.serialutil.SerialException as e:
log(str(e), True)
time.sleep(1)
ser.close()
</pre>

=== MyBool ===

<pre>
>>> class NewBool(int):
... def __new__(cls, value):
... return int.__new__(cls, bool(value))

>>> y = NewBool(2)
>>> y == 1
True
</pre>

https://jfine-python-classes.readthedocs.io/en/latest/subclass-int.html

Also check out the __cmp__

== md5 ==

import md5
md5.new("Nobody inspects the spammish repetition").hexdigest()
# 'bb649c83dd1ea5c9d9dec9a18df0ffe9'

<pre>
$ echo -n "Nobody inspects the spammish repetition" | md5sum
bb649c83dd1ea5c9d9dec9a18df0ffe9 -
<pre>

== External Libraries ==

=== execnet ===

Python Package Index - execnet - https://pypi.python.org/pypi/execnet

Examples - http://codespeak.net/execnet/examples.html
* basic local and remote communication - http://codespeak.net/execnet/example/test_info.html

Execute source code in subprocess, communicate through a channel:
<pre>
>>> import execnet
>>> gw = execnet.makegateway()
>>> channel = gw.remote_exec("channel.send(channel.receive()+1)")
>>> channel.send(1)
>>> channel.receive()
2
</pre>

Compare current working directories
<pre>
>>> import execnet, os
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec("import os; channel.send(os.getcwd())")
>>> res = ch.receive()
>>> assert res == os.getcwd()
</pre>

Get information from remote ssh account
<pre>
>>> import execnet, os
>>> gw = execnet.makegateway("ssh=codespeak.net")
>>> channel = gw.remote_exec("""
... import sys, os
... channel.send((sys.platform, sys.version_info, os.getpid()))
... """)
>>> platform, version_info, remote_pid = channel.receive()
>>> platform
'linux2'
>>> version_info
(2, 4, 2, 'final', 0)
</pre>

remote-exec a function (avoiding inlined source part I)
<pre>
import execnet

def multiplier(channel, factor):
while not channel.isclosed():
param = channel.receive()
channel.send(param * factor)

gw = execnet.makegateway()
channel = gw.remote_exec(multiplier, factor=10)

for i in range(5):
channel.send(i)
result = channel.receive()
assert result == i * 10

gw.exit()
</pre>

remote-exec a module (avoiding inlined source part II)
:"You can pass a module object to remote_exec in which case its source code will be sent. No dependencies will be transferred so the module must be self-contained or only use modules that are installed on the “other” side. Module code can detect if it is running in a remote_exec situation by checking for the special __name__ attribute."

<pre>
# content of a module remote1.py

if __name__ == '__channelexec__':
channel.send('initialization complete')
</pre>

<pre>
>>> import execnet, remote1
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec(remote1)
>>> print (ch.receive())
initialization complete
</pre>

Use a callback instead of receive() and wait for completion
<pre>
>>> import execnet
>>> gw = execnet.makegateway()
>>> channel = gw.remote_exec("for i in range(10): channel.send(i)")
>>> l = []
>>> channel.setcallback(l.append, endmarker=None)
>>> channel.waitclose() # waits for closing, i.e. remote exec finish
>>> l
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, None]
</pre>

a simple command loop pattern
<pre>
# contents of: remotecmd.py
import os

def simple(arg):
return arg + 1

def listdir(path):
return os.listdir(path)

if __name__ == '__channelexec__':
for item in channel:
channel.send(eval(item))
</pre>

<pre>
>>> import execnet, remotecmd
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec(remotecmd)
>>> ch.send('simple(10)') # execute func-call remotely
>>> ch.receive()
11
</pre>

=== evdev ===

Install:
apt-get install python-setuptools python-dev
easy_install pip
pip install evdev

Simple read usage:
<pre>
from evdev import InputDevice
from evdev import ecodes

# look for a /dev/input/by-id/usb...kbd or similar
DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd"

dev = InputDevice(DEVICE)

while True:
r, w, x = select([dev], [], [], 5)
for event in dev.read():
if event.type == 1 and event.value == 1:
print event.code
</pre>

List LED states:
dev.leds()
dev.leds(verbose=True)

Inject usage: (does not appear to really work)
<pre>
from evdev import InputDevice
from evdev import ecodes
from evdev import UInput

DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd"
dev = InputDevice(DEVICE)

ui = UInput()

ui.write(ecodes.EV_KEY, ecodes.KEY_4, 1)
ui.write(ecodes.EV_KEY, ecodes.KEY_4, 0)
ui.syn()

ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 1)
ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 0)
ui.syn()

ui.close()
</pre>

Tutorial:
* evdev documentation — python-evdev 0.4.3 documentation - http://pythonhosted.org/evdev/index.html
References:
* python-evdev/doc/tutorial.rst at master · gvalkov/python-evdev · GitHub - https://github.com/gvalkov/python-evdev/blob/master/doc/tutorial.rst

=== ldap-python ===

python-ldap: LDAP client API for Python - http://www.python-ldap.org/

Index of Packages : Python Package Index - https://pypi.python.org/pypi/python-ldap/

yum install python-devel openldap-devel
pip install python-ldap

Articles:
* Python LDAP Applications: Part 1 - Installing and Configuring the Python-LDAP Library and Binding to an LDAP Directory | Packt Publishing - http://www.packtpub.com/article/installing-and-configuring-the-python-ldap-library-and-binding-to-an-ldap-directory

=== xmmp ===

PDF:
* XMPP: The Definitive Guide - Oreilly - http://oriolrius.cat/blog/wp-content/uploads/2009/10/Oreilly.XMPP.The.Definitive.Guide.May.2009.pdf

xmmp is the Jabber protocol

Best options:

MIT libraries.
* sleekxmpp (was: sleekxmpp) is pretty popular and is used for examples in Peter Saint-Andre's XMPP book from O'Reilly. It reportedly works well, and finally got an email list in April 2010, and has a chat room at sleek@conference.jabber.org.

GPL libraries.
* xmpppy is used by gajim.

LPGL libraries.
* pyxmpp is pretty good and uses libxml2 internally for xml parsing.
* pyxmpp2 is the next version of pyxmpp, runs on python 2.7 and 3.2, and removes the libxml2 requirement. Like many, it requires dnspython.
* jabberpy is the original and still works for a lot of tasks, but is thoroughly unmaintained.

Other libraries.
* Wokkel, mentioned in another post. That's a new one for me, based on twisted.

chat - Best python XMPP / Jabber client library? - Stack Overflow - http://stackoverflow.com/questions/1901828/best-python-xmpp-jabber-client-library

=== xmpppy ===

xmpppy: the jabber python project - http://xmpppy.sourceforge.net/

"xmpppy is a Python library that is targeted to provide easy scripting with Jabber. Similar projects are Twisted Words and jabber.py.

This library was not designed from scratch. It inherits some code from jabberpy and have very similar API in many places. Though it is separate project since it have almost completely different architecture and primarily aims to work with jabberd2 - the new Open Source Jabber Server.

xmpppy is distributed under the terms of GNU General Public License and can be freely redistributed without any charge."

wget http://sourceforge.net/projects/xmpppy/files/xmpppy/0.4.1/xmpppy-0.4.1.tar.gz/download -O xmpppy-0.4.1.tar.gz
tar -zvxf xmpppy-0.4.1.tar.gz
pip install pydns

--- basic connection ---

<pre>
#!/usr/bin/env python

import xmpp

user="username@gmail.com"
password="password"
server="gmail.com"

jid = xmpp.JID(user)
connection = xmpp.Client(server,debug=[])
connection.connect()
result = connection.auth(jid.getNode(), password,"LFY-client")

connection.sendInitPresence()

while connection.Process(1):
pass
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- basic receive ---

Print messages when received:
<pre>
#!/usr/bin/env python

import xmpp

user="username@gmail.com"
password="password"
server="gmail.com"

def message_handler(connect_object, message_node):
message = "Welcome to my first Gtalk Bot :)"
connect_object.send( xmpp.Message( message_node.getFrom() ,message))

jid = xmpp.JID(user)
connection = xmpp.Client(server)
connection.connect()
result = connection.auth(jid.getNode(), password, "LFY-client")
connection.RegisterHandler('message', message_handler)

connection.sendInitPresence()

while connection.Process(1):
pass
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- remote-control shell bot ---

Replace the simple bot’s message_handler() function with this new one:

<pre>
def message_handler(connect_object,message_node):
command = str(message_node.getBody())

process = subprocess.Popen(command,shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
message = process.stdout.read()
if message=="":
message=process.stderr.read()

connect_object.send( xmpp.Message( message_node.getFrom() ,message))
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- roster ---

<pre>
import xmpp
import sys

userID = 'myname@gmail.com'
password = 'mypassword'
ressource = 'Script'

jid = xmpp.protocol.JID(userID)
jabber = xmpp.Client(jid.getDomain(), debug=[])

connection = jabber.connect(('talk.google.com',5222))
auth = jabber.auth(jid.getNode(), password, ressource)

jabber.sendInitPresence(requestRoster=1)
myroster = jabber.getRoster()
</pre>

To get status, you have to wait for the other clients to send you their status:
<pre>
jabber.RegisterHandler('presence', myPresenceHandler)

def myPresenceHandler(con, event):
fromjid = event.getFrom().getStripped()
status = myroster.getStatus(fromjid)
</pre>

References:
python - XMPP chat: accessing contacts' status messages with xmppPy's Roster - Stack Overflow - http://stackoverflow.com/questions/2381597/xmpp-chat-accessing-contacts-status-messages-with-xmpppys-roster

--- full roster script ---

kparal/jabber-roster · GitHub - https://github.com/kparal/jabber-roster

A simple Python tool for listing your Jabber roster contacts. You can use it to easily backup list of your buddies.

This program is maintained, but not further developed. Bugs will be fixed, but no new features will be added. If you want to work on this program, don't hesitate to contact me, I will gladly assign you to the development team.

git clone https://github.com/kparal/jabber-roster
python jabber_roster.py

https://pypi.python.org/pypi/jabber-roster

--- vcard ---

This information is not in the roster. You will need to query the clients individually and get their vCard by sending this IQ :

<pre>
<iq from='stpeter@jabber.org/roundabout'
id='v1'
type='get'>
<vCard xmlns='vcard-temp'/>
</iq>
</pre>

My solution:
<pre>
def vcard(disp, jid):
msg = xmpp.protocol.Iq()
msg.setType('get')
msg.setTo(jid)
qc = msg.addChild('vCard')
qc.setAttr('xmlns', 'vcard-temp')
rep = disp.SendAndWaitForResponse(msg)
# to see what other fields are available in the XML output:
# print rep
userid=fname=lname=title=department=region=None
for i in rep.getChildren():
for j in i.getChildren():
if j.getName() == "TITLE":
title = j.getData().encode('utf-8')
for k in j.getChildren():
if k.getName() == "GIVEN":
fname = k.getData().encode('utf-8')
if k.getName() == "FAMILY":
lname = k.getData().encode('utf-8')
if k.getName() == "ORGUNIT":
department = k.getData().encode('utf-8')
if k.getName() == "REGION":
region = k.getData().encode('utf-8')
return fname, lname, title, department, region
</pre>

References:
* xmpppy - Retrieve gtalk nickname in python xmpp - Stack Overflow - http://stackoverflow.com/questions/4539477/retrieve-gtalk-nickname-in-python-xmpp
* XEP-0054: vcard-temp - http://xmpp.org/extensions/xep-0054.html

=== google-api-python-client ===

Google APIs Client Library for Python, which includes:
* google-api-python-client: The core Python library for accessing Google APIs.
* oauth2client: A Python client library for OAuth 2.0.
* Sample applications using google-api-python-client and oauth2client

-

https://developers.google.com/api-client-library/python/start/installation
* Calendar API - Python/Command Line

-
Download:
wget https://code.google.com/p/google-api-python-client/downloads/detail?name=google-api-python-client-1.2.zip
wget https://code.google.com/p/google-api-python-client/downloads/detail?name=oauth2client-1.2.zip

Git:
git clone https://github.com/google/google-api-python-client
git clone https://github.com/google/oauth2client

References:
* google-api-python-client - Google APIs Client Library for Python - Google Project Hosting - https://code.google.com/p/google-api-python-client/

== Documentation Tools ==

=== pydoc ===

pydoc - http://docs.python.org/library/pydoc.html
: Documentation generator and online help system

=== Doxygen and Doxypy ===

See [[Python/Doxygen]]

=== Sphinx ===

Sphinx - http://sphinx.pocoo.org/

"Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license.

It was originally created for the new Python documentation, and it has excellent facilities for the documentation of Python projects, but C/C++ is already supported as well, and it is planned to add special support for other languages as well. Of course, this site is also created from reStructuredText sources using Sphinx!"

=== epydoc ===

epydoc - http://epydoc.sourceforge.net/

"Epydoc is a tool for generating API documentation for Python modules, based on their docstrings. For an example of epydoc's output, see the API documentation for epydoc itself (html, pdf). A lightweight markup language called epytext can be used to format docstrings, and to add information about specific fields, such as parameters and instance variables. Epydoc also understands docstrings written in reStructuredText, Javadoc, and plaintext. For a more extensive example of epydoc's output, see the API documentation for Python 2.5. "

== Python 3 ==

=== Python 3 Wall of Superpowers ===

Python 3 Wall of Superpowers - https://python3wos.appspot.com/

"As listed on PyPI - packages in red don't support python 3, packages in green do. Hopefully one day everything will be greener. "

=== New in Python 3 ===

==== qualnames ====

New in 3.3
<pre>
>>> def f():
... def g():
... class H: pass
... return H
... return g
...

>>> H=f()()

>>> # 3.3:
>>> H
<class '__main__.f.<locals>.g.<locals>.H'>
>>> H.__qualname__
'f.<locals>.g.<locals>.H'

>>> # 2.6:
>>> H
<class __main__.H at 0x2b885044b890>
>>> H.__qualname__
AttributeError: class H has no attribute '__qualname__'
</pre>

== Python 2 or Python 3 ==

Python2orPython3 - PythonInfo Wiki - http://wiki.python.org/moin/Python2orPython3
:Short version: Python 2.x is the status quo, Python 3.x is the shiny new thing.
:At the time of writing (July 4, 2010), the final 2.7 release is out, with a statement of extended support for this end-of-life release. The 2.x branch will see no new major releases after that. 3.x is under active and continued development, with 3.1 already available and 3.2 due for release around the turn of the year.

Python 2.7.1 Release - http://www.python.org/download/releases/2.7.1/

What’s New In Python 3.0 — Python v3.0.1 documentation - http://docs.python.org/release/3.0.1/whatsnew/3.0.html
* print() requires parenthesis
* raw_input() is now input()

== Python for Windows ==

Python for Windows - http://www.python.org/download/windows/

=== Windows IDEs ===

==== PyCharm ====

http://www.pycharm.com/

=== Hide Console ===

fork() does not work on windows. So what else can we do?

How to hide console window in python? - http://stackoverflow.com/questions/764631/how-to-hide-console-window-in-python
* "Simply save it with a .pyw extension. This will prevent the console window from opening. Explanation at the bottom of section [http://docs.python.org/2/tutorial/interpreter.html#executable-python-scripts]"

=== 32bit vs 64bit ===

IDLE:

32bit:
<pre>
C:\Python27>python
Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
</pre>

64bit: (note: default path is C:\Python27)
<pre>
C:\Python27-64bit>python.exe
Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
</pre>

Task Manager:
python.exe *32

python.exe

Platform:
>>> import platform
>>> platform.python_compiler()
'MSC v.1500 32 bit (Intel)'

>>> import platform
>>> platform.python_compiler()
'MSC v.1500 64 bit (AMD64)'

Code: [http://stackoverflow.com/questions/2208828/detect-64bit-os-windows-in-python]
import struct
def is_python_64bit():
return (struct.calcsize("P") == 8)

=== Stand-Alone Executable ===

==== Python ZIP Application ====

Python can be run from a single bundled zip file - similar to a Java WAR! Python has been able to (Allow interpreter to execute a zip file http://bugs.python.org/issue1739468)

"Python quietly added a new feature in 2.6 that makes it possible to bundle up a directory full of Python code into a single executable file." [http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html]

When a python zip application is executed, it will execute '__main__.py' [http://docs.python.org/using/cmdline.html#interface-options]

mkdir app
touch app/__main__.py
cd app
zip -r ../app.zip *
cd ..
python app.zip

Load a resource - The easiest way to find and load a program bundle like this is to use the pkg_resources module. - http://packages.python.org/distribute/pkg_resources.html

<pre>
# file: app/__main__.py

import pkg_resources

def main():
print('The rain in Spain falls mainly in the plain.')
print(pkg_resources.resource_string('resources', 'inFrance.txt'))

if __name__ == '__main__':
main()
</pre>

Now use a bit of UNIX magic to turn app2.zip into an executable:
echo '#!/usr/bin/env python' | cat - app2.zip > app2
chmod +x app2

"The zip file format is designed to allow a small executable program to be inserted at the front (that's how self-extracting zip files are created), so this is kosher and doesn't corrupt the zip file." [http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html]

Now you can simply run app2 like any executable.
$ ./app2

References:
* Able Pear Software: Bundling Python files into a stand-alone executable - http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html
* PEP 441 -- Improving Python ZIP Application Support - http://www.python.org/dev/peps/pep-0441/
* Issue 1739468: Allow interpreter to execute a zip file - Python tracker - http://bugs.python.org/issue1739468
* Package Discovery and Resource Access using pkg_resources — Distribute documentation - http://packages.python.org/distribute/pkg_resources.html

==== Shell Python Loader ====

Came before the Python ZIP Application support.

Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe

"This takes your Python application and packages it into a single executable shell script. You application can be made of up multiple python scripts. The executable shell script is also compressed, so the resulting script is smaller than the original. This should work on any system that has Python and a Bourne shell (works on Cygwin too). Note that this does not package the Python interpreter into the executable. This packages only you python code into a zip file, but this still allows you to distribute your application as a single shell script that will run on any system with Python installed. In theory, you could zip the .pyc files, but the byte code is guaranteed to run only on exactly the same version Python interpreter (including minor version). It is safer to just package the .py files."

First you need to zip all your python files:
zip main.zip main.py spam.py eggs.py

zipheader.sh:
<pre>
#!/bin/sh
# This is a self-extracting executable.
# Execute this like any normal executable.
# You may need to "chmod a+x" this file.
# This is a binary ZIP file with a Python loader header.
#
# Bourne shell loader:
PYTHON=$(which python 2>/dev/null)
if [ ! -x "$PYTHON" ] ; then
echo "Python not found!"
exit 1
fi
exec $PYTHON -c "
# Python loader:
import sys, os
if int(sys.version[0])<2:
print 'Python version 2.3 final or greater is required.'
print 'Your version is', sys.version
os._exit(1)
major = sys.version_info[0]
minor = sys.version_info[1]
releaselevel = sys.version_info[3]
if (major==2 and minor<3) or (major==2 and minor==3 and releaselevel!='final'):
print 'Python version 2.3 final or greater is required.'
print 'Your version is', sys.version
os._exit(1)
sys.path.insert(0, sys.argv[1])
del sys.argv[0:1]
import main
main.main()
" $0 $@
# Zip file:
</pre>

Concatinate zipheader.sh with the zip file.
cat zipheader.sh main.zip > main

Set main as executable.
chmod +x main

"This is an improvement on Joerg Raedler's Python recipe, [http://code.activestate.com/recipes/215301-binding-main-skript-and-modules-to-one-executable-/ 215301]. The original closes stdin. The herefile in the shell script redirects stdin before python gets a chance to start. This disables raw_input() and anything else that reads sys.stdin. Unfortunately, Raedler's boot script closes stdin, which is a fairly big limitation. The herefile in the shell script (END_OF_PYTHON_CODE) redirects stdin before python starts; stdin is closed at the end of the herefile. This disables raw_input() and anything else that reads sys.stdin. "

References:
* Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe
* Build a compressed self-extracting executable script on UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/497000-build-a-compressed-self-extracting-executable-scri/
* binding main skript and modules to one executable with python-2.3 under UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/215301-binding-main-skript-and-modules-to-one-executable-/

==== py2exe ====

py2exe - http://www.py2exe.org/
: "py2exe is a Python Distutils extension which converts Python scripts into executable Windows programs, able to run without requiring a Python installation. "

Tutorial - http://www.py2exe.org/index.cgi/Tutorial

Note: Install the version that matches your version of python

Note: run installer "as administrator"

setup.py
<pre>
from distutils.core import setup
import py2exe

setup(console=['hello.py'])
</pre>

No console window version: [http://ubuntuforums.org/showthread.php?t=728170] [http://stackoverflow.com/questions/3275293/hiding-console-window-of-python-gui-app-with-py2exe]
<pre>
from distutils.core import setup
import py2exe

setup(windows=['hello.py'])
</pre>

Build:
python setup.py py2exe

pygame is more complex: http://www.pygame.org/wiki/Pygame2exe

==== py2app ====

==== pyinstaller ====

==== cx_Freeze ====

=== Windows Libraries ===

==== Python for Windows Extensions - PyWin32 ====

Python for Windows Extensions - http://starship.python.net/~skippy/win32/
* There is a build for each python version and architecture (32bit vs 64bit)

Follow event viewer
<pre>
import win32evtlog # install Python for Windows Extensions - http://starship.python.net/~skippy/win32/

# open event viewer
server = "localhost"
source = "System"
handle = win32evtlog.OpenEventLog(server, source)

# seek to end of log
flags = win32evtlog.EVENTLOG_BACKWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
events = win32evtlog.ReadEventLog(handle, flags, 0)

# return to normal reading direction
flags = win32evtlog.EVENTLOG_FORWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
win32evtlog.ReadEventLog(handle, flags, 0) # ignore initial logs

# loop through events
while True:
events = win32evtlog.ReadEventLog(handle, flags, 0)
if events:
for event in events:
data = event.StringInserts
if data:
# Example: 07/03/14 12:34:35 - MYEVENTSOURCE - 0 - 1 - 4 "HELLO"
print event.TimeGenerated, '-', event.SourceName, '-', event.EventCategory, '-', event.EventID, '-', event.EventType, '-', '"%s"' % " ".join(data)
time.sleep(.5) # yield cpu
</pre>

References:
* PyWin32: Getting Windows Event Logs « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/07/27/pywin32-getting-windows-event-logs/

==== PyCrypto for Windows ====

Download Install: http://www.voidspace.org.uk/python/modules.shtml#pycrypto

Installation: [http://stackoverflow.com/questions/11405549/how-do-i-install-pycrypto-on-windows]
easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win32-py2.7.exe
easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win-amd64-py2.7.exe

==== Paramiko for Windows ====

# Install PyCrypto (above)
# pip install paramiko

==== PyGame for Windows ====

Download - http://www.pygame.org/download.shtml
* http://pygame.org/ftp/pygame-1.9.1.win32-py2.7.msi

== Indentation ==

Python: Myths about Indentation - http://www.secnetix.de/olli/Python/block_indentation.hawk
:There are quite some prejudices and myths about Python's indentation rules among people who don't really know Python. I'll try to address a few of these concerns on this page.
*"Whitespace is significant in Python source code."
*"Python forces me to use a certain indentation style."
*"You cannot safely mix tabs and spaces in Python."
*"I just don't like it."
*"How does the compiler parse the indentation?"

== PEP - Python Enhancement Proposals ==

Python Enhancement Proposals (PEPs)

* Index of Python Enhancement Proposals (PEPs) - http://www.python.org/dev/peps/

== Convert Tabs to Spaces ==

Convert tabs to 4 spaces:
sed -i 's/\t/ /g' script.py

== Linting ==

* pep8 - python style guide checker
* pychecker - more verbose, imports code, slower
* pylint - most verbose, configurable, "rates" code
* pyflakes - least verbose (dead/redundant code)

rpmforge:
yum install pyflakes pychecker
# pylint - not sure which pacakge includes

=== pep8 ===

See [[Python/pep8]]

=== pychecker ===

See [[Python/pychecker]]

=== pylint ===

See [[Python/pylint]]

== PEP8 Style Guidlines ==

PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/

* Use spaces - Use 4 spaces per indentation level.
* Never mix tabs and spaces.
* Limit all lines to a maximum of 79 characters.
* Line wrapping - The preferred way of wrapping long lines is by using Python's implied line continuation inside parentheses, brackets and braces. Long lines can be broken over multiple lines by wrapping expressions in parentheses.
* Encoding - For Python 3.0 and beyond, UTF-8 is preferred over Latin-1, see PEP 3120.
* Imports should usually be on separate lines
* Avoid extraneous whitespace in most situations
* Always use 'self' for the first argument to instance methods.

=== PEP8 Style Checker ===

See [[#pep8]]

== Benefits of Python ==

What is the return on investment for this group by learning Python? The list of benefits below sums up what the language has to offer:
* Python is designed from the ground up to encourage maintainable, readable and reusable code. It has great support for particular paradigms such as object-oriented programming.
* Python code is usually around a quarter of the length of code generated by C++ or Java - less typing, less debugging and less maintenance.
* Python does not need to be compiled or linked, which allows for a more iterative programming style and a quicker development cycle.
* Python is extremely portable. Code can be easily exchanged between Windows, Mac OS and Linux with little to no modification.
* Python comes with a staggering array of standard libraries and mature third-party projects. Web development, scientific computing, operator system access and network scripting are all supported.
* Python can talk with many other languages easily. It has support for invoking C and C++ code and can itself be called from libraries written in these languages.

Source: http://quantstart.com/articles/Quant-Reading-List-Python-Programming/ - Quant Reading List Python Programming

Benefits of Python:
* Python is available for most operating systems, including Windows, UNIX, Linux, and Mac OS.
* Python's clean object-oriented design and extensive support libraries offer two to ten fold the programmer productivity seen with languages like C, C++, C#, Java, VB, and Perl.

Source: Python Development - Benefits of Python - Digital Mesh - http://www.digitalmesh.com/offshore-development-center/python-development/benefits-of-python.html

== Python Package Management ==

=== Install Module ===

Installing Python Modules — Python v2.7.1 documentation - http://docs.python.org/install/index.html

The new standard: Distutils
python setup.py install

Download and extract target package, then:
sudo python setup.py install

Note: Running setup.py install builds and installs all modules in one run

Help:
python setup.py --help

Incremental method:
python setup.py build
python setup.py install
python setup.py build --help # help

Force rebuild and install:
python setup.py build --force # if have to repeat
python setup.py install --force # if have to repeat

Default installation paths:
<pre>
Platform Standard installation location Default value
Unix (pure) prefix/lib/pythonX.Y/site-packages /usr/local/lib/pythonX.Y/site-packages
Unix (non-pure) exec-prefix/lib/pythonX.Y/site-packages /usr/local/lib/pythonX.Y/site-packages
Windows prefix C:\Python
</pre>

Alternate install path:
python setup.py install --home=<dir>

Alternate base folders:
<pre>
Type of file Installation Directory Override option
pure module distribution home/lib/python --install-purelib
non-pure module distribution home/lib/python --install-platlib
scripts home/bin --install-scripts
data home/share --install-data
</pre>

Modifying Python’s Search Path:
import sys
sys.path.append('/www/python/')

=== Python Package Index ===

user-submitted Python packages:
http://pypi.python.org/pypi

sudo python setup.py install

Package Site:
http://pypi.python.org/pypi/[PACKAGE]
http://pypi.python.org/pypi/test

If they are on http://pypi.python.org/:
pip install foo

[http://stackoverflow.com/questions/410163/for-python-programmers-is-there-anything-equivalent-to-perls-cpan For Python programmers, is there anything equivalent to Perl's CPAN? - Stack Overflow]

=== Easy Installation ===

EasyInstall - The PEAK Developers' Center - http://peak.telecommunity.com/DevCenter/EasyInstall
: Easy Install is a python module (easy_install) bundled with setuptools that lets you automatically download, build, install, and manage Python packages.

CheeseShopTutorial - PythonInfo Wiki - http://wiki.python.org/moin/CheeseShopTutorial
: EasyInstall (easy_install) gives you a quick and painless way to install packages remotely by connecting to the Package Index or even other websites via HTTP. It is somewhat analogous to the CPAN and PEAR tools for Perl and PHP, respectively.

=== pip ===

See [[Python/pip]]

==== Create PIP Package ====

See [[Python/pip#Create PIP Package]]

=== Distribute ===

distribute - Easily download, build, install, upgrade, and uninstall Python packages

distribute 0.6.24 : Python Package Index - http://pypi.python.org/pypi/distribute

Installation:
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/d/distribute/distribute-0.6.24.tar.gz#md5=17722b22141aba8235787f79800cc452
tar -zvxf distribute-0.6.24.tar.gz
cd distribute-0.6.24
sudo python setup.py install

Test:
echo "import distribute" | grep python

== Named Parameters ==

http://diveintopython.org/power_of_introspection/optional_arguments.html

Python allows function arguments to have default values; if the function is called without the argument, the argument gets its default value. Futhermore, arguments can be specified in any order by using named arguments. Stored procedures in SQL Server Transact/SQL can do this, so if you're a SQL Server scripting guru, you can skim this part.

Here is an example of info, a function with two optional arguments:

def info(object, spacing=10, collapse=1):

spacing and collapse are optional, because they have default values defined. object is required, because it has no default value. If info is called with only one argument, spacing defaults to 10 and collapse defaults to 1. If info is called with two arguments, collapse still defaults to 1.

Say you want to specify a value for collapse but want to accept the default value for spacing. In most languages, you would be out of luck, because you would need to call the function with three arguments. But in Python, arguments can be specified by name, in any order.

http://www.wellho.net/mouth/1871_Optional-and-named-parameters-in-Python.html

If you add a parameter with two asterixes (**) on the end of a function definition, that parameter is taken as being the name of a dictionary into which all otherwise unidentified parameters are stored.

<pre>
def getnet(gross, taxrate=17.5, **others):
net = gross / (1.0 + taxrate*0.01)
for more in others.keys():
print "GETNET:",more," - ",others[more]
return neto
</pre>

== Python vs other languages ==

[http://www.strombergers.com/python/ Python is Cool (and Perl is not), Especially for C/C++ Programmers]

==Experimenting with Memory Remanence==
[http://citp.princeton.edu/memory/exp/ Experimenting with Memory Remanence]

<pre>
#!/usr/bin/env python

# a pirate's favorite chemical element
a = ""
while 1: a += "ARGON"
</pre>

sudo strings /dev/mem | less

==UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)==
See [[UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)]]

== Program Layout Best Practices Presentation ==

Program layout best practices - Seth House <seth@eseth.com> - Utah Django User Group - 2011-06-23
* Slides - https://github.com/whiteinge/presentations/tree/master/upyug_2011-06-09_program-layout
* Video - http://blip.tv/utah-open-source/python-layout-best-practices-for-modules-django-cli-utah-django-user-group-5315808

Module skeleton:
<pre>
#!/usr/bin/env python
# coding: utf-8

"""Module docstring"""

__author__ = ’Mr. Me <me@example.net>’
__version__ = ’1.2.3’

if __name__==’__main__’:
main()
</pre>

Executing modules as scripts
# PEP 338:
python -m SimpleHTTPServer

== CLI package ==

cli — command line tools — pyCLI v1.1.1 documentation - http://packages.python.org/pyCLI/

== Python IDE ==

IntegratedDevelopmentEnvironments - PythonInfo Wiki - http://wiki.python.org/moin/IntegratedDevelopmentEnvironments
* IDEs with introspection-based code completion and integrated debugger
* IDEs with introspection-based code completion /or/ integrated debugger
* IDEs with integrated gui builder
* Editors
* Reviews

Is there a good, free Python IDE for Windows? - Stack Overflow - http://stackoverflow.com/questions/126753/is-there-a-good-free-python-ide-for-windows
* Eclipse with PyDev
* Aptana Studio 3
* Active State also offers Komodo Edit
* WingWare - http://www.wingware.com/
* Eric IDE - http://www.die-offenbachs.de/eric/index.html
* MMM-Experts - Products - http://mmm-experts.com/Products.aspx?ProductId=4
** PyScripter is a Python IDE built in Python for Delphi(P4D) components and created with the ambition to become competitive in functionality with commercial Windows-based IDEs available for other languages. Being built in a compiled language is rather snappier than some of the other Python IDEs and provides an extensive blend of features that make it a productive Python development environment.
* ActiveState Python comes with PythonWin which is reasonably good
* PyCharm from JetBrains is not free in general, but has free early access program.
* SPE - Stani's Python Editor, is pretty awesome. See http://pythonide.blogspot.com/.
* You may want to take a look at the pdb module - http://docs.python.org/lib/module-pdb.html
* Geany (http://www.geany.org/) is a fairly light text editor that does a lot of IDE-like stuff.
* SciTE: http://www.scintilla.org/SciTE.html
* Editra: http://editra.org/
* Netbeans has support for python development in their Python Early Access version. - http://www.netbeans.org/features/python/
* I use Notepad++ with the appropriate settings.
* The Eric Python IDE - http://www.die-offenbachs.de/eric/index.html
* BOa Constructor www.boa-constructor.sourceforge.net really nice! drag and drop stuff!
* DrPtyhon is also very nice, and has many plug-ins. http://drpython.sourceforge.net/
* Vim
* I will make a plug for my preference of Python development environments: SPE. SPE is an integrated development environment that provides code completion, integrated Python shell, calltips, and various helper tools like notes and todo lists. It also includes wxGlade so you can make wxPython applications easily. Plus, it includes the python debugger WinPdb.
* http://groups.google.com.tr/group/python_opengl this software is my sample visual python ide.its demo version.
* I use PyScripter, and find it pretty awesome.
* There is one Python IDE written in Java: jHepWork: http://jwork.org/jhepwork/ it can detect Python's indentation rules rather nicely (+ detects structure of a code and syntax)
* gvim built with your python version support and with this plugins:
** NERDTree (don't read he's blog ;)
** matchit (it's builtin in new versions just put a symlink or junction)
** pyflakes
** pysmell
* Try Python Tools for Visual Studio. Supports CPython and IronPython. - http://pytools.codeplex.com/

PythonEditors - PythonInfo Wiki - http://wiki.python.org/moin/PythonEditors
* List of editors

Python Ide - http://c2.com/cgi/wiki?PythonIde
: There are a lot of Ides for PythonLanguage springing up. Many of them written in Python

Is there a better Python IDE? « Lennart Regebro: Python, Plone, Web - http://regebro.wordpress.com/2010/10/09/is-there-a-better-python-ide/
: I’m currently a user of WingIDE, but sometimes I feel the need for something better, mainly something that is a bit more stable. WingIDE often hangs when switching projects, sometimes the file listing decides that you must double-click on the expand arrow instead of just click on it, etc. It still doesn’t have macros despite this being a much requested feature (how hard can it be to add, really?) and I never really get the hang of the SVN integration, so I just ignore it. And they keep recommending me to use WingIDE’s separate instance of GTK instead of the System GTK, which is daft. Their designs are ugly, and it should work with the system GTK so it looks the same. Anything else is just silly.

=== Open Komodo ===

Open Komodo - http://www.openkomodo.com/
: Open Komodo is the source code repository for Komodo Edit - a free multi-language editor for dynamic languages, based on the award-winning Komodo IDE. ActiveState created the Open Komodo code repository in August 2007 and it has been used since then for building all Komodo Edit releases.

=== Eclipse ===

Build an Eclipse development environment for Perl, Python, and PHP - http://www.eclipse.org/resources/resource.php?id=485
: This tutorial shows how Eclipse's DLTK makes it possible to build development tools for scripting languages. In particular, it explains how to implement syntax coloring, user preferences, and interpreter integration in a plug-in-based project.

See [[#PyDev]]

=== EasyEclipse ===

EasyEclipse - http://www.easyeclipse.org/
:EasyEclipse packages together Eclipse, the open-source software
development platform, and selected open source plugins.
:We select, assemble, test, patch, build installers and document
a full IDE, offered as reliable distributions and plugins.
:EasyEclipse is:
* Free and open-source,
* Easy to download and install, and
* Simple to maintain, without version and dependency issues.

EasyEclipse for Python - http://www.easyeclipse.org/site/distributions/python.html

EclipsePythonIntegration - PythonInfo Wiki - http://wiki.python.org/moin/EclipsePythonIntegration
: Using Eclipse as a Python editor

=== PyDev ===

PyDev - http://pydev.org/
: PyDev is a Python IDE for Eclipse, which may be used in Python, Jython and IronPython development.

---

Pydev - http://pydev.sourceforge.net/ (old link)

How to Configure Eclipse for Python - http://www.rose-hulman.edu/class/csse/resources/Eclipse/eclipse-python-configuration.htm

Python Development with PyDev and Eclipse - Tutorial - http://www.vogella.de/articles/Python/article.html
: This article describes how to write and debug Python programs with Eclipse

PyDev Getting Started - http://pydev.org/manual_101_root.html

Running a program - http://pydev.org/manual_101_run.html

---

Supported Task Tags:
* TODO:
* TASK:
* FIXME:

---

Installation:
# Platform Runtime Binary (download around 45-50 MB) - http://download.eclipse.org/eclipse/downloads/ or http://www.eclipse.org/downloads/
#* Example: Eclipse Classic 4.2.2
# Eclipse Software Add - PyDev: http://pydev.org/updates
# Eclipse Software Add - MercurialEclipse: http://cbes.javaforge.com/update [http://javaforge.com/project/HGE] [http://mercurial.selenic.com/wiki/MercurialEclipse]
# Switch perspectives to PyDev

=== The Eric Python IDE ===

The Eric Python IDE - http://eric-ide.python-projects.org/
:Eric is a full featured Python and Ruby editor and IDE, written in python. It is based on the cross platform Qt gui toolkit, integrating the highly flexible Scintilla editor control. It is designed to be usable as everdays' quick and dirty editor as well as being usable as a professional project management tool integrating many advanced features Python offers the professional coder. eric4 includes a plugin system, which allows easy extension of the IDE functionality with plugins downloadable from the net.

Eric Python IDE - Wikipedia - http://en.wikipedia.org/wiki/Eric_Python_IDE

=== Wingware Python IDE ===

The Intelligent Development Environment for Python Programmers - Wingware Python IDE - http://www.wingware.com/
:Wingware's Python IDE is an Integrated Development Environment designed specifically for the Python programming language. Since 1999, Wingware has focused on Python.

=== VIM ===

Python .vimrc:
<pre>
" Kenneth's Preferences

set nobackup " don't create backup files
"set nohlsearch " don't highlight search results
set hlsearch " do highlight search results
set incsearch " incremental search
set ignorecase " ignore case during search
set smartcase " do case sensitive search when upper case is typed
set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
set showmatch " briefly highlight matching bracket
set matchtime=1 " highlight match for 0.1 seconds
set background=dark " black background
set shiftwidth=4 " shift blocks by 4 spaces
set shortmess=at " use short messages so we don't have to hit enter
set expandtab " expand tabs into spaces
set scrolloff=5 " always show 5 lines before & after cursor
set shell=bash " use bash for the command shell
set shellcmdflag=-lc " make bash startup as a login shell for ~/.bashrc
"set nowrap " do not wrap long lines
set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set wildmenu " enable filename tab completion
set ruler " display the ruler: shows cursor position
set cpoptions+=$ " display '$' at end of cw sequence
set title " display filename and vim info in window title bar
set modelines=5 " search the first and last 5 lines for "vim:" modeline
set backspace=2 " backspace deletes

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

"if v:version > 700
" set cursorline " underline the current line
"endif

if &t_Co > 2 || has( "gui_running" )
syntax on
endif

if has( 'mouse' )
set mouse=a " enable mouse
endif

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

"folding settings
set foldmethod=indent "fold based on indent
set foldnestmax=10 "deepest fold is 10 levels
set nofoldenable "dont fold by default
set foldlevel=10 "this is just what i use

" Auto clear trailing white space in Python files
func! DeleteTrailingWS()
exec "normal mz"
%s/\s\+$//ge
exe "normal `z"
endfunc
autocmd BufWrite *.py :call DeleteTrailingWS()
</pre>

----

Older versions:
<pre>
" KENNETH SETTINGS
set nocp " turn off compatibility mode
set nomodeline " do not allow files to dictate vim behavior

set incsearch " incremental search
set ignorecase " ignore case during search

set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set background=dark " black background
syntax on " turn on syntax highlighting
set hlsearch " highligh last search pattern

set nowrap " do not wrap long lines
set scrolloff=5 " always show 5 lines before & after cursor
set ruler " show the cursor position all the time
set backspace=2 " allow backspace deletion
set bs=indent,eol,start " allow backspacing over everything in insert mode

set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set expandtab " expand tab into spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
filetype indent on " turn on auto indent for specific file types
</pre>

Other options:
<pre>
set number " show line numbers
set numberwidth=1 " remove space before largest line numbers

set smartindent " ?? - maybe included in 'filetype indent on'

set ai " always auto indent

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

</pre>

Kenneth's fio .vimrc:
<pre>

"source $VIM/mswin.vim

set nobackup " don't create backup files
"set nohlsearch " don't highlight search results
set incsearch " incremental search
set ignorecase " ignore case during search
set smartcase " do case sensitive search when upper case is typed
set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
set showmatch " briefly highlight matching bracket
set matchtime=1 " highlight match for 0.1 seconds
set background=dark " black background
set shiftwidth=4 " shift blocks by 4 spaces
set shortmess=at " use short messages so we don't have to hit enter
set expandtab " expand tabs into spaces
set scrolloff=5 " always show 5 lines before & after cursor
set shell=bash " use bash for the command shell
set shellcmdflag=-lc " make bash startup as a login shell for ~/.bashrc
"set nowrap " do not wrap long lines
set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set wildmenu " enable filename tab completion
set ruler " display the ruler: shows cursor position
set cpoptions+=$ " display '$' at end of cw sequence
set title " display filename and vim info in window title bar
set modelines=5 " search the first and last 5 lines for "vim:" modeline

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

if v:version > 700
set cursorline " underline the current line
endif

if &t_Co > 2 || has( "gui_running" )
syntax on
endif

if has( 'mouse' )
set mouse=a " enable mouse
endif

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

"folding settings
set foldmethod=indent "fold based on indent
set foldnestmax=10 "deepest fold is 10 levels
set nofoldenable "dont fold by default
set foldlevel=10 "this is just what i use
</pre>

== Interpreter vs Script ==

Interpreter:
>>> import os
>>> os.uname()
('Linux', 'demo.oeey.com', '2.6.18-194.8.1.el5', '#1 SMP Thu Jul 1 19:07:06 EDT 2010', 'i686')

Script:
#!/usr/bin/env python
import os
print os.uname()

== Web Development ==

WebProgramming - PythonInfo Wiki - http://wiki.python.org/moin/WebProgramming
: This topic guide attempts to cover every aspect of programming Web applications (both clients and servers) using Python.

HOWTO Use Python in the web — Python v2.7.2 documentation - http://docs.python.org/howto/webservers.html
: This document shows how Python fits into the web. It presents some ways to integrate Python with a web server, and general practices useful for developing web sites.

=== Plug Simple Python Web App ===

<pre>
-------- Original Message --------
Subject: Re: Writing a simple python web app
Date: Mon, 25 Oct 2010 19:03:31 -0400
From: Roberto Mello <roberto.mello@gmail.com>
To: Provo Linux Users Group <plug@plug.org>

On Sat, Sep 11, 2010 at 5:40 PM, Dave Smith <dave@thesmithfam.org> wrote:
> I'm creating a very simple web application, and I want to use python. My first guess was that mod_python
would provide the easiest entry point. Boy, was I wrong. All the mod_python tutorials spend 80% of their
time extolling the virtues of mod_python over CGI, but they are quite lean on specific examples. It's quite
disappointing. A natural progression then leads me to mod_wsgi, and since that has equally poor
documentation, to Django. However, Django is overkill for my app (I don't even have a database).

For something as simple as you have described, and to run on an
embedded platform, I would probably stay away from frameworks designed
to run on full hardware, and go with something more minimalistic.

Take a look at CherryPy [1] or web.py [2].

[1] http://www.cherrypy.org/
[2] http://webpy.org/

Roberto Mello
</pre>

---

See also [[web.py]]

See also [[CherryPy]]

See also [[Django]]

=== Apache CGI ===

See [[CGI]]

Great tutorial: https://www.tutorialspoint.com/python/python_cgi_programming.htm

My favorite option, make all .cgi files executable: (or .py)
<pre>
DirectoryIndex index.html index.php index.py
<Location />
# AddHandler cgi-script .cgi
AddHandler cgi-script .py
Options +ExecCGI
</Location>
</pre>

If you want to allow .htaccess:
* see [[Apache#.htaccess]]

To enable on Ubuntu:
sudo a2enmod cgi
service apache2 restart

Directory index:
DirectoryIndex index.py

Or if you really want to go crazy, this will try to run *every* file: [http://blog.kapsobor.de/archives/2006/09/05/apache_addhandler_multiple_extensions/]
<pre>
# this has problems with / directory being run as a script, instead of index too!
#<Location />
# SetHandler cgi-script
# Options +ExecCGI
#</Location>

<Files ~ ".">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>

or:
<pre>
<Files ~ "^.[a-zA-Z]*$">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>

Or just have index file:
<Files "index">

Note: Place within a <VirtualHost> section if needed.

Note: Make sure to 'chmod +x' the files.

Start all HTML output with:
print "Content-Type: text/html\n"

If you want just text output:
print "Content-Type: text/plain\n"

GET (from Environment):
<pre>
print "<pre>"
for env in os.environ:
print "%s = %s" % (env, os.environ[env])
print "</pre>"
</pre>

<pre>
# sample for: http://oeey.com/py/test.py/test?var1=ken&var2=barbie

REQUEST_URI = /py/index.py/test?var1=ken&var2=barbie
QUERY_STRING = var1=ken&var2=barbie
PATH_INFO = /test
REQUEST_METHOD = GET

SCRIPT_NAME = /py/test.py
SERVER_NAME = oeey.com
REMOTE_ADDR = 216.51.42.66
SERVER_ADDR = 192.168.108.4
DOCUMENT_ROOT = /www/py
SCRIPT_FILENAME = /www/py/test.py
HTTP_HOST = cgi.oeey.com
</pre>

POST (from stdin):
<pre>
post = sys.stdin.readline()
if post:
print post
</pre>

<pre>
CONTENT_TYPE = application/x-www-form-urlencoded 

yin=one+two+three&yang=four+five+six
</pre>

cgi helper class:
<pre>
import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage()

# field keys
print "fields keys: ", form.keys() # list of field keys

# fild values
if "name" in form: # if form.has_key('name'):
print "field 'name' is: ", form["name"].value
print "field 'name' is: ", form.getvalue('name')

# headers
print "dictionary of headers: ", form.headers
print form.type
print form.headers['content-type']
</pre>

To prevent XSS escape the characters:
print cgi.escape(form.getvalue('name'))

<pre>
import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage()
if (form.has_key("action") and form.has_key("name") \
and form.has_key("age")):
if (form["action"].value == "display"):
print "%s = %s" % (form["name"].value form["age"].value)
</pre>

Note: cgi helper class also does sessions and cookies.

References:
* CGI Tutorial | Web Python - http://webpython.codepoint.net/cgi_tutorial
* Python CGI: An Interactive Instruction - http://www.cs.virginia.edu/~lab2q/

See also:
* Web Python - http://webpython.codepoint.net/

---

REST Example:

<pre>
import os, re
def get_rest():
if not "PATH_INFO" in os.environ:
return ()
# filter for garbage
#print os.environ["PATH_INFO"]
path_info = re.findall('[/A-Za-z0-9_-]*', os.environ["PATH_INFO"])
if not path_info:
return ()
path_info = ''.join(path_info)
# collapse empty paths
paths = path_info.split('/')
final_paths = []
for path in paths:
if path:
final_paths.append(path)
return tuple(final_paths)
rest = get_rest();
</pre>

=== mod_python ===

AVOID! AVOID! AVOID! - if you want this simple, just use CGI!

See [[Apache mod_python]]

mod_python - http://docs.python.org/howto/webservers.html#mod-python

Install:
yum install mod_python

Config:
/etc/httpd/conf.d/python.conf

Simple:
<pre>

</pre>

:"People coming from PHP often find it hard to grasp how to use Python in the web. Their first thought is mostly mod_python, because they think that this is the equivalent to mod_php. Actually, there are many differences. What mod_python does is embed the interpreter into the Apache process, thus speeding up requests by not having to start a Python interpreter for each request. On the other hand, it is not “Python intermixed with HTML” in the way that PHP is often intermixed with HTML. The Python equivalent of that is a template engine. mod_python itself is much more powerful and provides more access to Apache internals. It can emulate CGI, work in a “Python Server Pages” mode (similar to JSP) which is “HTML intermingled with Python”, and it has a “Publisher” which designates one file to accept all requests and decide what to do with them.

:mod_python does have some problems. Unlike the PHP interpreter, the Python interpreter uses caching when executing files, so changes to a file will require the web server to be restarted. Another problem is the basic concept – Apache starts child processes to handle the requests, and unfortunately every child process needs to load the whole Python interpreter even if it does not use it. This makes the whole web server slower. Another problem is that, because mod_python is linked against a specific version of libpython, it is not possible to switch from an older version to a newer (e.g. 2.4 to 2.5) without recompiling mod_python. mod_python is also bound to the Apache web server, so programs written for mod_python cannot easily run on other web servers.

:These are the reasons why mod_python should be avoided when writing new programs. In some circumstances it still might be a good idea to use mod_python for deployment, but WSGI makes it possible to run WSGI programs under mod_python as well."

=== WSGI ===

WSGI - http://docs.python.org/howto/webservers.html#mod-wsgi

: "The Web Server Gateway Interface, or WSGI for short, is defined in PEP 333 and is currently the best way to do Python web programming. While it is great for programmers writing frameworks, a normal web developer does not need to get in direct contact with it. When choosing a framework for web development it is a good idea to choose one which supports WSGI."

See [[mod_wsgi]]

=== Django ===

See [[Django]]

=== CherryPy ===

See [[CherryPy]]

=== web.py ===

web.py: makes web apps

Think about the ideal way to write a web app. Write the code to make it happen.

web.py : Python Package Index - http://pypi.python.org/pypi/web.py

Home Page: http://webpy.org/

"Django lets you write web apps in Django. TurboGears lets you write web apps in TurboGears. Web.py lets you write web apps in Python."
:— Adam Atlas

NOTE: Can modify source code without having to restart app! :-)

---

PIP Install:
pip install web.py

---

Hello World: [http://webpy.org/]
<pre>
import web

urls = (
'/(.*)', 'hello'
)
app = web.application(urls, globals())

class hello:
def GET(self, name):
if not name:
name = 'World'
return 'Hello, ' + name + '!'

if __name__ == "__main__":
app.run()
</pre>

Note: app.py can be renamed.

Running:
python app.py

Note: You can add an IP address/port after the "app.py" bit to control where web.py launches the server. You can also tell it to run a fastcgi or scgi server.
python app.py 1234

---

Templates

<pre>
# templates/hello.html
$def with (name)

$if name:
I just wanted to say hello to $name.
$else:
Hello, world!
</pre>

app = ...
render = web.template.render('templates/')

index.GET:
name = 'Bob'
return render.hello(name) # render.[TEMPLATE_NAME](vars)

get variable: (/?name=Joe)
i = web.input(name=None)
return render.hello(i.name)

variable in url:
urls = ('/(.*)', 'hello')

class index:
def GET(self, name):
return render.hello(name)

# visit: /Joe

multiple variables in url:
urls = ('/(.*)/(.*)', 'hello')

class hello:
def GET(self, fname, lname):
return render.hello(fname, lname)

# visit: /Joe/Smith

=== lpthw.web ===

lpthw.web: a fork of web.py for Learn Python The Hard Way

Locks web.py at a specific version and removes some magic. Thanks to Aaron Swartz for making it originally.

lpthw.web: a fork of web.py for Learn Python The Hard Way - http://pypi.python.org/pypi/lpthw.web

Home Page: http://learnpythonthehardway.org/

This conflicts with web.py, so only install one.

=== web ===

web - Web modules for CGI and WSGI web programming

web : Python Package Index - http://pypi.python.org/pypi/web

Warning: This is an alpha release for those who are interested in
checking progress. No documentation and not for production use.

Auth, session, mail, form and templating tools for writing CGI scripts or WSGI web applications.

This version incompatible with previous versions becuase large portions of the code have been refactored into separate components.

== Issues ==

=== Hide DeprecationWarning ===

==== Import DeprecationWarning ====

Error:
<pre>
/opt/xmpppy-0.4.1/xmpp/auth.py:24: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import sha,base64,random,dispatcher,re
/opt/xmpppy-0.4.1/xmpp/auth.py:26: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5
</pre>

Hide deprecation warnings with code: [http://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python]
<pre>
import warnings

with warnings.catch_warnings():
warnings.filterwarnings("ignore",category=DeprecationWarning)
import md5, sha
</pre>

Hide with header:
#!/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line:
python -W ignore::DeprecationWarning script.py

References:
* How to ignore deprecation warnings in Python - Stack Overflow - http://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python

==== Funciton DeprecationWarning ====

Error:
<pre>
/opt/xmpppy-0.4.1/xmpp/transports.py:307: DeprecationWarning: socket.ssl() is deprecated. Use ssl.wrap_socket() instead.
tcpsock._sslObj = socket.ssl(tcpsock._sock, None, None)
</pre>

Hide with header:
#!/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line:
python -W ignore::DeprecationWarning script.py

==== Ubuntu 24 pip packages ====

<pre>
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
</pre>

== keywords ==

[[Category:Programming]]
[[Category:Python]]

Python

2026-04-27T19:08:42Z

Kenneth: /* Apache CGI */

{{subtop}}

== Python ==

Python (programming language) - Wikipedia - http://en.wikipedia.org/wiki/Python_(programming_language)
: Python is an interpreted, general-purpose high-level programming language whose design philosophy emphasizes code readability. Python aims to combine "remarkable power with very clear syntax", and its standard library is large and comprehensive. Its use of indentation for block delimiters is unique among popular programming languages.
: Python supports multiple programming paradigms, primarily but not limited to object-oriented, imperative and, to a lesser extent, functional programming styles. It features a fully dynamic type system and automatic memory management, similar to that of Scheme, Ruby, Perl, and Tcl. Like other dynamic languages, Python is often used as a scripting language, but is also used in a wide range of non-scripting contexts.
: The reference implementation of Python ([http://en.wikipedia.org/wiki/CPython CPython]) is free and open source software and has a community-based development model, as do all or nearly all of its alternative implementations. CPython is managed by the non-profit Python Software Foundation.
: Python interpreters are available for many operating systems, and Python programs can be packaged into stand-alone executable code for many systems using various tools.

Author:
:"Python was conceived in the late 1980s and its implementation was started in December 1989 by '''[http://www.python.org/~guido/ Guido van Rossum]''' at CWI in the Netherlands as a successor to the ABC programming language (itself inspired by SETL) capable of exception handling and interfacing with the Amoeba operating system. Van Rossum is Python's principal author, and his continuing central role in deciding the direction of Python is reflected in the title given to him by the Python community, Benevolent Dictator for Life (BDFL)." [http://en.wikipedia.org/wiki/Python_(programming_language)]

Name:
:"The language is named after the BBC show “Monty Python’s Flying Circus” and has nothing to do with reptiles. Making references to Monty Python skits in documentation is not only allowed, it is encouraged!" [http://docs.python.org/py3k/tutorial/appetite.html]

We as people are "Pythonites" [http://raspi.tv/2013/how-to-use-interrupts-with-python-on-the-raspberry-pi-and-rpi-gpio-part-2]

== [http://www.python.org/dev/peps/pep-0020/ The Zen of Python] ==

"The core philosophy of the language is summarized by the document 'PEP 20 (The Zen of Python)'" [http://en.wikipedia.org/wiki/Python_%28programming_language%29]
* PEP 20 -- The Zen of Python - http://www.python.org/dev/peps/pep-0020/

<pre>
Abstract

Long time Pythoneer Tim Peters succinctly channels the BDFL's
guiding principles for Python's design into 20 aphorisms, only 19
of which have been written down.

The Zen of Python

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Easter Egg

>>> import this
</pre>

Other Sources:
* http://www.python.org/doc/humor/
* The Zen of Python – as a Poster « CODE POETRY - http://codepoetry.wordpress.com/2010/03/23/the-zen-of-python-as-a-poster/
** Zen of Python (PDF) http://codepoetry.files.wordpress.com/2010/03/zen-of-python.pdf

== Documentation ==

* Overview — Python v2.7.1 documentation - http://docs.python.org/index.html
** Download — Python v2.7.1 documentation - http://docs.python.org/download.html

* Overview — Python v3.2 documentation - http://docs.python.org/py3k/
** Download — Python v3.2 documentation - http://docs.python.org/py3k/download.html

* PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/

== Tutorials ==

* The Python Tutorial — Python v2.7.1 documentation - http://docs.python.org/tutorial/

* Python 101 -- Introduction to Python - http://www.rexx.com/~dkuhlman/python_101/python_101.html

* 4. More Control Flow Tools — Python v2.7.1 documentation - http://docs.python.org/tutorial/controlflow.html

== Linux Installation ==

RHEL Distribution installation:
yum -y install python python-setuptools

Ubuntu Distribution installation: [https://help.ubuntu.com/community/Mercurial]
# enable 'universe' in the sources.list
# "Mercurial is included in the Universe repository - activate this repo first"
sudo apt-get install mercurial

CentOS 5 Option Recommended Dependencies:
yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel

# this leaves: 2.6.6: _bsddb, bsddb185, dl, imageop, sunaudiodev
# this leaves: 2.7.1: _bsddb, bsddb185, dl, imageop, sunaudiodev
# this leaves: 3.2: none

CentOS 9 Option Recommended Dependencies:
# gdbm-devel not found in centos9?
yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel sqlite-devel bzip2-devel tk-devel tcl-devel

Ubuntu 12.04 LTS 64bit dependencies:
apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev

# maybe: lib64readline-dev libsqlite3-dev tk-dev

# this leaves: 2.7: _bsddb, bsddb185, dl, imageop, sunaudiodev

=== 2.x ===

CentOS dependencies:
sudo yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel

Ubuntu 12.04 LTS 64bit dependencies:
sudo apt-get update
sudo apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev

Ubuntu 24.04 LTS 64bit dependencies:
sudo apt-get update
sudo apt-get install gcc make sudo libbz2-dev tcl-dev libncurses-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev libffi-dev
# optional install
sudo apt-get install checkinstall

2.7.15:
<pre>
# PYVER=2.6.6
# PYVER=2.7.5
# PYVER=2.7.6
# PYVER=2.7.9
# PYVER=2.7.10
# PYVER=2.7.15
PYVER=2.7.18
mkdir -p ~/.src ; cd ~/.src
wget --no-check-certificate http://www.python.org/ftp/python/$PYVER/Python-$PYVER.tgz
tar -zvxf Python-$PYVER.tgz
cd Python-$PYVER
# ./configure --prefix=/opt/python-$PYVER
# make clean && make
./configure --prefix=/opt/python-$PYVER && make clean && make
sudo make install

sudo /opt/python-$PYVER/bin/python -m ensurepip --upgrade
sudo /opt/python-$PYVER/bin/pip install --upgrade pip
sudo /opt/python-$PYVER/bin/pip install requests

export PATH=/opt/python-$PYVER/bin:$PATH
echo -e "\n\nexport PATH=/opt/python-$PYVER/bin:\$PATH\n" >> ~/.bash_profile
# echo -e "\n\nexport PATH=/opt/python27/bin:\$PATH\n" >> ~/.bashrc

# NOTE: remove any old versions from your .bash_profile
</pre>

Bootstrapping the pip installer - https://docs.python.org/2/library/ensurepip.html

=== 3.x ===

3.8.16:
<pre>
#VER=3.7.0
#VER=3.8.15
#VER=3.10.9
VER=3.8.16
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/${VER}/Python-${VER}.tgz
tar -zvxf Python-${VER}.tgz
cd Python-${VER}
./configure --prefix=/opt/python${VER}
make

sudo make install

## OPTIONAL alternative install path:
sudo checkinstall
## note: 'checkinstall' makes a .deb, which you can uninstall with easier, but if you are using --prefix, you do need to
## premake the folders
sudo mkdir -p /opt/python${VER}/bin
sudo mkdir -p /opt/python${VER}/lib
shortVER=`echo $VER | cut -d '.' -f 1-2` # get short version eg. 3.10 from 3.10.9
sudo mkdir -p /opt/python${VER}/lib/python${shortVER}
sudo mkdir -p /opt/python${VER}/lib/python${shortVER}/test
# sudo checkinstall # hit Y, and enter twice
## note: older versions have problems with the virtualization of the file system, so do FS Translation = no
checkinstall --fstrans=no

export PATH=/opt/python${VER}/bin:$PATH
echo -e "\n\nexport PATH=/opt/python${VER}/bin:\$PATH\n" >> ~/.bash_profile

# have to manually update version
rm /opt/python${VER}/bin/python
ln -sfn python3.8 /opt/python${VER}/bin/python
#cp /opt/python${VER}/bin/python3.8 /opt/python${VER}/bin/python

# DEPENDENCIES
sudo apt install ...
yum install libffi-devel openssl-devel
</pre>

Note, if you want the libraries to be shared (like python-dev):
./configure --enable-shared

---

VER=3.6.9

VER=3.6.8

---

3.2.0:
<pre>
mkdir -p ~/src ; cd ~/src
wget http://www.python.org/ftp/python/3.2/Python-3.2.tgz
tar -zvxf Python-3.2.tgz
cd Python-3.2
./configure --prefix=/opt/python32
make
sudo make install

export PATH=/opt/python32/bin:$PATH
echo -e "\n\nexport PATH=/opt/python32/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.3.0:
<pre>
mkdir -p ~/src ; cd ~/src
wget http://www.python.org/ftp/python/3.3.0/Python-3.3.0.tgz
tar -zvxf Python-3.3.0.tgz
cd Python-3.3.0
./configure --prefix=/opt/python3.3.0
make
sudo make install

export PATH=/opt/python3.3.0/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.3.0/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.3.4:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/3.3.4/Python-3.3.4.tgz
tar -zvxf Python-3.3.4.tgz
cd Python-3.3.4
./configure --prefix=/opt/python3.3.4
make clean
make
sudo make install

export PATH=/opt/python3.3.4/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.3.4/bin:\$PATH\n" >> ~/.bash_profile
</pre>

3.4.0:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz
tar -zvxf Python-3.4.0.tgz
cd Python-3.4.0
./configure --prefix=/opt/python3.4.0
make
sudo make install

export PATH=/opt/python3.4.0/bin:$PATH
echo -e "\n\nexport PATH=/opt/python3.4.0/bin:\$PATH\n" >> ~/.bash_profile
</pre>

=== setuptools ===

Purpose: to install pip

Note: required for several 'python setup.py install' installations.

Python Package Index : setuptools 0.6c11 - http://pypi.python.org/pypi/setuptools

By Package Manager:
yum install python-setuptools

By Source:
#VER=2.0.1
#VER=2.2
VER=3.4.4
mkdir -p ~/.src ; cd ~/.src
wget --no-check-certificate http://pypi.python.org/packages/source/s/setuptools/setuptools-$VER.tar.gz
tar -zvxf setuptools-$VER.tar.gz
cd setuptools-$VER
# sudo python setup.py install
sudo /opt/python-2.7.9/bin/python setup.py install

Fix header:
# Only needed on older versions
sed -i 's#!/usr/bin/python#!/usr/bin/env python#g' /opt/python279/bin/easy_install

Install pip:
sudo /opt/python-2.7.9/bin/easy_install pip

Sample pip usage:
sudo /opt/python-2.7.9/bin/pip install paramiko

== IDLE and Interactive Mode ==

Interactive Mode - "When commands are read from a tty, the interpreter is said to be in interactive mode. In this mode it prompts for the next command with the primary prompt, usually three greater-than signs (>>>); for continuation lines it prompts with the secondary prompt, by default three dots (...)." [http://docs.python.org/2/tutorial/interpreter.html#invoking-the-interpreter]

Can use '_' to reference last output:
>>> 5 + 6
11
>>> print _
11

References:
* 2. Using the Python Interpreter — Python v2.7.3 documentation - http://docs.python.org/2/tutorial/interpreter.html

== Python 2.7 EOL ==

<blockquote>
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
</blockquote>

== Show All Python in Path ==

compgen -c | grep -i "python"

== Code ==

=== Hello World ===

[http://en.wikipedia.org/wiki/Shebang_%28Unix%29 shebang] (also called a hashbang) of Python:
#!/usr/bin/env python
print "Hello World"

NOTE: can also just use "#!/usr/bin/python"

=== Standard Script ===

Standard Script:
<pre>
def main(args):
# do logic

if __name__ == "__main__":
sys.exit(main(sys.argv))
</pre>

Needed for things like pylint

=== White Space ===

Python uses white space to indicate code blocks. Instead of ''{'' use a '':'' and indent consistently (4 spaces).

if a == b:
print 'equal'

Convert tabs to spaces:
sed -i 's/\t/ /g' *.py

Remove trailing whitespace:
sed -i 's/\s*$//' *.py
sed --in-place 's/[[:space:]]\+$//' fileName

Clean project:
# find . -name "*py" -exec dos2unix {} \;
# find . -name "*py" -exec sed -i 's/\t/ /g' {} \;
# find . -name "*py" -exec sed -i 's/\s*$//' {} \;
find . -name "*py" -exec dos2unix {} \; -exec sed -i 's/\t/ /g' {} \; -exec sed -i 's/\s*$//' {} \;

==== Trim Tailing White Space ====

<pre>
sed -e 's/[[:space:]]*$//'
</pre>

<pre>
find . -name "*.py" -exec sed -i -e 's/[[:space:]]*$//' {} \;
</pre>

ref: [https://stackoverflow.com/questions/369758/how-to-trim-whitespace-from-a-bash-variable]

=== Print ===

print - http://docs.python.org/tutorial/inputoutput.html

Print: # include new line
print("Hello World")
print "Hello World" # python 2 only

Suppress new line: [http://stackoverflow.com/questions/493386/how-to-print-in-python-without-newline-or-space]
<pre>
print('#', end='') # python 3
print "Hello World", # python 2 only (does produce a single separating white space though)

import sys
sys.stdout.write('.') # no separating white space
</pre>

# flush output
# optional, if new line won't come for some time
import sys
sys.stdout.flush()

Multiple with separator:
print("hello", "world") # separated by spaces
print("hello", "world", sep = '.') # separated by period

Formatting:
<pre>
mystr = "abc"
myint = 5
myflt = 3.141592
print( "%s %d %.3f" % (mystr, myint, myflt) )
print( "{0} {1} {2}".format(mystr, myint, myflt) ) # string's format function (python > 2.7)
print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name
print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding
</pre>

Padded floating point decimal number: (rounds)
print "%06.2f" % 2.319 # "002.32"
print "%6.2f" % 2.311 # " 2.31"
print "{:06.2f}".format(2.319) # "002.32"
print "{:6.2f}".format(2.319) # " 2.32"

# Python 3 version where var=2.321234
print(f"{var:.2f}") # "2.32"
print(f"{var:6.2f}") # " 2.32"
print(f"{var:06.2f}") # "002.32"
print(f"{var:08.2f}") # "00002.32"
print(f"{var:08.4f}") # "002.3212"
# Python 3 version where var=2.329 - rounding
print(f"{var:06.2f}") # "002.33"

=== Comments ===

# this is a comment
# this is a comment
x = y # this is a comment

Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended)
"""
multi line comment
"""

'''
multi line comment can comment out """
'''

This can be used like a C "#if 0" concept to comment out whole sections of code.

Using false block to comment out a block code: (does not ignore bad syntax though)
if False:
# code in this block is ignored

=== doc strings ===

Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended)
"""
multi line comment
"""

=== Help ===

dir - list functions and properties of objects:
import math
dir(math) # module
dir(dir) # function
dir("") # object - "", [], {}, etc...
dir(__builtins__) # built in functions
dir() # list imported objects, and dunder methods

Note: dunder (double under) methods determine special functionality (eg. __add__ for +)

help - manual printout: (class doc string, function doc string, data, etc)
import math
help(math) # full module info
help(len) # full function info
help(["foo"]) # full object info

Pydoc: (command line, same as 'import math, help(math)'
$ pydoc math

Documentation String: (aka docstring, doc string)
print(math.pow.__doc__) # module doc string
print(len.__doc__) # function doc string

Object or function type:
type(x)

mylist = []
isinstance(mylist, list) # True
type(s) is list # True

Note: isinstance() and type() not equivalent when objects and subclass get involved

IDLE interactive help:
help()

=== Variables ===

Basic Data Types
str = "string" # string
num = 5 # integer
flt = 5.0 # float
nothing = None # None (not the same as undefined)

Undefined. Variables start off in an 'undefined' state. It is always better to initially set the variable to None to avoid this state. To check if undefined: [http://code.activestate.com/recipes/59892-testing-if-a-variable-is-defined/] [http://code.activestate.com/recipes/59892/]
<pre>
try:
x
except NameError:
x = None
</pre>

Rules: [http://www.python.org/dev/peps/pep-0008/ PEP8 - Method Names and Instance Variables]
* any length
* letters, numbers, underscore
* first character can't be number
* case sensitive
* cannot use keywords

Multiple assignment:
x = y = z = 0 # Zero x, y and z
x, y, z = 1, "two", 3.0 # x = 1, y = "two", z = 3.0

Swap variables: (done in parallel)
a, b = b, a

Unpack variables: (requires equal items on each side)
a, b, c = [1, 2, 3]

Other Data Types:
* tuple
* list
* dictionary
* set
* objects

Everything in Python is an object that has:
* identity: id(obj)
* type: type(obj)
* value: obj
** mutable - id() stays the same on modification (dictionary, lists, objects)
** immutable - id() changes on modification as new object is created (string, integer, tuple)

==== Check if Variable Exists ====

<pre>
try:
a # does a exist in the current namespace
except NameError:
a = 10 # nope
</pre>

'a' in vars() or 'a' in globals()
'a' in vars(__builtins__)

<pre>
if hasattr(a, 'property'):
doStuff(a.property)
else:
otherStuff()
</pre>

Ref: [https://stackoverflow.com/questions/1592565/determine-if-variable-is-defined-in-python] [https://stackoverflow.com/questions/610883/how-to-know-if-an-object-has-an-attribute-in-python]

=== Numbers ===

num = 5
print "num: %d" % num

>>> print "Today's stock price: %f" % 50.4625 1
50.462500
>>> print "Today's stock price: %.2f" % 50.4625 2
50.46
>>> print "Change since yesterday: %+.2f" % 1.5 3
+1.50

Arithmetic:
+ - * /
// # integer division
% # modulus
** # exponent

Compact Arithmetic:
i+=1 # Increment - Note, there is no 'i++'
i*=2 # Multiply
i%=2 # modulus
...

Math Functions:
import math
math.ceil(x)
math.floor(x)
math.trunc(x)
math.pow(x,y)

abs(x) # built in

String interpolation:
%% % character
%d integer
%x hex
%X hex (upper)
%f float
%s string

x = 1/81
print('value: %.2f' % x) # 0.01
print('value: %.5f' % x) # 0.012345
print('%d %d %d' % (1, 2, 3))

Type check:
if not isinstance(myvar, (int, float)): ...

=== Strings ===

STRINGS ARE IMMUTABLE! (to modify convert to a list)

Strings: (String Constants)
str = "Hello"
str = 'Hello'
str = """
Hello
World
""" # can also use '''...'''

String literals can span multiple lines: (no wrapping when \ used)
hello = "hello\
world"

Length:
len("str")

Raw string: (back slashes are not escaped)
myrawstr = r"hello\nworld\
how are you"
myrawstr = r"C:\" # ERROR - limitation of what won't be escaped

Concatenation:
str = 'hot' + 'dog'
print "Hello" "World"

Can't concat numbers
str = 5 + 'test' # error
num = 5
str = num + 'test' # error
str = str(5) + str(num) + 'test' # ok

Repeated Concatenation:
str = 10 * 'ha'
str = 'ha' * 10 # same
print '-'*60 # --------------------

String Indexing:
s = 'hello'
print( s[0] ) # is 'h'
print( s[-1] ) # is 'o'
print( s[len(s) - 1 ] ) # is also 'o'

Modify Strings: [http://stackoverflow.com/questions/1228299/change-one-character-in-a-string-in-python]
s[1] = 'x' # ERROR!
ERROR: can't modify string by index

# solution:
s = list("Hello World")
s[0] = 'C'
print( "".join(s) )

String Slicing:
s = 'hello'
print( s[0:3] ) # 'hel'
i = 1
print( s[i:i+1] ) # 'e'
print( s[:3] ) # 0:3 'hel'
print( s[4:] ) # 4:end 'o'
print( s[:] ) # start:end 'hello'
print( s[-2:] ) # -2:end 'lo'

For loop:
s = 'hello'
for c in s:
print c # c = 'h' first time

Escape characters:
\\ \' \" \n \r \t

Alternate loop:
s = 'hello'
for i in range( len(s) ):
print s[i]

Ord and Char (ASCII):
ord('a') # 97
chr(97) # 'a'

0-9 : 48-57
A-Z : 65-90
a-z : 97-122

List Functions: (dir on any string)
dir('')
dir(str)

Help:
help('')
help(str)

Padding:
"a".rjust(2, '0') # '0a'
"1".ljust(4, '0') # '1000'

String Test Functions:
"HELLO".isupper() # True
"hello".islower() # True
"hello".isalpha() # True
"12.22".isdigit() # False (because of '.')
"12.22".isnumeric() # False (because of '.')
"hel2".isalnum() # True
"e" in "hello" # True
"hello".startswith('he') # True
"hello".endswith('lo') # True

Note: To check for floats: [http://stackoverflow.com/questions/354038/how-do-i-check-if-a-string-is-a-number-in-python]
<pre>
def is_number(s):
try:
float(s)
return True
except ValueError:
return False
</pre>

String Modify Functions: (sort of... Strings are immutable)
"HELLO".lower() # 'hello'
"hello".upper() # 'HELLO'
" hello ".strip() # 'hello' remove leading/tailing whitespace
" hello ".lstrip() # 'hello ' remove left whitespace
" hello ".rstrip() # ' hello' remove right whitespace
"www.com".split('.') # ['www', 'com'] <- list
"www.com".partition('.') # ('www', '.', 'com') <- tuple
"".join( ('a','b','c') ) # "abc" join strings, list or tuple
"-".join( ('a','b','c') ) # "a-b-c" join with separator
"23".zfill(4) # "0023"

String Search Functions:
"hello".find('lo') # 3 (-1 on fail)
"hello".index('lo') # 3 (exception on fail)
"hello".rfind('lo') # 3 right to left (-1 on fail)
"hello".rindex('lo') # 3 right to left (exception on fail)

Formatting: (form of concatenation)
<pre>
mystr = "abc"
myint = 5
myflt = 3.141592
print( "%s %d %.3f" % (mystr, myint, myflt) )
print( "{0} {1} {2}".format(mystr, myint, myflt) ) # pep 3101 style - string's format function (python > 2.7)
print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name
print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding
</pre>

Split multi line string:
for line in str.split('\n'): print(line)
lines = str.splitlines() # returns list
lines = str.splitlines(True) # returns list (each includes new line characters)

Input a string:
name = input('What is your name: ').strip() # python 3
name = raw_input('What is your name: ').strip() # python 2

Leet Speak:
leet_table = ''.maketrans('EIOBT', '31087')
'BE COOL. SPEAK LEET!'.translate(leet_table)

String Functions:
newstr = mystr.strip() # remove white space on both sides
newstr = mystr.rstrip() # remove white space on right side
newstr = mystr.lstrip() # remove white space on left side
newstr = mystr.replace('\n', ' ') # replace string with another
newstr = mystr.replace('jello', 'hello') # replace string with another

==== String Formatting ====

PyFormat: Using % and .format() for great good! - https://pyformat.info/

===== Zero Padding =====

3 zero padding:
"{:03}".format(1) # 003

3 zero padding, specify position:
"{1:03}".format(1, 2) # 002

Old method:
"%03d" % 2 # 002

=== chr and ord ===

Ord and Char (ASCII):
ord('a') # 97
chr(97) # 'a'

=== Range ===

Creates a list of numbers. Remember: range always ends one less than you want!

Count:
ten = 10
one = 1
range(ten) # [0,1,...,9] starts with zero by default
range(one, ten + 1) # [1,...,10] actually get 1 to 10
range(ten, one - 1, -1) # [10,9,...,1] count down
range(2, 10 + 1, 2) # [2,4,...,10] even numbers

=== drange ===

Solution for decimal range() step value:
<pre>
def drange(start, stop, step):
r = start
while r < stop:
yield r
r += step

sub_ten = drange(0.0, 1.0, 0.1)
</pre>

References:
* floating point - Python decimal range() step value - Stack Overflow - http://stackoverflow.com/questions/477486/python-decimal-range-step-value

=== Casting ===

Convert:
float(3)
str(85)
int(8.6)
round(5.5)

Check Type:
type(5) # <class 'int'>
type(5.0) # <class 'float'>
type('5') # <class 'str'>
type(None) # <class 'NoneType'>
type(print) # <class 'builtin_function_or_method'>
type(()) # <class 'tuple'>
type([]) # <class 'list'>

mytype = int
if type(1) is mytype: ...

=== Boolean Logic ===

True and False:
bol = True
bol = False

Boolean Operators: (in order precedence)
a == b
a != b
not a
a and b
a or b

Note: Short-circuit evaluated - 'and' and 'or'

Return Boolean:
return myint >= 16

Get Boolean from string: [http://stackoverflow.com/questions/715417/converting-from-a-string-to-boolean-in-python]
# Note: no built in way to do this

myString = "false"
val = (myString == "true")

True if myString=="True" else False

s in ['true', '1', 't', 'y', 'yes', 'yeah', 'yup', 'certainly', 'uh-huh']

def str2bool(v):
return v.lower() in ("yes", "true", "t", "1")

=== None ===

None is Python's version of NULL or NIL.

None is a singleton (Python only has one copy of None in the interpreter).

if myvar is None:
...

=== If Conditionals ===

Operators:
==, !=, not, and, or, <, >, <>, <=, >=, in, not in, is, is not

If statement:
<pre>
if a == b:
...
elif c < age < d:
...
else:
...
</pre>

Shorthand conditional: (ternary operator)
val = True if a == b else False

is - checks for identical objects - same id()
# None is a singleton, so this works great for None
if myvar is None:
...

Other shorthand:
<pre>
def mycmp(x, y):
if x > y: return True
else: return False
</pre>

Switch/Case statement:
* there is no built in switch/case statement, use "if ... elif ... elif ..." sequence instead

=== For Loop ===

''for'' is used for iterating over an iterable sequence (eg. lists, dictionaries, tuples). (for each)

# range(start, end + 1, skip)

for i in range(10):
print(i) # 0 .. 9

for i in range(5, 10):
print(i) # 5 .. 9

for i in range(10, 0, -1):
print(i) # 10 .. 1

for i in [1, 2, 3]: # iterate over list
print(i) # 1 2 3

Discard variable: [http://bytes.com/topic/python/answers/757444-loop-without-variable]
for _ in range(10): # repeat and discard variable (good for pep8 check)
print "hello world"

for key in my_dict.keys(): ...
for value in my_dict.values(): ...
for key, value in my_dict.items(): ...

Compact version:
for i in range(10): print(i) # 0 .. 9

Access index: [http://stackoverflow.com/questions/522563/accessing-the-index-in-python-for-loops]
# using enumeration
for idx, val in enumerate(mylist):
print idx, val

# using range
for ix in range(len(mylist)):
print mylist[ix]

# list comprehension
[ (ix, mylist[ix]) for ix in range(len(mylist))]

=== While Loop ===

i = 0
while i < 10:
i = i + 1

while True:
... # infinite loop

=== Breaking Loops ===

break statement:
while True:
if s == 'done':
break # jump out of the loop
...

continue statement: (works with while too)
for i in range(10):
if i == 2:
continue # skip 2
...

=== Functions ===

No return value: (None)
x = print('')
# x is 'None'

Note: specifying no return value is the same as this:
return None

No Operation - 'pass':
def noop():
pass

Function:
<pre>
def area(radius):
""" This is a doc string """ # can be " ... " also...
import math
return math.pi * radius ** 2
</pre>

Function Doc string:
print(area.__doc__)
foo.func_doc

Function name:
foo.func_name

Local variable:
name = "old"
def cname():
name = "new"
print(name) # is still 'old'

Global variable:
name = "old"
def cname():
global name
name = "new"
cname()
print(name) # is now 'new'

By convention starting point is: (not required)
def main():

Parameters: (with default)
def foo(x, y, z = "default"):

Note: don't use mutable for defaults, as the object is remembered across runs! (unless that is what you want)
<pre>
# each call to t() will increase list
def t(foo=[]):
foo.append('1')
print(foo)
</pre>

<pre>
# fix mutable issue
def named_param(a, foo=None):
foo = foo or []
if not foo:
foo.append(a)
</pre>

Parameters by keyword:
def foo(x = '1', y = '2', z = '3'):
...
foo(y = 'a', x = 'b') # order is not important with keywords

Trick for optional parameter:
def do(x = None):
if x == None:

Function tests - functions are callable
callable(myfunc)

=== Type Hints ===

: "Note The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers, IDEs, linters, etc."

def surface_area_of_cube(edge_length: float) -> str:
return f"The surface area of the cube is {6 * edge_length ** 2}."

Without the hints:
def surface_area_of_cube(edge_length):
...

ref: https://docs.python.org/3/library/typing.html

=== Pass ===

Pass is a null operation (noop):

Do nothing, noop, take no action:
pass

Great for creating minimal classes:
class MyEmptyClass:
pass

Greate for making stub functions/methods:
def initlog(*args):
pass # remember to implement this!

=== Slice ===

Take take a slice of strings, lists, tuples, etc...
a = [0,1,2,3,4,5]
a[0] # 0 - not a slice
a[-1] # 5 - not a slice
a[0:1] # [0]
a[1:3] # [1, 2]
a[:2] # [0, 1]
a[:-1] # [0, 1, 2, 3, 4] - all but last
a[2:] # [2, 3, 4, 5]
a[::2] # [0, 2, 4] - stride
a[::-1] # [5, 4, 3, 2, 1, 0] - reverse
range(0,10)[::2] # [0, 2, 4, 6, 8] - stride

a = '012345'
a[1:3] # '12'
a[::2] # '024'

=== [http://docs.python.org/reference/compound_stmts.html Compound Statements] ===

Compound statements: (generally discouraged)
fun1(); fun2(); fun3()

Compound with statements: [http://docs.python.org/reference/compound_stmts.html#the-with-statement]
with A() as a, B() as b: ...

#is equivalent to
with A() as a:
with B() as b:
...

[http://www.python.org/dev/peps/pep-0343 PEP 0343] - The “with” statement

f = open('file')
with f: ...

Try: (quick ignore all exceptions, very dangerous)
try: my_broken_function()
except: pass

=== Passing Command Line Arguments ===

Parameters are stored in the 'sys.argv' variable:
import sys
print( sys.argv[0] ) # name of script
print( sys.argv[1] ) # first argument
for arg in sys.argv: print( arg ) # list all arguments

Print script name:
os.path.basename(sys.argv[0])

Print path to script:
print sys.path[0]

Paths: [http://code.activestate.com/recipes/474083-get-the-path-of-the-currently-executing-python-scr/]
<pre>
import os,sys
print "CWD: ",os.getcwd()
print "Script: ",sys.argv[0]
print ".EXE: ",os.path.dirname(sys.executable)
print "Script dir: ", os.path.realpath(os.path.dirname(sys.argv[0]))
pathname, scriptname = os.path.split(sys.argv[0])
print "Relative script dir: ",pathname
print "Script dir: ", os.path.abspath(pathname)
</pre>

os.path.realpath(__file__)

print 'sys.argv[0] =', sys.argv[0]
pathname = os.path.dirname(sys.argv[0])
print 'path =', pathname
print 'full path =', os.path.abspath(pathname)

=== Environment Variables ===

import os
a = os.environ.get('MYVAR') # get environment var
a = os.environ.get('MYVAR','default') # get environment var
a = os.environ['MYVAR'] # get environment var
os.environ['MYVAR'] = 'something' # set environment var
os.environ.update({'MYVAR':'something'}) # set environment var

for env in os.environ:
print "%s = %s" % (env, os.environ[env])

with open("environment.txt", "w") as f:
for env in os.environ:
f.write("%s = %s\n" % (env, os.environ[env]))

=== PYTHONPATH ===

Environment Search path:
PYTHONPATH=...

From Code:
sys.path.append('...')

Get site-package directory from shell: [http://stackoverflow.com/questions/122327/how-do-i-find-the-location-of-my-python-site-packages-directory]
python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

=== Import and Modules ===

Import and Modules - http://docs.python.org/tutorial/modules.html

Import module:
import package # standard import
import package.code # nested name space
from package import code # import specifics from module
import module # standard import
from math import sin # import specifics from module
from module import myfunc as fun1 # alias function/object
import math as other_math # alias module
import longname as ln # alias module
from module import * # BAD BAD BAD!

import math
math.pow(x,y)

Group and organize by:
* stdlib libraries
* local libraries
* 3rd party libraries

Remove import:
del math

from math import *
pow(x,y)

from math import pow
pow(x,y)

import math as math_lib # rename
math_lib.pow(x,y)

List functions in module:
import math
dir(math)

List all built in functions:
dir(__builtins__)

Note: Modules have the extension .py

Simply module Example:

# mymod.py:
def hello():
""" this is a doc string """
print "Hello World"

# Use the module:
import mymod
mymod.hello():

# Use everything in the the module, except for names beginning with '_': (bad practice!)
from mymod import *
hello()

# Use specific functions in the module:
from mymod import hello, hi
hello()

# Function assignment:
h = mymod.hello
h()

# module name
mymod.__name__ # mymod
__name__ # used inside of module will print current module or "__main__" if directly called

Note: careful when building own modules, as the import literally imports and executes the code, so any code not in functions is executed!

Check if being called directly or indirectly:
if __name__ == "__main__":
# execute tests...

Standard Script Setup:
<pre>
def main(args):
# do logic
if __name__ == "__main__":
sys.exit(main(sys.argv))
</pre>

Easter egg:
# The Zen of Python
import this

Search path:
* environment variable 'PYTHONPATH' (on Unix, this is usually .:/usr/local/lib/python)
* sys.path (list of paths, initialized to PYTHONPATH)
** sys.path.append('path')

Module folder: [http://stackoverflow.com/questions/391879/organising-my-python-project] [http://stackoverflow.com/questions/279237/python-import-a-module-from-a-folder] [http://stackoverflow.com/questions/72852/how-to-do-relative-imports-in-python]
/kenlib/kenmod.py
# so you can do "import kenlib.kenmod"
# create file '__init__.py' in kenlib/

Check if package exists:
<pre>
try:
import argparse
except ImportError:
print >> sys.stderr, "Please install argparse from http://code.google.com/p/
argparse."
exit(1)
</pre>

=== ifmain ===

if __name__ == '__main__':
sys.exit(main(sys.argv))

if __name__ == '__main__':
sys.exit(main(sys.argv[1:]) or 0)

=== Lists ===

* Lists can be modified (mutable)
* Tuples are constant (immutable)

List: (mutable or modifiable)
x = [] # empty list
x = [5] # singleton list

s = list("Hello World")
print(s) # s = ['H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd']
s[0] = 'X'

s.append(x) # append x
s.insert(i, x) # insert x at i
del(s[i]) # delete item at i
s.sort() # sort (modify in place!)
s.reverse() # reverse
s.index(x) # index of x in s
new = old[:] # create copy of list
new = list(old) # create copy of list

Check if item in list:
if x in s: ...

Filter list of strings based on contents: [https://stackoverflow.com/questions/2152898/filtering-a-list-of-strings-based-on-contents]
res = [k for k in lst if 'ab' in k]

List comprehension
[n * n for n in range(1, 11)] # squares of numbers 1 to 10
[c for c in 'pizza'] # list of string 'pizza'
[c.upper() for c in 'pizza'] # list of string 'pizza' uppercase
[n for n in nums if n > 0] # filters out only positive numbers in nums list
[fname for fname in os.listdir(path) # on multiple lines
if os.path.isfile(fname)
if fname.endswith('.py')]

Enumeration
for index, value in enumerate(mylist): print index, value

Length:
len(x)

Concatenation:
x + s
x * 2

Convert list to string:
"".join(mylist) # no separator
" ".join(mylist) # space separator
",".join(mylist) # comma separator

Test Membership
x in s

Multi dimensional list:
mylist = [ [0,"hi"], [1,"bye"] ]
print( mylist[0][1] )

Sorting: (see lambda)
data.sort() # alpha numerica sort
data.reverse() # alpha numerica sort

<pre>
# custom sort
data = [1, 5, 3, 9]
def mylistcmp(x, y):
if x == y: return 0
if x < y:
return -1
else: # swap 1 and -1 here to reverse
return 1

data.sort(cmp=mylistcmp)
</pre>

Slices:
lst = ['1a', '2b', '3c', '4d', '5e', '6f']
lst[0] # '1a'
lst[-1] # '6f' last item
lst[0:2] # ['1a', '2b']
lst[0:2] = ['11', '22'] # lst = ['11', '22', '3c', '4d', '5e', '6f'] - assignment to slice
lst[0:2] = [] # remove items
lst[0:0] = ['1a', '2b'] # insert items (at beginning)
lst[-1:-1] = ['7g', '8h'] # insert items (at item BEFORE last), not really what you want, use append()
lst[-1:] = ['7g', '8h'] # replace last item
lst[:] = [] # clear entire list
del(lst[0:2]) # delete slice items

Note: it is not safe to modify a sequence while being iterated over. Create a copy first:
for x in mylst[:]: # make a slice copy of the entire list
if len(x) > 6: a.insert(0,x)

Make list from tuple:
list( range(10) )

Merge multiple lists: [http://www.testingreflections.com/node/view/4930]
<pre>
def merge(seq):
merged = []
for s in seq:
for x in s:
merged.append(x)
return merged

# sample usage:
foo = [['a', 'b'],['c'],['d', 'e', 'f']]
print merge(foo)
>>>['a', 'b', 'c', 'd', 'e', 'f']
</pre>

=== Tuple ===

Tuple: (immutable or constant)
x = () # empty tuple
x = (5,) # singleton tuple
x = (5) # NOTE: integer with parenthesis, not tuple

s = tuple("Hello World")
print(s) # s = ('H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd')
# ERROR: s[0] = 'X'

Make list from tuple:
list( range(10) )

Use in conditionals:
if answer in ('y', 'ye', 'yes'): ...

Convert tuple string to tuple: [http://stackoverflow.com/questions/3945856/converting-string-to-tuple-and-adding-to-tuple]
x = "(1,2,3)"
t = tuple(int(v) for v in re.findall("[0-9]+", x))

Another Convert tuple string to tuple: [http://stackoverflow.com/questions/3945856/converting-string-to-tuple-and-adding-to-tuple]
import ast
ast.literal_eval("(1,2,3,4)") # (1,2,3,4)

=== Named Tuple ===

What are "named tuples" in Python? - Stack Overflow - http://stackoverflow.com/questions/2970608/what-are-named-tuples-in-python

<pre>
from collections import namedtuple
Point = namedtuple('Point', 'x y')
pt1 = Point(1.0, 5.0)
pt2 = Point(2.5, 1.5)
print p1 # point(x=1, y=5)
print p1.x # 1.0
print p1[0] # 1.0
</pre>

=== Dictionary ===

NOTE: Dictionaries are also known as associative arrays, maps or hash tables.

Dictionary: (mutable or modifiable)

color = {} # empty dictionary
color = {'red' : 1, 'blue', 2}
print( color['red'] ) # 1
print( color.get('red') ) # 1
print( color.get('red', 'default') ) # red if found, or return 'default'
color['green'] = 3 # able to add new item on the fly!
'red' in color # True - check if index exists
del(color["red"]) # delete item
len(color) # 2 - count of items

List in dictionary
<pre>
d.setdefault(key,[]).append(member) # tricky!

# more tricky: set default type as list
from from collections import defaultdict
d = defaultdict(list)
d[key].append(member)
</pre>

Enumeration:
for index, key in enumerate(mylist): print index, key

functions:
d.clear()
d.copy()
d.get(key)
d.get(key, default_val)
d.update(e) # update with (key, value) pairs in e
d.setdefault(key, value) # only update if not exist, and return set value

Views: (adjust on the fly as dictionary changes)
d.items() (key, values)
for k, v in d.items(): print(k, v)

d.keys()
for k in d.keys(): print(k)

d.values()
for v in d.values(): print(v)

Sorting: (see also lambda)
<pre>
# custom sort of dictionaries in list
data = [dict(number=x) for x in '036149'] # create list of dictionaries
def mydictcmp(x, y):
if x['number'] == y['number']: return 0
if x['number'] < y['number']:
return -1
else: # swap 1 and -1 here to reverse
return 1

# data.sort(cmp=mydictcmp) # modifies data

new_data = sorted(data, cmp=mydictcmp) # better solution
</pre>

Sorting:
<pre>
my_dict = {'a':1, 'b':4, 'c':3, 'd':9}
my_dict_keys = my_dict.keys()

def my_dict_sort(x, y):
if my_dict[x] == my_dict[y]: return 0
if my_dict[x] < my_dict[y]: return -1
if my_dict[x] > my_dict[y]: return 1

#sorted_my_dict = sorted(my_dict_keys, cmp=my_dict_sort, reverse=True)
sorted_tuple = sorted(my_dict_keys, cmp=my_dict_sort)
print "Sorted Dictionary:"
for key in sorted_tuple:
print "%s: %d" % (key, my_dict[key])

# a: 1, c: 3, b: 4, d: 9,
# reverse: d: 9, b: 4, c: 3, a: 1,
</pre>

Sort and keys:
keys = sorted(mydict.keys())

Sort by value: [http://stackoverflow.com/questions/613183/python-sort-a-dictionary-by-value]
import operator
x = {1: 2, 3: 4, 4:3, 2:1, 0:0}
sorted_x = sorted(x.iteritems(), key=operator.itemgetter(1))

Sort options:
# LIST.sort(cmp=None, key=None, reverse=False) -- stable sort *IN PLACE*;
# sorted(iterable, cmp=None, key=None, reverse=False)

Another: (works well!)
sorted(my_dict.items(), key=lambda x: x[1])

See: http://wiki.python.org/moin/HowTo/Sorting/

=== Sets ===

Sets are special lists with unique values (good for removing duplicates)
lst = [1, 1, 6, 8, 1, 5]
s = set(lst)
print(s) # set([8, 1, 5, 6])
s.add(item)
item in s # true
s.remove(item)

Union (return combination of all):
s.union(t)

Intersection (return matching):
s.intersection(t)

Symmetric Difference (return only items not matching either, inverse of intersection):
s.symmetric_difference(t)
list( set(mylist).symmetric_difference(set(mylist2)) )

<pre>
Operation Equivalent Result
len(s) cardinality of set s
x in s test x for membership in s
x not in s test x for non-membership in s
s.issubset(t) s <= t test whether every element in s is in t
s.issuperset(t) s >= t test whether every element in t is in s
s.union(t) s | t new set with elements from both s and t
s.intersection(t) s & t new set with elements common to s and t
s.difference(t) s - t new set with elements in s but not in t
s.symmetric_difference(t) s ^ t new set with elements in either s or t but not both
s.copy() new set with a shallow copy of s
</pre>

References:
* sets — Unordered collections of unique elements - http://docs.python.org/library/sets.html

=== Files ===

Open file, get file descriptor:
fd = open('/dev/null')

File Modes:
'r' read (default)
'w' write (truncate and create if needed)
'a' append (create if needed)

'b' binary
't' text (default)

'+' read and write

Common Functions: (fd)

f.read() # read whole file to string
f.read(n) # read n bytes from file to string
f.readline() # read single line as string (while loop)
f.readlines() # read all lines to list

f.write(str) # write string to file (does not auto include new line)
f.writelines(list) # write list to file (does not auto include new lines)

f.seek(0) # seek to start of file
f.seek(pos) # seek to position
f.truncate() # truncate file

f.close() # close file (fd)
f.closed # True/False flag that reports if file is closed

Help:
help(open('/dev/null'))
pydoc file

Read Text File Line by Line
f = open(fname, 'r') # ('r' optional)
for line in f: # or f.readlines()
print(line)
f.close() # optional

Write list to file (with new lines):
f.writelines('\n'.join(mylist))

One liner read whole file:
print( open( fname, 'r' ).read() )

Write text file:
# f = open(fname, 'a') # append text file
f = open(fname, 'w')
f.write("Hello\n")
f.write( str(10) ) # convert numbers to strings first
f.writelines(["line one\n", "line two\n"])
f.close

Read binary file:
def is_gif(fname):
f = open(fname, 'br') # not sure 'b' is correct
first4 = tuple(f.read(4))
return first4 == (0x47, 0x49, 0x46, 0x38)

Read all characters of file:
c = f.read(1)
while c:
# if c == ... logic
c = f.read(1)

Reading file in while loop: [http://ubuntuforums.org/showthread.php?t=1092300]
<pre>
f=open("file")
while True:
line=f.readline()
if not line: break
print line
f.close()
</pre>

with - implicit close - [http://www.python.org/dev/peps/pep-0343/ PEP 343 -- The "with" Statement]: [http://docs.python.org/reference/compound_stmts.html#the-with-statement] (python >= 2.5)
with open("/etc/passwd") as f:
# ..., f.read(), etc

With statement: (promptly close open handle at end of block)
with open(fname, 'r') as f:
for ...

<pre>
# eqivalent to:
f = open("/etc/passwd")
f.__enter__()
try:
...
finally:
f.__exit__()
</pre>

cat: (for grabbing a configuration line)
<pre>
def cat(filename):
with open(filename) as f:
return f.readline().strip()
</pre>

=== With ===

PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/

'with' structure:
<pre>
with EXPR as VAR:
BLOCK
</pre>

translates to:
<pre>
VAR = EXPR
VAR.__enter__()
try:
BLOCK
finally:
VAR.__exit__()
</pre>

Note: The "as VAR" part is optional. Good for locking:
<pre>
with locking(myLock):
BLOCK
</pre>

with - implicit close - [http://www.python.org/dev/peps/pep-0343/ PEP 343 -- The "with" Statement]: [http://docs.python.org/reference/compound_stmts.html#the-with-statement] (python >= 2.5)
with open("/etc/passwd") as f:
...

With statement: (promptly close open handle at end of block)
with open(fname, 'r') as f:
for ...

See also [[#Files]]

References:
* PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/
* PEP 340 -- Anonymous Block Statements - http://www.python.org/dev/peps/pep-0340/

=== Try Exception Handling ===

Exception Handling - http://docs.python.org/release/2.6.6/tutorial/errors.html

throw execption:
raise [EXCEPTION]
raise IOError("This is a test!")

Catch exceptions:
try:
except ( [EXCEPTION], [EXCEPTION] ):
except [EXCEPTION]:
except [EXCEPTION] as err: # use "[EXCEPTION], err" in python < 2.6
print("Error: {}".format(err))
except:
raise # reraise caught error that is not handled
else: # if no exception caught
finally: # always executed

Note: Usually a good idea to re-raise exception if you don't handle it (just 'raise')

General Exception: (all exceptions inherit from Exception)
try:
raise Exception('spam', 'eggs')
except Exception as e:
pass

Custom Exception Class:
<pre>
class MyError(Exception):
"""My Error Exception"""

class MyException(Exception):
pass

rasie MyError("My Error")
</pre>

<pre>
class ValidationError(Exception):
def __init__(self, message, Errors):

# Call the base class constructor with the parameters it needs
Exception.__init__(self, message)

# Now for your custom code...
self.Errors = Errors
def __str__(self):
return "MyErrors - %s" % self.Errors
</pre>

Rename standard exception:
<pre>
class Failure(StandardError):
"""Exception for unexpected failures."""
</pre>

Note: a good place to hide the exception classes in the __init__.py file.

==== exception line number ====

Python When I catch an exception, how do I get the type, file, and line number? - Stack Overflow [http://stackoverflow.com/questions/1278705/python-when-i-catch-an-exception-how-do-i-get-the-type-file-and-line-number]
<pre>
import sys, os

try:
raise NotImplementedError("No error")
except Exception as e:
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print(exc_type, fname, exc_tb.tb_lineno)
</pre>

==== traceback ====

<pre>
import traceback
try:
...
except Exception as E:
...
print 'Traceback: %s', traceback.format_exc()
</pre>

Catching multiple exceptions and getting properties from the exception:
<pre>
import traceback
import sys

class MyError(Exception):
"""My Error Exception"""

try:
#raise ValueError("a value error") # <type 'exceptions.ValueError'>
raise MyError("a my error") # <class '__main__.MyError'>
raise Exception("won't be caught") # <type 'exceptions.Exception'>
except (ValueError, MyError) as e:
print type(e) # <class '__main__.MyError'>
print e # __str__() = a my error
msg = str(e) # msg = 'a my error'
print e.__doc__ # My Error Exception
print e.__str__() # a my error
print e.__repr__() # MyError('a my error',)
print e.args # ('a my error',)
x = e # __getitem__() allows args to be unpacked directly
# x, y = e # if two args were passed, use len(e.args) to see count

# manually print stack trace (import traceback, sys)
print "Exception in user code:"
print '-'*60
traceback.print_exc(file=sys.stdout)
# stack_trace = traceback.format_exc()
print '-'*60
</pre>

=== Object Oriented Programming ===

Terms:
* encapsulates
* inheritance
* polymorphism
* method overriding

==== class ====

class animal.py:
<pre>
class Animal(object): # default "(object)" not required
" Animal class "
def __init__(self, name):
self.name = name # name is private variable
self.__iam = "animal" # __iam is private variable
def __str__(self): # string representation
return "name = '%s'" % (self.name)
def __repr__(self): # object representation (just typing instance in IDLE, or repr())
return 'Animal(%s)' % str(self) # returns __str__
def speak(self, msg='Nothing to say'):
print(msg)
</pre>

Note: if no __str__ defined, but __repr__ is, then: __str__ = __repr__

Usage:
<pre>
from animal import Animal
a = Animal('good')
print(a.__doc__) # Animal class
print(a) # name = 'good'
a # Animal(name = 'good')
print(a.name) # good
a.speak('hi') # hi
print(a._Animal__iam) # animal - access to private variable (shouldn't do)
</pre>

subclass dog.py:
<pre>
from animal import Animal
class Dog(Animal):
" Dog class "
def __init__(self, name):
self.name = name # name is private variable
self.__iam = "dog" # __iam is private variable
# NOTICE: no __str__, inherited from super class
def __repr__(self): # override to reflect "dog"
return 'Dog(%s)' % str(self) # returns __str__
def speak(self, msg='Nothing to say'): # override method
super(Dog, self).speak('Dog says: ' + msg) # call super class method
</pre>

Subclass usage:
<pre>
from dog import Dog
d = Dog('bad')
print(d.__doc__) # Dog class
print(d) # name = 'bad'
d # Dog(name = 'bad')
print(d.name) # bad
d.speak('hi') # Dog says: hi
print(d._Dog__iam) # dog - access to private variable (shouldn't do)
</pre>

Super: [http://stackoverflow.com/questions/576169/understanding-python-super-with-init-methods]
# python 3.0
super().__init__()

# python 2.7
super(self.__class__, self).__init__()
super([CLASSNAME], self).__init__()

<pre>
class Rectangle(Polygon):
def __init__(self, id, width, height):
# super(Rectangle, self).__init__(id)
super(self.__class__, self).__init__(id)
self.shape = (width, height)
</pre>

Get class name:
self.__class__.__name__
obj.__class__.__name__

Full name:
obj.__module__ + "." + obj.__class__.__name__

Private variables: (uses name mangling)
self.__age
# direct access: _classname__varname (eg. p._Person__age = 44)

Tests:
isinstance(myobj, myclass) # check if myobj is a class or subclass of myclass
issubclass(A, B) # check if A is subclass of B
issubclass(Exception, BaseException) # True

Multiple Inheritance: (methods searched in order)
class DerviedClass(Base1, Base2, Base3):
pass

Standard startup script:
class MyClass():
...
@classmethod
def main(cls, argv=None):
...
# return cls() # creator
if __name__ == "__main__";
sys.exit(MyClass.main(sys.argv))

Raise an error if a method is not overridden (not implemented)
def _get_dist(self):
"""Find linux disto. Implemented in subclasses."""
raise NotImplementedError

==== Multiple Constructors ====

<pre>
class Animal(object):

def __init__(self, name):
self.name = name

@classmethod
def create_fido(cls):
return cls("fido")
</pre>

==== Descriptor Classes ====

"In general, a descriptor is an object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol: __get__(), __set__(), and __delete__(). If any of those methods are defined for an object, it is said to be a descriptor." [http://docs.python.org/2/reference/datamodel.html#implementing-descriptors]

<pre>
object.__get__(self, instance, owner)

Called to get the attribute of the owner class (class attribute access) or of an instance of that
class (instance attribute access). owner is always the owner class, while instance is the instance
that the attribute was accessed through, or None when the attribute is accessed through the owner.
This method should return the (computed) attribute value or raise an AttributeError exception.

object.__set__(self, instance, value)

Called to set the attribute on an instance instance of the owner class to a new value, value.
</pre>

Example: [http://stackoverflow.com/questions/3798835/understanding-get-and-set-and-python-descriptors]
<pre>
class Celsius(object):
def __init__(self, value=0.0):
self.value = float(value)
def __get__(self, instance, owner):
return self.value
def __set__(self, instance, value):
self.value = float(value)

class Temperature(object):
celsius = Celsius()

t = Temperature()
t.celsius = 5 # calls Celsius.__set__
print t.celsius # calls Celsius.__get__
</pre>

Note: Also appears in 'pydoc' under "Data descriptors defined here" with object's docstring. Trick: the __doc__ can be overridden in the __init__ method, which makes for dynamic docstrings!

References:
* Descriptor HowTo Guide — Python v2.7.5 documentation - http://docs.python.org/2/howto/descriptor.html
* 3.4.2.2. Implementing Descriptors¶ - 3. Data model — Python v2.7.5 documentation - http://docs.python.org/2/reference/datamodel.html#implementing-descriptors

=== [http://docs.python.org/reference/simple_stmts.html#the-exec-statement exec code] ===

DANGEROUS!

Execute code in string:
exec( 'print("hi")' )

Example of executing a startup script for IDLE:
if os.path.isfile('.pythonrc.py'): exec(open('.pythonrc.py').read())

import os
filename = os.environ.get('PYTHONSTARTUP')
if filename and os.path.isfile(filename):
exec(open(filename).read())

=== eval ===

VERY DANGEROUS

Will execute a string as though it were pure python code!

eval("print 'hello'")

=== Common Header Format ===

<pre>
__author__ = "software team"
__date__ ="$Feb 15, 2011 1:42:17 PM$"
__version__ = "1.1.0.441 bandelier@66d9fe1edf3e"
</pre>

<pre>
Next should be authorship information. This information should follow this format:

__author__ = "Rob Knight, Gavin Huttley, and Peter Maxwell"
__copyright__ = "Copyright 2007, The Cogent Project"
__credits__ = ["Rob Knight", "Peter Maxwell", "Gavin Huttley",
"Matthew Wakefield"]
__license__ = "GPL"
__version__ = "1.0.1"
__maintainer__ = "Rob Knight"
__email__ = "rob@spot.colorado.edu"
__status__ = "Production"
</pre>
Source: Python Coding Guidelines - http://bayes.colorado.edu/PythonGuidelines.html

These will show up in the help() like such:
<pre>
Help on module test:

NAME
test

FILE
/home/kenneth/test.py

FUNCTIONS
test()

DATA
__author__ = 'test author'
__version__ = '1.0'

VERSION
1.0

AUTHOR
test author
</pre>

References:
* Python: What is the common header format? - Stack Overflow - http://stackoverflow.com/questions/1523427/python-what-is-the-common-header-format

=== Common Environment Variables ===

Command line and environment — Python v2.7.2 documentation - http://docs.python.org/using/cmdline.htm

PYTHONUNBUFFERED
: If this is set to a non-empty string it is equivalent to specifying the -u option. [http://docs.python.org/using/cmdline.html#envvar-PYTHONUNBUFFERED]
: -u - Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
* See also stdout - Python output buffering - Stack Overflow - http://stackoverflow.com/questions/107705/python-output-buffering
export PYTHONUNBUFFERED=1
set PYTHONUNBUFFERED=1

PYTHONVERBOSE
:If this is set to a non-empty string it is equivalent to specifying the -v option. If set to an integer, it is equivalent to specifying -v multiple times.
: -v - Print a message each time a module is initialized, showing the place (filename or built-in module) from which it is loaded.
export PYTHONVERBOSE=1
set PYTHONVERBOSE=1

PYTHONPATH
: Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored. [http://docs.python.org/using/cmdline.html#envvar-PYTHONPATH]
export PYTHONPATH=/path1/:path2/
set PYTHONPATH=C:\path1\;C:\path2\

=== Queue ===

Good for threading.

"The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics." [http://docs.python.org/2/library/queue.html]

<pre>
import Queue

bucket = Queue.Queue()
bucket.put("something")

while True:
try:
item = bucket.get(block=False)
except Queue.Empty:
pass
else:
# do stuff with queue
</pre>

Important methods:
Queue.qsize()
Queue.empty()
Queue.put(item[, block[, timeout]])
Queue.get([block[, timeout]])

-

Example: http://docs.python.org/2/library/queue.html
<pre>
def worker():
while True:
item = q.get()
do_work(item)
q.task_done()

q = Queue()
for i in range(num_worker_threads):
t = Thread(target=worker)
t.daemon = True
t.start()

for item in source():
q.put(item)

q.join() # block until all tasks are done
</pre>

-

"The module implements three types of queue, which differ only in the order in which the entries are retrieved. In a FIFO queue, the first tasks added are the first retrieved. In a LIFO queue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the heapq module) and the lowest valued entry is retrieved first."

Queue Types:
class Queue.Queue(maxsize=0) # FIFO Queue
class Queue.LifoQueue(maxsize=0) # LIFO Queue
class Queue.PriorityQueue(maxsize=0) # Priority Queue

References:
* 8.10. Queue — A synchronized queue class — Python v2.7.5 documentation - http://docs.python.org/2/library/queue.html

== Recipes ==

=== Remove Numbers from String ===

Using list comprehension: [http://stackoverflow.com/questions/12851791/removing-numbers-from-string]
<pre>
result = ''.join([i for i in s if not i.isdigit()])
</pre>

Using pattern matching:
<pre>
>>> re.compile(r'\D+').findall("test 1234 test 1234")
['test ', ' test ']

>>> re.compile(r"\d").sub("", "test 1234 test 1234")
'test test '
</pre>

=== Retry Wrapper ===

<pre>
import functool
class SomethingRetry:

def retry_with_backoff(retries: int, base_delay: float, exceptions: tuple[type[Exception], ...] = (Exception,)) -> callable:
"""
Retries a function with exponential backoff, doubling on each attempt.
Arguments:
retries: Number of retry attempts.
base_delay: Initial delay between retries in seconds.
exceptions: Tuple of exceptions to catch and retry.
"""

def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(1, retries + 1):
try:
return func(*args, **kwargs)
except exceptions as e:
print(f"Warning: Attempt {attempt} failed with error: {e}")
if attempt == retries:
print("Error: All retry attempts failed.")
raise
sleep_time = base_delay * (2 ** (attempt - 1))
print(f"Notice: Retrying in {sleep_time} seconds...")
time.sleep(sleep_time)
return wrapper
return decorator

@retry_with_backoff(retries=6, base_delay=10, exceptions=(Exception,))
def do_something(self, sample_param: str):
</pre>

== Standard Libraries ==

=== user input ===

<pre>
line = raw_input() # Python 2
# line = input() # Python 2
line = input('What is your name: ')
print(": " + line.strip() + " :")
</pre>

=== sys - stdout stderr stdin ===

STDOUT:
import sys
sys.stdout.write("Hello ") # does not write new line
sys.stdout.write("World\n")

STDERR:
import sys
sys.stderr.write("ERROR!\n")

STDIN:
import sys
line = sys.stdin.readline()

line = input() # use raw_input() in Python 2
line = input('What is your name: ')
print(": " + line.strip() + " :")

Read piped stdin line by line: [http://stackoverflow.com/questions/1450393/how-do-you-read-from-stdin-in-python]
# cat /etc/passwd | ./readlines.py # linux
# type file.txt | python readlines.py # windows

# fileinput.input() - returns fileinput object which is iterative
import fileinput
for line in fileinput.input():
print(": " + line + " :")

# sys.stdin.readlines() - returns list of strings
import sys
for line in sys.stdin.readlines():
print(": " + line + " :")

# sys.stdin - returns object which is iterative
import sys
for line in sys.stdin:
print(": " + line + " :")

# ugly while loop: [http://ubuntuforums.org/showthread.php?t=1092300]
while True:
try:
s = raw_input("Say something ")
except EOFError:
break
print ": %s" % s

Check if stdin has any waiting data: [http://stackoverflow.com/questions/3762881/how-do-i-check-if-stdin-has-some-data]
<pre>
import sys
import select

# if select.select([sys.stdin,],[],[],0.0)[0]: # alternative old method

if not sys.stdin.isatty():
print "Have data!"
print sys.stdin.readlines()
else:
print "No data"
</pre>

<pre>
if len(sys.argv) > 1:
message = ' '.join(sys.argv[1:])
elif not sys.stdin.isatty():
# read from pipe in
message = sys.stdin.readline()
else:
# read from user
sys.stdout.write("Message: ")
message = raw_input().strip()
</pre>

=== [http://docs.python.org/library/re.html Regular Expressions] ===

Import:
import re # user regular expressions

Match vs Search:
re.match("cde", "abcdef") # no match (match searches beginning of string, first match)
re.match("abc", "abcdef") # match (match searches beginning of string, first match)
re.search("cd", "abcdef") # match (matches any location, first match)

Both re.match() and re.search() return objects that are not intuitive to use. You can check the object's group() method for what matched, but I prefer re.findall().

Match:
s = 'done'
if re.match('done|quit', s) != None # True

Search:
s = 'done'
if re.search('do', s) != None # True

Replace:
msg = 'jello world'
msg = re.sub('jello', 'hello', msg)

For case intensive searches append 3rd parameter "re.I".

Find all:
re.findall("[a-z]+", "aaa111bbb222") # ['aaa', 'bbb']
re.findall("111([a-z]+)", "aaa111bbb222") # ['bbb']
re.findall("hello", "HELLO", re.I) # ['HELLO'] - Case Insensitive

Split:
>>> re.split('[a-z]+','111aaa222bbb')
['111', '222', '']

>>> re.split('([a-z]+)','111aaa222bbb') # parenthesis cause return of all
['111', 'aaa', '222', 'bbb', '']

Replacement Substitution:
# re.sub(pattern, repl, string[, count, flags])
re.sub(r'\\', r'/', r'\\data\files')

Compiled regular express for repeated use: (performance)
valid = re.compile(r"[0-9]+")
valid.match("aaa111bbb") # not valid
match = valid.search("aaa111bbb222") # valid
print(match.group()) # what matched: '111'

=== Date and Time ===

Sleep in seconds:
import time
time.sleep(seconds)

Use the following functions to convert between time representations:
<table border="1">
<tr><th>From</th><th>To</th><th>Use</th></tr>
<tr><td>seconds since the epoch</td><td>struct_time in UTC</td><td>gmtime()</td></tr>
<tr><td>seconds since the epoch</td><td>struct_time in local time</td><td>localtime()</td></tr>
<tr><td>struct_time in UTC</td><td>seconds since the epoch</td><td>calendar.timegm()</td></tr>
<tr><td>struct_time in local time</td><td>seconds since the epoch</td><td>mktime()</td></tr>
</table>

Date formats:
<pre>
>>> time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
'2014-04-04 12:22:43'

>>> time.time()
1396635763.0581281

>>> datetime.datetime.now()
datetime.datetime(2014, 4, 4, 12, 22, 43, 58456)
# datetime(year, month, day, hour, minute, second, microsecond)

>>> time.localtime()
time.struct_time(tm_year=2014, tm_mon=4, tm_mday=4, tm_hour=12, tm_min=22, tm_sec=43, tm_wday=4, tm_yday=94, tm_isdst=1)
</pre>

Current epoch time:
int(time.time())

Time zone offset:
standard_tz_offset = - time.timezone / 3600 # 7
daylight_tz_offset = - time.altzone / 3600 # 6

Is daylight saving time: [http://stackoverflow.com/questions/2881025/python-daylight-savings-time]
import time
time.localtime()
_.tm_isdst # 0 or 1

mst_epoch_time = time.time() # server in MST/MDT
if time.localtime().tm_isdst: # are we in daylight saving time?
utc_epoch_time = mst_epoch_time + time.altzone
else:
utc_epoch_time = mst_epoch_time + time.timezone

Format current time: [http://docs.python.org/2/library/time.html#time.strftime]
time.strftime('%d%b%y') # '12May11'
time.strftime('%Y.%m.%d %H:%M:%S') # '2013.08.09 14:01:28'

Time conversions:
<pre>
# epoch to str
epoch_time = time.time() # 1375992088.7867229
dt = datetime.datetime.fromtimestamp( epoch_time ) # datetime.datetime(2013, 8, 8, 14, 1, 28, 786723)
date_str = dt.strftime("%Y-%m-%d %H:%M:%S") # '2013-08-08 14:01:28'
tz_offset = - time.altzone / 3600
if tz_offset < 0:
tz_offset_str = "-%02d00" % abs(tz_offset)
else:
tz_offset_str = "+%02d00" % abs(tz_offset)
print date_str + " " + tz_offset_str # '2013-08-08 14:01:28 -0600'
</pre>

<pre>
# UTC epoch to str
gm_epoch_time = calendar.timegm(time.gmtime()) # 1375992114
dt = datetime.utcfromtimestamp( gm_epoch_time ) # datetime.datetime(2013, 8, 8, 20, 1, 54)
dt.strftime("%Y-%m-%d %H:%M:%S +0000") # '2013-08-08 20:01:54 +0000'
dt.strftime("%Y-%m-%dT%H:%M:%SZ") # '2013-08-08 20:01:54 +0000'
</pre>

<pre>
# Local time str to epoch
iso_string='2013-08-08T14:01:28'
epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S' ) ) # 1375992088.0

iso_string='2013-08-08T14:01:28 MDT'
epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S %Z' ) ) # 1375992088.0

# not sure how to specify numerical time zone and pull it in!

# UTC (Zulu) str to epoch, and back again
iso_string = '2013-08-08T20:01:54Z'
epoch_time = calendar.timegm( time.strptime( iso_string.replace('Z', 'GMT'), '%Y-%m-%dT%H:%M:%S%Z' ) )
print epoch_time # 1375992114
print "%sZ" % datetime.utcfromtimestamp(timestamp).isoformat() # 2013-08-08T20:01:54Z
</pre>

<pre>
# Convert epoch from Local to UTC
epoch_time = 1375992114
tz_offset = - time.altzone / 3600
gm_epoch_time = epoch_time - tz_offset * 3600 # 1376013714
</pre>

<pre>
# Convert epoch from UTC to Local
gm_epoch_time = 1376013714
tz_offset = - time.altzone / 3600
epoch_time = gm_epoch_time + tz_offset * 3600 # 1375992114
</pre>

struct_time:
time.localtime() # local zime zone
time.gmtime() # UTC time

today = time.localtime()
today_year = today.tm_year
today_month = today.tm_mon
today_day = today.tm_mday

time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime())

Zulu time to epoch time: [http://stackoverflow.com/questions/127803/how-to-parse-iso-formatted-date-in-python]
s = '2008-09-03T20:56:35.450686Z'
event_time_struct = time.strptime(s.split('.')[0]+ "UTC", "%Y-%m-%dT%H:%M:%S%Z")
epoch_time = calendar.timegm(event_time_struct)

Parse string to time.struct_time (like gmtime()): [http://docs.python.org/2/library/time.html#time.strptime]
time.strptime(time.ctime()) # current time - default format is ctime format: "%a %b %d %H:%M:%S %Y"
time.strptime(time.ctime(time.time())) # default format is ctime format: "%a %b %d %H:%M:%S %Y"
time.strptime("30 Nov 00", "%d %b %y")

Build from ticks: [http://www.tutorialspoint.com/python/python_date_time.htm]
localtime = time.localtime( time.time() ) # build time.struct_time from ticks
print time.asctime( localtime ) # Tue Jan 13 10:17:09 2009

dt = datetime.datetime.fromtimestamp( time.time() ) # build datetime.datetime struct from ticks
dt.strftime("%Y-%m-%d %H:%M:%S") # '2013-07-31 23:07:48'
time.strptime( time.time(), ")

<pre>
from datetime import date
now = date.today() # now = datetime.date(2003, 12,2)
now.strftime("%m-%d-%y") # '12-02-03'
# dates support calendar arthimetic
birthday = date(1964, 7, 31)
age = now - birthday
age.days # 14368
</pre>

ts = (datetime.datetime.now().strftime("%Y%m%d%H%M")) # 201105191118
tstxt = (datetime.datetime.now().strftime("%m-%d-%Y %H:%M")) # 05-19-2011 11:18

Seconds since epoch:
int(time.time()) # 1305842304

Zulu and back again: ([http://aboutsimon.com/2013/06/05/datetime-hell-time-zone-aware-to-unix-timestamp/ Datetime hell])
<pre>
import time
from calendar import timegm
from datetime import datetime

if __name__ == '__main__':
iso_string = '2013-06-05T15:19:10Z'

timestamp = timegm(
time.strptime(
iso_string.replace('Z', 'GMT'),
'%Y-%m-%dT%H:%M:%S%Z'
)
)

print iso_string
print timestamp
print '-> %sZ' % datetime.utcfromtimestamp(timestamp).isoformat()
</pre>

http://wiki.python.org/moin/TimeTransitionsImage?action=AttachFile&do=get&target=v1.png [http://wiki.python.org/moin/WorkingWithTime]

References:
* WorkingWithTime - PythonInfo Wiki - http://wiki.python.org/moin/WorkingWithTime
* Datetime hell. Time zone aware to UNIX timestamp. | About me and the things I do - http://aboutsimon.com/2013/06/05/datetime-hell-time-zone-aware-to-unix-timestamp/
* WorkingWithTime - Python Wiki - http://wiki.python.org/moin/WorkingWithTime

---

Something I cooked up...

<pre>
def utc_time(self, years_offset=60):
mst_epoch_time = time.time() # server is in MST
if time.localtime().tm_isdst: # are we daylight saving time?
utc_epoch_time = mst_epoch_time + time.altzone
else:
utc_epoch_time = mst_epoch_time + time.timezone
return utc_epoch_time

def tz_time(self, tz_offset, daylight_adjust=True):
""" return the future time based on timezone offset
@tz_offset is timezone offset, eg -7 for MST
@daylight_adjust is flag to auto adjust for daylight time"""
if daylight_adjust:
if time.localtime().tm_isdst: # are we daylight saving time?
_tz_offset = tz_offset - 1
else:
_tz_offset = tz_offset
_utc_epoch_time = self.utc_time()
_adj_time = _utc_epoch_time + _tz_offset * 60 * 60
return _adj_time

def future_time(self, tz_offset=-7, daylight_adjust=True, years_offset=60):
""" Calculate some future year """
_tz_time = self.tz_time(tz_offset)
future_utc_epoch_time = _tz_time + years_offset * 365 * 24 * 60 * 60
# account for leap years
cur_year = datetime.datetime.now().year
cur_year_leap = cur_year / 4
future_year = cur_year + years_offset
future_year_leap = future_year / 4
leap_days = future_year_leap - cur_year_leap
future_utc_epoch_time += leap_days * 24 * 60 * 60
return future_utc_epoch_time
</pre>

=== Math ===

Math Functions:
import math
math.ceil(x)
math.floor(x)
math.trunc(x)
math.pow(x,y)

abs(x) # built in

=== Random ===

import random
random.choice(['a', 'b', 'c'])
random.sample(range(100), 10)
random.random() # random float 0 < x < 1
random.randrange(3) # integer in [0, 1, 2]
random.randrange(1, 3) # integer in [1, 2]
random.randint(1, 3) # integer in [1, 2, 3]

=== Operating System Operations ===

Import OS Library:
import os

Common Functions:
# directory functions
os.getcwd() # get current directory
os.listdir(p) # get list of directory contents
os.chdir(p) # change directory
os.mkdir(p) # make directory
os.makedirs(p) # make directory and all intermediate
os.rmdir(p) # remove empty directory, see shutil.rmtree()
os.removedirs(p) # remove empty directories, see shutil.rmtree()
os.path.exists(p) # does file/directory exist?
os.path.isdir(p) # does directory exist
os.path.join(folder1, folder2, file, etc) # portable path join

# file functions
os.remove(p) # remove file
os.unlink(p) # same as os.remove()
os.path.exists(p) # does file/directory exist?
os.path.isfile(p) # does file exist
os.stat(fname) # get file stats
os.stat(fname).st_size # file size
os.link(src, dst) # create hard link
os.readlink(p) # read symbolic link
os.rename(old, new) # rename/move file or directory
os.tempnam(p) # temporary unique name for target path
os.tmpfile() # create temporary file
os.tmpnam() # temporary file name
os.path.getsize(filename) # get file size

# system functions
os.fork() # fork process
os.getpid() # get process id
os.getppid() # get parent's process id
os.kill(pid, sig) # kill a process
os.getlogin() # get user's login name

# execution
os.system(cmd) # output to stdout, return exit_status
os.popen(cmd) # returns a pipe to command

Note: to get path to script:
sys.path[0]

Get directory and file names:
os.path.dirname(path) # '/root/test.txt' -> '/root'
os.path.basename(path) # '/root/test.txt' -> 'test.txt'
os.path.splitext(path) # '/root/test.test.txt' -> ('/root/test.test', '.txt') - extract extention
filename.split(".")[-1] # '/root/test.test.txt' -> 'txt' - extract extention

Normalize path, eliminating double slashes, etc.
os.path.normpath(path) # '/root/..' -> '/' or 'test/..' -> '.'
os.path.abspath(path) # '/root/..' -> '/' or 'test/..' -> '/root/test'

List dir: (returns list)
os.listdir( os.getcwd() )

List of files:
[p for p in os.listdir(os.getcwd())
if os.path.isfile(p)]

List of folders:
[p for p in os.listdir(os.getcwd())
if os.path.isdir(p)]

Build path for local OS: (using appropriate slash)
pathstr = os.path.join("folder", "folder", "file_or_folder")

File and directory management with shutil
import shutil
shutil.copyfile('file1', 'file2')
shutil.move('file1', 'folder2/file1')

File Wildcards:
import glob
glob.glob('*.py') # return list of files

Walk directory tree: [http://stackoverflow.com/questions/120656/directory-listing-in-python]
<pre>
import os
for dirname, dirnames, filenames in os.walk('.'):
for subdirname in dirnames:
print os.path.join(dirname, subdirname)
for filename in filenames:
print os.path.join(dirname, filename)
</pre>

ugly 'deltree' (by kenneth)
<pre>
import os
def deltree(path):
files = os.listdir(path)
for file in files:
file_path = os.path.join(path, file)
if os.path.isdir(file_path):
deltree(file_path)
os.rmdir(file_path)
else:
os.remove(file_path)
</pre>

Cleaner 'deltree' (by Kep)
<pre>
import shutil
# Blow away (if needed), and reclone to proper version
if os.path.isdir("xxx"):
shutil.rmtree("xxx")
</pre>

== System Platform ==

<pre>
import platform

if platform.system() == "Linux":
# ...
if platform.system() == "Windows":
# ...
else:
# ...
</pre>

=== Temp Files ===

import tempfile
filename = tempfile.mktemp()
f = open(filename, 'w')
...
f.close()

import os
os.remove(filename) # delete temp file

=== Pickling ===

Pickling (serializing, making objects persistent):
import pickle
grades = {'bob' : [4, 5, 6],
'sue' : [3, None, 7, 7]}
outfile = open(fname, 'wb')
pickle.dump(grades, outfile)

infile = open(fname, 'rb')
grades = pickle.load(infile)

=== [http://docs.python.org/library/logging.html Logging] ===

* logging — Logging facility for Python — Python 3 documentation - https://docs.python.org/3/library/logging.html
* Logging HOWTO — Python Dev Version documentation - http://docs.python.org/dev/howto/logging.html
* Logging Cookbook — Python Dev Version documentation - http://docs.python.org/dev/howto/logging-cookbook.html

Setup logger:
import logging
logging.basicConfig(level=logging.DEBUG,format='[%(asctime)s.%(msecs)03d] %(levelname)s %(filename)s(line:%(lineno)d): %(message)s')

Log Levels:
import logging
msg = "i broke"
logging.debug(msg)
logging.info(msg)
logging.warning(msg) # WARNING:root:i broke
logging.error(msg) # ERROR:root:i broke
logging.critical(msg) # CRITICAL:root:i broke

log.setLevel(level) # debug, info, warn/warning, error, critical
log.setLevel(logging.DEBUG)

A good convention to use when naming loggers is to use a module-level logger, in each module which uses logging, named as follows: [http://docs.python.org/dev/howto/logging.html]
logger = logging.getLogger(__name__)

Simple example: [http://stackoverflow.com/questions/345991/python-no-handlers-could-be-found-for-logger-opengl-error]
<pre>
log_format='%(asctime)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
logging.basicConfig(format=log_format, datefmt=log_date_format)
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)

log.info('hi')
</pre>

Formatter function:
<pre>
def makeFormatter(format_str="", date_format="%Y-%m-%d %H:%M:%S"):
if not format_str:
format_str = "%(asctime)s | %(levelname)s | %(filename)s:%(lineno)d:%(funcName)s | %(message)s"
return logging.Formatter(fmt=format_str, datefmt=date_format)
</pre>

Medium example:
<pre>
# Create logger
log_format='%(asctime)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
log_formatter = logging.Formatter(fmt=log_format, datefmt=log_date_format)
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)

# Log to console
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(log_formatter)
log.addHandler(ch)

# Log to file
fh = logging.FileHandler('my_logger.log')
fh.setLevel(logging.DEBUG)
fh.setFormatter(log_formatter)
log.addHandler(fh)
</pre>

Log all loggers to file - attach to root logger
<pre>
import logging

logger = logging.getLogger(__name__)

# Configure Logger
log_format='%(asctime)s | %(name)s | %(levelname)s | %(message)s'
log_date_format='%Y-%m-%d %H:%M:%S'
log_formatter = logging.Formatter(fmt=log_format, datefmt=log_date_format)
logging.basicConfig(level=logging.DEBUG, format=log_format, datefmt=log_date_format)

# Setup Log to File handler, and attach to root logger
root_logger = logging.getLogger() # Calling getLogger() without a name returns the root logger
fh = logging.FileHandler('logger.txt')
fh.setLevel(logging.DEBUG)
fh.setFormatter(log_formatter)
root_logger.addHandler(fh)
</pre>

Application Setup (not basic log configuration carries across imported modules):
<pre>
import logging
def main():
logging.basicConfig(filename='myapp.log', level=logging.INFO)
logging.info('Started')
...
logging.info('Finished')

if __name__ == '__main__':
main()
</pre>

Log multiple lines:
<pre>
for line in output.splitlines():
log.debug(line)
</pre>

Set log level: (warning is default)
logging.basicConfig(level=logging.DEBUG)

Set log level from string value: [http://docs.python.org/dev/howto/logging.html]
<pre>
# assuming loglevel is bound to the string value obtained from the
# command line argument. Convert to upper case to allow the user to
# specify --log=DEBUG or --log=debug
numeric_level = getattr(logging, loglevel.upper(), None)
if not isinstance(numeric_level, int):
raise ValueError('Invalid log level: %s' % loglevel)
logging.basicConfig(level=numeric_level, ...)
</pre>

Log to file:
logging.basicConfig(filename='example.log',level=logging.DEBUG)
logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG) # do not append

Format output: (once set, can't be changed?)
# WARNING:MESSAGE
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)

# 05/23/2011 12:50:02 PM MESSAGE
logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')

# 05/23/2011 12:55:28 PM : WARNING : MESSAGE
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')

# 2011-05-23 13:05:30 : WARNING : MESSAGE
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

# 2011-05-23 13:05:30 | WARNING | FILE:FUNC | MESSAGE
logging.basicConfig(format='%(asctime)s | %(levelname)s | %(filename)s:%(funcName)s | %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

Log record attributes - http://docs.python.org/library/logging.html#logrecord-attributes

Format Example:
<pre>
FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' # doesn't work!
logging.basicConfig(format=FORMAT)
d = { 'clientip' : '192.168.0.1', 'user' : 'fbloggs' }
logger = logging.getLogger('tcpserver')
logger.warning('Protocol problem: %s', 'connection reset', extra=d)
</pre>

<pre>
import logging

# create logger with 'spam_application'
logger = logging.getLogger('spam_application')
logger.setLevel(logging.DEBUG)

# create file handler which logs even debug messages
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)

# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)

# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(fh)
logger.addHandler(ch)

logger.info('Message to loggers and handlers')
</pre>

=== Templating ===

from string import Template
t = Template('${name}'s fav dish is $food')
t.substitute(name = 'ken', food = 'beans') # throws exception if missing one, but can have more
t.safe_substitute(name = 'ken') # will fill in what it can

Change delimiter:
class MYT(Template):
delimiter = '%'
minput = input("File name format (%d - date, %s - size): ")
myt = MYT(minput)
myt.substitute(%d=date, %s=size)

=== [http://docs.python.org/library/threading.html Threading] ===

Basic thread: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import thread

def someFunc():
print "someFunc was called"

thread.start_new_thread(someFunc, ())
</pre>

More complete basic thread:
<pre>
import thread
import threading
import time

def run_thread (threadname, sleeptime):
print "Thread start: " + threadname
time.sleep(sleeptime)
print "Thread end: " + threadname
thread.interrupt_main() # throw keyboard interrupt to kill main program

if __name__ == "__main__":
thread.start_new_thread(run_thread, ("Thread1", 3))
try:
while 1:
pass
except:
print "Thread1 exited...."
</pre>

Can use a global counter instead of exception to track number of threads:
<pre>
import thread
import threading
import time

threadcount = 3

def run_thread (threadname, sleeptime):
global threadcount
print "Thread start: " + threadname
time.sleep(sleeptime)
print "Thread end: " + threadname
threadcount -= 1

if __name__ == "__main__":
thread.start_new_thread(run_thread, ("Thread1", 3))
thread.start_new_thread(run_thread, ("Thread2", 4))
thread.start_new_thread(run_thread, ("Thread3", 2))
while 1:
if threadcount == 0:
break
else:
pass
print "completed"
</pre>

---

Thread object: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import threading

t1 = threading.Thread(target=someFunc)
# with parameters: Thread(target=myfunction, args=('MyStringHere',1)).start()
t1.start()
t1.join()
</pre>

Basic threading with parameters:
<pre>
import threading
t = threading.Thread(None, someFunc, None, (param1, param2))
t.start()
t.join()
</pre>

<pre>
t = threading.Thread(target=client_recv, args=(s, name))
while True:
t.join(timeout=2)
if t.isAlive():
print "Thread is dead..."
break
</pre>

Threading as a class: [http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html]
<pre>
import threading

class FuncThread(threading.Thread):
def __init__(self, target, *args):
self._target = target
self._args = args
threading.Thread.__init__(self)

def run(self):
self._target(*self._args)

# Example usage
def someOtherFunc(data, key):
print "someOtherFunc was called : data=%s; key=%s" % (str(data), str(key))

t1 = FuncThread(someOtherFunc, [1,2], 6)
t1.start()
t1.join()
</pre>

Threading as a class #2:
<pre>
import threading
import time
import sys

class MyT(threading.Thread):
def __init__(self, name, time):
threading.Thread.__init__(self)
self.name = name
self.time = time
def run(self):
print("Thread Start: " + self.name)
for i in range(self.time):
print("[" + self.name + "]", end='')
sys.stdout.flush()
time.sleep(1)
print("Thread Stop: " + self.name)

myt1 = MyT('1', 10)
myt1.start()

myt2 = MyT('2', 15)
myt2.start()

myt3 = MyT('3', 5)
myt3.start()

myt4 = MyT('4', 10)
myt4.start()

myt1.join()
myt2.join()
myt3.join()
myt4.join()
</pre>

Locking:
<pre>
lock=thread.allocate_lock()

# pass lock to thread, and use in thread as such:
lock.acquire()
#some operation
lock.release()
</pre>

Returning data from a thread (using a [http://docs.python.org/library/queue.html queue]): [http://stackoverflow.com/questions/1886090/return-value-from-thread] [http://stackoverflow.com/questions/2829329/catch-a-threads-exception-in-the-caller-thread-in-python]
<pre>
# Pass exception back to parent
import sys
import threading
import Queue

class ExcThread(threading.Thread):
def __init__(self, bucket):
threading.Thread.__init__(self)
self.bucket = bucket
def run(self):
try:
raise Exception('An error occured here.')
except Exception:
self.bucket.put(sys.exc_info())

def main():
bucket = Queue.Queue()
thread_obj = ExcThread(bucket)
thread_obj.start()
while True:
try:
exc = bucket.get(block=False)
except Queue.Empty:
pass
else:
exc_type, exc_obj, exc_trace = exc
# deal with the exception
print "Exception Caught:", exc_type, exc_obj
print "Exception Trace:", exc_trace
thread_obj.join(0.1)
if thread_obj.isAlive():
continue
else:
break

if __name__ == '__main__':
main()
</pre>

<pre>
Exception Caught: <type 'exceptions.Exception'> An error occured here.
Exception Trace: <traceback object at 0x2b2230a285f0>
</pre>

Another Exception catching queue example: (kenneth's example)
<pre>
# thread function
def run_thread(queue, item):
...
queue.put( Exception("Failure on item:" + item) )

threads = []
queue = Queue.Queue()

# start threads
for item in items:
t = Thread(None, run_thread, None, (queue, item))
t.start()
threads.append(t)

# wait for threads
for t in threads:
t.join()

# catch and reraise exceptions
while not queue.empty():
raise(queue.get())
</pre>

Returning data from a thread (using a callback function): [http://stackoverflow.com/questions/1886090/return-value-from-thread]
:"Another approach is to pass a callback function to the thread. This gives a simple, safe and flexible way to return a value to the parent, anytime from the new thread."
<pre>
import threading
import time
import sys

class MyThread(threading.Thread):
def __init__(self, cb):
threading.Thread.__init__(self)
self.callback = cb
def run(self):
for i in range(10):
self.callback(i)
time.sleep(1)

def count(x):
print x
sys.stdout.flush()

t = MyThread(count)
t.start()
</pre>

References:
* 16.2. threading — Higher-level threading interface — Python v2.7.1 documentation - http://docs.python.org/library/threading.html
* An Introduction to Thread Programming - http://www.prasannatech.net/2008/08/introduction-to-thread-programming.html
* Python - Multithreaded Programming - http://www.tutorialspoint.com/python/python_multithreading.htm
* Running Functions as Threads in Python « Software Ramblings - http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html
* Understanding Threading in Python LG #107 - http://linuxgazette.net/107/pai.html
* multithreading - Simple threading in Python 2.6 using thread.start_new_thread() - Stack Overflow - http://stackoverflow.com/questions/849674/simple-threading-in-python-2-6-using-thread-start-new-thread
* python - Return value from thread - Stack Overflow - http://stackoverflow.com/questions/1886090/return-value-from-thread

---

Interrupt main thread:
<pre>
# source: http://stackoverflow.com/questions/4591917/python-how-to-quit-cli-when-stuck-in-blocking-raw-input

import thread
import time

def main():
try:
m = thread.start_new_thread(killable_input, tuple())
while 1:
time.sleep(0.1)
except KeyboardInterrupt:
print "exception"

def killable_input():
w = thread.start_new_thread(normal_input, tuple())
i = thread.start_new_thread(wait_sometime, tuple())

def normal_input():
s = raw_input("input:")

def wait_sometime():
time.sleep(4) # or any other condition to kill the thread
print "too slow, killing imput"
thread.interrupt_main()

if __name__ == '__main__':
main()
</pre>

=== urllib ===

See [[Python/urllib]]

=== Command line arguments ===

<pre>
import sys
for x in sys.argv:
print x
</pre>

=== Execute other python script ===

import sys
sys.argv = ['', 'my', 'args', 'here']
execfile('myscript.py') # does not support arguments directly

=== Exit Code ===

sys.exit(n) http://docs.python.org/library/sys.html:
import sys
sys.exit(1) # exit code of 1 (default is 0)
sys.exit("some error") # exit code of 1, and message displayed

=== [http://docs.python.org/library/syslog.html syslog] ===

<pre>
import syslog
syslog.openlog('[WATER]')
syslog.syslog('Processing started')
if error:
syslog.syslog(syslog.LOG_ERR, 'Processing started')
</pre>

<pre>
import syslog
syslog.openlog('[WATER]')
def log(msg):
syslog.syslog(str(msg))
log("Starting...")
</pre>

=== Subprocess ===

Subprocess - http://docs.python.org/library/subprocess.html

Methods of subprocess (or external command execution):
* subprocess (only method to rule/replace all others)
* os.system
* os.spawn*
* os.popen*
* popen2.*
* commands.*

My New Favorite:
<pre>
import subprocess
class Run(object):
class RC(int):
pass
def __new__(cls, cmd, *args, **kwargs):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
rc = cls.RC(p.wait())
rc.output = "\n".join(p.stdout.readlines())
rc.error = "\n".join(p.stderr.readlines())
return rc

if Run("some command"): print "do something" # on rc of not 0 (negative or positive)
</pre>

My Favorite:
<pre>
def exe(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
#output = " | ".join(p.stdout.readlines())
output = "\n".join(p.stdout.readlines())
rc = p.wait()
return (rc, output)

def exe_checked(cmd, checked=True):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
#output = " | ".join(p.stdout.readlines())
output = "\n".join(p.stdout.readlines())
rc = p.wait()
if rc and checked:
raise Exception("Command returned non zero [{0}]: {1}".format(rc, cmd)
return (rc, output)

def exe_with_stderr(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subproc
ess.PIPE)
output = " | ".join(p.stdout.readlines())
outputerr = " | ".join(p.stderr.readlines())
rc = p.wait()
return (rc, output, outputerr)
</pre>

<pre>
def exec_program(cmd):
s = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = s.communicate()
result = s.wait()
return stdout, stderr, result
</pre>

Pass along output and return true error code:
import subprocess
retcode = subprocess.call(["ls", "-l"]) # true return code, output sent to STDOUT

retcode = subprocess.call(["foo.sh","args"],shell=True) # Execute in shell, with shell processing
retcode = subprocess.call(['C:\\Temp\\a b c\\Notepad.exe'])
retcode = subprocess.call(["D:/Python24/Python.exe", "Skripta.py", "arg1", "arg2"])

Note: shell=True is equivalent to: subprocess.Popen(['/bin/sh', '-c', args[0], args[1], ...])

Process STDOUT and STDERR and get return code:
<pre>
#!/usr/bin/env python
import subprocess
import string

cmd = "cat /etc/passwd | grep ken"
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) # stderr=subprocess.STDOUT to combine

out = string.join(p.stdout.readlines())
outerr = string.join(p.stderr.readlines())
retcode = str(p.wait())

print("STDOUT: " + out)
print("STDERR: " + outerr)
print("RETCODE: " + retcode)
</pre>

Above with stderr redirected to stdout (combined):
<pre>
import subprocess
p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
print line,
retval = p.wait()
</pre>

Replacing shell backtick: [http://docs.python.org/library/subprocess.html]
# shell code:
# OUTPUT=`mycmd myarg`
from subprocess import Popen
output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]

Replacing shell pipeline [http://docs.python.org/library/subprocess.html]
# shell code:
# OUTPUT=`dmesg | grep hda`
from subprocess import Popen
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

Replacing os.system(): [http://docs.python.org/library/subprocess.html]
# old:
sts = os.system("mycmd" + " myarg")

# new:
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)[1]

Control an executed process:
<pre>
cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

# cycle while process continues alive
while p.poll() == None:

# check if input
if select.select([p.stdout.fileno()], [], [], 1)[0]:
while select.select([p.stdout.fileno()], [], [], .1)[0]:
line = p.stdout.readline()
# ... process

# write something to process
p.stdin.write("tp %s %s\n" % (user1, user2))

...

# time to force a quit: (choose which you need)
p.kill() # SIGKILL
p.terminate() # SIGTERM
os.kill(p.pid, signal.SIGTERM) # same as first, but not as clean
</pre>

==== Other Command Execution ====

Old method with os.system(): [http://docs.python.org/library/os.html]
import os
encoded_wait_code = os.system("some command") # output sent to STDOUT
# Note: returns code of 0 if command returns 0, but wait() encoded value for all other values

Quick and dirty old method (prints stderr, captures stdout)
<pre>
import os
out = string.join( os.popen(cmd).readlines() )
</pre>

Quick and dirty old method (prints stderr, captures stdout, captures ugly wait() encoded return code)
<pre>
import os
p = os.popen(cmd)
stdout = "".join( p.readlines() )
encoded_wait_code = p.close() # None if 0
rcode = os.WEXITSTATUS(ugly_rcode) # fix ugly wait() encoded rcode
print(stdout)
print(rcode)

if os.popen(cmd).close():
print "error with cmd: %s" % cmd
sys.exit(1)
</pre>

???:
<pre>
os.spawnvp(os.P_WAIT,"/usr/bin/mplayer",["","-ao","pcm",file])
# or
os.spawnvp(os.P_NOWAIT,"/usr/bin/mplayer",["","-ao","pcm",file])
</pre>

Replacing os.system(): [http://docs.python.org/library/subprocess.html]
# old:
sts = os.system("mycmd" + " myarg")

# new:
p = Popen("mycmd" + " myarg", shell=True)
sts = os.waitpid(p.pid, 0)[1]

More realistic example replacing os.system(): [http://docs.python.org/library/subprocess.html]
<pre>
try:
retcode = call("mycmd" + " myarg", shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e
</pre>

==== select ====

Select:

select.select(rlist, wlist, xlist[, timeout])
: This is a straightforward interface to the Unix select() system call. The first three arguments are sequences of ‘waitable objects’: either integers representing file descriptors or objects with a parameterless method named fileno() returning such an integer
: The optional timeout argument specifies a time-out as a floating point number in seconds. When the timeout argument is omitted the function blocks until at least one file descriptor is ready. A time-out value of zero specifies a poll and never blocks.

if select.select([sys.stdin.fileno()], [], [], .1)[0]:
line = p.stdout.readline()

==== full wrapped execution ====

Process timeout (by Kenneth):
<pre>
import subprocess
import time
import os
import signal

def cmd(cmd, timeout=300, retry=0):
# Execute command, time progress, and kill on timeout

# (status, output) = commands.getstatusoutput(cmd)
start_time = None
status = None
output = None
for retry_count in range(retry+1):
if retry_count > 0:
print("RETRY: " + str(retry_count) + " " + cmd)
start_time = time.time()
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
timeout_fail = False
while p.poll() == None:
if time.time() - start_time > timeout:
# kill process
print("TIMEOUT: " + str(timeout) + "sec - killing (" + str(p.pid) + ") - " + cmd)
try:
os.kill(p.pid, signal.SIGTERM)
except OSError as e:
print("Kill exception: " + str(e))
timeout_fail = True
break
#print(".")
time.sleep(1)
status = p.poll()
output = p.stdout.read()
if timeout_fail == False:
break
seconds = int(time.time() - start_time)
return (status, output, seconds)
</pre>

---

Process input and output wrapper (from minecraft):
<pre>
import subprocess
import select
import re
import sys

cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)

# cycle while process continues alive
while p.poll() == None:

if select.select([p.stdout.fileno()], [], [], 1)[0]:
while select.select([p.stdout.fileno()], [], [], .1)[0]:
line = p.stdout.readline()
# <kiloforce> -tp joe
match = re.findall("<(.*)> -tp (.*)", line)
if match:
user1 = match[0][0]
user2 = match[0][1]
print ">> ACTION: Teleporting %s to %s" % (user1, user2)
p.stdin.write("tp %s %s\n" % (user1, user2))
if line:
print ">> SERVER: %s" % line.strip()
else:
print "The server died!"
sys.exit(1)

if select.select([sys.stdin.fileno()], [], [], .1)[0]:
line = sys.stdin.readline()
print ">> CONSOLE: %s" % line.strip()
p.stdin.write("%s" % line)
</pre>

==== Command With Timeout ====

python - Using module 'subprocess' with timeout - Stack Overflow - http://stackoverflow.com/questions/1191374/using-module-subprocess-with-timeout

<pre>
import subprocess, threading

class Command(object):
def __init__(self, cmd):
self.cmd = cmd
self.process = None

def run(self, timeout):
def target():
print 'Thread started'
self.process = subprocess.Popen(self.cmd, shell=True)
self.process.communicate()
print 'Thread finished'

thread = threading.Thread(target=target)
thread.start()

thread.join(timeout)
if thread.is_alive():
print 'Terminating process'
self.process.terminate()
thread.join()
print self.process.returncode

command = Command("echo 'Process started'; sleep 2; echo 'Process finished'")
command.run(timeout=3)
command.run(timeout=1)
</pre>

My version:
<pre>
import subprocess
import threading

class Command(object):
def __init__(self, cmd, capture=False, shell=True):
self.cmd = cmd
self.process = None
self.thread = None
self.shell = shell
if type(cmd) == list:
self.shell = False
self.rc = None
self.pid = None
self.stdout = None
self.stderr = None
self.capture = capture # capture stdout?

def run(self, timeout=None, wait=True):
def target():
print '>> Thread started (%s)' % self.cmd
if self.capture:
self.process = subprocess.Popen(self.cmd, shell=self.shell, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
else:
self.process = subprocess.Popen(self.cmd, shell=self.shell)
print '>> Thread PID: %d' % self.process.pid
self.pid = self.process.pid
self.stdout, self.stderr = self.process.communicate()
self.rc = self.process.returncode
print '>> Thread finished: %d' % self.rc

self.thread = threading.Thread(target=target)
self.thread.start()

if wait:
self.wait(timeout)
return self # so you can do: c = Command().run()

def wait(self, timeout=None):
print '>> Waiting for thread...'
if timeout:
self.thread.join(timeout)
else:
self.thread.join()
if self.thread.is_alive():
print '>> Terminating process'
self.process.terminate()
self.rc = self.process.returncode
self.thread.join()
if self.thread.is_alive():
print '>> Failed to terminate'
else:
print '>> Thread completed'
#print self.process.returncode

def is_alive(self ):
return self.thread.is_alive()

command = Command('do something').run()

command2 = Command(['do', 'something', 'long']).run(wait=False)
command2.wait(timeout=10)

command3 = Command('want output', capture=True).run()
print command3.stdout
</pre>

=== System call ===

os.system("some_command with args")
os.system("some_command < input_file | another_command > output_file")
stream = os.popen("some_command with args")

Here's a summary of the ways to call external programs and the advantages and disadvantages of each:

1.
: os.system("some_command with args") passes the command and arguments to your system's shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example,
: os.system("some_command < input_file | another_command > output_file")
: However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs.
: http://docs.python.org/lib/os-process.html

2.
: stream = os.popen("some_command with args") will do the same thing as os.system except that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything.
: http://docs.python.org/lib/os-newstreams.html

3.
: The Popen class of the subprocess module. This is intended as a replacement for os.popen but has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you'd say
: print Popen("echo Hello World", stdout=PIPE, shell=True).stdout.read()
: instead of
: print os.popen("echo Hello World").read()
: but it is nice to have all of the options there in one unified class instead of 4 different popen functions.
: http://docs.python.org/lib/node528.html

4.
: The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply wait until the command completes and gives you the return code. For example:
: return_code = call("echo Hello World", shell=True)
: http://docs.python.org/lib/node529.html

5.
: The os module also has all of the fork/exec/spawn functions that you'd have in a C program, but I don't recommend using them directly.

: The subprocess module should probably be what you use.

Source: [http://stackoverflow.com/questions/89228/how-to-call-external-command-in-python]

== Advanced Code ==

=== Executing Code from Command Line ===

python -c [cmd]
# -c cmd : program passed in as string (terminates option list)

<pre>
python -c "import sys;exec(sys.stdin.read())" << "EOF"
def test1():
print "test1"
test1()
print "test"
EOF
</pre>

=== Integer Return Code Class ===

Python3 version:
<pre>
class RC(int):
def __new__(cls, rc, stdout='', stderr=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, stdout='', stderr=''):
super().__init__()
self.rc = rc # redundant
self.stdout = stdout
self.stderr = stderr
def __repr__(self):
"""Optional string output"""
return "(%s, '%s', '%s')" % (self.rc, self.stdout, self.stderr)
</pre>

Python2 version:
<pre>
class RC(int):
def __new__(cls, rc, stdout='', stderr=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, stdout='', stderr=''):
super(self.__class__, self).__init__(rc)
self.rc = rc # redundant
self.stdout = stdout
self.stderr = stderr
def __repr__(self):
"""Optional string output"""
return "(%s, '%s', '%s')" % (self, self.stdout, self.stderr)
</pre>

sample:
<pre>
>>> rc1 = RC(5)
>>> rc2 = RC(0, 'out', 'err')
>>> rc1
(5, '', '')
>>> rc2
(5, 'out', 'err')
>>> rc2.stdout
'out'
>>> int(rc1)
5
>>> 0 + rc1
5
>>> if rc1 <= 5: print "yes"
...
yes
>>> if rc2: print "non-zero"
...
</pre>

=== Smart Execute ===

<pre>
class RC(int):
def __new__(cls, rc, output=''):
"""We can't use use __init__ because int is immutable."""
return super(RC, cls).__new__(cls, rc)
def __init__(self, rc, output=''):
super().__init__()
self.rc = rc # redundant
self.output = output
def __repr__(self):
"""Optional string output"""
return "(%s, '%s')" % (self.rc, self.output)

def smart_execute(cmd, display=True, checked=False):
output = ""
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, shell=True, encoding='utf-8')
for c in iter(lambda: process.stdout.read(1), ""):
if display:
sys.stdout.buffer.write(c.encode('utf-8'))
output += c
process.stdout.close()
rc = process.wait()
if checked and rc:
raise subprocess.CalledProcessError(ret, cmd)
return RC(rc, output.strip())
</pre>

Not smart version:
<pre>
def execute(cmd, return_output=False):
out = ""
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, shell=True, encoding='utf-8')
for c in iter(lambda: process.stdout.read(1), ""):
sys.stdout.buffer.write(c.encode('utf-8'))
if return_output:
out += c
process.stdout.close()
ret = process.wait()
if ret:
raise subprocess.CalledProcessError(ret, cmd)
if return_output:
return out.strip()
</pre>
</pre>

=== select ===

usage: select.select(rlist, wlist, xlist[, timeout]) [http://docs.python.org/2/library/select.html]
* rlist: wait until ready for reading
* wlist: wait until ready for writing
* xlist: wait for an “exceptional condition” (see the manual page for what your system considers such a condition)
* The optional timeout argument specifies a time-out as a floating point number in seconds.

if select.select([p.stdout.fileno()], [], [], 1)[0]:

=== ANSI Color ===

* ANSI escape code - http://en.wikipedia.org/wiki/ANSI_escape_code

---

unicode - Print in terminal with colors using Python? - Stack Overflow - https://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python

<pre>
class bcolors:
HEADER = '\033[95m'
OKBLUE = '\033[94m'
OKGREEN = '\033[92m'
WARNING = '\033[93m'
FAIL = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'

print bcolors.WARNING + "Warning: No active frommets remain. Continue?" + bcolors.ENDC
</pre>

<pre>
class colors:
black = 0
red = 1
green = 2
yellow = 3
blue = 4
magenta = 5
cyan = 6
white = 7
fg = 30 # foreground add
bg = 40 # background add
bfg = 90 # bold foreground add
bbg = 100 # bold background add

print '\033[%dm red foreground \033[0m' % (colors.red + colors.fg)
print '\033[%dm bold red foreground \033[0m' % (colors.red + colors.bfg)
print '\033[%dm blue background \033[0m' % (colors.blue + colors.bg)
print '\033[%dm bold blue background \033[0m' % (colors.blue + colors.bbg)
</pre>

---

Before you start spouting escape sequences, you should check that stdout is a tty. You can do this with sys.stdout.isatty()

<pre>
sys.stdout.isatty()
</pre>

<pre>
def hilite(string, status, bold):
attr = []
if status:
# green
attr.append('32')
else:
# red
attr.append('31')
if bold:
attr.append('1')
return '\x1b[%sm%s\x1b[0m' % (';'.join(attr), string)
</pre>

Python | change text color in shell - Stack Overflow - https://stackoverflow.com/questions/2330245/python-change-text-color-in-shell

---

For reference, a small bash script which prints a table of escape sequences: colortable16.sh: ([http://www.frexx.de/xterm-256-notes/ The 256 color mode of xterm], [http://plug.org/pipermail/plug/2008-March/016821.html The 90s called; they want their 256 colors back])
<pre>
#!/bin/bash

# prints a color table of 8bg * 8fg * 2 states (regular/bold)
echo
echo Table for 16-color terminal escape sequences.
echo Replace ESC with \\033 in bash.
echo
echo "Background | Foreground colors"
echo "---------------------------------------------------------------------"
for((bg=40;bg<=47;bg++)); do
for((bold=0;bold<=1;bold++)) do
echo -en "\033[0m"" ESC[${bg}m | "
for((fg=30;fg<=37;fg++)); do
if [ $bold == "0" ]; then
echo -en "\033[${bg}m\033[${fg}m [${fg}m "
else
echo -en "\033[${bg}m\033[1;${fg}m [1;${fg}m"
fi
done
echo -e "\033[0m"
done
echo "--------------------------------------------------------------------- "
done

echo
echo
</pre>

Python version: (modified by me)
<pre>
#!/usr/bin/env python

import sys

# prints a color table of 8bg * 8fg * 2 states (regular/bold)
print
print "Table for 16-color terminal escape sequences."
print "Replace ESC with \\033 in bash."
print
print "Background | Foreground colors"
print "---------------------------------------------------------------------"
for bg in range(40, 47+1):
for bold in range(0, 1+1):
sys.stdout.write("\033[0m"" ESC[{bg}m | ".format(bg=bg))
for fg in range(30, 37+1):
if bold == 0:
sys.stdout.write("\033[{bg}m\033[{fg}m [{fg}m ".format(bg=bg, fg=fg))
else:
sys.stdout.write("\033[{bg}m\033[1;{fg}m [1;{fg}m".format(bg=bg, fg=fg))
sys.stdout.write("\033[0m\n")
print "--------------------------------------------------------------------- "

print
print
</pre>

=== Execute External Application ===

Execute program: (and send output to STDOUT)
encoded_wait_code = os.system('mkdir mydir') # output sent to STDOUT

Get output and ugly return code: (STDERR goes to STDOUT)
p = os.popen(cmd)
print p.readlines()
encoded_wait_code = p.close()

Execute other python script
import sys
sys.argv = [, 'my', 'args', 'here']
execfile('myscript.py') # does not support arguments directly

Subprocess:
* See [[#Subprocess]]

=== Program Layout ===

<pre>
PackageName/
README
setup.py
bin/
script
docs/
test/ # some include in package_name
pacakgename/
__init__.py
code1.py
subpackage/
__init__.py
</pre>

=== __init__.py ===

Purpose: Makes the folder into a package library

NOTE: This file is most commonly blank

Good place to put your package functions.
<pre>
# ken/__init__.py
def load():
...
</pre>

<pre>
import ken
ken.load()
</pre>

Can import other modules to then use as package modules:
<pre>
# ken/__init__.py
from ken.client import Server
</pre>

<pre>
import ken
ken.Server()
</pre>

Good place to put your Exceptions.
<pre>
class MyException(Exception):
""" My Error Exception """
</pre>

Use the __all__ to specify what gets imported on 'from package import *'
__all__ = ['func1', 'func2']

=== Metadata ===

VERSION = (0, 2, 1)
or
__version__ = '0.2.1'

Also See [[#pydoc]]

---

__version__ = '1.10'
__author__ = 'Joshua Roesslein'
__license__ = 'MIT'

=== setup.py ===

See [[#Distributing Python Modules with Distutils]]

=== Distributing Python Modules with Distutils ===

Distributing Python Modules — http://docs.python.org/distutils/index.html

Uses setup.py for package installation.
python setup.py install

==== Simple example ====

To install foo.py:
<pre>
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
</pre>

Source: http://docs.python.org/distutils/introduction.html

==== klib example ====

klib setup.py:
<pre>
from distutils.core import setup

setup(name = "klib",
version = "101",
description = "klib library",
url = "http://www.oeey.com/klib",
author = "Kenneth",
author_email = "kenneth@oeey.com",
license = "Kenneth License",
long_description = """Really long text here.""",
platforms = "Any Platform",

packages = ['klib','klib.messenger'],
)
</pre>

Installed to:
/opt/python26/lib/python2.6/site-packages/klib/
/opt/python26/lib/python2.6/site-packages/klib-100-py2.6.egg-info

Metadata: (/opt/python26/lib/python2.6/site-packages/klib-101-py2.6.egg-info)
<pre>
Metadata-Version: 1.0
Name: klib
Version: 101
Summary: klib library
Home-page: http://www.oeey.com/klib
Author: Kenneth
Author-email: kenneth@oeey.com
License: Kenneth License
Description: Really long text here.
Platform: Any platform
</pre>

Any fields not included will be marked:
Platform: UNKNOWN

References:
* 2. Writing the Setup Script — Python v2.7.2 documentation - http://docs.python.org/distutils/setupscript.html
* http://wiki.python.org/moin/Distutils/Tutorial

==== Uninstall Package ====

Collect files during install:
python setup.py install --record install-record.txt
sudo rm $(cat install-record.txt)

References:
* installation - python setup.py uninstall - Stack Overflow - http://stackoverflow.com/questions/1550226/python-setup-py-uninstall

=== Distributing Python Modules with Setuptools ===

<pre>
#!/usr/bin/env python
import setuptools

setuptools.setup(name="tweepy",
version="1.10",
description="Twitter library for python",
license="MIT",
author="Joshua Roesslein",
author_email="tweepy@googlegroups.com",
url="http://github.com/tweepy/tweepy",
packages = setuptools.find_packages(),
keywords= "twitter library",
zip_safe = True)

</pre>

=== Decorators ===

Decorators wrap functions

Function Counter: [http://stackoverflow.com/questions/739654/understanding-python-decorators]
<pre>
def fcount(func):
def wrapper(*args, **kw):
wrapper.count = wrapper.count + 1
return func(*args, **kw)
wrapper.count = 0
return wrapper

@fcount
def f(n):
return n*n

print "f.count = ", f.count
print "f(10) = ", f(10)
print "f.count = ", f.count
print "f(3) = ", f(3)
print "f.count = ", f.count
print "f(12) = ", f(12)
print "f.count = ", f.count
print "f(15) = ", f(15)
print "f.count = ", f.count

# Output:
# f.count = 0
# f(10) = 100
# f.count = 1
# f(3) = 9
# f.count = 2
# f(12) = 144
# f.count = 3
# f(15) = 225
# f.count = 4
</pre>

Great reference: Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

Example 1:
@dec1
def func(arg1, arg2, ...): pass

# equivalent to:
def func(arg1, arg2, ...): pass
func = dec1(func)

Example 2:
<pre>
@dec2
@dec1
def func(arg1, arg2, ...):
pass
</pre>

<pre>
# equivalent to:
def func(arg1, arg2, ...):
pass
func = dec2(dec1(func))
</pre>

Example 3:
@decomaker(argA, argB, ...)
def func(arg1, arg2, ...):
pass

# equivalent to:
func = decomaker(argA, argB, ...)(func)

# example:
@accepts(int,int)
@returns(float)
def bar(low,high):
pass

Example 4: (overrides and ignores passed in function - bad, unless desired)
<pre>
def spamrun(fn):
def sayspam(*args):
print "spam, spam, spam"
return sayspam

@spamrun
def useful(a, b):
print a**2 + b**2
</pre>

Example 5: (Decorator that modifies behavior of undecorated func)
<pre>
def addspam(fn):
def new(*args):
print "spam, spam, spam"
return fn(*args)
return new

@addspam
def useful(a, b):
print a**2 + b**2
</pre>

My Example 1: (print args and continue to function)
<pre>
def showargs(fn):
def doshowargs(*args, **kw):
print "Args for:", fn.__name__, args, kw
return fn(*args, **kw)
return doshowargs

@showargs
def useful(a, b):
print a**2 + b**2

</pre>

Another Example: [http://stackoverflow.com/questions/739654/understanding-python-decorators]
<pre>
def makebold(fn):
def wrapped(*args, **kw):
return "" + fn(*args, **kw) + ""
return wrapped

def makeitalic(fn):
def wrapped(*args, **kw):
return "" + fn(*args, **kw) + ""
return wrapped

@makebold
@makeitalic
def hello(msg):
return msg

print hello("hello world")
# output: hello world
</pre>

References:
* PEP 318 -- Decorators for Functions and Methods - http://www.python.org/dev/peps/pep-0318/
* Charming Python: Decorators make magic easy - http://www.ibm.com/developerworks/linux/library/l-cpdecor/index.html
* What are some common uses for Python decorators? - Stack Overflow - http://stackoverflow.com/questions/489720/what-are-some-common-uses-for-python-decorators
* Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

---

Utah Python July 2013 Meeting - Decorator Examples: [https://www.youtube.com/watch?v=uh9uhzitdvw]

Example of uses:
* Count function calls
* Add additional output (debugging)
* Timing code
* Thread synchronization (locks)
* Type checking

<pre>
## Simple wrapping call
def shout(f):
def wrapper(*args):
print('This is before {name}'.format(name=f.__name__))
ret = f(*args)
print('This is after {name}'.format(name=f.__name__))
return ret
wrapper.__name__ = f.__name__
return wrapper

@shout
def echo(text):
print(text)

echo('this is a test')
</pre>

<pre>
## Counting function calls
def count(f):
def wrapper(*args):
wrapper.counter += 1
print("Function called {count} times".format(count=wrapper.counter))
return f(*args)
wrapper.counter = 0
# change wrapper name, or it will just be 'wrapper' (for inspection)
wrapper.__name__ = f.__name__
return wrapper

@count
def donothing():
pass

donothing()
donothing()
print("Function called a total of {count} times".format(count=donothing.counter))
</pre>

<pre>
## Timing your code
import time

def timer(f):
def wrapper(*arg):
t = time.time()
ret = f(*arg)
print("Took {elapsedtime} seconds".format(elapsedtime=time.time()-t))
return ret
return wrapper

@timer
def fact(n):
prod = 1
while n > 0:
prod *= n
n -= 1
return prod

total = fact(10000)
</pre>

<pre>
## Synchronization
import threading

def synchronized(lock):
def wrap(f):
def wrapper(*args, **kwargs):
lock.acquire()
try:
return f(*args, **kwargs)
finally:
lock.release()
return wrapper
return wrap

mylock = threading.Lock()

@synchronized(mylock)
def myfunc():
pass
</pre>

<pre>
## Type checking
def myMethod(ID, name):
''' Old way '''
if not (isinstance(ID, 'uint') and isinstance(name, 'utf8string')):
raise Exception()

def accepts(*typeargs):
''' Decorator way '''
def wrap(f):
def wrapper(*args):
for a, t in zip(args, typeargs):
if not isinstance(a, t):
raise Exception('{a} not of type {t}'.format(a=a, t=t))
return f(*args)
return wrapper
return wrap

@accepts(int)
def add1(n):
return n + 1
</pre>

<pre>
## Memoize
def memoize(f):
def wrapper(*args):
if not wrapper.ret:
wrapper.ret = f(*args)
return wrapper.ret
wrapper.ret = False
wrapper.__name__ = f.__name__
return wrapper

import time

@memoize
def sumto10():
count = 0
sum = 0
while count < 10:
sum += count
time.sleep(1)
count += 1
return sum

# first time will count slow
print sumto10()
# second time will return cached value instantly
print sumto10()
</pre>

Memoize: "Caches a function's return value each time it is called. If called later with the same arguments, the cached value is returned (not reevaluated)" [http://wiki.python.org/moin/PythonDecoratorLibrary#Memoize]

<pre>
## Nested decorators
import time
@shout
@memoize
def test1():
print "calculating"
time.sleep(3)
return 100
</pre>

Order is important! For tested decorators the execution order is from the top to the bottom. If you reverse @shout and @memoize, the @shout will not run because of the way @memoize was written.

<pre>
## registry
registry = []
# can also decorate decorators:
@shout
def registry(f):
registry.append(f)
return f

@register
def test2():
pass
</pre>

Decorator is just a function that returns a function.

Can use functools to automatically update decorator wrappers with __name__, etc [http://docs.python.org/2/library/functools.html]
<pre>
## Auto syntax sygar the wrapper for you
import functools
def my_decorator(f):
@functools.wraps(f)
def wrapper(*args, **kwds):
print 'Calling decorated function'
return f(*args, **kwds)
return wrapper
</pre>

=== Class Method Decorators ===

Static methods: [http://docs.python.org/library/functions.html#staticmethod]
: Similar to C++ and Java static methods
<pre>
class C:
@staticmethod
def say(msg):
print(msg)

C.say('hi')
</pre>

Class method: [http://docs.python.org/library/functions.html#classmethod]
: A class method receives the class as implicit first argument, just like an instance method receives the instance.
: Class methods are different than C++ or Java static methods. If you want those, see staticmethod() in this section.
<pre>
@classmethod
def setup_parser(cls, parser):
</pre>

See also [[#Class Properties]]

=== Class Properties ===

Properties: (not using decorators)
<pre>
class C(object):
def __init__(self):
self._x = None
def getx(self):
print("get x")
return self._x
def setx(self, value):
print("set x")
self._x = value
def delx(self):
print("del x")
del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")
</pre>

<pre>
#usage
c = C()
c.x = 5
print(c.x)
del(c.x)
</pre>

Getter and Setter Property Decorators: [http://docs.python.org/library/functions.html#property]
<pre>
class C(object): # make sure to inherit from object for these properties to work!
def __init__(self):
self._x = None

@property
def x(self):
"""I'm the 'x' property."""
print("get x")
return self._x

@x.setter # python >= 2.6
def x(self, value):
print("set x")
self._x = value

@x.deleter # python >= 2.6
def x(self):
print("del x")
del self._x
</pre>

<pre>
#usage
c = C()
c.x = 5
print(c.x)
del(c.x)
</pre>

Note: If you don't add the @x.setter, the property will be read only, and will throw a "AttributeError: can't set attribute" if attempted to be set.

Using x.getter example: [http://wiki.python.org/moin/AlternativeDescriptionOfProperty]
<pre>
class Cls(object):
def __init__(self):
self._x = 100

x = property()

@x.getter
def x(self):
"""I'm the 'x' property."""
return self._x

@x.setter
def x(self, value):
self._x = value

@x.deleter
def x(self):
del self._x
</pre>

=== Custom Decorators ===

decorator
@dec1
def func(arg1, arg2, ...): pass

equivalent to:
def func(arg1, arg2, ...): pass
func = dec1(func)

Basic layout (names are arbitrary):
<pre>
def my_decorator(function_name):
# update wrapper.__doc__ and .func_name
# or @functools.wraps(my_wrapper)
def mywrapper(*args, **kw):
# do something before
result = function_name(*args, **kw)
# do something after
return result
return my_wrapper
</pre>

Example:
<pre>
def limit(length):
def decorator(function):
def wrapper(*args, **kw):
result = function(*args, **kw)
result = result[:length]
return result
return wrapper
return decorator
</pre>

<pre>
@limit(3)
def echo(foo): return foo

echo('123456') # '123'
</pre>

=== assert ===

assert True
assert False # exception
assert 1 == 1
assert 1 == 2, "some message" # AssertionError: some message

=== Snake Case ===

Convert Camel Case to Snake Case: [http://stackoverflow.com/questions/1175208/elegant-python-function-to-convert-camelcase-to-snake-case]

<pre>
def snake_case(name):
s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name)
return re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1).lower()
</pre>

<pre>
>>> convert('CamelCase')
'camel_case'
</pre>

=== Camel Case ===

Camel Case: [http://stackoverflow.com/questions/8347048/camelcase-every-string-any-standard-library]

>>> ''.join(x for x in 'make IT camel CaSe'.title() if not x.isspace())
'MakeItCamelCase'

To titleize a sentence:
>>> "this is a test".title()
"This Is A Test"

=== Lambda ===

Simple functions
mul = lambda a, b: a * b

# equivalent to:
def mul(a, b):
return a * b

Sort usage:
data = [dict(number=x) for x in '036149']
data.sort(key=lambda x: float(x['number']))

Compare usage:
data = [dict(number=x) for x in '036149']
data.sort(cmp=lambda x,y: cmp(x['number'], y['number']))

Boolean:
lambda x: x % 3 == 0

Multiple parameter:
lambda x, y: x + y

Call another function:
lambda word: len(word)

Strings:
lambda line: line.split()[2]

Embedded:
def transform(n):
return lambda x: x + n

No parameters:
lambda: 1

Default parameters:
lambda x=1: x+1

References:
* Python: Lambda Functions - http://www.secnetix.de/olli/Python/lambda_functions.hawk

=== Map ===

Apply a function to items of a sequence:
map(str, range(3)) # ['0', '1', '2']

Note: list comprehensions can replace map and filter

=== Reduce ===

Apply a function to pairs of the sequence:
import operator
reduce(operator.mul, [1,2,3,4]) # 24 - ((1*2)*3)*4)

Note: sum or for loop can replace reduce

=== Filter ===

Return a sequence of items for which function(item) is True
filter(lambda x: x >= 0, [0, -1, 3]) # [0, 3]

Note: list comprehensions can replace map and filter

Find matching items in two lists:
a = ['a','b','c']
b = ['c','d']
filter(lambda x: x in a, b) # returns only 'c' as it is in both lists

=== List comprehension ===

Nothing more than Syntactic Sugar

5. Data Structures — Python v2.7.5 documentation - http://docs.python.org/2/tutorial/datastructures.html#list-comprehensions
: "List comprehensions provide a concise way to create lists."
: "A list comprehension consists of '''brackets''' containing an '''expression''' followed by a '''for clause''', then '''zero or more for or if clauses'''. "

Remove newlines from each item in a list:
mylist = [x.strip() for x in mylist]

Simple:
doubled = [x*2 for x in mylist]
squares = [x**2 for x in range(10)]

If:
[ 2*x for x in range(1,10) if x >= 5 ]

Complex:
[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]

Expanding PI:
[str(round(pi, i)) for i in range(1, 6)]

Return list of words
my_list = [current_word.lower() for current_word in words]

Utah Python July 2013 Meeting: (http://basepi.net)
<pre>
# Iterative
squares = []
for x in range(10):
squares.append(x**2)
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

## Functional
squares = map(lambda x: x**2, range(10))

## List comprehension
squares = [x**2 for x in range(10)]
even_squares = [x**2 for x in range(10) if x%2 == 0]

# get hash of strings in list:
import sha
strings = ['Python is cool!', 'I love python']
hashes = [sha.sha(x).digest() for x in strings]

# flatten matrix:
matrix = [[1, 2, 3], [4, 5, 6]]
[x for row in matrix for x in row]
# [1, 2, 3, 4, 5, 6]

# build matrix:
[(x,y) for x in range(10) for y in range(10)]
# matrix of not both odd and not both even
[(x,y) for x in range(10) for y in range(10) if x%2 != y%2]

# suck lines of files into list, ignoring commented lines
[line.strip().split("\t") for line in open("my_file.tab")
if not line.startswith('#')]
</pre>

References:
* 5.1.4. List Comprehensions - Data Structures — Python v2.7.2 documentation - http://docs.python.org/tutorial/datastructures.html#list-comprehensions
* Utah Python July 2013 Meeting - YouTube - https://www.youtube.com/watch?v=uh9uhzitdvw [http://basepi.net]

=== [http://docs.python.org/reference/expressions#calls Mass Parameter Passing] ===

Using: *args and **kw
* *args a tuple of remaining unnamed parameters
* **kw a dictionary of remaining named parameters

def param_func(a, b='b', *args, **kw):
print( [a, b, args, kw] )

WARNING: If not careful, this can result in this error: "TypeError: param_func() got multiple values for keyword argument 'myvar'". Might be better to just use:

def param_func(*args, **kw):
print( [args, kw] )

param_func(1, 2, 3, 4, 5) # [1, 2, (3, 4, 5), {}]
param_func(1, 2, 3, 4, 5, joe='test') # [1, 2, (3, 4, 5), {'joe': 'test'}]
mylist = ("a", "c", "g", "h", 5)
param_func(mylist) # [('a', 'c', 'g', 'h', 5), 'b', (), {}]
param_func(*mylist) # ['a', 'c', ('g', 'h', 5), {}] *tricky*
Note: *mydict will strip out just keys and treat like list
Note: **mydict will treat like named parameters

def mywrapper(*args, **kw):
# do something before
result = function_name(*args, **kw)

Passing arbitrary parameters:
<pre>
def fun2(**kw):
print kw # {'a': '1', 'b': '2', 'c': '3'}

def fun1(c='3', **kw):
print kw # {'a': '1', 'b': '2'}
fun2(c='3', **kw)

fun1(a='1', b='2')
</pre>

=== [http://www.python.org/dev/peps/pep-0234/ Iteration Protocol] ===

[http://www.python.org/dev/peps/pep-0234/ PEP 234 -- Iterators]

Making a class iterable:
<pre>
class mycount(object):
def __init__(self, max):
self.x = 0
self.max = max
def __iter__(self): # return object that implements __next__() or next()
return self
def next(self):
if self.x < self.max:
self.x += 1
return self.x
else:
self.x = 0
raise StopIteration()
</pre>

for i in mycount(10): print i # 1 ... 10
print [x for x in mycount(10) ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10)
for i in c: print i # 1 ... 10
[ x for x in c ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

=== Generators ===

Generators are functions that return to their previous state (using 'yield') when iterated over, which can be used in for loops.

==== Yield ====

Improve on Iterators:
* Infinite data (infinite iterators)
* Memory-intensive iteration
* Cooperative Multi-threading

Generators are functions that return to their previous state when iterated over, which can be used in for loops.

Note: Generators throw StopIteration when completed

Note: the 'yield' keyword will stash away the state of the function and return the value. (Continuations)

<pre>
def gen_range(end):
cur = 0
while cur < end:
yield cur
# return here next time
cur += 1

# for loop iteration
for i in gen_range(10): print i

# manual iteration
my_gen = gen_range(10)
try:
while True:
print my_gen.next()
except StopIteration:
pass

# list comprehension
numbers = [x for x in gen_range(10)]
</pre>

Note: one can place an empty 'return' in a generator if needed. A 'return' with argument is invalid.

Making a class generate:
<pre>
class mycount(object):
def __init__(self, max):
self.max = max
def __iter__(self):
return self.next() # change here
def next(self): # change here
x = 0
while x < self.max:
x += 1
yield x

for i in mycount(10): print i # 1 ... 10
print [x for x in mycount(10) ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10)
for i in c: print i # 1 ... 10
[ x for x in c ] # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

itr = mycount(10).next()
itr.next() # 1
itr.next() # 2 - eventually throws StopIteration exception
</pre>

Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
<pre>
* Iterators are used to traverse a list
* Generators are more powerful Iterators

# simple 'n' number in functional form
def firstn(n):
num = 0
nums = []
while num < n:
nums.append(num)
num += 1
return nums

# simple 'n' number in generator form
def firstn(n):
num = 0
while num < n:
yield num
num += 1

-

# Infinite list generator
def counter():
num = 0
while True:
yield num
num += 1

# Fibbonacci generator
def fib():
num1 = 0
num2 = 1
yield num1
while True:
num1, num2 = num2, num1 + num2
yield num1

# Fibbonacci generator 2
def fib(n):
num1 = 0
num2 = 1
count = 0
yield num1
while count < n:
count += 1
num1, num2 = num2, num1 + num2
yield num1
</pre>

Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
<pre>
## Cooperative Multi-Threading:
# Cooperative multithreading relies on the threads themselves to relinquish control once they are at a stopping point.

def mythread(name):
print('Starting Thread: {name}...'.format(name=name))
counter = 10
while counter > 0:
print('Thread: {name} doing some work!'.format(name=name))
counter -= 1
yield

thread1 = mythread('Alice')
thread2 = mythread('Bob')
thread3 = mythread('Trudy')

try:
while True:
thread1.next()
thread2.next()
thread3.next()
except StopIteration:
pass
</pre>

References:
* Utah Python July 2013 Meeting: [https://www.youtube.com/watch?v=uh9uhzitdvw] [http://basepi.net]
** http://files.basepi.net/talks/pyarmory.zip

=== dunder methods ===

Some dunder (double under) methods determine what will happen when + (__add__) or other similar numerical symbols used.

<pre>
class str_add(object):
def __init__(self, name):
self.name = name
def __add__(self, other):
self.name += other.name
return self
# note: only need to return self if you want to go beyond 'a + b'

a = str_add('a_')
b = str_add('b_')
c = str_add('c_')
d = a + b + c
print(a.name) # a_b_c_
print(b.name) # b_
print(c.name) # c_
print(d.name) # a_b_c_
</pre>

Other common methods to override:
* __add__ for +
* __sub__ for -
* __mul__ for *
* __div__ for /
* __mod__ for %

References:
* Emulating numeric types - http://docs.python.org/2/reference/datamodel.html#emulating-numeric-types

=== [http://docs.python.org/library/doctest.html doctest] ===

test.py:
<pre>
#!/usr/bin/env python
"""
This is the "example" module.

>>> double(5)
10
"""

def double(n):
"""
This is double function

>>> double(3)
6
"""
return n * 2

if __name__ == "__main__":
import doctest
doctest.testmod()
</pre>

=== unittest ===

unittest.TestCase

Here is a short script to test three functions from the random module:
<pre>
import random
import unittest

class TestSequenceFunctions(unittest.TestCase):

def setUp(self):
self.seq = range(10)

def test_shuffle(self):
# make sure the shuffled sequence does not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, range(10))

# should raise an exception for an immutable sequence
self.assertRaises(TypeError, random.shuffle, (1,2,3))

def test_choice(self):
element = random.choice(self.seq)
self.assertTrue(element in self.seq)

def test_sample(self):
with self.assertRaises(ValueError):
random.sample(self.seq, 20)
for element in random.sample(self.seq, 5):
self.assertTrue(element in self.seq)

if __name__ == '__main__':
unittest.main()
</pre>

The unittest module can be used from the command line to run tests from modules, classes or even individual test methods:
python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method
python -m unittest -v test_module # verbose
python -m unittest -h # help

References:
* 25.3. unittest — Unit testing framework — Python v2.7.4 documentation - http://docs.python.org/2/library/unittest.html
* Python Debugging Tips - Google Drive - https://docs.google.com/presentation/d/130Wl--IzbDh9ymF6KHMu0WrsIwz6k32pav1gFbsw7NI/edit?usp=sharing

=== Internet Access ===

<pre>
# Python 3
from urllib.request import urlopen
for line in urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'):
line = line.decode('utf-8')
if 'EST' in line or 'EDT' in line: # look for eastern time
print(line)
</pre>

=== Simple SMTP ===

Requires local STMP server.

<pre>
import smtplib
server = smtplib.SMTP("localhost")
server.sendmail('mrken@demo.oeey.com', 'kenneth@demo.oeey.com',
"""To: kenneth@demo.oeey.com
From: mrken@demo.oeey.com
Subject: This is fun

Beware of the dog.
""")
server.quit()
</pre>

<pre>
s = smtplib.SMTP(smtp_server)
</pre>

with TLS:
<pre>
s = smtplib.SMTP("smtp.gmail.com",587)
s.ehlo()
s.starttls()
s.ehlo()
</pre>

SSL:
<pre>
s = smtplib.SMTP_SSL('smtp.mail.yahoo.com', 465)
</pre>

=== Data Compress ===

zlib, gzip, bz2, zipfile, tarfile

<pre>
import zlib
s = b'hello world' # only in python >3
len(s)
t = zlib.compress(s)
zlib.decompress(t)
zlib.crc32(s)
</pre>

=== gzip ===

<pre>
import gzip
content = "Lots of content here"
f = gzip.open('file.txt.gz', 'wb')
f.write(content)
f.close()
</pre>

<pre>
import gzip
f_in = open('file.txt', 'rb')
f_out = gzip.open('file.txt.gz', 'wb')
f_out.writelines(f_in)
f_out.close()
f_in.close()
</pre>

=== Performance Timer ===

Time a task repeated 10 times:
<pre>
from timeit import Timer
Timer('t=a; a=b; b=t', 'a=1; b=2').timeit()
Timer('print("hi")').timeit(10) # repeat 10 times and time
</pre>

Simple method:
<pre>
start_time = int(time.time())
... task ...
compute_time = int(time.time()) - start_time
</pre>

=== Python Version Check ===

Solution:

__init__.py:
<pre>
# Check Python Version:
import sys
assert sys.version_info >= (2, 6, 0), 'requires Python 2.6 or higher'
del sys
</pre>

resulting error if not > 2.6:
<pre>
>>> import nucleon
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "nucleon/__init__.py", line 9, in ?
assert sys.version_info >= (2, 6, 0), 'nucleon requires Python 2.6 or higher'
AssertionError: nucleon requires Python 2.6 or higher
</pre>

Other solutions:

Note: this will throw syntax error on 2.4 due to ternary
<pre>
import sys
if sys.version_info < (2, 4):
raise "must use python 2.5 or greater"
else:
# syntax error in 2.4, ok in 2.5
x = 1 if True else 2
print x
</pre>

maybe solution:
<pre>
try:
eval("1 if True else 2")
except SyntaxError:
# doesn't have ternary
</pre>

<pre>
>>> try:
... from my_module import twoPointSixCode
... except Exception:
... print "can't import, probably because your python is too old!"
>>>
</pre>

<pre>
#!/usr/bin/env/python
import sys
if sys.version_info<(2,6,0):
sys.stderr.write("You need python 2.6 or later to run this script\n")
exit(1)
</pre>

<pre>
>>> import sys
>>> sys.hexversion # hex printed as integer
33817584
>>> '%x' % sys.hexversion # output as hex, not zero fill 2.4.3
'20403f0'
>>> sys.hexversion < 0x02060000 # test if version less than 2.6
True
>>> print "0x%08x" % sys.hexversion # print as hex and zero fill and prepend '0x'
0x020403f0
</pre>

<pre>
import sys
if sys.hexversion < 0x02060500:
print >> sys.stderr, "This script requires python 2.6.5 or higher."
exit(1)
</pre>

<pre>
>>> import sys
>>> sys.version_info
(2, 6, 4, 'final', 0)
>>> if not sys.version_info[:2] == (2, 6):
... print "Error, I need python 2.6"
... else:
... from my_module import twoPointSixCode
>>>
</pre>

References:
* http://stackoverflow.com/questions/446052/python-best-way-to-check-for-python-version-in-program-that-uses-new-language-fe
* http://stackoverflow.com/questions/1093322/how-do-i-check-what-version-of-python-is-running-my-script

=== [http://docs.python.org/library/struct.html Struct - Binary Data Record Layout] ===

Format Characters:
<pre>
Format C Type Python type Standard size
x pad byte no value
c char str of len 1 1
b signed char integer 1
B unsigned char integer 1
? _Bool bool 1
h short integer 2
H unsigned short integer 2
i int integer 4
I unsigned int integer 4
l long integer 4
L unsigned long integer 4
q long long integer 8
Q unsigned long long integer 8
f float float 4
d double float 8
s char[] string
p char[] string
P void * integer
</pre>

H # two byte unassigned number
I # four byte unassigned number
< # standard size and in little-endian byte order

import struct
data = struct.pack('<IIIHH', 1, 2, 3, 4, 5)
# data = '\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x05\x00'

fields = struct.unpack('<IIIHH', data)
# fields = (1L, 2L, 3L, 4, 5)
# using splice: fields = struct.unpack('<IIIHH', data[start:start+16])

f1, f2, f3, f4, f5 = fields

hex(f1) # hex a field:
# '0x1L'

>>> struct.pack("<5sh", "helloo", 11)
'hello\x0b\x00'
>>> struct.unpack("<5sh", 'hello\x0b\x00')
('hello', 11)

=== pdb interactive debugger ===

# uncomment the following to use the interactive debugger:
# import pdb; pdb.set_trace()

=== [http://docs.python.org/library/telnetlib.html Telnet] ===

create:
class telnetlib.Telnet([host[, port[, timeout]]])

Note: The instance is initially not connected by default; the open() method must be used to establish a connection. Alternatively, the host name and optional port number can be passed to the constructor, to, in which case the connection to the server will be established before the constructor returns.

open if not initially done so:
Telnet.open(host[, port[, timeout]])

Functions:
Telnet.read_until(expected[, timeout]) # Read until a given string, expected, is encountered or until timeout seconds have passed.
# on timeout return whatever is in queue, possibly empty string
Telnet.close() # close connection
Telnet.read_all() # Read all data until EOF; block until connection closed.
Telnet.write(buffer) # Write a string to the socket, doubling any IAC characters. This can block if the connection is blocked.
Telnet.expect(list[, timeout]) # Read until one from a list of a regular expressions matches.
# Return a tuple of three items: the index in the list of matches; the match object; and the text read to match.

Read until end of line:
line = tn.read_until("\n")

<pre>
import getpass
import sys
import telnetlib

HOST = "localhost"
user = raw_input("Enter your remote account: ")
password = getpass.getpass()

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ")
tn.write(user + "\n")
if password:
tn.read_until("Password: ")
tn.write(password + "\n")

tn.write("ls\n")
tn.write("exit\n")

print tn.read_all()
</pre>

<pre>
#!/usr/bin/env python

import sys
import telnetlib

HOST = "216.119.193.252"
USER = "admin"
PASS = "@lt12345"

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ")
tn.write(USER + "\n")
tn.read_until("Password: ")
tn.write(PASS + "\n")
tn.read_until("# ")

tn.write("terminal length 0\n")
tn.read_until("# ")

tn.write("show running-config\n")
configlines = tn.read_until("# ")

tn.write("exit\n")
tn.read_all()

print(configlines)
</pre>

=== SSH ===

Cheap method:
os.system('ssh user@server command')

SecureShell - PythonInfo Wiki - http://wiki.python.org/moin/SecureShell
: There are several ways to use Secure Shell (SSH) in Python
* pyssh - http://pyssh.sourceforge.net/
* paramiko - Paramiko is a native Python implementation of SSH. - http://www.lag.net/paramiko/
* conch - conch is another native implementation of SSH and part of the Twisted Matrix project - http://twistedmatrix.com/projects/conch
* SSH wrapper - Some tools just wrap around existing ssh/sftp implementations
** keyphrene - http://www.keyphrene.com/products/org.keyphrene/
** keyphrene SSH wrapper - http://membres.lycos.fr/fredp/python/pyscp.html
* Fabric - http://docs.fabfile.org/en/1.4.0/index.html
* pexpect - Noah.org - http://www.noah.org/wiki/Pexpect

SFTP:
* SFTP in Python: Really Simple SSH | The best of Zeth - http://zeth.net/post/332/

pexpect:
wget http://pexpect.sourceforge.net/pexpect-2.3.tar.gz
tar xzf pexpect-2.3.tar.gz
cd pexpect-2.3
sudo python ./setup.py install

=== SFTP ===

SFTP in Python? (platform independent) - Stack Overflow - http://stackoverflow.com/questions/432385/sftp-in-python-platform-independent

paramiko: ssh2 protocol for python - http://www.lag.net/paramiko/

pip install paramiko

mkdir -p ~/.src ; cd ~/.src
wget http://www.lag.net/paramiko/download/paramiko-1.7.7.1.zip
unzip paramiko-1.7.7.1.zip
cd paramiko-1.7.7.1
python setup.py install

SFTP Methods:
* chdir
* chmod
* chown
* close
* file
* from_transport
* get
* get_channel
* getcwd
* listdir
* listdir_attr
* logger
* lstat
* mkdir
* normalize
* open
* put
* readlink
* remove
* rename
* request_number
* rmdir
* sock
* stat
* symlink
* truncate
* ultra_debug
* unlink
* utime

<pre>
import paramiko

host = "THEHOST.com" #hard-coded
port = 22
transport = paramiko.Transport((host, port))

password = "THEPASSWORD" #hard-coded
username = "THEUSERNAME" #hard-coded
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

import sys
path = './THETARGETDIRECTORY/' + sys.argv[1] #hard-coded
localpath = sys.argv[1]
sftp.put(localpath, path)

sftp.close()
transport.close()
print 'Upload done.'
</pre>

VMware Example:
<pre>
#!/usr/bin/env python

import paramiko
import sys

host = "sftp2.engx.vmware.com"
port = 443
transport = paramiko.Transport((host, port))

username = "xxxx"
password = "xxx"
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

print dir(sftp)

print sftp.chdir("build-uploader")
print sftp.listdir()

sftp.close()
transport.close()
</pre>

=== paramiko ===

See [[Python/paramiko]]

=== [http://docs.python.org/library/getpass.html Get Password] ===

Prompt for password:
getpass.getpass(prompt='Password: ')

import getpass
user = raw_input("Enter your remote account: ")
password = getpass.getpass()

=== [http://docs.python.org/library/configparser.html ini files] ===

<pre>
ini_filename = 'my.ini'

import ConfigParser
parser = ConfigParser.SafeConfigParser()
parser.read(ini_filename)

sections = parser.sections() # get sections
for section in sections:
print "[" + section + "]"
for name, value in parser.items(section):
print name + " = " + value
</pre>

<pre>
if parser.has_option(section, option):
print( parser.get(section, option) )
</pre>

To have the option names not convert to lower case:
parser.optionxform = str

Ignore section: [http://stackoverflow.com/questions/2885190/using-pythons-configparser-to-read-a-file-without-section-name]
ini_str = '[root]\n' + open(ini_path, 'r').read()
ini_fp = StringIO.StringIO(ini_str)
config = ConfigParser.RawConfigParser()
config.readfp(ini_fp)

Get option:
config.has_option('root', 'some_key')
config.get('root', 'some_key')

config.items('root') # list of Key Value tuples

=== optparse - Option Parser ===

[http://docs.python.org/library/optparse.html - Option Parser

Note: Deprecated since version 2.7: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

<pre>
import optparse
parser = optparse.OptionParser()

parser.add_option('--name', default=None)

parser.add_option(
'-l', '--loglevel',
help='Log Level (default: info)',
default="info", action='store', type='string')

(options, args) = parser.parse_args()

# if no parameters passed show help
if len(sys.argv) == 1:
parser.print_help()
sys.exit(1)

# if a stand alone parameter passed, override option
if args:
options.target = args[0]

# show option:
print(options.loglevel)
</pre>

Note: Order is important when listing add_options!

Pretty Description:
parser = optparse.OptionParser(description="Purpose: to do something")
# or
parser.description = "Purpose: to do something"

Parse arguments:
(options, args) = parser.parse_args()
Note: If you like, you can pass a custom argument list to parse_args(), but that’s rarely necessary: by default it uses sys.argv[1:].

Change Usage Message:
<pre>
usage = "%prog [options]" # default message
# Usage: <prog> [options]" # where <prog> is replaced with the script name

parser = OptionParser(usage=usage)
# or
parser.usage = usage # default message
# or
parser.set_usage(usage)
</pre>

Remove option: (does not remove default value from 'options' variable!!! - stupid!)
parser.remove_option('--loglevel')

Modify an option: (changes are saved to parser automatically, unless in an option group)
option = parser.get_option('--loglevel')
option.help = 'something else'
option.default = 'debug'

# or
parser.remove_option("--loglevel")
parser.add_option("--loglevel", help="something else", default="info")

Modify an option in an option group:
opt_group = parser.get_option_group("--loglevel")
opt_group.remove_option("--loglevel")
opt_group.add_option("--loglevel", help="something else", default="info")

Types: "string", "int", "long", "choice", "float" and "complex" [http://docs.python.org/library/optparse.html#optparse-standard-option-types]
Defaults:
* Default actions is 'store'.
* Default type is 'string'.
* Default dest is the long name of the parameter (or short name if no long name provided)
* Default value is None

<pre>
from optparse import OptionParser
[...]
parser = OptionParser()

# string
parser.add_option("-f", "--file", dest="filename",
help="write report to FILE", metavar="FILE")

# boolean - action is what to do when flag found
parser.add_option("-q", "--quiet",
action="store_true", dest="verbose", default=False,
help="don't print status messages to stdout")

# integer
parser.add_option(
"-m", "--minutes",
help="Minutes (default: 60)",
default=60, type='int')

(options, args) = parser.parse_args()

print "Filename: " + options.filename
</pre>

fake handling of required option:
<pre>
if not option.filename:
parser.print_help()
parser.error("filename is required")
</pre>

change default usage:
parser = OptionParser(usage="Usage: %prog [args] arg1 arg2")

Really ugly way to print help usage :-(
<pre>
def Usage():
pname = sys.argv[0]
cmd='%s -h' % (pname)
os.system(cmd)

if (len(sys.argv) < 2):
Usage()
sys.exit(1)
</pre>

Groups:
<pre>
# OptionGroup(parser, title, description=None)
group = OptionGroup(parser, "Dangerous Options",
"Caution: use these options at your own risk. "
"It is believed that some of them bite.")
group.add_option("-g", action="store_true", help="Group option.")
parser.add_option_group(group)
</pre>

Hide an option:
* Option.help
: "Help text to print for this option when listing all available options after the user supplies a help option (such as --help). If no help text is supplied, the option will be listed without help text. To hide this option, use the special value optparse.SUPPRESS_HELP."
<pre>
parser.add_option("--secret", help=SUPPRESS_HELP)
</pre>

Issues with Option Parser:
* No handling for positional arguments
* No adding help for positional arguments
* No handling for required field

=== argparse ===

argparse - http://docs.python.org/library/argparse.html

argparse — Parser for command-line options, arguments and sub-commands

NOTE: MUCH MORE POWERFUL THAN OPTION PARSER!

Note: New in version 2.7.

Installation on <= 2.6
<pre>
# Install argparse: - http://code.google.com/p/argparse/
# REQUIRES SETUPTOOLS
mkdir -p ~/.src ; cd ~/.src
wget http://argparse.googlecode.com/files/argparse-1.2.1.tar.gz
tar -zvxf argparse-1.2.1.tar.gz
cd argparse-1.2.1
sudo python setup.py install
</pre>

The argparse module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse will figure out how to parse those out of sys.argv. The argparse module also automatically generates help and usage messages and issues errors when users give the program invalid arguments.

<pre>
import argparse

parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
const=sum, default=max,
help='sum the integers (default: find the max)')

args = parser.parse_args()
print args.accumulate(args.integers)
</pre>

<pre>
import argparse

argparser = demoproject.util.argparse.ArgumentParser(
usage='Kick off demoproject benchmarks.')
argparser.add_argument(
'--card',
help='What cards to run on. Depends on Jenkins NODE_LABEL environment'
'variable for correct mappings.')
argparser.add_argument(
'--suite', metavar='SUITE',
help='Specify the appropriate benchmark suite as defined in '
'the benchmark ini file. Defaults to standard_perf',
default='standard_perf')
args = argparser.parse_args()

print args.card
</pre>

Remove an argument: [http://stackoverflow.com/questions/32807319/disable-remove-argument-in-argparse]
<pre>
def remove_options(parser, options):
""" remove_options(parser, ['--some-option', '--some-option2']) """
for option in options:
for action in parser._actions:
if vars(action)['option_strings'][0] == option:
parser._handle_conflict_resolve(None,[(option,action)])
break
</pre>

=== [http://docs.python.org/library/commands.html commands] ===

35.16. commands — Utilities for running commands — Python v2.7.1 documentation - http://docs.python.org/library/commands.html

Note: Deprecated since version 2.6: The commands module has been removed in Python 3.0. Use the subprocess module instead.

The commands module contains wrapper functions for os.popen() which take a system command as a string and return any output generated by the command and, optionally, the exit status.

Get status and output:
status, output = commands.getstatusoutput(cmd)

Get output ignoring status:
output = commands.getoutput(cmd)

NOTE: DO NOT USE getstatus() AS IT DOES NOT DO WHAT YOU THINK IT DOES!
status = commands.getstatus(file)

<pre>
>>> import commands

>>> commands.getstatusoutput('ls /bin/ls')
(0, '/bin/ls')

>>> commands.getstatusoutput('cat /bin/junk')
(256, 'cat: /bin/junk: No such file or directory')

>>> commands.getstatusoutput('/bin/junk')
(256, 'sh: /bin/junk: not found')

>>> commands.getoutput('ls /bin/ls')
'/bin/ls'

>>> commands.getstatus('/bin/ls')
'-rwxr-xr-x 1 root 13352 Oct 14 1994 /bin/ls'
</pre>

=== [http://docs.python.org/library/_winreg.html Windows Registry] ===

Note: The _winreg module has been renamed to winreg in Python 3.0.

* 32.3. winreg – Windows registry access — Python v3.2 documentation - http://docs.python.org/py3k/library/winreg.html
* 34.3. _winreg – Windows registry access — Python v2.7.1 documentation - http://docs.python.org/library/_winreg.html

Read from registry:
<pre>
from _winreg import *
key = OpenKey(HKEY_LOCAL_MACHINE, r'Software\Microsoft\Outlook Express', 0, KEY_ALL_ACCESS)
QueryValueEx(key, "InstallRoot")
</pre>

Write to registry:
<pre>
keyVal = r'Software\Microsoft\Internet Explorer\Main'
try:
key = OpenKey(HKEY_CURRENT_USER, keyVal, 0, KEY_ALL_ACCESS)
except:
key = CreateKey(HKEY_CURRENT_USER, keyVal)
SetValueEx(key, "Start Page", 0, REG_SZ, "http://www.blog.pythonlibrary.org/")
CloseKey(key)
</pre>

Enumerate registry keys:
<pre>
from _winreg import EnumKey, HKEY_USERS

try:
i = 0
while True:
subkey = EnumKey(HKEY_USERS, i)
print subkey
i += 1
except WindowsError:
# WindowsError: [Errno 259] No more data is available
pass
</pre>

Another enumeration example:
<pre>
# File: winreg-example-1.py

import _winreg

explorer = _winreg.OpenKey(
_winreg.HKEY_CURRENT_USER,
"Software\\Microsoft\\Windows\\CurrentVersion\\Explorer"
)

# list values owned by this registry key
try:
i = 0
while 1:
name, value, type = _winreg.EnumValue(explorer, i)
print repr(name),
i += 1
except WindowsError:
print

value, type = _winreg.QueryValueEx(explorer, "Logon User Name")

print
print "user is", repr(value)
</pre>

Enable Remote Desktop Example: (by Kenneth)
<pre>
from _winreg import *
key = OpenKey(HKEY_LOCAL_MACHINE, r'SYSTEM\CurrentControlSet\Control\Terminal Server', 0, KEY_ALL_ACCESS)
SetValueEx(key, "fDenyTSConnections", 0, REG_DWORD, 0)
ValueTypeTuple = QueryValueEx(key, "fDenyTSConnections") # (0, 4)
print("Please reboot the system to complete the changes")
</pre>

References:
* Python’s _winreg: Editing the Windows Registry « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/03/20/pythons-_winreg-editing-the-windows-registry/

=== JSON ===

JSON - http://docs.python.org/library/json.html

ENCODING TO JSON STRING (type str)

import json
json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
# '["foo", {"bar": ["baz", null, 1.0, 2]}]'

compact encoding:
import json
json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',',':'))
# '[1,2,3,{"4":5,"6":7}]'

Pretty printing:
import json
print json.dumps({'4': 5, '6': 7}, sort_keys=True, indent=4)
# {
# "4": 5,
# "6": 7
# }

DECODING JSON STRING TO PYTHON LIST OR DICTIONARY OBJECT

import json
json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]')
# [u'foo', {u'bar': [u'baz', None, 1.0, 2]}]

Load from file:
f = open(filename)
j = json.load(f)
print j["kind"] # "dual-controller-adapter"
#{
# "kind": "dual-controller-adapter",
#}

From Command Line:
echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool

# get value
echo '{"test":1,"test2":2}' | python -c 'import sys,json;data=json.loads(sys.stdin.read()); print data["test"]'

# read large file
python -c "import json;f=open('file.json');data=json.load(f);print json.dumps(data,indent=2)"
python -c "import json;print json.dumps(json.load(open('file.json')),indent=2)"

Load from web:
<pre>
import httplib
import json

def get_http_json(server, path):
""" This performs an HTTP GET against a server and path and returns the
resulting page data.

server The server we're hitting, e.g. demo.com
path The path we're hitting, e.g. api/part/MA12345/
"""
connection = httplib.HTTPConnection(server)
try:
url = 'http://%s/%s' % (server, path)
print 'Querying url %s' % url
connection.request('GET', url)
response = connection.getresponse()
if response.status == httplib.OK:
return json.loads(response.read())
else:
print 'Unable to get %s. %s %s' % (url, response.reason,
response.status)
return None
finally:
connection.close()

j = get_http_json(server, path)
</pre>

Pretty printing from the command line:
echo '{"json":"obj"}' | python -mjson.tool
#{
# "json": "obj"
#}

References:
* 18.2. json — JSON encoder and decoder — Python v2.7.1 documentation - http://docs.python.org/library/json.html

=== Sockets ===

See [[Python/Sockets]]

=== [http://docs.python.org/library/base64.html base64] ===

import base64

Encode:
encoded = base64.b64encode('data to be encoded')

Decode:
base64.b64decode(encoded)

=== daemonize ===

Creating a daemon the Python way « Python recipes « ActiveState Code - http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/

<pre>
import os
import sys

try:
pid = os.fork()
except OSError, e:
raise Exception, "%s [%d]" % (e.strerror, e.errno)

if (pid != 0): # parent
os._exit(0)

# create new process session
os.setsid()

# redirect stdin, stdout, stderr
os.open('/dev/null', os.O_RDWR)
os.dup2(0, 1)
os.dup2(0, 2)

#... tasks ...
</pre>

My Solution:
<pre>
import os
import sys
import signal
import syslog

HUP = False

def log(msg):
syslog.syslog(str(msg))
syslog.openlog('[WATER]')

def kill_signal_handler(signal, frame):
log("Being killed off: {0} {1}".format(signal, frame))
log("Shutting Down!")
sys.exit(0)
signal.signal(signal.SIGTERM, kill_signal_handler)
signal.signal(signal.SIGALRM, kill_signal_handler)

def hup_signal_handler(signal, frame):
global HUP
log("Signal reload received: {0} {1}".format(signal, frame))
HUP=True
signal.signal(signal.SIGHUP, hup_signal_handler)

def main_loop():
global HUP

while True:
# ... handle HUP=True ...

# ... main code ...

# wait 60 seconds before we start the next loop
time.sleep(60) # signals will interrupt this

if __name__ == "__main__":
log("Starting Daemon...")
child_id = os.fork()
if child_id:
log("Daemonized!")
sys.exit(0)
os.setsid()
# redirect stdin, stdout, stderr
os.open('/dev/null', os.O_RDWR)
os.dup2(0, 1)
os.dup2(0, 2)

# write pid
with open("water.pid", "w") as f:
f.write(str(os.getpid()))

while True:
try:
main_loop()
except Exception as e:
log("Critical Failure Caught: %s" % str(e))
log("Critical Failure - Restarting!")
time.sleep(30)
#raise e
</pre>

ln -s /opt/pyton27/bin/python pyton-water
./python-water water.py
killall python-water

==== fork ====

<pre>
import os
import sys
import time

child_id = os.fork()
if child_id:
print "parent exiting"
sys.exit(0)

while True:
time.sleep(3)
print "child"
</pre>

For windows, to hide the console, see [[#Hide Console]]

=== csv ===

13.1. csv — CSV File Reading and Writing — Python v2.7.2 documentation - http://docs.python.org/library/csv.html

Read:
<pre>
import csv
spamReader = csv.reader(open('eggs.csv', 'rb'), delimiter=' ', quotechar='|')
for row in spamReader:
print ', '.join(row)
</pre>

Write:
<pre>
import csv
spamWriter = csv.writer(open('eggs.csv', 'wb'), delimiter=' ',
quotechar='|', quoting=csv.QUOTE_MINIMAL)
spamWriter.writerow(['Spam'] * 5 + ['Baked Beans'])
spamWriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam'])
</pre>

=== SQLite ===

11.13. sqlite3 — DB-API 2.0 interface for SQLite databases — Python v2.7.2 documentation - http://docs.python.org/library/sqlite3.html

sqlite3 - New in version 2.5.

<pre>
import sqlite3

conn = sqlite3.connect('/tmp/example')

c = conn.cursor()

# Create table
c.execute('''create table stocks
(date text, trans text, symbol text,
qty real, price real)''')

# Insert a row of data
c.execute("""insert into stocks
values ('2006-01-05','BUY','RHAT',100,35.14)""")

# Show affected row count
print c.rowcount

# Save (commit) the changes
conn.commit()

# We can also close the cursor if we are done with it
c.close()
</pre>

parameter substitution:
<pre>
# Never do this -- insecure!
symbol = 'IBM'
c.execute("... where symbol = '%s'" % symbol)

# Do this instead
t = (symbol,)
c.execute('select * from stocks where symbol=?', t)
c.execute('select * from stocks where symbol=?', (symbol,))

# Larger example
for t in [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00),
('2006-04-06', 'SELL', 'IBM', 500, 53.00),
]:
c.execute('insert into stocks values (?,?,?,?,?)', t)
</pre>

select results:
<pre>
c = conn.cursor()
c.execute('select * from stocks order by price')

for row in c:
print row # row is a tuple

# OR (but not both)

rows = c.fetchall() # rows is list of tuples
print len(rows)
for row in rows:
print row
</pre>

Show which version of sqlite3: [http://zetcode.com/db/sqlitepythontutorial/]
<pre>
#!/usr/bin/python
# -*- coding: utf-8 -*-

import sqlite3 as lite
import sys

con = None

try:
con = lite.connect('test.db')

cur = con.cursor()
cur.execute('SELECT SQLITE_VERSION()')

data = cur.fetchone()

print "SQLite version: %s" % data

except lite.Error, e:

print "Error %s:" % e.args[0]
sys.exit(1)

finally:

if con:
con.close()
</pre>

Or simply: [http://stackoverflow.com/questions/1545479/force-python-to-forego-native-sqlite3-and-use-the-installed-latest-sqlite3-ver]
# echo -e "import sqlite3\nprint sqlite3.sqlite_version_info" | python
(3, 3, 6)

=== MySQL Database ===

Ubuntu packages:
* python-mysqldb - Python interface to MySQL
* python-mysql.connector - pure Python implementation of MySQL Client/Server protocol

apt-get install python-mysqldb
>>> import _mysql

apt-get install python-mysql.connector

PIP:
* https://pypi.python.org/pypi/MySQL-python
* https://pypi.python.org/pypi/mysqlclient

pip install MySQL-python

pip install mysqlclient

---

_mysql - http://mysql-python.sourceforge.net/MySQLdb.html

apt-get install python-mysqldb

<pre>
import _mysql

db=_mysql.connect(host="localhost",user="user",
passwd="pass",db="db")

db.query("""SELECT spam, eggs, sausage FROM breakfast
WHERE price < 5""")

r=db.store_result()
# ...or...
r=db.use_result()

row = r.fetch_row()
</pre>

=== couchdb ===

Getting started with couchdb-python — couchdb-python v0.8 documentation - http://packages.python.org/CouchDB/getting-started.html

<pre>
couch = couchdb.Server('http://127.0.0.1:5984/')

# database
db = couch.create('test') # create new
db = couch['mydb'] # or get existing

# print databases:
for db in couch:
print db

doc = {'foo': 'bar'}
doc_id, doc_rev = db.save(doc)
# (u'9278f7691c3421227a9f8436540008f9', u'1-4c6114c65e295552ab1019e2b046b10e')
# note: this also updates doc with _rev and _id
doc = db['9278f7691c3421227a9f8436540008f9']

# better to create a document using an id you want, instead of the randomly generated one
db[doc.id] = doc # this will write immediately to the database

# print all docs:
for id in db:
print id

# delete doc
db.delete(doc)

# delete database
couch.delete('test')
</pre>

Search database:
<pre>
# show all records
map_fun = '''function(doc) { emit(doc.name, null); }'''

# row = <Row id=u'12341234123412341234', key=None, value=None>
# emit(key, value) - returns the values for key and value

map_fun = '''
function(doc) {
if (doc.username == 'virt-01')
emit(doc.username, null);
}
'''

# row = <Row id=u'12341234123412341234', key=None, value=None>
for row in db.query(map_fun):
print row.id

for row in db.query(map_fun, descending=True):
print row.id
</pre>

couchdb-python - CouchDB Python Library - Google Project Hosting - http://code.google.com/p/couchdb-python/

Installation:
<pre>
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/C/CouchDB/CouchDB-0.8.tar.gz#md5=e32b39e459e1fa2586ea252712a11a59
tar -zvxf CouchDB-0.8.tar.gz
cd CouchDB-0.8
python setup.py install
</pre>

Test:
echo "import couchdb" | python

=== nose ===

nose: nose extends unittest to make testing easier

nose : Python Package Index - http://pypi.python.org/pypi/nose

Install:
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/n/nose/nose-1.1.2.tar.gz#md5=144f237b615e23f21f6a50b2183aa817
tar -zvxf nose-1.1.2.tar.gz
cd nose-1.1.2
sudo python setup.py install

skeleton:
mkdir skeleton
cd skeleton
mkdir bin NAME tests docs
touch NAME/__init__.py
touch tests/__init__.py
touch tests/NAME_tests.py

nose test skeleton:
<pre>
# tests/NAME_tests.py
from nose.tools import *
import klib

def setup():
print "SETUP!"

def teardown():
print "TEAR DOWN!"

def test_basic():
print "I RAN!"

# def test_something_else():
</pre>

Usage:
nosetests

See Also:
* [[#doctest]]
* [[#unittest]]

=== pydoc ===

25.1. pydoc — Documentation generator and online help system — Python v2.7.2 documentation - http://docs.python.org/library/pydoc.html

pydoc - The pydoc module automatically generates documentation from Python modules.

pydoc - same as 'import [library], help([library])'
$ pydoc [library]

Note: lists some special 'module metadata variables' separately (e.g. version, author).

special pydoc recognized metadata sections:
__version__
__author__
__credits__
__date__

==== pydoc function module ====

<pre>
Help on package klib:

NAME
klib - This is the name

FILE
/root/klib/klib/__init__.py

DESCRIPTION
These are module doc strings
that give helpful details

PACKAGE CONTENTS
markone

FUNCTIONS
func1()
This is a doc string
for func1

DATA
PI = 3.141
__all__ = ['func1', 'PI']
__author__ = 'Kenneth <kenneth@oeey.com>'
__copyright__ = 'Copyright 2007, The Project'
__credits__ = 'Credits go to me'
__date__ = '26 February 2001'
__email__ = 'ken@oeey.com'
__license__ = 'GPL'
__maintainer__ = 'Ken'
__status__ = 'Production'
__version__ = '$Revision: 83492 $'

VERSION
83492

DATE
26 February 2001

AUTHOR
Kenneth <kenneth@oeey.com>

CREDITS
Credits go to me
</pre>

Source:
<pre>
"""
This is the name

These are module doc strings
that give helpful details
"""

#__version__ = "1.0"
__version__ = "$Revision: 83492 $"
__author__ = "Kenneth <kenneth@oeey.com>"

__date__ = "26 February 2001"
__credits__ = "Credits go to me"

__all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license'
__license__ = "GPL"
__copyright__ = "Copyright 2007, The Project"
__maintainer__ = "Ken"
__email__ = "ken@oeey.com"
__status__ = "Production"

def func1():
"""
This is a doc string
for func1
"""
</pre>

==== pydoc class module ====

Sample:
<pre>
Help on module class_pydoc:

NAME
class_pydoc - This is the name

FILE
/root/class_pydoc.py

DESCRIPTION
These are module doc strings
that give helpful details

CLASSES
Toad

class Toad
| doc string for object
|
| Methods defined here:
|
| __init___(self, name)
| Constructor
|
| instance_function(self)
| Instance Function
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| class_function(cls) from __builtin__.classobj
| Class Function
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| my_class_attribute = 'sample class attribute'

DATA
PI = 3.141
__author__ = 'Kenneth <kenneth@oeey.com>'
__copyright__ = 'Copyright 2007, The Project'
__credits__ = 'Credits go to me'
__date__ = '26 February 2001'
__email__ = 'ken@oeey.com'
__license__ = 'GPL'
__maintainer__ = 'Ken'
__status__ = 'Production'
__version__ = '1.0'

VERSION
1.0

DATE
26 February 2001

AUTHOR
Kenneth <kenneth@oeey.com>

CREDITS
Credits go to me
</pre>

Code:
<pre>
"""
This is the name

These are module doc strings
that give helpful details
"""

__version__ = "1.0"
#__version__ = "$Revision: 83492 $"
__author__ = "Kenneth <kenneth@oeey.com>"

__date__ = "26 February 2001"
__credits__ = "Credits go to me"

#__all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license'
__license__ = "GPL"
__copyright__ = "Copyright 2007, The Project"
__maintainer__ = "Ken"
__email__ = "ken@oeey.com"
__status__ = "Production"

class Toad:
""" doc string for object """

my_class_attribute = 'sample class attribute'

def __init___(self, name):
""" Constructor """

def instance_function(self):
""" Instance Function """

@classmethod
def class_function(cls):
""" Class Function """
</pre>

=== username ===

User ID:
os.getuid()

User Name:
# fails it stdin piped to program
os.getlogin()

User Name:
import sys
import pwd
def get_username():
return pwd.getpwuid( os.getuid() )[ 0 ]

Environment:
os.environ['USER']

=== hostname ===

import socket
print(socket.gethostname())

os.uname()[1]

hostname = os.system('hostname')

=== Networking ===

==== Get Hostname ====

import socket
print(socket.gethostname())

==== Convert hostname to ip address ====

>>> socket.gethostbyaddr('demo')
('demo.oeey.com', [], ['10.10.10.10'])

/usr/local/bin/whatip: (or nametoip)
<pre>
#!/usr/bin/env python

#
# whatip - convert hostname to ip address
# author: Kenneth Burgener <kenneth@oeey.com> (FEB 2012)
#

import socket
import sys

if len(sys.argv) != 2:
print "Convert Hostname to IP Address"
print "Usage: %s HOSTNAME" % sys.argv[0].split('/')[-1]
sys.exit(1)

hostname = sys.argv[1]
try:
result = socket.gethostbyaddr(hostname)
print result[2][0]
except (socket.gaierror, socket.herror) as e:
print "Error: Unable to resolve '%s'" % hostname
sys.exit(1)
</pre>

==== Get Public IP Address ====

<pre>
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("gmail.com",80))
print(s.getsockname()[0])
s.close()
</pre>

networking - Finding local IP addresses using Python's stdlib - Stack Overflow - http://stackoverflow.com/questions/166506/finding-local-ip-addresses-using-pythons-stdlib

==== Python Network Server ====

<pre>
#!/usr/bin/python # This is server.py file

import socket # Import socket module

s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.
s.bind((host, port)) # Bind to the port

s.listen(5) # Now wait for client connection.
while True:
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
c.send('Thank you for connecting')
c.close() # Close the connection
</pre>

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
* 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

==== Python Network Client ====

<pre>
#!/usr/bin/python # This is client.py file

import socket # Import socket module

s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.

s.connect((host, port))
print s.recv(1024)
s.close # Close the socket when done
</pre>

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
* 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

=== Run Library Module as Script ===

Run python with -m option:
-m mod : run library module as a script (terminates option list)

Example:
cat [json_file] | python -m json.tool
# same as:
cat [json_file] | python /opt/python26/lib/python2.6/json/tool.py

Good for when library is buried deep in the library directories

=== Hex ===

<pre>
>>> hex(10) # print integer as hex string
'0xa'
>>> sys.hexversion # python version as integer
33817584
>>> hex(sys.hexversion) # python version as hex
'0x20403f0'
>>> print "0x%08x" % sys.hexversion # python version as hex
'0x20403f0'
>>> int('0xa', 16) # hex string to integer
10
>>> int('0a', 16) # hex string to integer
10
>>> int('0xa', 0) # hex string to integer auto detected du to '0x'
10
</pre>

=== Binary ===

Integer to Binary String with Negative Number Support: (2's complement)
<pre>
# source: http://www.gossamer-threads.com/lists/python/python/645216
def bin(x):
if x < -32768 or x > 65535:
raise Exception("Out of Range Error: %d" % x)
if x < 0:
x = 65535 + x + 1
bin_str = ''
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(15, 11, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(11, 7, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(7, 3, -1))
bin_str += ' '
bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(3, -1, -1))
return bin_str
</pre>

Integer to Binary String:
<pre>
# source: http://www.gossamer-threads.com/lists/python/python/645216
def bin(x):
if x < 0 or x > 65535:
raise Exception("Out of Range Error: %d" % x)
bin_str = ''
bin_str = ''.join(x & (1 << i) and '1' or '0' for i in range(15, 7, -1))
bin_str = bin_str + ' '
bin_str = bin_str + ''.join(x & (1 << i) and '1' or '0' for i in range(7, -1, -1))
return bin_str

print bin(0) # '00000000 00000000'
print bin(1) # '00000000 00000001'
print bin(255) # '00000000 11111111'
print bin(256) # '00000001 00000000'
print bin(43690) # '10101010 10101010'
print bin(65535) # '11111111 11111111'
print bin(65536) # '00000000 00000000' - rolled over to next set!
</pre>

Integer to Binary:
bin(170) # '0b10101010'

Binary to Integer:
int('10101010', 2) # 170

=== bitwise ===

See [[#Binary]]

BitwiseOperators - Python Wiki - http://wiki.python.org/moin/BitwiseOperators

<pre>
The Operators:

x << y
Returns x with the bits shifted to the left by y places (and new bits on the right-hand-side are zeros). This is the same as multiplying x by 2**y.
x >> y
Returns x with the bits shifted to the right by y places. This is the same as //'ing x by 2**y.
x & y
AND - Does a "bitwise and". Each bit of the output is 1 if the corresponding bit of x AND of y is 1, otherwise it's 0.
x | y
OR - Does a "bitwise or". Each bit of the output is 0 if the corresponding bit of x AND of y is 0, otherwise it's 1.
~ x
Returns the complement of x - the number you get by switching each 1 for a 0 and each 0 for a 1. This is the same as -x - 1.
x ^ y
XOR - Does a "bitwise exclusive or". Each bit of the output is the same as the corresponding bit in x if that bit in y is 0, and it's the complement of the bit in x if that bit in y is 1.
</pre>

=== Script Path ===

<pre>
if __name__ == "__main__":
# Change directory to where this script lives so we can find relative
# paths from here regardless of where the working directory was when we
# ran the script.
# APP_DIR = os.path.dirname(sys.argv[0])
APP_DIR = os.path.dirname(os.path.abspath(sys.argv[0))
if APP_DIR:
os.chdir(APP_DIR)
</pre>

normpath(path) # Normalize path, eliminating double slashes, etc.
abspath(path) # Return an absolute path

Path to a module: [http://stackoverflow.com/questions/247770/retrieving-python-module-path] (couldn't fine a_module)
import a_module
import os
path = os.path.dirname(a_module.__file__)

Path to module: [http://stackoverflow.com/questions/50499/in-python-how-do-i-get-the-path-and-name-of-the-file-that-is-currently-executin]
import inspect, os
print inspect.getfile(inspect.currentframe()) # script filename (usually with path)
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))) # script directory

Even better:
print os.path.realpath(__file__)
print os.path.dirname(__file__)
print os.path.basename(__file__)

references:
* http://stackoverflow.com/questions/595305/python-path-of-script

keywords: scriptpath script path program path pathname

=== Singleton ===

GLOBAL OBJECT!

None is a singleton (Python only has one copy of None in the interpreter).

Make your own singleton:

override the new method like this:
<pre>
class Singleton(object):
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Singleton, cls).__new__(
cls, *args, **kwargs)
return cls._instance

if __name__ == '__main__':
s1=Singleton()
s2=Singleton()
if(id(s1)==id(s2)):
print "Same"
else:
print "Different"
</pre>

Using decorator -implementation from PEP318? Implementing the singleton pattern with a decorator:
<pre>
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance

@singleton
class MyClass:
...
</pre>

* Python and the Singleton Pattern - Stack Overflow - http://stackoverflow.com/questions/42558/python-and-the-singleton-pattern

=== XML ===

<pre>
from xml.dom.minidom import parse, parseString

dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name

datasource = open('c:\\temp\\mydata.xml')
dom2 = parse(datasource) # parse an open file

dom3 = parseString('<myxml>Some data<empty/> some more data</myxml>')
</pre>

Pretty Print: [http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python]
<pre>
import xml.dom.minidom

xml = xml.dom.minidom.parse(xml_fname) # or xml.dom.minidom.parseString(xml_string)
pretty_xml_as_string = xml.toprettyxml()
</pre>

Pretty Print with etree [http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python]
<pre>
import lxml.etree as etree

x = etree.parse("filename")
print etree.tostring(x, pretty_print = True)
</pre>

References:
* 19.7. xml.dom.minidom — Lightweight DOM implementation — Python v2.7.3 documentation - http://docs.python.org/library/xml.dom.minidom.html
* Pretty printing XML in python - Stack Overflow - http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python

---

Simply don't include self.output.startElement and self.output.endElement for elements you don't want:

simple_filter_parser.py: (my version of [http://freegeographytools.com/2010/removin-time-data-out-of-a-gpx-or-kml-file-to-make-it-work-right-in-google-earth])
<pre>
# this parses out the time element of a gpx for Google Earth
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class my_filter_xml(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self.indent = 0
self.ignore = False
self.output.startDocument()

def startElement(self, name, attrs):
print " " * self.indent + "start: " + name
if attrs.keys():
keys = " " * (self.indent + 2) + "attrs:"
for key in attrs.keys():
keys = keys + " " + key + "=" + attrs[key]
print keys
self.indent = self.indent + 2
#self.output.startElement(name, {})
if name == "time":
self.ignore = True
#if not self.ignore:
self.output.startElement(name, attrs)

def characters(self, ch):
if ch.strip():
print " " * self.indent + "ch: " + ch.strip()
if not self.ignore:
self.output.characters(ch)
else:
if not self.ignore:
self.output.characters("\n")

def endElement(self, name):
self.indent = self.indent - 2
print " " * self.indent + "/end: " + name
#if not self.ignore:
self.output.endElement(name)
if name == "time":
self.ignore = False

if __name__ == '__main__':
source = open("tracks.gpx")
dest = open("tracks_out.gpx", 'w+')
xml.sax.parse(source, my_filter_xml(dest, True))
</pre>

---

fixdescriptor.py:
<pre>
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self._outfile = outfile
self._verbose = verbose
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self._filter = [ 'payload',
'payloads',
'file-list',
'file',
'checksum' ]
self._keep = [ 'vib',
'maintenance-mode' ]
self._inside = None

def startElement(self, name, attrs):
# Certain elements are kept unchanged
if name in self._keep:
if self._verbose:
print '%s element start found' % (name)
self.output.startElement(name, {})
if name == 'maintenance-mode':
if self._verbose:
print 'inside maintenance-mode'
self._inside = name
return
# Other elements are filtered (removed)
elif name in self._filter:
if self._verbose:
print '%s element start found' % (name)
self._inside = name
return
self.output.startElement(name, attrs)

def characters(self, ch):
if self._inside == 'maintenance-mode':
if self._verbose:
print 'new content for maintenance-mode'
self.output.characters('false')
elif self._inside not in self._filter:
self.output.characters(ch)

def endElement(self, name):
# If it's not a checksum element, only print it
if name in self._filter + self._keep:
self._inside = None
if self._verbose:
print '%s element end found' % (name)
if name in self._filter:
return
self.output.endElement(name)

if __name__ == '__main__':
#
parser = optparse.OptionParser(usage='%prog ...')
parser.add_option('-v', '--verbose',
action='store_true', default=False,
help='Print more information.')
parser.add_option('--input-xml',
type='str', default='descriptor.xml',
help='Name of input XML [default: %default]')
parser.add_option('--output-xml',
type='str', default='fixed_descriptor.xml',
help='Name of output XML [default: %default]')
opts, args = parser.parse_args(argv[1:])
source = open(opts.input_xml)
dest = open(opts.output_xml, 'w+')
xml.sax.parse(source, fix_descriptor(dest, opts.verbose))
</pre>

--

Simple read and output parsing:
<pre>
import os
import sys
import optparse
import fnmatch
from sys import argv
import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler):
def __init__(self, outfile, verbose=False):
self._outfile = outfile
self._verbose = verbose
self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8')
self._inside = None

def startElement(self, name, attrs):
# attrs.items() is a list of tuples (name, value)
print "start:", name, "/", attrs.items()
self.output.startElement(name, attrs)

def characters(self, ch):
print "char:", self._inside, "/", ch
self.output.characters(ch)

def endElement(self, name):
print "end:", name
self.output.endElement(name)

if __name__ == '__main__':
parser = optparse.OptionParser(usage='%prog ...')
parser.add_option('-v', '--verbose',
action='store_true', default=False,
help='Print more information.')
parser.add_option('--input-xml',
type='str', default='descriptor.xml',
help='Name of input XML [default: %default]')
parser.add_option('--output-xml',
type='str', default='fixed_descriptor.xml',
help='Name of output XML [default: %default]')
opts, args = parser.parse_args(argv[1:])
source = open(opts.input_xml)
dest = open(opts.output_xml, 'w+')
xml.sax.parse(source, fix_descriptor(dest, opts.verbose))
</pre>

=== Unicode ===

Unicode string:
u'Hello World !'

Raw Unicode string:
>>> ur'Hello\u0020World !'
u'Hello World !'

Convert:
u = u'hello\xe4'
str(u) # UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 5: ordinal not in range(128)
s = u.encode('utf8') # 's' is now type str

References:
* Introduction to Unicode Strings - http://docs.python.org/tutorial/introduction.html#unicode-strings

=== Zip ===

See [[Python/Zip]]

=== signals ===

signal.SIGTERM : kill [pid]
signal.SIGALRM : kill -ALRM [pid]
signal.SIGHUP : kill -HUP [pid]

<pre>
import signal, os

def handler(signum, frame):
print 'Signal handler called with signal', signum
raise IOError("Couldn't open device!")

# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)

# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)

signal.alarm(0) # Disable the alarm
</pre>

Note: use signal.SIGTERM to handle "kill" command

References:
* 17.4. signal — Set handlers for asynchronous events — Python v2.7.5 documentation - http://docs.python.org/2/library/signal.html

==== Ctrl+C ====

<pre>
#!/usr/bin/env python
import signal
import sys
def signal_handler(signal, frame):
print 'You pressed Ctrl+C!'
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
print 'Press Ctrl+C'

# linux only:
signal.pause()

# windows alternative:
#import time
#time.sleep(60)
</pre>

References:
* control - How do I capture SIGINT in Python? - Stack Overflow - http://stackoverflow.com/questions/1112343/how-do-i-capture-sigint-in-python

==== Non-blocking raw_input for Python ====

Non-blocking raw_input for Python:
<pre>
# source: http://www.garyrobinson.net/2009/10/non-blocking-raw_input-for-python.html

import signal

class AlarmException(Exception):
pass

def alarmHandler(signum, frame):
raise AlarmException

def nonBlockingRawInput(prompt='', timeout=20):
signal.signal(signal.SIGALRM, alarmHandler)
signal.alarm(timeout)
try:
text = raw_input(prompt)
signal.alarm(0)
return text
except AlarmException:
print '\nPrompt timeout. Continuing...'
signal.signal(signal.SIGALRM, signal.SIG_IGN)
return ''
</pre>

=== email ===

<pre>
import smtplib
import email.mime.text

msg_text = "Hello World"
msg = email.mime.text.MIMEText(msg_text)

# me == the sender's email address
# you == the recipient's email address
msg['Subject'] = 'The contents of %s' % textfile
msg['From'] = me
msg['To'] = you

# Send the message via our own SMTP server, but don't include the
# envelope header.
s = smtplib.SMTP('localhost')
s.sendmail(me, [you], msg.as_string())
s.quit()
</pre>

my nagios fake /bin/mail:
<pre>
#!/usr/bin/env python

import smtplib
import email.mime.text
import optparse
import os
import sys
import pwd

os.system('logger mail called')

parser = optparse.OptionParser()
parser.add_option(
'-s', '--subject',
help='Subject',
default="")
parser.add_option(
'-t', '--to',
help='To',
dest="send_to",
default=None)
parser.add_option(
'-f', '--from',
help='From',
dest="send_from",
default=None)
parser.add_option(
'-S', '--server',
help='server',
default="prime")
(options, args) = parser.parse_args()

def get_username():
return pwd.getpwuid( os.getuid() )[ 0 ]

if options.send_to:
send_to = options.send_to
else:
if len(args) == 1:
send_to = args[0]
else:
parser.print_help()
sys.exit(1)

if options.send_from:
send_from = options.send_from
else:
username = get_username()
hostname = os.uname()[1]
send_from = username + "@" + hostname

msg_text = sys.stdin.read()
msg = email.mime.text.MIMEText(msg_text)

msg['Subject'] = options.subject
msg['From'] = send_from
msg['To'] = send_to

s = smtplib.SMTP(options.server)
s.sendmail(send_from, [send_to], msg.as_string())
s.quit()
</pre>

Authenticated (with gmail):
<pre>
import smtplib
import email.mime.text

# from_email = "kenneth@demo.oeey.com"
# to_email = "kenneth@demo.oeey.com"
from_email = "Kenneth Burgener <kenneth@demo.oeey.com>"
to_emails = ["Kenneth Burgener <kenneth@demo.oeey.com>", "Kenneth Cell <xxx@xxx.com>"]

msg_text = "hello world"
msg = email.mime.text.MIMEText(msg_text)
msg['Subject'] = 'Test text'
msg['From'] = from_email
# msg['To'] = to_email
msg['To'] = ", ".join(to_emails)
# s = smtplib.SMTP('localhost')
print "sending..."
s = smtplib.SMTP_SSL('smtp.gmail.com', '465')
s.login('xxx', 'xxx')
# s.sendmail(from_email, [to_email], msg.as_string())
s.sendmail(from_email, to_emails, msg.as_string())
s.quit()
print "sent!"
</pre>

Source:
* 20.12. smtplib — SMTP protocol client — Python v2.7.5 documentation - http://docs.python.org/2/library/smtplib.html
* 18.1.11. email: Examples — Python v2.7.5 documentation - http://docs.python.org/2/library/email-examples.html

=== pprint ===

8.18. pprint — Data pretty printer — Python v2.7.5 documentation - http://docs.python.org/2/library/pprint.html

import pprint
pprint.pprint(myVar)

import pprint
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(myVar)

Note: Useless on objects!

Dump object:
<pre>
def dumpobject(obj):
print "<<<---"
print type(obj), obj.__class__.__name__
for attr in dir(obj):
if attr[0] == "_": continue # ignore under/dunder properties
if str(type(getattr(obj, attr))) == "<type 'instancemethod'>":
print attr, ": METHOD"
else:
print attr, ":", type(getattr(obj, attr)), ":", getattr(obj, attr)
print "END --->>>"
</pre>

=== Curses ===

Curses Programming with Python — Python v3.3.2 documentation - http://docs.python.org/3/howto/curses.html

"The curses library supplies a terminal-independent screen-painting and keyboard-handling facility for text-based terminals; such terminals include VT100s, the Linux console, and the simulated terminal provided by various programs. Display terminals support various control codes to perform common operations such as moving the cursor, scrolling the screen, and erasing areas. Different terminals use widely differing codes, and often have their own minor quirks.

In a world of graphical displays, one might ask “why bother”? It’s true that character-cell display terminals are an obsolete technology, but there are niches in which being able to do fancy things with them are still valuable. One niche is on small-footprint or embedded Unixes that don’t run an X server. Another is tools such as OS installers and kernel configurators that may have to run before any graphical support is available."

---

Thread from Utah Python User's Group mailing list:

-

Hello, I am trying to learn more about curses in python for a desktop application. Does anybody have any recommendations on tutorials or documentation?

-Corban

-

We've use urwid at work. Seems to be a good approach if you aren't already tied to something. Learning curve can be a bit steep (well, compared to what you'd think it should be). Good luck!

http://excess.org/urwid/

Gabriel Gunderson

-

This video series looks pretty good:

http://www.youtube.com/user/pythoncursestutorial?feature=watch

Official HOWTOs from the python.org site:

http://docs.python.org/2/howto/curses.html
http://docs.python.org/3/howto/curses.html

As Gabe mentioned, there are other libraries that provide similar or
greater functionality, so if you aren't doing this specifically to
learn curses, it's worth at least looking at a few of them to compare.

There's a brief comparison of three possibilities here:

http://stackoverflow.com/questions/8349085/python-ncurses-cdk-urwid-difference

Daniel Fackrell

-

Here's another option that looks promising:

https://pypi.python.org/pypi/blessings/

Daniel Fackrell

=== dialog ===

pythondialog - http://pythondialog.sourceforge.net/
: "pythondialog is a Python wrapper for the dialog utility originally written by Savio Lam, and later rewritten by Thomas E. Dickey. Its purpose is to provide an easy to use, pythonic and comprehensive Python interface to dialog. This allows one to make simple text-mode user interfaces on Unix-like systems (including Linux). "

Note: The latest versions (2.12 and later) only support Python 3; users who really want to stick to Python 2 should use version 2.11 for now.

Install:
# latest:
# wget http://sourceforge.net/projects/pythondialog/files/latest/download?source=dlp

wget http://sourceforge.net/projects/pythondialog/files/pythondialog/2.11/python2-pythondialog-2.11.tar.gz/download
tar -zvxf python2-pythondialog-2.11.tar.gz
cd pythondialog-2.11

import dialog
d = dialog.Dialog()
if d.yesno("Continue?") == d.DIALOG_OK:
print "Continuing..."

---

My solution

kiloforce / dialog / source / — Bitbucket - https://bitbucket.org/kiloforce/dialog

<pre>
#!/usr/bin/env python

""" Wrapper for Linux 'dialog' command """

__author__ = "Kenneth Burgener <kenneth@k.ttak.org>"
__copyright__ = "(c) 2013"
__date__ = "$Sep 27, 2013$"
__version__ = "1.0"
__credits__ = r"""
XXXXXXXXXXXXXXXXXX
X Dialog Wrapper #
##################
"""

import subprocess
import time
import sys

class Dialog():
""" Python wrapper for Linux 'dialog' command

dialog - display dialog boxes from shell scripts

>>> import dialog
>>> dlg = dialog.Dialog()
>>> rc = dlg.msgbox('hello world')

Return codes:
YES = 0
OK = 0
EXIT = 0
NO = 1
CANCEL = 1
ESC = 255
"""

YES = 0
OK = 0
EXIT = 0
NO = 1
CANCEL = 1
ESC = 255

def __init__(self):
rc = self._exe('which dialog > /dev/null')[0]
if rc != 0:
print "Error: failed to find Linux dialog command"
sys.exit(1)

def _exe(self, cmd, strip=True):
p = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE)
output = "\n".join(p.stderr.readlines())
if strip:
output = output.strip()
rc = p.wait()
return (rc, output)

def rc_to_str(self, rc):
if rc == 0:
return "YES"
elif rc == 1:
return "NO"
elif rc == 255:
return "ESX"
else:
raise Exception("Unknown return code: %s" % rc)

def clear(self):
"""
Clear screen

Example:
>>> dialog.clear()"""
rc, output = self._exe('clear')
assert rc in (0,), "clear rc"
return rc

def msgbox(self, msg):
"""
Simple message box

@param msg (str) = display message
@return: rc (int)

Example:
>>> rc = dialog.msgbox('hello world')"""
rc, output = self._exe('dialog --msgbox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.ESC), "msgbox rc"
return rc

def yesno(self, msg):
"""
Yes/No dialog

@param msg (str) = display message
@return: rc (int)

Example:
>>> rc = dialog.yesno('continue')"""
rc, output = self._exe('dialog --yesno "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.NO, self.ESC), "yesno rc"
return rc

def infobox(self, msg):
"""
Dialog that does not wait for user's response
(do your own pause)

@param msg (str) = display message
@return: rc (int) always zero

Example:
>>> for i in range(10):
>>> dialog.infobox('counting: ' + str(i))
>>> time.sleep(3)"""
rc, output = self._exe('dialog --infobox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES,), "infobox rc"
return rc

def inputbox(self, msg):
"""
Get user input

@param msg (str) = display message
@return: rc (int), output (str/None)

Example:
>>> rc, output = dialog.inputbox('name:')"""
rc, output = self._exe('dialog --inputbox "{msg}" 0 0'.format(msg=msg))
assert rc in (self.YES, self.NO, self.ESC), "inputbox rc"
if not output:
output = None
return rc, output

def textbox(self, filename):
"""
Display a file

@param filename (str) = file to display
@return: rc (int)

Example:
>>> dialog.textbox('/etc/passwd')"""
rc, output = self._exe('dialog --textbox {filename} 0 0'.format(filename=filename))
assert rc in (self.YES, self.ESC), "textbox rc"
return rc

def menu(self, msg, item_list):
"""
Single selection checkbox menu

@param msg (str) = display message
@param item_list (str list) = list of items to display
@return: rc (int), selection (int/None)

Example:
>>> rc, selection = dialog.menu('Select multiple options:',
... ('option 1', 'option 2'))"""
item_str = ""
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
rc, output = self._exe('dialog --menu "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
if output:
selection = int(output)
else:
selection = None
assert rc in (self.YES, self.NO, self.ESC), "menu rc"
return rc, selection

def checklist(self, msg, item_list, checked_list=None):
"""
Multiple selection checkbox menu

@param msg (str) = display message
@param item_list (str list) = list of items
@param checked_list (int list) = default items to check
@return: rc (int), selections (int list/None)

Example:
>>> rc, selections = dialog.checklist('Select multiple options:',
... ('option 1', 'option 2')
"""
item_str = ""
assert type(item_list) in (type(None), type(()), type([])), "checklist: item_list must be a list"
assert type(checked_list) in (type(None), type(()), type([])), "checklist: checked_list must be a list"
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
if checked_list and i+1 in checked_list:
item_str += " on"
else:
item_str += " off"
rc, output = self._exe('dialog --checklist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
print item_str
selections = []
for choice in output.split('"'):
if choice.strip():
selections.append(int(choice))
if selections:
selections = tuple(selections)
else:
selections = None
assert rc in (self.YES, self.NO, self.ESC), "checklist rc"
return rc, selections

def radiolist(self, msg, item_list, checked_item=None):
"""
Single selection radio menu, with default

@param msg (str) = display message
@param item_list (str list) = list of items
@param checked_item (int) = default item to select
@return: rc (int), selection (int)

Example:
>>> rc, selection = dialog.radiolist('Select one option:',
>>> ('option 1', 'option 2'), 2)"""
item_str = ""
for i in range(len(item_list)):
item_str += " " + str(i+1) + " \"" + item_list[i] + "\""
if checked_item and i+1 == int(checked_item):
item_str += " on"
else:
item_str += " off"
rc, selection = self._exe('dialog --radiolist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))
if not selection:
selection = None
assert rc in (self.YES, self.NO, self.ESC), "radiolist rc"
return rc, selection

def gauge(self, msg, percent):
"""
Progress bar (does not wait for user's input)

@param msg (str) = display message
@param percent (int) = progress
@return: rc (int) always zero

Example:
>>> for i in range(0, 100+1, 2):
>>> dialog.gauge('Progress:', i)
>>> time.sleep(.1)"""
rc, output = self._exe('echo {percent} | dialog --gauge "{msg}" 0 0'.format(percent=percent, msg=msg))
assert rc in (self.YES,), "guage rc"
return rc

def progress(self, msg, percent):
"""
Alias for self.guage()
"""
return self.gauge(msg=msg, percent=percent)

def form(self, msg, item_list):
"""
Multiple item entry form

@param msg (str) = display message
@param item_list (str list) = list of items
@return: rc (int), answers (str list)

Example:
>>> rc, answers = dialog.form('Enter details',
... ('first name', 'last name'))"""
item_str = ""
name_len = 1
for i in range(len(item_list)):
if len(item_list[i])+3 > name_len:
name_len = len(item_list[i])+3
for i in range(len(item_list)):
item_str += " \"" + item_list[i] + ":\" " + str(i+1) + " 1 \"\" " + str(i+1) +\
" " + str(name_len) + " 40 0"
rc, answers = self._exe('dialog --form "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str),
strip=False)
answers = answers.split('\n\n')
if not answers:
answers = None
assert rc in (self.YES, self.NO, self.ESC), "form rc"
return rc, answers

def test(self):
"""
Test features

>>> import dialog
>>> dlg = dialog.Dialog()
>>> dlg.test()
"""

def wait_user(msg):
self.msgbox("%s" % msg)

# rc_to_str
wait_user("Testing return code...")
print self.rc_to_str(self.YES) # YES
print self.rc_to_str(self.NO) # NO
print self.rc_to_str(self.ESC) # ESX
try:
print self.rc_to_str(111) # exception
raise Exception("Did not throw exception")
except Exception as rc:
pass

# msgbox
wait_user("Testing msgbox...")
rc = self.msgbox("msgbox")
if rc not in (self.YES, self.ESC):
raise Exception("Unknown return code for msgbox: %s" % rc)

# yesno
wait_user("Testing yesno...")
rc = self.yesno("yesno")
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for yesno: %s" % rc)

# infobox
wait_user("Testing infobox...")
rc = self.infobox("infobox sleep 1 sec")
if rc not in (self.YES,):
raise Exception("Unknown return code for infobox %s" % rc)
time.sleep(1)

# inputbox
wait_user("Testing inputbox...")
rc, output = self.inputbox("inputbox")
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for inputbox: %s" % rc)
wait_user("Result: %s" % output)

# textbox
wait_user("Testing textbox...")
rc = self.textbox("/etc/passwd")
if rc not in (self.YES, self.ESC):
raise Exception("Unknown return code for textbox: %s" % rc)

# menu
wait_user("Testing menu...")
rc, selection = self.menu("menu", ("option 1", "option 2"))
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for menu: %s" % rc)
wait_user("Result: %s" % selection)

# checklist
wait_user("Testing checklist...")
rc, selection = self.checklist("menu", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for checklist: %s" % rc)
wait_user("Result: %s" % str(selection))
wait_user("Testing checklist with default selection...")
rc, selection = self.checklist("menu", ["option 1", "option 2"], [1, 2])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for checklist: %s" % rc)
wait_user("Result: %s" % str(selection))

# radiolist
wait_user("Testing radiolist...")
rc, selection = self.radiolist("menu", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for radiolist: %s" % rc)
wait_user("Result: %s" % str(selection))
wait_user("Testing radiolist with default selection...")
rc, selection = self.radiolist("menu", ["option 1", "option 2"], 2)
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for radiolist: %s" % rc)
wait_user("Result: %s" % str(selection))

# guage
wait_user("Testing guage...")
for i in range(4+1):
rc = self.gauge("test guage", i*25)
# rc = self.progress("test progress", i*25)
if rc not in (self.YES,):
raise Exception("Unknown return code for guage: %s" % rc)
time.sleep(1)

# form
wait_user("Testing form...")
rc, answers = self.form("form", ["option 1", "option 2"])
if rc not in (self.YES, self.NO, self.ESC):
raise Exception("Unknown return code for form: %s" % rc)
wait_user("Result: %s" % str(answers))

# done with test, cleanup
wait_user("Done with tests...")
self.clear()

if __name__ == "__main__":
# dialog = Dialog()
# dialog.test()
pass
</pre>

=== Busy Loop and Thread.yield() ===

"I want to tell my Python threads to yield, and so avoid hogging the CPU unnecessarily. In Java, you could do that using the Thread.yield() function"

Solution:
time.sleep(0)
time.sleep(0.00001)
time.sleep(epsilon)

References:
* multithreading - In there something similar to Java's Thread.yield() in Python? Does that even make sense? - Stack Overflow - http://stackoverflow.com/questions/1908206/in-there-something-similar-to-javas-thread-yield-in-python-does-that-even-ma
* multithreading - How does a threading.Thread yield the rest of its quantum in Python? - Stack Overflow - http://stackoverflow.com/questions/787803/how-does-a-threading-thread-yield-the-rest-of-its-quantum-in-python

=== USB ===

Programming with PyUSB 1.0 - http://pyusb.sourceforge.net/docs/1.0/tutorial.html

PyUSB - http://pyusb.sourceforge.net/

git clone https://github.com/walac/pyusb

import usb.core
dev = usb.core.find()

# find our device
dev = usb.core.find(idVendor=0xfffe, idProduct=0x0001)

"'''Dealing with multiple identical devices''' - Sometimes you may have two identical devices connected to the computer. How can you differentiate them? Device objects come with two additional attributes which are not part of the USB Spec, but are very useful: bus and address attributes. First of all, it is worth to say that these attributes come from the backend and a backend is free to not support them, in which case they are set to None. That said, these attributes represent the bus number and bus address of the device and, as you might already have imagined, can be used to differentiate two devices with the same idVendor and idProduct attributes."

=== Tkinter ===

Python GUI Bitcoin: [http://bitcoin.stackexchange.com/questions/10594/simplest-way-of-obtaining-mtgox-exchange-rate-via-an-api]
<pre>
import json
import urllib2
from Tkinter import *

url='http://api.bitcoincharts.com/v1/markets.json'
req=urllib2.Request(url)
response=urllib2.urlopen(req).read()
output=json.loads(response)

mtusd_price = output[-10]['avg']
mteur_price = output[31]['avg']
mtgbp_price = output[44]['avg']

MTUSD= 'USD' + ' ' + str(mtusd_price)
MTEUR= 'EUR' + ' ' + str(mteur_price)
MTGBP = 'GBP' + ' ' + str(mtgbp_price)

root= Tk()
root.title("Bitcoin Price")
root.geometry("250x100")

USD = Label(root, text=MTUSD)
EUR = Label(root, text=MTEUR)
GBP = Label(root, text=MTGBP)
USD.pack()
EUR.pack()
GBP.pack()
mainloop()
</pre>

=== ctypes ===

"ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python."
:-- 15.17. ctypes — A foreign function library for Python — Python v2.7.7 documentation - https://docs.python.org/2/library/ctypes.html

Use "c scope" to build a database of symbols to see what options are available.

---

testlib.c
<pre>
#include <stdio.h>

void myprint(void);

void myprint()
{
printf("hello world\n");
}
</pre>

Compile:
gcc -shared -Wl,-soname,testlib -o testlib.so -fPIC testlib.c

testlibwrapper.py

Python:
<pre>
import ctypes

testlib = ctypes.CDLL('/full/path/to/testlib.so')
testlib.myprint()
</pre>

Source: python - ctypes - Beginner - Stack Overflow - http://stackoverflow.com/questions/5081875/ctypes-beginner

---

nzjrs's Gists - https://gist.github.com/nzjrs

---

<pre>
#include <stdio.h>

int sum(double *x,int n)
{
int i;
double counter;
counter = 0;
for(i=0;i<n;i++)
{
counter=counter+x[i];

}
return counter;
}
</pre>

<pre>
gcc -c sum.c
gcc -shared -o sum.so sum.o
</pre>

<pre>
from ctypes import *
my_sum=CDLL('sum.so')
a=numpy.array(range(10),dtype=float)
my_sum.sum(a.ctypes.data_as(c_void_p),int(10))
</pre>

<pre>
a=(c_double*10)()
for i in range(10):
a[i]=i
my_sum.sum(a,int(10))
</pre>

Source: Ctypes — Numerical Computing with Sage v6.2 - http://www.sagemath.org/doc/numerical_sage/ctypes.html

---

=== Warnings ===

if sys.version_info >= (1, 0, 0):
warnings.warn("Danger Will Robinson: trying Phi on untested Python 3.0!")

=== enum ===

Python 3: https://docs.python.org/3/library/enum.html

<pre>
from enum import Enum

# class syntax
class Color(Enum):
RED = 1
GREEN = 2
BLUE = 3

# functional syntax
Color = Enum('Color', ['RED', 'GREEN', 'BLUE'])
</pre>

---

Python 2: [http://stackoverflow.com/questions/36932/how-can-i-represent-an-enum-in-python]
<pre>
class Enum(set):
""" Basic enumerated type

Example
>>> Animals = phi.Enum(["DOG", "CAT", "HORSE"])
>>> Animals.DOG
'DOG'
>>> 'HORSE' in Animals
True
"""
def __getattr__(self, name):
if name in self:
return name
raise AttributeError("Missing Enum property: %s" % name)
def __setattr__(self, name, value): # this makes it read-only
raise AttributeError("Unable to modify Enum property: %s (%s)" % (name, value))

Animals = Enum(["DOG", "CAT", "HORSE"])

print(Animals.DOG) # 'DOG'
</pre>

----------

Alternative:
<pre>
class Stationery:
Pen, Pencil, Eraser = range(0, 3)
</pre>

One liner:
class Enum(tuple): __getattr__ = tuple.index

<pre>
>>> State = Enum(['Unclaimed', 'Claimed'])
>>> State.Claimed
1
>>> State[1]
'Claimed'
>>> State
('Unclaimed', 'Claimed')
>>> range(len(State))
[0, 1]
>>> [(k, State[k]) for k in range(len(State))]
[(0, 'Unclaimed'), (1, 'Claimed')]
>>> [(k, getattr(State, k)) for k in State]
[('Unclaimed', 0), ('Claimed', 1)]
</pre>

Python 3 supports enum natively with the enum.Enum library:

=== pySerial ===

https://pythonhosted.org/pyserial/

"This module encapsulates the access for the serial port. It provides backends for Python running on Windows, OSX, Linux, BSD (possibly any POSIX compliant system) and IronPython. The module named “serial” automatically selects the appropriate backend."

https://github.com/pyserial/

Intro - https://pythonhosted.org/pyserial/shortintro.html

---

<pre>
import serial
import time

ser = serial.Serial(
port='COM4',\
baudrate=9600,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)

print("connected to: " + ser.portstr)

#this will store the line
seq = []
count = 1

while True:
for c in ser.read():
seq.append(chr(c)) #convert from ANSII
joined_seq = ''.join(str(v) for v in seq) #Make a string from array

if chr(c) == '\n':
print("Line " + str(count) + ': ' + joined_seq)
seq = []
count += 1
break

ser.close()
</pre>

Ref: http://stackoverflow.com/questions/16077912/python-serial-how-to-use-the-read-or-readline-function-to-read-more-than-1-char

---

I modified the code to monitor a COM port: (Windows)

Monitor Serial:

<pre>
import serial, time, sys

logger = open("serial.txt", "w")
timer = time.time()

def log(msg, head=False):
msg = msg.strip()
if head:
msg = ">>> " + msg
logger.write(msg + "\n")
sys.stdout.write(msg + "\n")
newtime = time.time()
if newtime > timer + 1:
logger.flush()
sys.stdout.flush()

while True:

ser = None
try:
ser = serial.Serial(
port='COM1',\
baudrate=115200,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)
except serial.serialutil.SerialException as e:
log(str(e), True)
time.sleep(1)
continue

log("Connected to: " + ser.portstr, True)
count=1

try:
line = []
while True:
for c in ser.readline():
line.append(c)
if c == "\n":
log("".join(line))
line = []
except serial.serialutil.SerialException as e:
log(str(e), True)
time.sleep(1)
ser.close()
</pre>

=== MyBool ===

<pre>
>>> class NewBool(int):
... def __new__(cls, value):
... return int.__new__(cls, bool(value))

>>> y = NewBool(2)
>>> y == 1
True
</pre>

https://jfine-python-classes.readthedocs.io/en/latest/subclass-int.html

Also check out the __cmp__

== md5 ==

import md5
md5.new("Nobody inspects the spammish repetition").hexdigest()
# 'bb649c83dd1ea5c9d9dec9a18df0ffe9'

<pre>
$ echo -n "Nobody inspects the spammish repetition" | md5sum
bb649c83dd1ea5c9d9dec9a18df0ffe9 -
<pre>

== External Libraries ==

=== execnet ===

Python Package Index - execnet - https://pypi.python.org/pypi/execnet

Examples - http://codespeak.net/execnet/examples.html
* basic local and remote communication - http://codespeak.net/execnet/example/test_info.html

Execute source code in subprocess, communicate through a channel:
<pre>
>>> import execnet
>>> gw = execnet.makegateway()
>>> channel = gw.remote_exec("channel.send(channel.receive()+1)")
>>> channel.send(1)
>>> channel.receive()
2
</pre>

Compare current working directories
<pre>
>>> import execnet, os
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec("import os; channel.send(os.getcwd())")
>>> res = ch.receive()
>>> assert res == os.getcwd()
</pre>

Get information from remote ssh account
<pre>
>>> import execnet, os
>>> gw = execnet.makegateway("ssh=codespeak.net")
>>> channel = gw.remote_exec("""
... import sys, os
... channel.send((sys.platform, sys.version_info, os.getpid()))
... """)
>>> platform, version_info, remote_pid = channel.receive()
>>> platform
'linux2'
>>> version_info
(2, 4, 2, 'final', 0)
</pre>

remote-exec a function (avoiding inlined source part I)
<pre>
import execnet

def multiplier(channel, factor):
while not channel.isclosed():
param = channel.receive()
channel.send(param * factor)

gw = execnet.makegateway()
channel = gw.remote_exec(multiplier, factor=10)

for i in range(5):
channel.send(i)
result = channel.receive()
assert result == i * 10

gw.exit()
</pre>

remote-exec a module (avoiding inlined source part II)
:"You can pass a module object to remote_exec in which case its source code will be sent. No dependencies will be transferred so the module must be self-contained or only use modules that are installed on the “other” side. Module code can detect if it is running in a remote_exec situation by checking for the special __name__ attribute."

<pre>
# content of a module remote1.py

if __name__ == '__channelexec__':
channel.send('initialization complete')
</pre>

<pre>
>>> import execnet, remote1
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec(remote1)
>>> print (ch.receive())
initialization complete
</pre>

Use a callback instead of receive() and wait for completion
<pre>
>>> import execnet
>>> gw = execnet.makegateway()
>>> channel = gw.remote_exec("for i in range(10): channel.send(i)")
>>> l = []
>>> channel.setcallback(l.append, endmarker=None)
>>> channel.waitclose() # waits for closing, i.e. remote exec finish
>>> l
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, None]
</pre>

a simple command loop pattern
<pre>
# contents of: remotecmd.py
import os

def simple(arg):
return arg + 1

def listdir(path):
return os.listdir(path)

if __name__ == '__channelexec__':
for item in channel:
channel.send(eval(item))
</pre>

<pre>
>>> import execnet, remotecmd
>>> gw = execnet.makegateway()
>>> ch = gw.remote_exec(remotecmd)
>>> ch.send('simple(10)') # execute func-call remotely
>>> ch.receive()
11
</pre>

=== evdev ===

Install:
apt-get install python-setuptools python-dev
easy_install pip
pip install evdev

Simple read usage:
<pre>
from evdev import InputDevice
from evdev import ecodes

# look for a /dev/input/by-id/usb...kbd or similar
DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd"

dev = InputDevice(DEVICE)

while True:
r, w, x = select([dev], [], [], 5)
for event in dev.read():
if event.type == 1 and event.value == 1:
print event.code
</pre>

List LED states:
dev.leds()
dev.leds(verbose=True)

Inject usage: (does not appear to really work)
<pre>
from evdev import InputDevice
from evdev import ecodes
from evdev import UInput

DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd"
dev = InputDevice(DEVICE)

ui = UInput()

ui.write(ecodes.EV_KEY, ecodes.KEY_4, 1)
ui.write(ecodes.EV_KEY, ecodes.KEY_4, 0)
ui.syn()

ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 1)
ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 0)
ui.syn()

ui.close()
</pre>

Tutorial:
* evdev documentation — python-evdev 0.4.3 documentation - http://pythonhosted.org/evdev/index.html
References:
* python-evdev/doc/tutorial.rst at master · gvalkov/python-evdev · GitHub - https://github.com/gvalkov/python-evdev/blob/master/doc/tutorial.rst

=== ldap-python ===

python-ldap: LDAP client API for Python - http://www.python-ldap.org/

Index of Packages : Python Package Index - https://pypi.python.org/pypi/python-ldap/

yum install python-devel openldap-devel
pip install python-ldap

Articles:
* Python LDAP Applications: Part 1 - Installing and Configuring the Python-LDAP Library and Binding to an LDAP Directory | Packt Publishing - http://www.packtpub.com/article/installing-and-configuring-the-python-ldap-library-and-binding-to-an-ldap-directory

=== xmmp ===

PDF:
* XMPP: The Definitive Guide - Oreilly - http://oriolrius.cat/blog/wp-content/uploads/2009/10/Oreilly.XMPP.The.Definitive.Guide.May.2009.pdf

xmmp is the Jabber protocol

Best options:

MIT libraries.
* sleekxmpp (was: sleekxmpp) is pretty popular and is used for examples in Peter Saint-Andre's XMPP book from O'Reilly. It reportedly works well, and finally got an email list in April 2010, and has a chat room at sleek@conference.jabber.org.

GPL libraries.
* xmpppy is used by gajim.

LPGL libraries.
* pyxmpp is pretty good and uses libxml2 internally for xml parsing.
* pyxmpp2 is the next version of pyxmpp, runs on python 2.7 and 3.2, and removes the libxml2 requirement. Like many, it requires dnspython.
* jabberpy is the original and still works for a lot of tasks, but is thoroughly unmaintained.

Other libraries.
* Wokkel, mentioned in another post. That's a new one for me, based on twisted.

chat - Best python XMPP / Jabber client library? - Stack Overflow - http://stackoverflow.com/questions/1901828/best-python-xmpp-jabber-client-library

=== xmpppy ===

xmpppy: the jabber python project - http://xmpppy.sourceforge.net/

"xmpppy is a Python library that is targeted to provide easy scripting with Jabber. Similar projects are Twisted Words and jabber.py.

This library was not designed from scratch. It inherits some code from jabberpy and have very similar API in many places. Though it is separate project since it have almost completely different architecture and primarily aims to work with jabberd2 - the new Open Source Jabber Server.

xmpppy is distributed under the terms of GNU General Public License and can be freely redistributed without any charge."

wget http://sourceforge.net/projects/xmpppy/files/xmpppy/0.4.1/xmpppy-0.4.1.tar.gz/download -O xmpppy-0.4.1.tar.gz
tar -zvxf xmpppy-0.4.1.tar.gz
pip install pydns

--- basic connection ---

<pre>
#!/usr/bin/env python

import xmpp

user="username@gmail.com"
password="password"
server="gmail.com"

jid = xmpp.JID(user)
connection = xmpp.Client(server,debug=[])
connection.connect()
result = connection.auth(jid.getNode(), password,"LFY-client")

connection.sendInitPresence()

while connection.Process(1):
pass
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- basic receive ---

Print messages when received:
<pre>
#!/usr/bin/env python

import xmpp

user="username@gmail.com"
password="password"
server="gmail.com"

def message_handler(connect_object, message_node):
message = "Welcome to my first Gtalk Bot :)"
connect_object.send( xmpp.Message( message_node.getFrom() ,message))

jid = xmpp.JID(user)
connection = xmpp.Client(server)
connection.connect()
result = connection.auth(jid.getNode(), password, "LFY-client")
connection.RegisterHandler('message', message_handler)

connection.sendInitPresence()

while connection.Process(1):
pass
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- remote-control shell bot ---

Replace the simple bot’s message_handler() function with this new one:

<pre>
def message_handler(connect_object,message_node):
command = str(message_node.getBody())

process = subprocess.Popen(command,shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
message = process.stdout.read()
if message=="":
message=process.stderr.read()

connect_object.send( xmpp.Message( message_node.getFrom() ,message))
</pre>

References:
* Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- roster ---

<pre>
import xmpp
import sys

userID = 'myname@gmail.com'
password = 'mypassword'
ressource = 'Script'

jid = xmpp.protocol.JID(userID)
jabber = xmpp.Client(jid.getDomain(), debug=[])

connection = jabber.connect(('talk.google.com',5222))
auth = jabber.auth(jid.getNode(), password, ressource)

jabber.sendInitPresence(requestRoster=1)
myroster = jabber.getRoster()
</pre>

To get status, you have to wait for the other clients to send you their status:
<pre>
jabber.RegisterHandler('presence', myPresenceHandler)

def myPresenceHandler(con, event):
fromjid = event.getFrom().getStripped()
status = myroster.getStatus(fromjid)
</pre>

References:
python - XMPP chat: accessing contacts' status messages with xmppPy's Roster - Stack Overflow - http://stackoverflow.com/questions/2381597/xmpp-chat-accessing-contacts-status-messages-with-xmpppys-roster

--- full roster script ---

kparal/jabber-roster · GitHub - https://github.com/kparal/jabber-roster

A simple Python tool for listing your Jabber roster contacts. You can use it to easily backup list of your buddies.

This program is maintained, but not further developed. Bugs will be fixed, but no new features will be added. If you want to work on this program, don't hesitate to contact me, I will gladly assign you to the development team.

git clone https://github.com/kparal/jabber-roster
python jabber_roster.py

https://pypi.python.org/pypi/jabber-roster

--- vcard ---

This information is not in the roster. You will need to query the clients individually and get their vCard by sending this IQ :

<pre>
<iq from='stpeter@jabber.org/roundabout'
id='v1'
type='get'>
<vCard xmlns='vcard-temp'/>
</iq>
</pre>

My solution:
<pre>
def vcard(disp, jid):
msg = xmpp.protocol.Iq()
msg.setType('get')
msg.setTo(jid)
qc = msg.addChild('vCard')
qc.setAttr('xmlns', 'vcard-temp')
rep = disp.SendAndWaitForResponse(msg)
# to see what other fields are available in the XML output:
# print rep
userid=fname=lname=title=department=region=None
for i in rep.getChildren():
for j in i.getChildren():
if j.getName() == "TITLE":
title = j.getData().encode('utf-8')
for k in j.getChildren():
if k.getName() == "GIVEN":
fname = k.getData().encode('utf-8')
if k.getName() == "FAMILY":
lname = k.getData().encode('utf-8')
if k.getName() == "ORGUNIT":
department = k.getData().encode('utf-8')
if k.getName() == "REGION":
region = k.getData().encode('utf-8')
return fname, lname, title, department, region
</pre>

References:
* xmpppy - Retrieve gtalk nickname in python xmpp - Stack Overflow - http://stackoverflow.com/questions/4539477/retrieve-gtalk-nickname-in-python-xmpp
* XEP-0054: vcard-temp - http://xmpp.org/extensions/xep-0054.html

=== google-api-python-client ===

Google APIs Client Library for Python, which includes:
* google-api-python-client: The core Python library for accessing Google APIs.
* oauth2client: A Python client library for OAuth 2.0.
* Sample applications using google-api-python-client and oauth2client

-

https://developers.google.com/api-client-library/python/start/installation
* Calendar API - Python/Command Line

-
Download:
wget https://code.google.com/p/google-api-python-client/downloads/detail?name=google-api-python-client-1.2.zip
wget https://code.google.com/p/google-api-python-client/downloads/detail?name=oauth2client-1.2.zip

Git:
git clone https://github.com/google/google-api-python-client
git clone https://github.com/google/oauth2client

References:
* google-api-python-client - Google APIs Client Library for Python - Google Project Hosting - https://code.google.com/p/google-api-python-client/

== Documentation Tools ==

=== pydoc ===

pydoc - http://docs.python.org/library/pydoc.html
: Documentation generator and online help system

=== Doxygen and Doxypy ===

See [[Python/Doxygen]]

=== Sphinx ===

Sphinx - http://sphinx.pocoo.org/

"Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license.

It was originally created for the new Python documentation, and it has excellent facilities for the documentation of Python projects, but C/C++ is already supported as well, and it is planned to add special support for other languages as well. Of course, this site is also created from reStructuredText sources using Sphinx!"

=== epydoc ===

epydoc - http://epydoc.sourceforge.net/

"Epydoc is a tool for generating API documentation for Python modules, based on their docstrings. For an example of epydoc's output, see the API documentation for epydoc itself (html, pdf). A lightweight markup language called epytext can be used to format docstrings, and to add information about specific fields, such as parameters and instance variables. Epydoc also understands docstrings written in reStructuredText, Javadoc, and plaintext. For a more extensive example of epydoc's output, see the API documentation for Python 2.5. "

== Python 3 ==

=== Python 3 Wall of Superpowers ===

Python 3 Wall of Superpowers - https://python3wos.appspot.com/

"As listed on PyPI - packages in red don't support python 3, packages in green do. Hopefully one day everything will be greener. "

=== New in Python 3 ===

==== qualnames ====

New in 3.3
<pre>
>>> def f():
... def g():
... class H: pass
... return H
... return g
...

>>> H=f()()

>>> # 3.3:
>>> H
<class '__main__.f.<locals>.g.<locals>.H'>
>>> H.__qualname__
'f.<locals>.g.<locals>.H'

>>> # 2.6:
>>> H
<class __main__.H at 0x2b885044b890>
>>> H.__qualname__
AttributeError: class H has no attribute '__qualname__'
</pre>

== Python 2 or Python 3 ==

Python2orPython3 - PythonInfo Wiki - http://wiki.python.org/moin/Python2orPython3
:Short version: Python 2.x is the status quo, Python 3.x is the shiny new thing.
:At the time of writing (July 4, 2010), the final 2.7 release is out, with a statement of extended support for this end-of-life release. The 2.x branch will see no new major releases after that. 3.x is under active and continued development, with 3.1 already available and 3.2 due for release around the turn of the year.

Python 2.7.1 Release - http://www.python.org/download/releases/2.7.1/

What’s New In Python 3.0 — Python v3.0.1 documentation - http://docs.python.org/release/3.0.1/whatsnew/3.0.html
* print() requires parenthesis
* raw_input() is now input()

== Python for Windows ==

Python for Windows - http://www.python.org/download/windows/

=== Windows IDEs ===

==== PyCharm ====

http://www.pycharm.com/

=== Hide Console ===

fork() does not work on windows. So what else can we do?

How to hide console window in python? - http://stackoverflow.com/questions/764631/how-to-hide-console-window-in-python
* "Simply save it with a .pyw extension. This will prevent the console window from opening. Explanation at the bottom of section [http://docs.python.org/2/tutorial/interpreter.html#executable-python-scripts]"

=== 32bit vs 64bit ===

IDLE:

32bit:
<pre>
C:\Python27>python
Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
</pre>

64bit: (note: default path is C:\Python27)
<pre>
C:\Python27-64bit>python.exe
Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
</pre>

Task Manager:
python.exe *32

python.exe

Platform:
>>> import platform
>>> platform.python_compiler()
'MSC v.1500 32 bit (Intel)'

>>> import platform
>>> platform.python_compiler()
'MSC v.1500 64 bit (AMD64)'

Code: [http://stackoverflow.com/questions/2208828/detect-64bit-os-windows-in-python]
import struct
def is_python_64bit():
return (struct.calcsize("P") == 8)

=== Stand-Alone Executable ===

==== Python ZIP Application ====

Python can be run from a single bundled zip file - similar to a Java WAR! Python has been able to (Allow interpreter to execute a zip file http://bugs.python.org/issue1739468)

"Python quietly added a new feature in 2.6 that makes it possible to bundle up a directory full of Python code into a single executable file." [http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html]

When a python zip application is executed, it will execute '__main__.py' [http://docs.python.org/using/cmdline.html#interface-options]

mkdir app
touch app/__main__.py
cd app
zip -r ../app.zip *
cd ..
python app.zip

Load a resource - The easiest way to find and load a program bundle like this is to use the pkg_resources module. - http://packages.python.org/distribute/pkg_resources.html

<pre>
# file: app/__main__.py

import pkg_resources

def main():
print('The rain in Spain falls mainly in the plain.')
print(pkg_resources.resource_string('resources', 'inFrance.txt'))

if __name__ == '__main__':
main()
</pre>

Now use a bit of UNIX magic to turn app2.zip into an executable:
echo '#!/usr/bin/env python' | cat - app2.zip > app2
chmod +x app2

"The zip file format is designed to allow a small executable program to be inserted at the front (that's how self-extracting zip files are created), so this is kosher and doesn't corrupt the zip file." [http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html]

Now you can simply run app2 like any executable.
$ ./app2

References:
* Able Pear Software: Bundling Python files into a stand-alone executable - http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html
* PEP 441 -- Improving Python ZIP Application Support - http://www.python.org/dev/peps/pep-0441/
* Issue 1739468: Allow interpreter to execute a zip file - Python tracker - http://bugs.python.org/issue1739468
* Package Discovery and Resource Access using pkg_resources — Distribute documentation - http://packages.python.org/distribute/pkg_resources.html

==== Shell Python Loader ====

Came before the Python ZIP Application support.

Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe

"This takes your Python application and packages it into a single executable shell script. You application can be made of up multiple python scripts. The executable shell script is also compressed, so the resulting script is smaller than the original. This should work on any system that has Python and a Bourne shell (works on Cygwin too). Note that this does not package the Python interpreter into the executable. This packages only you python code into a zip file, but this still allows you to distribute your application as a single shell script that will run on any system with Python installed. In theory, you could zip the .pyc files, but the byte code is guaranteed to run only on exactly the same version Python interpreter (including minor version). It is safer to just package the .py files."

First you need to zip all your python files:
zip main.zip main.py spam.py eggs.py

zipheader.sh:
<pre>
#!/bin/sh
# This is a self-extracting executable.
# Execute this like any normal executable.
# You may need to "chmod a+x" this file.
# This is a binary ZIP file with a Python loader header.
#
# Bourne shell loader:
PYTHON=$(which python 2>/dev/null)
if [ ! -x "$PYTHON" ] ; then
echo "Python not found!"
exit 1
fi
exec $PYTHON -c "
# Python loader:
import sys, os
if int(sys.version[0])<2:
print 'Python version 2.3 final or greater is required.'
print 'Your version is', sys.version
os._exit(1)
major = sys.version_info[0]
minor = sys.version_info[1]
releaselevel = sys.version_info[3]
if (major==2 and minor<3) or (major==2 and minor==3 and releaselevel!='final'):
print 'Python version 2.3 final or greater is required.'
print 'Your version is', sys.version
os._exit(1)
sys.path.insert(0, sys.argv[1])
del sys.argv[0:1]
import main
main.main()
" $0 $@
# Zip file:
</pre>

Concatinate zipheader.sh with the zip file.
cat zipheader.sh main.zip > main

Set main as executable.
chmod +x main

"This is an improvement on Joerg Raedler's Python recipe, [http://code.activestate.com/recipes/215301-binding-main-skript-and-modules-to-one-executable-/ 215301]. The original closes stdin. The herefile in the shell script redirects stdin before python gets a chance to start. This disables raw_input() and anything else that reads sys.stdin. Unfortunately, Raedler's boot script closes stdin, which is a fairly big limitation. The herefile in the shell script (END_OF_PYTHON_CODE) redirects stdin before python starts; stdin is closed at the end of the herefile. This disables raw_input() and anything else that reads sys.stdin. "

References:
* Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe
* Build a compressed self-extracting executable script on UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/497000-build-a-compressed-self-extracting-executable-scri/
* binding main skript and modules to one executable with python-2.3 under UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/215301-binding-main-skript-and-modules-to-one-executable-/

==== py2exe ====

py2exe - http://www.py2exe.org/
: "py2exe is a Python Distutils extension which converts Python scripts into executable Windows programs, able to run without requiring a Python installation. "

Tutorial - http://www.py2exe.org/index.cgi/Tutorial

Note: Install the version that matches your version of python

Note: run installer "as administrator"

setup.py
<pre>
from distutils.core import setup
import py2exe

setup(console=['hello.py'])
</pre>

No console window version: [http://ubuntuforums.org/showthread.php?t=728170] [http://stackoverflow.com/questions/3275293/hiding-console-window-of-python-gui-app-with-py2exe]
<pre>
from distutils.core import setup
import py2exe

setup(windows=['hello.py'])
</pre>

Build:
python setup.py py2exe

pygame is more complex: http://www.pygame.org/wiki/Pygame2exe

==== py2app ====

==== pyinstaller ====

==== cx_Freeze ====

=== Windows Libraries ===

==== Python for Windows Extensions - PyWin32 ====

Python for Windows Extensions - http://starship.python.net/~skippy/win32/
* There is a build for each python version and architecture (32bit vs 64bit)

Follow event viewer
<pre>
import win32evtlog # install Python for Windows Extensions - http://starship.python.net/~skippy/win32/

# open event viewer
server = "localhost"
source = "System"
handle = win32evtlog.OpenEventLog(server, source)

# seek to end of log
flags = win32evtlog.EVENTLOG_BACKWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
events = win32evtlog.ReadEventLog(handle, flags, 0)

# return to normal reading direction
flags = win32evtlog.EVENTLOG_FORWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ
win32evtlog.ReadEventLog(handle, flags, 0) # ignore initial logs

# loop through events
while True:
events = win32evtlog.ReadEventLog(handle, flags, 0)
if events:
for event in events:
data = event.StringInserts
if data:
# Example: 07/03/14 12:34:35 - MYEVENTSOURCE - 0 - 1 - 4 "HELLO"
print event.TimeGenerated, '-', event.SourceName, '-', event.EventCategory, '-', event.EventID, '-', event.EventType, '-', '"%s"' % " ".join(data)
time.sleep(.5) # yield cpu
</pre>

References:
* PyWin32: Getting Windows Event Logs « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/07/27/pywin32-getting-windows-event-logs/

==== PyCrypto for Windows ====

Download Install: http://www.voidspace.org.uk/python/modules.shtml#pycrypto

Installation: [http://stackoverflow.com/questions/11405549/how-do-i-install-pycrypto-on-windows]
easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win32-py2.7.exe
easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win-amd64-py2.7.exe

==== Paramiko for Windows ====

# Install PyCrypto (above)
# pip install paramiko

==== PyGame for Windows ====

Download - http://www.pygame.org/download.shtml
* http://pygame.org/ftp/pygame-1.9.1.win32-py2.7.msi

== Indentation ==

Python: Myths about Indentation - http://www.secnetix.de/olli/Python/block_indentation.hawk
:There are quite some prejudices and myths about Python's indentation rules among people who don't really know Python. I'll try to address a few of these concerns on this page.
*"Whitespace is significant in Python source code."
*"Python forces me to use a certain indentation style."
*"You cannot safely mix tabs and spaces in Python."
*"I just don't like it."
*"How does the compiler parse the indentation?"

== PEP - Python Enhancement Proposals ==

Python Enhancement Proposals (PEPs)

* Index of Python Enhancement Proposals (PEPs) - http://www.python.org/dev/peps/

== Convert Tabs to Spaces ==

Convert tabs to 4 spaces:
sed -i 's/\t/ /g' script.py

== Linting ==

* pep8 - python style guide checker
* pychecker - more verbose, imports code, slower
* pylint - most verbose, configurable, "rates" code
* pyflakes - least verbose (dead/redundant code)

rpmforge:
yum install pyflakes pychecker
# pylint - not sure which pacakge includes

=== pep8 ===

See [[Python/pep8]]

=== pychecker ===

See [[Python/pychecker]]

=== pylint ===

See [[Python/pylint]]

== PEP8 Style Guidlines ==

PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/

* Use spaces - Use 4 spaces per indentation level.
* Never mix tabs and spaces.
* Limit all lines to a maximum of 79 characters.
* Line wrapping - The preferred way of wrapping long lines is by using Python's implied line continuation inside parentheses, brackets and braces. Long lines can be broken over multiple lines by wrapping expressions in parentheses.
* Encoding - For Python 3.0 and beyond, UTF-8 is preferred over Latin-1, see PEP 3120.
* Imports should usually be on separate lines
* Avoid extraneous whitespace in most situations
* Always use 'self' for the first argument to instance methods.

=== PEP8 Style Checker ===

See [[#pep8]]

== Benefits of Python ==

What is the return on investment for this group by learning Python? The list of benefits below sums up what the language has to offer:
* Python is designed from the ground up to encourage maintainable, readable and reusable code. It has great support for particular paradigms such as object-oriented programming.
* Python code is usually around a quarter of the length of code generated by C++ or Java - less typing, less debugging and less maintenance.
* Python does not need to be compiled or linked, which allows for a more iterative programming style and a quicker development cycle.
* Python is extremely portable. Code can be easily exchanged between Windows, Mac OS and Linux with little to no modification.
* Python comes with a staggering array of standard libraries and mature third-party projects. Web development, scientific computing, operator system access and network scripting are all supported.
* Python can talk with many other languages easily. It has support for invoking C and C++ code and can itself be called from libraries written in these languages.

Source: http://quantstart.com/articles/Quant-Reading-List-Python-Programming/ - Quant Reading List Python Programming

Benefits of Python:
* Python is available for most operating systems, including Windows, UNIX, Linux, and Mac OS.
* Python's clean object-oriented design and extensive support libraries offer two to ten fold the programmer productivity seen with languages like C, C++, C#, Java, VB, and Perl.

Source: Python Development - Benefits of Python - Digital Mesh - http://www.digitalmesh.com/offshore-development-center/python-development/benefits-of-python.html

== Python Package Management ==

=== Install Module ===

Installing Python Modules — Python v2.7.1 documentation - http://docs.python.org/install/index.html

The new standard: Distutils
python setup.py install

Download and extract target package, then:
sudo python setup.py install

Note: Running setup.py install builds and installs all modules in one run

Help:
python setup.py --help

Incremental method:
python setup.py build
python setup.py install
python setup.py build --help # help

Force rebuild and install:
python setup.py build --force # if have to repeat
python setup.py install --force # if have to repeat

Default installation paths:
<pre>
Platform Standard installation location Default value
Unix (pure) prefix/lib/pythonX.Y/site-packages /usr/local/lib/pythonX.Y/site-packages
Unix (non-pure) exec-prefix/lib/pythonX.Y/site-packages /usr/local/lib/pythonX.Y/site-packages
Windows prefix C:\Python
</pre>

Alternate install path:
python setup.py install --home=<dir>

Alternate base folders:
<pre>
Type of file Installation Directory Override option
pure module distribution home/lib/python --install-purelib
non-pure module distribution home/lib/python --install-platlib
scripts home/bin --install-scripts
data home/share --install-data
</pre>

Modifying Python’s Search Path:
import sys
sys.path.append('/www/python/')

=== Python Package Index ===

user-submitted Python packages:
http://pypi.python.org/pypi

sudo python setup.py install

Package Site:
http://pypi.python.org/pypi/[PACKAGE]
http://pypi.python.org/pypi/test

If they are on http://pypi.python.org/:
pip install foo

[http://stackoverflow.com/questions/410163/for-python-programmers-is-there-anything-equivalent-to-perls-cpan For Python programmers, is there anything equivalent to Perl's CPAN? - Stack Overflow]

=== Easy Installation ===

EasyInstall - The PEAK Developers' Center - http://peak.telecommunity.com/DevCenter/EasyInstall
: Easy Install is a python module (easy_install) bundled with setuptools that lets you automatically download, build, install, and manage Python packages.

CheeseShopTutorial - PythonInfo Wiki - http://wiki.python.org/moin/CheeseShopTutorial
: EasyInstall (easy_install) gives you a quick and painless way to install packages remotely by connecting to the Package Index or even other websites via HTTP. It is somewhat analogous to the CPAN and PEAR tools for Perl and PHP, respectively.

=== pip ===

See [[Python/pip]]

==== Create PIP Package ====

See [[Python/pip#Create PIP Package]]

=== Distribute ===

distribute - Easily download, build, install, upgrade, and uninstall Python packages

distribute 0.6.24 : Python Package Index - http://pypi.python.org/pypi/distribute

Installation:
mkdir -p ~/.src ; cd ~/.src
wget http://pypi.python.org/packages/source/d/distribute/distribute-0.6.24.tar.gz#md5=17722b22141aba8235787f79800cc452
tar -zvxf distribute-0.6.24.tar.gz
cd distribute-0.6.24
sudo python setup.py install

Test:
echo "import distribute" | grep python

== Named Parameters ==

http://diveintopython.org/power_of_introspection/optional_arguments.html

Python allows function arguments to have default values; if the function is called without the argument, the argument gets its default value. Futhermore, arguments can be specified in any order by using named arguments. Stored procedures in SQL Server Transact/SQL can do this, so if you're a SQL Server scripting guru, you can skim this part.

Here is an example of info, a function with two optional arguments:

def info(object, spacing=10, collapse=1):

spacing and collapse are optional, because they have default values defined. object is required, because it has no default value. If info is called with only one argument, spacing defaults to 10 and collapse defaults to 1. If info is called with two arguments, collapse still defaults to 1.

Say you want to specify a value for collapse but want to accept the default value for spacing. In most languages, you would be out of luck, because you would need to call the function with three arguments. But in Python, arguments can be specified by name, in any order.

http://www.wellho.net/mouth/1871_Optional-and-named-parameters-in-Python.html

If you add a parameter with two asterixes (**) on the end of a function definition, that parameter is taken as being the name of a dictionary into which all otherwise unidentified parameters are stored.

<pre>
def getnet(gross, taxrate=17.5, **others):
net = gross / (1.0 + taxrate*0.01)
for more in others.keys():
print "GETNET:",more," - ",others[more]
return neto
</pre>

== Python vs other languages ==

[http://www.strombergers.com/python/ Python is Cool (and Perl is not), Especially for C/C++ Programmers]

==Experimenting with Memory Remanence==
[http://citp.princeton.edu/memory/exp/ Experimenting with Memory Remanence]

<pre>
#!/usr/bin/env python

# a pirate's favorite chemical element
a = ""
while 1: a += "ARGON"
</pre>

sudo strings /dev/mem | less

==UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)==
See [[UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)]]

== Program Layout Best Practices Presentation ==

Program layout best practices - Seth House <seth@eseth.com> - Utah Django User Group - 2011-06-23
* Slides - https://github.com/whiteinge/presentations/tree/master/upyug_2011-06-09_program-layout
* Video - http://blip.tv/utah-open-source/python-layout-best-practices-for-modules-django-cli-utah-django-user-group-5315808

Module skeleton:
<pre>
#!/usr/bin/env python
# coding: utf-8

"""Module docstring"""

__author__ = ’Mr. Me <me@example.net>’
__version__ = ’1.2.3’

if __name__==’__main__’:
main()
</pre>

Executing modules as scripts
# PEP 338:
python -m SimpleHTTPServer

== CLI package ==

cli — command line tools — pyCLI v1.1.1 documentation - http://packages.python.org/pyCLI/

== Python IDE ==

IntegratedDevelopmentEnvironments - PythonInfo Wiki - http://wiki.python.org/moin/IntegratedDevelopmentEnvironments
* IDEs with introspection-based code completion and integrated debugger
* IDEs with introspection-based code completion /or/ integrated debugger
* IDEs with integrated gui builder
* Editors
* Reviews

Is there a good, free Python IDE for Windows? - Stack Overflow - http://stackoverflow.com/questions/126753/is-there-a-good-free-python-ide-for-windows
* Eclipse with PyDev
* Aptana Studio 3
* Active State also offers Komodo Edit
* WingWare - http://www.wingware.com/
* Eric IDE - http://www.die-offenbachs.de/eric/index.html
* MMM-Experts - Products - http://mmm-experts.com/Products.aspx?ProductId=4
** PyScripter is a Python IDE built in Python for Delphi(P4D) components and created with the ambition to become competitive in functionality with commercial Windows-based IDEs available for other languages. Being built in a compiled language is rather snappier than some of the other Python IDEs and provides an extensive blend of features that make it a productive Python development environment.
* ActiveState Python comes with PythonWin which is reasonably good
* PyCharm from JetBrains is not free in general, but has free early access program.
* SPE - Stani's Python Editor, is pretty awesome. See http://pythonide.blogspot.com/.
* You may want to take a look at the pdb module - http://docs.python.org/lib/module-pdb.html
* Geany (http://www.geany.org/) is a fairly light text editor that does a lot of IDE-like stuff.
* SciTE: http://www.scintilla.org/SciTE.html
* Editra: http://editra.org/
* Netbeans has support for python development in their Python Early Access version. - http://www.netbeans.org/features/python/
* I use Notepad++ with the appropriate settings.
* The Eric Python IDE - http://www.die-offenbachs.de/eric/index.html
* BOa Constructor www.boa-constructor.sourceforge.net really nice! drag and drop stuff!
* DrPtyhon is also very nice, and has many plug-ins. http://drpython.sourceforge.net/
* Vim
* I will make a plug for my preference of Python development environments: SPE. SPE is an integrated development environment that provides code completion, integrated Python shell, calltips, and various helper tools like notes and todo lists. It also includes wxGlade so you can make wxPython applications easily. Plus, it includes the python debugger WinPdb.
* http://groups.google.com.tr/group/python_opengl this software is my sample visual python ide.its demo version.
* I use PyScripter, and find it pretty awesome.
* There is one Python IDE written in Java: jHepWork: http://jwork.org/jhepwork/ it can detect Python's indentation rules rather nicely (+ detects structure of a code and syntax)
* gvim built with your python version support and with this plugins:
** NERDTree (don't read he's blog ;)
** matchit (it's builtin in new versions just put a symlink or junction)
** pyflakes
** pysmell
* Try Python Tools for Visual Studio. Supports CPython and IronPython. - http://pytools.codeplex.com/

PythonEditors - PythonInfo Wiki - http://wiki.python.org/moin/PythonEditors
* List of editors

Python Ide - http://c2.com/cgi/wiki?PythonIde
: There are a lot of Ides for PythonLanguage springing up. Many of them written in Python

Is there a better Python IDE? « Lennart Regebro: Python, Plone, Web - http://regebro.wordpress.com/2010/10/09/is-there-a-better-python-ide/
: I’m currently a user of WingIDE, but sometimes I feel the need for something better, mainly something that is a bit more stable. WingIDE often hangs when switching projects, sometimes the file listing decides that you must double-click on the expand arrow instead of just click on it, etc. It still doesn’t have macros despite this being a much requested feature (how hard can it be to add, really?) and I never really get the hang of the SVN integration, so I just ignore it. And they keep recommending me to use WingIDE’s separate instance of GTK instead of the System GTK, which is daft. Their designs are ugly, and it should work with the system GTK so it looks the same. Anything else is just silly.

=== Open Komodo ===

Open Komodo - http://www.openkomodo.com/
: Open Komodo is the source code repository for Komodo Edit - a free multi-language editor for dynamic languages, based on the award-winning Komodo IDE. ActiveState created the Open Komodo code repository in August 2007 and it has been used since then for building all Komodo Edit releases.

=== Eclipse ===

Build an Eclipse development environment for Perl, Python, and PHP - http://www.eclipse.org/resources/resource.php?id=485
: This tutorial shows how Eclipse's DLTK makes it possible to build development tools for scripting languages. In particular, it explains how to implement syntax coloring, user preferences, and interpreter integration in a plug-in-based project.

See [[#PyDev]]

=== EasyEclipse ===

EasyEclipse - http://www.easyeclipse.org/
:EasyEclipse packages together Eclipse, the open-source software
development platform, and selected open source plugins.
:We select, assemble, test, patch, build installers and document
a full IDE, offered as reliable distributions and plugins.
:EasyEclipse is:
* Free and open-source,
* Easy to download and install, and
* Simple to maintain, without version and dependency issues.

EasyEclipse for Python - http://www.easyeclipse.org/site/distributions/python.html

EclipsePythonIntegration - PythonInfo Wiki - http://wiki.python.org/moin/EclipsePythonIntegration
: Using Eclipse as a Python editor

=== PyDev ===

PyDev - http://pydev.org/
: PyDev is a Python IDE for Eclipse, which may be used in Python, Jython and IronPython development.

---

Pydev - http://pydev.sourceforge.net/ (old link)

How to Configure Eclipse for Python - http://www.rose-hulman.edu/class/csse/resources/Eclipse/eclipse-python-configuration.htm

Python Development with PyDev and Eclipse - Tutorial - http://www.vogella.de/articles/Python/article.html
: This article describes how to write and debug Python programs with Eclipse

PyDev Getting Started - http://pydev.org/manual_101_root.html

Running a program - http://pydev.org/manual_101_run.html

---

Supported Task Tags:
* TODO:
* TASK:
* FIXME:

---

Installation:
# Platform Runtime Binary (download around 45-50 MB) - http://download.eclipse.org/eclipse/downloads/ or http://www.eclipse.org/downloads/
#* Example: Eclipse Classic 4.2.2
# Eclipse Software Add - PyDev: http://pydev.org/updates
# Eclipse Software Add - MercurialEclipse: http://cbes.javaforge.com/update [http://javaforge.com/project/HGE] [http://mercurial.selenic.com/wiki/MercurialEclipse]
# Switch perspectives to PyDev

=== The Eric Python IDE ===

The Eric Python IDE - http://eric-ide.python-projects.org/
:Eric is a full featured Python and Ruby editor and IDE, written in python. It is based on the cross platform Qt gui toolkit, integrating the highly flexible Scintilla editor control. It is designed to be usable as everdays' quick and dirty editor as well as being usable as a professional project management tool integrating many advanced features Python offers the professional coder. eric4 includes a plugin system, which allows easy extension of the IDE functionality with plugins downloadable from the net.

Eric Python IDE - Wikipedia - http://en.wikipedia.org/wiki/Eric_Python_IDE

=== Wingware Python IDE ===

The Intelligent Development Environment for Python Programmers - Wingware Python IDE - http://www.wingware.com/
:Wingware's Python IDE is an Integrated Development Environment designed specifically for the Python programming language. Since 1999, Wingware has focused on Python.

=== VIM ===

Python .vimrc:
<pre>
" Kenneth's Preferences

set nobackup " don't create backup files
"set nohlsearch " don't highlight search results
set hlsearch " do highlight search results
set incsearch " incremental search
set ignorecase " ignore case during search
set smartcase " do case sensitive search when upper case is typed
set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
set showmatch " briefly highlight matching bracket
set matchtime=1 " highlight match for 0.1 seconds
set background=dark " black background
set shiftwidth=4 " shift blocks by 4 spaces
set shortmess=at " use short messages so we don't have to hit enter
set expandtab " expand tabs into spaces
set scrolloff=5 " always show 5 lines before & after cursor
set shell=bash " use bash for the command shell
set shellcmdflag=-lc " make bash startup as a login shell for ~/.bashrc
"set nowrap " do not wrap long lines
set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set wildmenu " enable filename tab completion
set ruler " display the ruler: shows cursor position
set cpoptions+=$ " display '$' at end of cw sequence
set title " display filename and vim info in window title bar
set modelines=5 " search the first and last 5 lines for "vim:" modeline
set backspace=2 " backspace deletes

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

"if v:version > 700
" set cursorline " underline the current line
"endif

if &t_Co > 2 || has( "gui_running" )
syntax on
endif

if has( 'mouse' )
set mouse=a " enable mouse
endif

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

"folding settings
set foldmethod=indent "fold based on indent
set foldnestmax=10 "deepest fold is 10 levels
set nofoldenable "dont fold by default
set foldlevel=10 "this is just what i use

" Auto clear trailing white space in Python files
func! DeleteTrailingWS()
exec "normal mz"
%s/\s\+$//ge
exe "normal `z"
endfunc
autocmd BufWrite *.py :call DeleteTrailingWS()
</pre>

----

Older versions:
<pre>
" KENNETH SETTINGS
set nocp " turn off compatibility mode
set nomodeline " do not allow files to dictate vim behavior

set incsearch " incremental search
set ignorecase " ignore case during search

set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set background=dark " black background
syntax on " turn on syntax highlighting
set hlsearch " highligh last search pattern

set nowrap " do not wrap long lines
set scrolloff=5 " always show 5 lines before & after cursor
set ruler " show the cursor position all the time
set backspace=2 " allow backspace deletion
set bs=indent,eol,start " allow backspacing over everything in insert mode

set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set expandtab " expand tab into spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
filetype indent on " turn on auto indent for specific file types
</pre>

Other options:
<pre>
set number " show line numbers
set numberwidth=1 " remove space before largest line numbers

set smartindent " ?? - maybe included in 'filetype indent on'

set ai " always auto indent

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

</pre>

Kenneth's fio .vimrc:
<pre>

"source $VIM/mswin.vim

set nobackup " don't create backup files
"set nohlsearch " don't highlight search results
set incsearch " incremental search
set ignorecase " ignore case during search
set smartcase " do case sensitive search when upper case is typed
set shiftwidth=4 " shift blocks in 4 space increments
set tabstop=4 " the (typewriter) tab stops are every 4 spaces
set smarttab " tab at beginning of line inserts 4 spaces (shiftwidth)
set showmatch " briefly highlight matching bracket
set matchtime=1 " highlight match for 0.1 seconds
set background=dark " black background
set shiftwidth=4 " shift blocks by 4 spaces
set shortmess=at " use short messages so we don't have to hit enter
set expandtab " expand tabs into spaces
set scrolloff=5 " always show 5 lines before & after cursor
set shell=bash " use bash for the command shell
set shellcmdflag=-lc " make bash startup as a login shell for ~/.bashrc
"set nowrap " do not wrap long lines
set visualbell " flash the screen rather than beep
set noerrorbells " don't beep on errors
set wildmenu " enable filename tab completion
set ruler " display the ruler: shows cursor position
set cpoptions+=$ " display '$' at end of cw sequence
set title " display filename and vim info in window title bar
set modelines=5 " search the first and last 5 lines for "vim:" modeline

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

if v:version > 700
set cursorline " underline the current line
endif

if &t_Co > 2 || has( "gui_running" )
syntax on
endif

if has( 'mouse' )
set mouse=a " enable mouse
endif

if has( "autocmd" )
" open the file in the last place we were editing
autocmd BufReadPost *
\ if line("'\"") > 1 && line("'\"") <= line("$") |
\ exe "normal! g'\"" |
\ endif
endif

"folding settings
set foldmethod=indent "fold based on indent
set foldnestmax=10 "deepest fold is 10 levels
set nofoldenable "dont fold by default
set foldlevel=10 "this is just what i use
</pre>

== Interpreter vs Script ==

Interpreter:
>>> import os
>>> os.uname()
('Linux', 'demo.oeey.com', '2.6.18-194.8.1.el5', '#1 SMP Thu Jul 1 19:07:06 EDT 2010', 'i686')

Script:
#!/usr/bin/env python
import os
print os.uname()

== Web Development ==

WebProgramming - PythonInfo Wiki - http://wiki.python.org/moin/WebProgramming
: This topic guide attempts to cover every aspect of programming Web applications (both clients and servers) using Python.

HOWTO Use Python in the web — Python v2.7.2 documentation - http://docs.python.org/howto/webservers.html
: This document shows how Python fits into the web. It presents some ways to integrate Python with a web server, and general practices useful for developing web sites.

=== Plug Simple Python Web App ===

<pre>
-------- Original Message --------
Subject: Re: Writing a simple python web app
Date: Mon, 25 Oct 2010 19:03:31 -0400
From: Roberto Mello <roberto.mello@gmail.com>
To: Provo Linux Users Group <plug@plug.org>

On Sat, Sep 11, 2010 at 5:40 PM, Dave Smith <dave@thesmithfam.org> wrote:
> I'm creating a very simple web application, and I want to use python. My first guess was that mod_python
would provide the easiest entry point. Boy, was I wrong. All the mod_python tutorials spend 80% of their
time extolling the virtues of mod_python over CGI, but they are quite lean on specific examples. It's quite
disappointing. A natural progression then leads me to mod_wsgi, and since that has equally poor
documentation, to Django. However, Django is overkill for my app (I don't even have a database).

For something as simple as you have described, and to run on an
embedded platform, I would probably stay away from frameworks designed
to run on full hardware, and go with something more minimalistic.

Take a look at CherryPy [1] or web.py [2].

[1] http://www.cherrypy.org/
[2] http://webpy.org/

Roberto Mello
</pre>

---

See also [[web.py]]

See also [[CherryPy]]

See also [[Django]]

=== Apache CGI ===

See [[CGI]]

Great tutorial: https://www.tutorialspoint.com/python/python_cgi_programming.htm

My favorite option, make all .cgi files executable: (or .py)
<pre>
DirectoryIndex index.html index.php index.py
<Location />
# AddHandler cgi-script .cgi
AddHandler cgi-script .py
Options +ExecCGI
</Location>
</pre>

To enable on Ubuntu:
sudo a2enmod cgi
service apache2 restart

Directory index:
DirectoryIndex index.py

Or if you really want to go crazy, this will try to run *every* file: [http://blog.kapsobor.de/archives/2006/09/05/apache_addhandler_multiple_extensions/]
<pre>
# this has problems with / directory being run as a script, instead of index too!
#<Location />
# SetHandler cgi-script
# Options +ExecCGI
#</Location>

<Files ~ ".">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>

or:
<pre>
<Files ~ "^.[a-zA-Z]*$">
SetHandler cgi-script
Options +ExecCGI
</Files>
</pre>

Or just have index file:
<Files "index">

Note: Place within a <VirtualHost> section if needed.

Note: Make sure to 'chmod +x' the files.

Start all HTML output with:
print "Content-Type: text/html\n"

If you want just text output:
print "Content-Type: text/plain\n"

GET (from Environment):
<pre>
print "<pre>"
for env in os.environ:
print "%s = %s" % (env, os.environ[env])
print "</pre>"
</pre>

<pre>
# sample for: http://oeey.com/py/test.py/test?var1=ken&var2=barbie

REQUEST_URI = /py/index.py/test?var1=ken&var2=barbie
QUERY_STRING = var1=ken&var2=barbie
PATH_INFO = /test
REQUEST_METHOD = GET

SCRIPT_NAME = /py/test.py
SERVER_NAME = oeey.com
REMOTE_ADDR = 216.51.42.66
SERVER_ADDR = 192.168.108.4
DOCUMENT_ROOT = /www/py
SCRIPT_FILENAME = /www/py/test.py
HTTP_HOST = cgi.oeey.com
</pre>

POST (from stdin):
<pre>
post = sys.stdin.readline()
if post:
print post
</pre>

<pre>
CONTENT_TYPE = application/x-www-form-urlencoded 

yin=one+two+three&yang=four+five+six
</pre>

cgi helper class:
<pre>
import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage()

# field keys
print "fields keys: ", form.keys() # list of field keys

# fild values
if "name" in form: # if form.has_key('name'):
print "field 'name' is: ", form["name"].value
print "field 'name' is: ", form.getvalue('name')

# headers
print "dictionary of headers: ", form.headers
print form.type
print form.headers['content-type']
</pre>

To prevent XSS escape the characters:
print cgi.escape(form.getvalue('name'))

<pre>
import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage()
if (form.has_key("action") and form.has_key("name") \
and form.has_key("age")):
if (form["action"].value == "display"):
print "%s = %s" % (form["name"].value form["age"].value)
</pre>

Note: cgi helper class also does sessions and cookies.

References:
* CGI Tutorial | Web Python - http://webpython.codepoint.net/cgi_tutorial
* Python CGI: An Interactive Instruction - http://www.cs.virginia.edu/~lab2q/

See also:
* Web Python - http://webpython.codepoint.net/

---

REST Example:

<pre>
import os, re
def get_rest():
if not "PATH_INFO" in os.environ:
return ()
# filter for garbage
#print os.environ["PATH_INFO"]
path_info = re.findall('[/A-Za-z0-9_-]*', os.environ["PATH_INFO"])
if not path_info:
return ()
path_info = ''.join(path_info)
# collapse empty paths
paths = path_info.split('/')
final_paths = []
for path in paths:
if path:
final_paths.append(path)
return tuple(final_paths)
rest = get_rest();
</pre>

=== mod_python ===

AVOID! AVOID! AVOID! - if you want this simple, just use CGI!

See [[Apache mod_python]]

mod_python - http://docs.python.org/howto/webservers.html#mod-python

Install:
yum install mod_python

Config:
/etc/httpd/conf.d/python.conf

Simple:
<pre>

</pre>

:"People coming from PHP often find it hard to grasp how to use Python in the web. Their first thought is mostly mod_python, because they think that this is the equivalent to mod_php. Actually, there are many differences. What mod_python does is embed the interpreter into the Apache process, thus speeding up requests by not having to start a Python interpreter for each request. On the other hand, it is not “Python intermixed with HTML” in the way that PHP is often intermixed with HTML. The Python equivalent of that is a template engine. mod_python itself is much more powerful and provides more access to Apache internals. It can emulate CGI, work in a “Python Server Pages” mode (similar to JSP) which is “HTML intermingled with Python”, and it has a “Publisher” which designates one file to accept all requests and decide what to do with them.

:mod_python does have some problems. Unlike the PHP interpreter, the Python interpreter uses caching when executing files, so changes to a file will require the web server to be restarted. Another problem is the basic concept – Apache starts child processes to handle the requests, and unfortunately every child process needs to load the whole Python interpreter even if it does not use it. This makes the whole web server slower. Another problem is that, because mod_python is linked against a specific version of libpython, it is not possible to switch from an older version to a newer (e.g. 2.4 to 2.5) without recompiling mod_python. mod_python is also bound to the Apache web server, so programs written for mod_python cannot easily run on other web servers.

:These are the reasons why mod_python should be avoided when writing new programs. In some circumstances it still might be a good idea to use mod_python for deployment, but WSGI makes it possible to run WSGI programs under mod_python as well."

=== WSGI ===

WSGI - http://docs.python.org/howto/webservers.html#mod-wsgi

: "The Web Server Gateway Interface, or WSGI for short, is defined in PEP 333 and is currently the best way to do Python web programming. While it is great for programmers writing frameworks, a normal web developer does not need to get in direct contact with it. When choosing a framework for web development it is a good idea to choose one which supports WSGI."

See [[mod_wsgi]]

=== Django ===

See [[Django]]

=== CherryPy ===

See [[CherryPy]]

=== web.py ===

web.py: makes web apps

Think about the ideal way to write a web app. Write the code to make it happen.

web.py : Python Package Index - http://pypi.python.org/pypi/web.py

Home Page: http://webpy.org/

"Django lets you write web apps in Django. TurboGears lets you write web apps in TurboGears. Web.py lets you write web apps in Python."
:— Adam Atlas

NOTE: Can modify source code without having to restart app! :-)

---

PIP Install:
pip install web.py

---

Hello World: [http://webpy.org/]
<pre>
import web

urls = (
'/(.*)', 'hello'
)
app = web.application(urls, globals())

class hello:
def GET(self, name):
if not name:
name = 'World'
return 'Hello, ' + name + '!'

if __name__ == "__main__":
app.run()
</pre>

Note: app.py can be renamed.

Running:
python app.py

Note: You can add an IP address/port after the "app.py" bit to control where web.py launches the server. You can also tell it to run a fastcgi or scgi server.
python app.py 1234

---

Templates

<pre>
# templates/hello.html
$def with (name)

$if name:
I just wanted to say hello to $name.
$else:
Hello, world!
</pre>

app = ...
render = web.template.render('templates/')

index.GET:
name = 'Bob'
return render.hello(name) # render.[TEMPLATE_NAME](vars)

get variable: (/?name=Joe)
i = web.input(name=None)
return render.hello(i.name)

variable in url:
urls = ('/(.*)', 'hello')

class index:
def GET(self, name):
return render.hello(name)

# visit: /Joe

multiple variables in url:
urls = ('/(.*)/(.*)', 'hello')

class hello:
def GET(self, fname, lname):
return render.hello(fname, lname)

# visit: /Joe/Smith

=== lpthw.web ===

lpthw.web: a fork of web.py for Learn Python The Hard Way

Locks web.py at a specific version and removes some magic. Thanks to Aaron Swartz for making it originally.

lpthw.web: a fork of web.py for Learn Python The Hard Way - http://pypi.python.org/pypi/lpthw.web

Home Page: http://learnpythonthehardway.org/

This conflicts with web.py, so only install one.

=== web ===

web - Web modules for CGI and WSGI web programming

web : Python Package Index - http://pypi.python.org/pypi/web

Warning: This is an alpha release for those who are interested in
checking progress. No documentation and not for production use.

Auth, session, mail, form and templating tools for writing CGI scripts or WSGI web applications.

This version incompatible with previous versions becuase large portions of the code have been refactored into separate components.

== Issues ==

=== Hide DeprecationWarning ===

==== Import DeprecationWarning ====

Error:
<pre>
/opt/xmpppy-0.4.1/xmpp/auth.py:24: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import sha,base64,random,dispatcher,re
/opt/xmpppy-0.4.1/xmpp/auth.py:26: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5
</pre>

Hide deprecation warnings with code: [http://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python]
<pre>
import warnings

with warnings.catch_warnings():
warnings.filterwarnings("ignore",category=DeprecationWarning)
import md5, sha
</pre>

Hide with header:
#!/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line:
python -W ignore::DeprecationWarning script.py

References:
* How to ignore deprecation warnings in Python - Stack Overflow - http://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python

==== Funciton DeprecationWarning ====

Error:
<pre>
/opt/xmpppy-0.4.1/xmpp/transports.py:307: DeprecationWarning: socket.ssl() is deprecated. Use ssl.wrap_socket() instead.
tcpsock._sslObj = socket.ssl(tcpsock._sock, None, None)
</pre>

Hide with header:
#!/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line:
python -W ignore::DeprecationWarning script.py

==== Ubuntu 24 pip packages ====

<pre>
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
</pre>

== keywords ==

[[Category:Programming]]
[[Category:Python]]