Notes on Python


pyCharm

Change Navigate -> Declaration shortcut to Cmd+click by Adding Mouse Shortcut

  • Original Ctrl+Click conflicts Mac’s right click

Assign Cmd+F to Find

Pass command arguments to python

  • Run->Edit configurations->“+” (add new config)->Python -> add Script params

Use requirements.txt to install dependences

  • if failed, run ./venv/bin/pip install -r requirements.txt in Terminal
Use ToolBox to upgrade pyCharm
Inspect long collection
  • Evaluate expression: print(list)

Add folder as source folder

Python

Prefer to use str.format over %

is vs ==

  • == checks equality, is checks identities
sys.argv
  • list of strings representing the arguments
  • sys.argv[0]: name of the script

os.chdir(“../..”)

  • Don’t change current directory
  • Don’t change state, hard to test, bug

Execute external commands

Gotcha

  • Python doesn’t support ++

Tools

python -m SimpleHTTPServer
| python -mjson.tool

Installing a different python 2.7

sudo yum install python27
rpm -ql python2.7-stack-2.7.12-1.9.g680e0d1.x86_64
/opt/python-2.7/bin/python

Pip

PIP Install on Mac: Operation Not Permitted
  • pip install –user
    • install the package into your own home directory
pip install –ignore-installed six

Debugging

pudb

python2.7 -m pip install pudb
python2.7 -m pudb.run s.py

logging

*args and **kwargs

Modules

Unit test

Misc

Notes on Shell Programming


Notes on Shell Programming

Special Parameters

$#

The number of arguments passed to the program; or the number of parameters set by executing the set statement

$*

Collectively references all the positional parameters as $1, $2, ...

$@

Same as $*, except when double-quoted ("$@") collectively references all the positional parameters as "$1", "$2", ...

$0

The name of the program being executed

$$

The process id number of the program being executed

grep -v "$name" phonebook > /tmp/phonebook$$

$!

The process id number of the last program sent to the background for execution

$?

The exit status of the last command not executed in the background

$-

The current option flags in effect


Other Variables Used by the Shell Variable

CDPATH

The directories to be searched whenever cd is executed without a full path as argument.

ENV

The name of a file that the shell executes in the current environment when started interactively.

FCEDIT

The editor used by fc. If not set, ed is used.

HISTFILE

If set, it specifies a file to be used to store the command history. If not set or if the file isn't writable, $HOME/.sh_history is used.

HISTSIZE

If set, specifies the number of previously entered commands accessible for editing. T

HOME

IFS

The Internal Field Separator characters; used by the shell to delimit words when parsing the command line, for the read and set commands, when substituting the output from a back-quoted command, and when performing parameter substitution. Normally, it contains the three characters space, horizontal tab, and newline.

PATH

PPID

The process id number of the program that invoked this shell (that is, the parent process).

PS1: The primary command prompt, normally "$ ".

PS2: The secondary command prompt, normally "> ".

PWD

Parameter Substitution

$parameter or ${parameter}

To access the tenth and greater arguments, you must write ${10}.

shift

whatever was previously stored inside $2 will be assigned to $1, The old value of $1 will be irretrievably lost. When this command is executed, $# (the number of arguments variable) is also automatically decremented by one.

You can shift more than one "place" at once by writing a count immediately after shift, as in

shift 3

${parameter:-value}

Substitute the value of parameter if it's set and non-null; otherwise, substitute value.

${parameter-value}

Substitute the value of parameter if it's set; otherwise, substitute value.

${parameter:=value}

Substitute the value of parameter if it's set and non-null; otherwise, substitute value and also assign it to parameter.

${parameter=value}

Substitute the value of parameter if it's set; otherwise, substitute value and also assign it to parameter.

${parameter:?value}

Substitute the value of parameter if it's set and non-null; otherwise, write value to standard error and exit. If value is omitted, write parameter: parameter null or not set instead.

${parameter?value}

Substitute the value of parameter if it's set; otherwise, write value to standard error and exit. If value is omitted, write parameter: parameter null or not set instead.

${parameter:+value}

Substitute value if parameter is set and non-null; otherwise, substitute null.

${parameter+value}

Substitute value if parameter is set; otherwise, substitute null.

${#parameter}

Substitute the length of parameter. If parameter is * or @, the result is not specified.

${parameter#pattern}

Substitute the value of parameter with pattern removed from the left side. The smallest portion of the contents of parameter matching pattern is removed. Shell filename substitution characters (*, ?, [...], !, and @) may be used in pattern.

${parameter##pattern}

Same as #pattern except the largest matching pattern is removed.

${parameter%pattern}

Same as #pattern except pattern is removed from the right side.

${parameter%%pattern}

Same as ##pattern except the largest matching pattern is removed from the right side. Quoting

'…' Removes special meaning of all enclosed characters

"…" Removes special meaning of all enclosed characters except $, `, and \

\c

Removes special meaning of character c that follows; inside double quotes removes special meaning of $, `, ", newline, and \that follows, but is otherwise not interpreted; used for line continuation if appears as last character on line (newline is removed)

Using the Backslash for Continuing Lines

The Backslash Inside Double Quotes

The backslash inside these quotes removes the meaning of characters that otherwise would be interpreted inside double quotes (that is, other backslashes, dollar signs, back quotes, newlines, and other double quotes). If the backslash precedes any other character inside double quotes, the backslash is ignored by the shell and passed on to the program:

Command Substitution

The Back Quote `command`

The $(...) Construct $(command)

variable=$(command)

The expr Command(for older linux)

expr 10 + 20 / 2

Each operator and operand given to expr must be a separate argument,

expr 17 * 6

The shell saw the * and substituted the names of all the files in your directory!

expr 17 \* 6

only use the command substitution mechanism to assign the output from expr back to the variable:

$ i=$(expr $i + 1) Add 1 to i

like the shell's built-in integer arithmetic, expr only evaluates integer arithmetic expressions. You can use awk or bc if you need to do floating point calculations.

Decisions

user="$1"

if who | grep "^$user " > /dev/null

then

echo "$user is logged on"

fi

if command

then

command

else

command

fi


if command

then

command

elif command2

then

elif commandn

then

else

fi

The test Command

test String Operators

Operator Returns TRUE (exit status of 0) if

string1 = string2, string1 != string2, string

-n string

string is not null (and string must be seen by test).

-z string

string is null (and string must be seen by test).

An Alternative Format for test

[ expression ]

spaces must appear after the [ and before the ].

[ "$name" = julio ]

Integer Operators

int1 -eq int2

int1 -ge int2

int1 -gt int2

int1 -le int2

int1 -lt int2

int1 -ne int2

File Operators

-d file file is a directory.

-e file file exists.

-f file file is an ordinary file.

-r file file is readable by the process.

-s file file has nonzero length.

-w file file is writable by the process.

-x file file is executable.

-L file file is a symbolic link.

The Logical Negation Operator !

The Logical AND Operator –a

The Logical OR Operator –o

[ -f "$mailfile" -a -r "$mailfile" ]

Parentheses

Use parentheses in a test expression to alter the order of evaluation; and make sure that the parentheses are quoted because they have a special meaning to the shell.

[ \( "$count" -ge 0 \) -a \( "$count" -lt 10 \) ]

The exit Command

The case Command

hour=$(date +%H)

case "$hour"

in

0? | 1[01] ) echo "Good morning";;

1[2-7] ) echo "Good afternoon";;

* ) echo "Good evening";;

esac

The -x Option for Debugging Programs

You can trace the execution of any program by typing sh -x followed by the name of the program.

The Null Command :

The && and || constructs implement shortcut logic

enable you to execute a command based on whether the preceding command succeeds or fails.

sort bigdata > /tmp/sortout && mv /tmp/sortout bigdata

[ -z "$EDITOR" ] && EDITOR=/bin/ed

grep "$name" phonebook || echo "Couldn't find $name"


The for Command

for i in 1 2 3

do

echo $i

done

The $@ Variable

Whereas the shell replaces the value of $* with $1, $2, ..., if you instead use the special shell variable "$@" it will be replaced with "$1", "$2", ... .

The for Without the List

for var

do

command

done

Shell automatically sequences through all the arguments typed on the command line, just as if you had written

for var in "$@"

do

command

done

The while Command

The while loop is often used in conjunction with the shift command to process a variable number of arguments typed on the command line.

while [ "$#" -ne 0 ]

do

echo "$1"

shift

done

The until Command

until who | grep "^$user " > /dev/null

do

sleep 60

done

break

When the break is executed, control is sent immediately out of the loop, where execution then continues as normal with the command that follows the done.

break n the n innermost loops are immediately exited.

continue n causes the commands in the innermost n loops to be skipped; but execution of the loops then continues as normal.

Executing a Loop in the Background

An entire loop can be sent to the background for execution simply by placing an ampersand after the done.

for file in memo[1-4]

do

run $file

done &

I/O Redirection on a Loop

Input/Output redirected into the loop applies to all commands in the loop that read their data from standard input or write to standard output.you can also redirect the standard error output from a loop, simply by tacking on a 2> file after the done.

You can override redirection of the entire loop's input or output by explicitly redirecting the input and/or output of commands inside the loop. To force input or output of a command to come from or go to the terminal, use the fact that /dev/tty always refers to your terminal.

for file

do

echo "Processing file $file" > /dev/tty

done > output

Piping Data Into and Out of a Loop

A command's output can be piped into a loop, and the entire output from a loop can be piped into another command in the expected manner.

for i in 1 2 3 4

do

echo $i

done | wc -l

Typing a Loop on One Line

for i in 1 2 3 4; do echo $i; done

if commands can also be typed on the same line using a similar format:

if [ 1 = 1 ]; then echo yes; fi

The getopts Command

getopts options variable

The getopts command is designed to be executed inside a loop. Each time through the loop, getopts examines the next command line argument and determines whether it is a valid option. This determination is made by checking to see whether the argument begins with a minus sign and is followed by any single letter contained inside options. If it does, getopts stores the matching option letter inside the specified variable and returns a zero exit status.


If the letter that follows the minus sign is not listed in options, getopts stores a question mark inside variable before returning with a zero exit status. It also writes an error message to standard error.

If no more arguments are left on the command line or if the next argument doesn't begin with a minus sign, getopts returns a nonzero exit status.

To indicate to getopts that an option takes a following argument, you write a colon character after the option letter on the getopts command line.

getopts mt: option


If getopts doesn't find an argument after an option that requires one, it stores a question mark inside the specified variable and writes an error message to standard error. Otherwise, it stores the actual argument inside a special variable called OPTARG.


Another special variable called OPTIND is initially set to one and is updated each time getopts returns to reflect the number of the next command-line argument to be processed.

while getopts mt: option

do

case "$option"

in

m) mailopt=TRUE;;

t) interval=$OPTARG;;

\?) echo "Usage: mon [-m] [-t n] user"

exit 1;;

esac

done

shiftcount=$((OPTIND – 1))

shift $shiftcount

user=$1


Reading and Printing Data

The read Command read variables

You can use the –r option to read to prevent shell interpreting the backslash character.

while read –r line

Special echo Escape Characters \c

\c tells echo to leave the cursor right where it is after displaying the last argument and not to go to the next line.

The printf Command

printf "format" arg1 arg2 ...

printf Conversion Specification Modifiers

- Left justify value.

+ Precede integer with + or -.

(space) Precede positive integer with space character.

# Precede octal integer with 0, hexadecimal integer with 0x or 0X.

Width Minimum width of field; * means use next argument as width.

Precision

Minimum number of digits to display for integers; maximum number of characters to display for strings; * means use next argument as precision.


More on Subshells

A subshell can't change the value of a variable in a parent shell, nor can it change its current directory.

The . Command

Its purpose is to execute the contents of file in the current shell, A subshell is not spawned to execute the program.


The (...) and { ...; } Constructs

They are used to group a set of commands together.

The first form causes the commands to be executed by a subshell, the latter form by the current shell.

For {}., if the commands enclosed in the braces are all to be typed on the same line, a space must follow the left brace, and a semicolon must appear after the last command.

.profile File

/etc/profile and $HOME/.profile

I/O Redirection

< file, > file

>| file

Redirect standard output to file; file is created if it doesn't exist and zeroed if it does; the noclobber (-C) option to set is ignored.

>> file

Like >, only output is appended to file if it already exists.

Redirect the standard output for a command to standard error by writing

command >& 2

Collect the standard output and the standard error output from a program into the same file.

command >foo 2>>foo or command >foo 2>&1

Because the shell evaluates redirection from left to right on the command line, the last example cannot be written: command 2>&1 > foo

Inline Input Redirection

command <<word

word

the shell uses the lines that follow as the standard input for command, until a line that contains just word is found.

The shell performs parameter substitution for the redirected input data, executes back-quoted commands, and recognizes the backslash character. However, any other special characters, such as *, |, and ", are ignored. If you have dollar signs, back quotes, or backslashes in these lines that you don't want interpreted by the shell, you can precede them with a backslash character. Alternatively, if you want the shell to leave the input lines completely untouched, you can precede the word that follows the << with a backslash.

If the first character that follows the << is a dash (-), leading tab characters in the input will be removed by the shell.

<& digit Standard input is redirected from the file associated with file descriptor digit.

>& digit Standard output is redirected to the file associated with file descriptor digit.

<&- Standard input is closed.

>&- Standard output is closed.

<> file Open file for both reading and writing.

Shell Archives

One or more related shell programs can be put into a single file and then shipped to someone.

cat > program_name <<\THE-END-OF-DATA

THE-END-OF-DATA

The exec Command

Because the exec'ed program replaces the current one, there's one less process hanging around; also, startup time of an exec'ed program is quicker, due to the way the Unix system executes processes.


exec can be used to close standard input and reopen it with any file that you want to read. To change standard input to file:

exec < file

Redirection of standard output is done similarly: exec > report

If you use exec to reassign standard input and later want to reassign it someplace else, you can simply execute another exec. To reassign standard input back to the terminal, you would write

exec < /dev/tty


The set Command

General Format: set options args

This command is used to turn on or off options as specified by options. It is also used to set positional parameters, as specified by args.

Each single letter option in options is enabled if the option is preceded by a minus sign (-), or disabled if preceded by a plus sign (+).

set Options

--

-- option tells set not to interpret any subsequent arguments on the command line as options. It also prevents set from displaying all your variables if no other arguments follow.

-a

Automatically export all variables that are subsequently defined or modified.

-v

Print each shell command line as it is read.

-x

Print each command and its arguments as it is executed,

set with No Arguments

Using set to Reassign Positional Parameters

There is no way to directly assign a value to a positional parameter;

These parameters are initially set on execution of the shell program. The only way they may be changed is with the shift or the set commands. If words are given as arguments to set on the command line, those words will be assigned to the positional parameters $1, $2, and so forth. The previous values stored in the positional parameters will be lost forever. So

set a b c

assigns a to $1, b to $2, and c to $3. $# also gets set to 3.

The IFS Variable stands for Internal Field Separator.

echo "$IFS" | od -b

The readonly Command

readonly –p gets a list of your read-only variables

The unset Command

unset x Remove x from the environment

This causes the shell to erase definitions of the variables or functions listed in names. Read-only variables cannot be unset. The –v option to unset specifies that a variable name follows, whereas the –f option specifies a function name. If neither option is used, it is assumed that variable name(s) follow.

The eval Command

eval command-line

the net effect is that the shell scans the command line twice before executing it.

pipe="|"

ls $pipe wc -l

The shell takes care of pipes and I/O redirection before variable substitution, so it never recognizes the pipe symbol inside pipe.

eval ls $pipe wc –l

The first time the shell scans the command line, it substitutes | as the value of pipe. Then eval causes it to rescan the line, at which point the | is recognized by the shell as the pipe symbol.

Get the last parameter: eval echo \$$#

The wait Command

wait process-id

where process-id is the process id number of the process you want to wait for. If omitted, the shell waits for all child processes to complete execution.

The trap Command

trap commands signals

Commonly Used Signal Numbers

0 Exit from the shell

1 Hangup

2 Interrupt (for example, Delete, Ctrl+c key)

15 Software termination signal (sent by kill by default)

trap "rm $WORKDIR/work1$$ $WORKDIR/dataout$$; exit" 1 2

The value of WORKDIR and $$ will be substituted at the time that the trap command is executed. If you wanted this substitution to occur at the time that either signal 1 or 2 was received (for example, WORKDIR may not have been defined yet), you can put the commands inside single quotes:

trap 'echo logged off at $(date) >>$HOME/logoffs' 0

trap 'rm $WORKDIR/work1$$ $WORKDIR/dataout$$; exit' 1 2

trap with No Arguments

Executing trap with no arguments results in the display of any traps that you have changed.

Ignoring Signals

If the command listed for trap is null, the specified signal will be ignored when received.

trap "" 2

If you ignore a signal, all subshells also ignore that signal. However, if you specify an action to be taken on receipt of a signal, all subshells will still take the default action on receipt of that signal.


If you execute trap : 2

and then execute your subshells, then on receiving the interrupt signal the current shell will do nothing (it will execute the null command), but all active subshells will be terminated (they will take the default action—termination).

Resetting Traps

After you've changed the default action to be taken on receipt of a signal, you can change it back again with trap if you simply omit the first argument;

trap 1 2

Functions

name () { command; ... command; }

Removing a Function Definition

To remove the definition of a function from the shell, you use the unset command with the –f option.

unset –f nu


Material

Unix Shell Programming, Third Edition


Small Tips on Notepad++


                        Small Tips on Notepad++

Notepad++ is a great free source code editor and Notepad replacement, which supports several programming languages.

Useful Notepadd+ keyboard shortcuts
Ctrl+o      Open
Ctrl-Z      Undo
Ctrl-Y       Redo
Ctrl-D      Duplicate Current Line
Ctrl-L       Delete Current Line
F3      Fine Next
Shift-F3        Find Previous
Ctrl-G      Launch GoToLine Dialog
Ctrl-W      Close Current Document
F11         Toggle Full Screen Mode
Ctrl-Tab        Next Document
Ctrl-Shift-Tab Previous Document
Ctrl-Shift-Up  Move Current Line Up
Ctrl-Shift-Down   Move Current Line Down   
Ctrl-Shift-BackSpace     Delete to start of line
Ctrl-Shift-Delete      Delete to end of line
Ctrl-U      Convert to lower case
Ctrl-Shift-U    Convert to UPPER CASE
Shift-Tab (selection of several lines) 
Remove Tabulation or Space (outdent)

Formatting xml
1. Select the xml code
2. TextFX -> HTML Tidy -> Tidy: Reindent XML
 
Delete the first word/space/line no’s
TextFX > TextFX Tools > Delete Line Numbers or First word

Remove the trailing spaces
TextFX > TextFX Edit > Trim Trailing Spaces

Remove Blank Lines
TextFX > TextFX Edit > Delete Blank Lines
TextFX > TextFX Edit > Delete Surplus Blank Lines
Install notepad++ plugin
    Just search and download plugin from http://npp-plugins.sourceforge.net/, and extract it, you will find .dll files, copy them to notepad++/plugins/ directory, and restart np++.

Useful notepad++ plugins
Notepadd++ already includes:
Mime Tools   Base64 encode/decode,
TextFx          provides many practical function such as upper/lower case, escape/unescape, auto-close XML,
nppExport     Export file to RTF/HTML format.

Explorer Plugin – a file browser
Windows Manager - help to switch a file to another.

Multi Clipboard - add up to ten text phrases to the clipboard.
Compare Plugin - show the difference between 2 files.
XML Tools - XML Checker. Can auto-close tags and provides other useful features.

Customize Notepad++
    In this article, the author introduces a way to port ANY Textmate theme over to Notepad++.

Resources:

Labels

adsense (5) Algorithm (69) Algorithm Series (35) Android (7) ANT (6) bat (8) Big Data (7) Blogger (14) Bugs (6) Cache (5) Chrome (19) Code Example (29) Code Quality (7) Coding Skills (5) Database (7) Debug (16) Design (5) Dev Tips (63) Eclipse (32) Git (5) Google (33) Guava (7) How to (9) Http Client (8) IDE (7) Interview (88) J2EE (13) J2SE (49) Java (186) JavaScript (27) JSON (7) Learning code (9) Lesson Learned (6) Linux (26) Lucene-Solr (112) Mac (10) Maven (8) Network (9) Nutch2 (18) Performance (9) PowerShell (11) Problem Solving (11) Programmer Skills (6) regex (5) Scala (6) Security (9) Soft Skills (38) Spring (22) System Design (11) Testing (7) Text Mining (14) Tips (17) Tools (24) Troubleshooting (29) UIMA (9) Web Development (19) Windows (21) xml (5)