Ipython

Load a file into external editor

edit file

  • Use Emacs

EDITOR=emacsclient ipython
edit file

Run a file

run file

Save code from shell

save filename.py 1-10

Display available namespaces

who

  • Including data types

whos

Get documentation

  • Use ? after module, function, whatever

  • pinfo some module or function or whatever

Show source code of a function

psource <function>

  • Get whole source file

pfile <function>

Edit a function

  • -x will not execute the new code

edit <function>

Save shell command in variable

foo = !ps ax | grep something

Define aliases

%alias pids ps ax | grep app | cut -d " " -f 1
%store pids

Debugging

  • Switch on pdb on execptions

pdb

  • Run script in pdb

run -d file

  • Run script in pdb with breakpoint in line 23

run -d -b 23 file

  • Automatically start Ipython debugger on exception

try:
  ret = 1 / 0
except Exception, e:
  import sys, IPython
  IPython.Shell.IPShell(argv=[])
  IPython.Debugger.Pdb(IPython.ipapi.get().options.colors).set_trace(sys._getframe())

Profile

%time some_function

  • Run cProfile

%prun file or function

  • Filter output

%prun -l some_filter_string file or function

History

  • Show history

hist

  • Execute command nr x

_ix

  • Ranges (1-5)

In[1:6]

  • Print output of command 42

Out[42]

Bookmarks

  • Create a dir bookmark

bookmark name

  • Save bookmark

store name

  • List bookmarks

bookmark -l

  • Delete bookmark

bookmark -d name

Macros

  • Save history commands 1-5 in a macro muh

macro muh 1-5

  • Save the macro

store muh

  • Execute it by calling its name

  • Show the source

print muh

Background jobs

  • Start a statement in the background

bg some_func()

  • Show status of job

job[0].status

  • Get the result

job[0].result

Connect to an existing console

  • Start ipython with console or qtconsole

ipython kernel
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing kernel-26492.json

  • Now you can connect another console by executing

ipython console --existing kernel-26492.json

  • If the kernel is on another system you either have to setup ssh tunnel for all port specified in the json file

ssh user@remote -f -N -L 61947:127.0.0.1:61947

  • or append –ssh user@remote to the ipython console command

  • The kernel.json file can be found in /run/user/0/jupyter/ or ~/.ipython/profile_default/security

  • To manually create all ssh tunnels

for port in $(cat kernel-1234.json | grep '_port' | grep -o '[0-9]\+'); do ssh myremotehost -f -N -L $port:127.0.0.1:$port; done

Parallel computing

  • Its possible to do parallel computing in different ways like SSH, MPI, PBS and even Windows HPC

  • The controller opens a socket and allow engines to connect to it

  • First of all lets start an Ipython console locally for every core

ipcluster start
ipython

  • To run a simple hello world on all cores exec

from IPython.parallel import Client
c = Client()
c.ids
c[:].apply_sync(lambda : "Hello, World")

  • Manually start a controller on your head node and start engines on all computing nodes (must have access to the file ~/.ipython/profile_default/security/ipcontroller-engine.json generated by the controller)

ipcontroller --enginessh=user@host
ipengine

  • Now you can import the module os on every engine and print uname information

with c[:].sync_imports():
  import os
%px print os.uname()

  • If all nodes share a home directory for ipcontroller-engine.json file you can automatically start the controller on localhost and engines on remote

  • Create a new profile with

ipython profile create --parallel --profile=ssh

  • Edit ipcluster_config.py and add

c.IPClusterEngines.engine_launcher_class = 'SSHEngineSetLauncher'
c.SSHEngineSetLauncher.engines = { 'host1.example.com' : 2,
          'host2.example.com' : 5,
          'host3.example.com' : (1, ['--profile-dir=/home/different/location']),
          'host4.example.com' : 8 }

  • The key of the engines dict is the host and value number of engines to start

  • Now you can setup the cluster by executing

ipcluster start --profile=ssh

  • Use map to call a function on a dataset on all nodes

v = c[:]
result = v.map_sync(lambda x: x*x, data)

Save state of a console

  • You can start / stop logging of a sessions state with %logstart <file> / %logstop

  • To load the session state exec

ipython -lp <file>

  • To automatically save state edit ~/.ipython/profile_default/ipython_config.py and set

c.TerminalInteractiveShell.logstart = True
c.TerminalInteractiveShell.logfile = '~/ipython_session.log'

Extensions

  • To install an extension execute

%install_ext https://www...