####### Ipython ####### Load a file into external editor ================================ .. code-block:: bash edit file * Use Emacs .. code-block:: bash EDITOR=emacsclient ipython edit file Run a file ========== .. code-block:: bash run file Save code from shell ==================== .. code-block:: bash save filename.py 1-10 Display available namespaces ============================ .. code-block:: bash who * Including data types .. code-block:: bash whos Get documentation ================= * Use ? after module, function, whatever * pinfo some module or function or whatever Show source code of a function ============================== .. code-block:: bash psource * Get whole source file .. code-block:: bash pfile Edit a function =============== * -x will not execute the new code .. code-block:: bash edit Save shell command in variable ============================== .. code-block:: bash foo = !ps ax | grep something Define aliases ============== .. code-block:: bash %alias pids ps ax | grep app | cut -d " " -f 1 %store pids Debugging ========= * Switch on pdb on execptions .. code-block:: bash pdb * Run script in pdb .. code-block:: bash run -d file * Run script in pdb with breakpoint in line 23 .. code-block:: bash run -d -b 23 file * Automatically start Ipython debugger on exception .. code-block:: bash try: ret = 1 / 0 except Exception, e: import sys, IPython IPython.Shell.IPShell(argv=[]) IPython.Debugger.Pdb(IPython.ipapi.get().options.colors).set_trace(sys._getframe()) Profile ======= * http://pynash.org/2013/03/06/timing-and-profiling.html .. code-block:: bash %time some_function * Run cProfile .. code-block:: bash %prun file or function * Filter output .. code-block:: bash %prun -l some_filter_string file or function History ======= * Show history .. code-block:: bash hist * Execute command nr x .. code-block:: bash _ix * Ranges (1-5) .. code-block:: bash In[1:6] * Print output of command 42 .. code-block:: bash Out[42] Bookmarks ========= * Create a dir bookmark .. code-block:: bash bookmark name * Save bookmark .. code-block:: bash store name * List bookmarks .. code-block:: bash bookmark -l * Delete bookmark .. code-block:: bash bookmark -d name Macros ====== * Save history commands 1-5 in a macro muh .. code-block:: bash macro muh 1-5 * Save the macro .. code-block:: bash store muh * Execute it by calling its name * Show the source .. code-block:: bash print muh Background jobs =============== * Start a statement in the background .. code-block:: bash bg some_func() * Show status of job .. code-block:: bash job[0].status * Get the result .. code-block:: bash job[0].result Connect to an existing console =============================== * Start ipython with console or qtconsole .. code-block:: bash ipython kernel [IPKernelApp] To connect another client to this kernel, use: [IPKernelApp] --existing kernel-26492.json * Now you can connect another console by executing .. code-block:: bash ipython console --existing kernel-26492.json * If the kernel is on another system you either have to setup ssh tunnel for all port specified in the json file .. code-block:: bash ssh user@remote -f -N -L 61947:127.0.0.1:61947 * or append `--ssh user@remote` to the ipython console command * The kernel.json file can be found in ``/run/user/0/jupyter/`` or ``~/.ipython/profile_default/security`` * To manually create all ssh tunnels .. code-block:: bash for port in $(cat kernel-1234.json | grep '_port' | grep -o '[0-9]\+'); do ssh myremotehost -f -N -L $port:127.0.0.1:$port; done Parallel computing ================== * Its possible to do parallel computing in different ways like SSH, MPI, PBS and even Windows HPC * The controller opens a socket and allow engines to connect to it * First of all lets start an Ipython console locally for every core .. code-block:: bash ipcluster start ipython * To run a simple hello world on all cores exec .. code-block:: python from IPython.parallel import Client c = Client() c.ids c[:].apply_sync(lambda : "Hello, World") * Manually start a controller on your head node and start engines on all computing nodes (must have access to the file ~/.ipython/profile_default/security/ipcontroller-engine.json generated by the controller) .. code-block:: bash ipcontroller --enginessh=user@host ipengine * Now you can import the module ``os`` on every engine and print uname information .. code-block:: python with c[:].sync_imports(): import os %px print os.uname() * If all nodes share a home directory for ipcontroller-engine.json file you can automatically start the controller on localhost and engines on remote * Create a new profile with .. code-block:: bash ipython profile create --parallel --profile=ssh * Edit ``ipcluster_config.py`` and add .. code-block:: python c.IPClusterEngines.engine_launcher_class = 'SSHEngineSetLauncher' c.SSHEngineSetLauncher.engines = { 'host1.example.com' : 2, 'host2.example.com' : 5, 'host3.example.com' : (1, ['--profile-dir=/home/different/location']), 'host4.example.com' : 8 } * The key of the engines dict is the host and value number of engines to start * Now you can setup the cluster by executing .. code-block:: bash ipcluster start --profile=ssh * Use map to call a function on a dataset on all nodes .. code-block:: python v = c[:] result = v.map_sync(lambda x: x*x, data) Save state of a console ======================= * You can start / stop logging of a sessions state with ``%logstart `` / ``%logstop`` * To load the session state exec .. code-block:: bash ipython -lp * To automatically save state edit ``~/.ipython/profile_default/ipython_config.py`` and set .. code-block:: bash c.TerminalInteractiveShell.logstart = True c.TerminalInteractiveShell.logfile = '~/ipython_session.log' Extensions ========== * To install an extension execute .. code-block:: bash %install_ext https://www... * https://pypi.python.org/pypi/ipython-sql * Cython * http://blog.cloudera.com/blog/2014/08/how-to-use-ipython-notebook-with-apache-spark/