NumPy Cookbook

Chapter 1. Winding Along with IPython

In this chapter, we will cover the following topics:

Installing IPython
Using IPython as a shell
Reading manual pages
Installing Matplotlib
Running a web notebook
Exporting a web notebook
Importing a web notebook
Configuring a notebook server
Exploring the SymPy profile

Installing IPython

IPython can be installed in various ways depending on your operating system. For the terminal-based shell, there is a dependency on readline. The web notebook requires tornado and zmq.

In addition to installing IPython, we will install setuptools, which gives you the easy_install command. The easy_install command is the default, standard package manager for Python. pip can be installed once you have easy_install available. The pip command is similar to easy_install, and adds options such as uninstalling.

How to do it...

This section describes how IPython can be installed on Windows, Mac OS X, and Linux. It also describes how to install IPython and its dependencies with easy_install and pip, or from source.

Installing IPython and setup tools on Windows: A binary Windows installer for Python 2 or Python 3 is available on the IPython website. Also see http://ipython.org/ipython-doc/stable/install/install.html#windows.
Install setuptools with an installer from http://pypi.python.org/pypi/setuptools#files. Then install pip; for instance:
```
cd C:\Python27\scripts
python .\easy_install-27-script.py pip
```
Installing IPython On Mac OS X: Install the Apple Developer Tools (Xcode) if necessary. Xcode can be found on the OSX DVDs that came with your Mac or App Store. Follow the easy_install/pip instructions, or the installing from source instructions provided later in this section.
Installing IPython On Linux: Because there are so many Linux distributions, this section will not be exhaustive.
- On Debian, type the following command:
```
su – aptitude install ipython python-setuptools
```
- On Fedora, the magic command is as follows:
```
su – yum install ipython python-setuptools-devel
```
- The following command will install IPython on Gentoo:
```
su – emerge ipython
```
- For Ubuntu, the install command is as follows:
```
sudo apt-get install ipython python-setuptools
```
Installing IPython with easy_install or pip: Install IPython and all the dependencies required for the recipes in this chapter with easy_install, using the following command:
```
easy_install ipython pyzmq tornado readline
```
Alternatively, you can first install pip with easy_install, by typing the following command in your terminal:
```
easy_install pip
```
After that, install IPython using pip, with the following command:
```
sudo pip install ipython pyzmq tornado readline
```
Installing from source: If you want to use the bleeding edge development version, then installing from source is for you.
1. Download the latest tarball from https://github.com/ipython/ipython/downloads.
2. Unpack the source code from the archive:
```
tar xzf ipython-<version>.tar.gz
```
3. If you have Git installed, you can clone the Git repository instead:
```
$ git clone https://github.com/ipython/ipython.git
```
4. Go to the ipython directory:
```
cd ipython
```
5. Run the setup script. This may require you to run the command with sudo, as follows:
```
sudo setup.py install
```

How it works...

We installed IPython using several methods. Most of these methods install the latest stable release, except when you install from source, which will install the development version.

Using IPython as a shell

Scientists and engineers are used to experimenting. IPython was created by scientists with experimentation in mind. The interactive environment that IPython provides is viewed by many as a direct answer to Matlab, Mathematica, and Maple and R.

Following is a list of features of the IPython shell:

Tab completion
History mechanism
Inline editing
Ability to call external Python scripts with %run
Access to system commands
The pylab switch
Access to Python debugger and profiler

How to do it...

This section describes how to use the IPython shell.

The pylab switch: The pylab switch automatically imports all the Scipy, NumPy, and Matplotlib packages. Without this switch, we would have to import these packages ourselves.

All we need to do is enter the following instruction on the command line:

$ ipython -pylab
Type "copyright", "credits" or "license" for more information.

IPython 0.12 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

Welcome to pylab, a matplotlib-based Python environment [backend: MacOSX].
For more information, type 'help(pylab)'.
In [1]: quit()
quit() or Ctrl 
+ D quits the IPython shell.

Saving a session: We might want to be able to go back to our experiments. In IPython, it is easy to save a session for later use, with the following command:

In [1]: %logstart
Activating auto-logging. Current session state plus future input saved.
Filename       : ipython_log.py
Mode           : rotate
Output logging : False
Raw input log  : False
Timestamping   : False
State          : active

Logging can be switched off as follows:

In [9]: %logoff
Switching logging OFF

Executing system shell commands: Execute system shell commands in the default IPython profile by prefixing the command with the ! symbol. For instance, the following input will get the current date:
```
In [1]: !date
```
In fact, any line prefixed with ! is sent to the system shell. Also, we can store the command output, as shown here:
```
In [2]: thedate = !date
In [3]: thedate
```
Displaying history: We can show the history of commands with the %hist command () for example:
```
In [1]: a = 2 + 2

In [2]: a
Out[2]: 4

In [3]: %hist
a = 2 + 2
a
%hist
```
This is a common feature in Command Line Interface (CLI) environments. We can also search through the history with the -g switch
```
In [5]: %hist -g a = 2
    1: a = 2 + 2
```

Note

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

How it works...

We saw a number of so called "magic functions" in action. These functions start with the % character. If the magic function is used on a line by itself, the % prefix is optional.

Running a web notebook

The newest release of IPython introduced a new exciting feature – the web notebook. A so called "notebook server" can serve notebooks over the web. We can now start a notebook server and have a web-based IPython environment. This environment has most of the features in the regular IPython environment. The new features include the following:

Displaying images and inline plots
Using HTML and Markdown in text cells
Importing and exporting of notebooks

Getting ready

Before we start, we should make sure that all the required software is installed. There is a dependency on tornado and zmq. See the Installing IPython recipe in this chapter for more information.

How to do it...

Running a notebook: We can start a notebook with the following code:
```
$ ipython notebook

[NotebookApp] Using existing profile dir: u'/Users/ivanidris/.ipython/profile_default'
[NotebookApp] The IPython Notebook is running at: http://127.0.0.1:8888
[NotebookApp] Use Control-C to stop this server and shut down all kernels.
```
As you can see, we are using the default profile. A server started on the local machine at port 8888. We will learn how to configure these settings later on in this chapter. The notebook is opened in your default browser; this is configurable as well:
IPython lists all the notebooks in the directory where you started the notebook. In this example no notebooks were found. The server can be stopped with Ctrl + C.
Running a notebook in the pylab mode: Run a web notebook in the pylab mode with the following command:
```
$ ipython notebook --pylab
```
This loads the Scipy, NumPy, and Matplotlib modules.
Running notebook with inline figures: We can display inline Matplotlib plots with the inline directive, using the following command:
```
$ ipython notebook --pylab inline
```

Create a notebook: Click on the New Notebook button to create a new notebook:
Create an array: Create an array with the arange function. Type the command in the following screenshot, and press Enter:
Next, enter the following command and press Enter. You will see the output as shown in Out [2] in the following screenshot:
Plot the sinc function: Apply the sinc function to the array and plot the result, as shown in the following screenshot:

How it works...

The inline option lets you display inline Matplotlib plots. When combined with the pylab mode, you don't need to import the NumPy, SciPy, and Matplotlib packages.

Importing a web notebook

Python scripts can be imported as a web notebook. Obviously, we can also import previously exported notebooks.

How to do it...

The following steps show how a python script can be imported as a web notebook:

Import a python script by dragging it from Explorer or Finder into the notebook page. The following screenshot is an example of what we see after dragging the vectorsum.py from NumPy Beginner's Guide into the notebook page:
Click the Upload button to import the program. IPython does a decent job of importing the code. Unfortunately, as shown in the following screenshot, the code is all placed in one cell. At least that is how it worked at the time of writing:

Tag the script for multiple cells.

In order to split the code into multiple cells we need to use special tags. These tags are in fact Python comments, but they look a bit like XML tags. The code has to start with the following tag:

# <nbformat>2</nbformat>

This indicates the format of the notebook. Each new code cell is indicated with the following tag:

# <codecell>

The following is the tagged code:

# <nbformat>2</nbformat>
#!/usr/bin/env/python

from datetime import datetime
import numpy
"""
 Chapter 1 of NumPy Beginners Guide.
 This program demonstrates vector addition the Python way.
 Run from the command line as follows
     
  python vectorsum.py n
 
 where n is an integer that specifies the size of the vectors.

 The first vector to be added contains the squares of 0 up to n. 
 The second vector contains the cubes of 0 up to n.
 The program prints the last 2 elements of the sum and the elapsed time.
"""

def numpysum(n):
   a = numpy.arange(n) ** 2
   b = numpy.arange(n) ** 3
   c = a + b

   return c

def pythonsum(n):
   a = range(n)
   b = range(n)
   c = []

   for i in range(len(a)):
       a[i] = i ** 2
       b[i] = i ** 3
       c.append(a[i] + b[i])

   return c
   
# <codecell>
size = int(50)

# <codecell>
start = datetime.now()
c = pythonsum(size)
delta = datetime.now() - start
print "The last 2 elements of the sum", c[-2:]
print "PythonSum elapsed time in microseconds", delta.microseconds

# <codecell>
start = datetime.now()
c = numpysum(size)
delta = datetime.now() - start
print "The last 2 elements of the sum", c[-2:]
print "NumPySum elapsed time in microseconds", delta.microseconds

The code is split into several cells according to the tags, as shown in the following screenshot:

Configuring a notebook server

A public notebook server needs to be secure. You should set a password and use a SSL certificate to connect to it. We need the certificate to provide secure communication over https (for more information see https://en.wikipedia.org/wiki/Transport_Layer_Security).

How to do it...

The following steps describe how to configure a secure notebook server:

Generate a password: We can generate a password from IPython. Start a new IPython session, and type in the following commands:
```
In [1]: from IPython.lib import passwd

In [2]: passwd()
Enter password: 
Verify password: 
Out[2]: 'sha1:0e422dfccef2:84cfbcbb3ef95872fb8e23be3999c123f862d856'
```
At the second input line you will be prompted for a password. You need to remember this password. A long string is generated. Copy this string because we will need it later on.

Create a SSL certificate: To create a SSL certificate, you will need to have the openssl command in your path.

Setting up the openssl command is not rocket science, but can be tricky. Unfortunately, it is outside the scope of this book. On the bright side there are plenty of tutorials available online to help you further.

Execute the following command to create a certificate with the name mycert.pem:

$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
Generating a 1024 bit RSA private key
......++++++
........................++++++
writing new private key to 'mycert.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:
Email Address []:

The openssl utility prompts you to fill in some fields. For more information, check the relevant man page (short for manual page).

Create a server profile: Create a special profile for the server using the following command:
```
ipython profile create nbserver
```
Edit the profile configuration file: Edit the configuration file. In this example, it can be found in Edit in ~/.ipython/profile_nbserver/ipython_notebook_config.py.
The configuration file is pretty large, so we will omit many of the lines in it. The lines that we need to change at minimum are:
```
c.NotebookApp.certfile = u'/absolute/path/to/your/certificate'
c.NotebookApp.password = u'sha1:b...your password'
c.NotebookApp.port = 9999
```
Notice that we are pointing to the SSL certificate we created. We set a password and changed the port to 9999.
Start the server: Using the following command, start the server to check whether the changes worked.
```
ipython notebook --profile=nbserver
[NotebookApp] Using existing profile dir: u'/Users/ivanidris/.ipython/profile_nbserver'
[NotebookApp] The IPython Notebook is running at: https://127.0.0.1:9999
[NotebookApp] Use Control-C to stop this server and shut down all kernels.
```
The server is running on port 9999, and you need to connect to it via https. If everything goes well, we should see a login page. Also, you would probably need to accept a security exception in your browser.

How it works...

We created a special profile for our public server. There are some sample profiles that are already present, such as the default profile. Creating a profile adds a profile_<profilename> folder to the .ipython directory with, among others, a configuration file. The profile can then be loaded with the --profile=<profile_name> command-line option. We can list the profiles with the following command:

ipython profile list

Available profiles in IPython:
    cluster
    math
    pysh
    python3

    The first request for a bundled profile will copy it
    into your IPython directory (/Users/ivanidris/.ipython),
    where you can customize it.

Available profiles in /Users/ivanidris/.ipython:
    default
    nbserver
    sh

Filter reviews by

All

Amazon verified reviews

Sujit Pal Jun 22, 2014

Numpy is central to most scientific Python toolkits, and learning to write effective Numpy code can make your code more readable and faster. While the Numpy documentation is quite comprehensive, books provide a more structured learning path, and since there are not too many books on Numpy, this book hits a sweet spot. The book is aimed at intermediate level Python users. You will gain more from the book if you work out the code examples yourself rather than just read the examples. Also the examples are slightly mathy (its a book about arrays and matrices after all), so you may have to do some reading if you don't remember your linear algebra, for example.The book covers examples from famous algorithms (Fibonacci, Sieve of Eratosthenes, etc), finance, etc, mainly to show the usage for various NumPy functions, both simple and advanced. There is a full chapter of recipes on Audio and Image processing techniques. There is also discussion of using memory mapped files, sharing data with the Python Image Library (PIL) through the array interface, converting code to Cython for speed, universal functions (none on vectorize() strangely), masking, etc. There is other information, such as interfacing with R using RPy2, running Numpy on Google App Engine and PiCloud (I didn't pay too much attention to these since I didn't anticipate using them).The format of the recipes were a bit unusual. Generally it tells you what you can do with it (in the title), then gives a quick overview of the approach (in English), followed by the full code to do accomplish the recipe. The recipes in the book flips this around, putting partial code with some explanation first, then the full code, and then the overview. So the reading (or following along with a Python shell) is not linear, reader has to jump back and forth. Not a huge deal once you recognize it, but using a more linear style may enhance the reading and learning experience in future editions.I read this book after I read "Learning Numpy Arrays" by the same author, and there is quite a bit of overlap in the examples. Perhaps not surprising because the subject is mostly identical. However, I think its still worth purchasing both the books because each book has enough unique content.

Amazon Verified review

Tom Jensen Morgan Jan 15, 2013

I read this book during the holidays. [...] I found it quite useful and well-written. I am just learning Python and am primarily interested in how it can be used for geospatial processing. I recommend this book for anyone wanting to grow their Python skillset.

BeckyLEXI Jan 30, 2013

While this book has some advanced examples, the nice thing is that it is accessible for newer to intermediate users of python and NumPy. The recipes are well documented and are concrete, helping with the learning experience. The range of the examples also helps to expand your understanding of what NumPy is capable of - which is enlightening. I will continue to use this book as a reference into the future.

Amazon Customer Dec 28, 2012

When I first mentioned that I was getting this book, a colleague of mine wanted to know why I was even bothering - 'just go straight for Pandas' he said. Actually, I think he is missing the point of both this book and NumPy in general.Not only is this one of those well written cookbooks that sets out the problems and solutions neatly and succinctly, but it is one of those cookbooks that you turn to, not really expecting to find the answer to a problem you are having right now, but rather solutions to problems that give you insight into just how broad and wide the solutions that NumPy can be applied to.The NumPy Cookbook covers everything from getting started with IPython (worth the price of admission alone, trust me) - to re-sizing images, processing audio, performing statistical analysis (obviously), estimating stock returns and, well, err, installing Pandas.The very best cookbooks answer the questions you did not know you had, and show you to do things that you did not know were possible. That is what the 'NumPy Cookbook' does - and it does it exceptionally well.I spend my time working with Python, examining numbers from our data warehouse, and I have been using NumPy for a long while to manipulate the data, to slice, dice and fill in the blanks. But this book showed me some new tricks, some tricks that I will be able to apply directly to my work, and for that I am grateful.So, install Pandas and forget all about NumPy? No. NumPy is a pre-requisite for Pandas, and you really should know all about it, because the two are not mutually exclusive. Read this book - and learn about some of the really cool stuff that you can do with NumPy.

alan1955 Jan 15, 2013

I am a scientist just beginning to use Python, Numpy, Scipy, and Matplotlib on a Linux machine. There are a lot of things to learn to use with anyone of these. This book provides valuable insights for a person just beginning with Numpy that would take a long time to discover on your own through working examples. The examples are quite extensive covering many areas. All areas are not of interest to me, but the techniques used through working example can be easily adapted to whatever problem you are working on, saving a lot of time. I would say for me the examples in chapter 7 on Profiling and debugging chapter 2 on advanced indexing, and chapter 3 on commonly used functions were very useful. I would recommend this to anyone who is wanting to get started with Numpy and is planning to use it regularly. More information can be had at: [...]