Data Cookbook

- 16 mins

Data Cookbook

An ever-growing collection of code blocks to perform useful data manipulation and plotting functions with standard Python libraries. This is mostly for my own self-reference, but possibly useful to others.



Delete a line matching a pattern

sed '/some_string/d' $FILE


These code snacks describe useful features of Python 3+ that aren’t always emphasized.

Force only named arguments to functions

In the below example, arguments following the splat * must be supplied as named arguments.

This is somewhat intuitive if you’re used to Pythons splat operator for *args or **kwargs. Here, the lonely splat “catches” positional arguments passed to the function after its introduction in the definition string.

def function(positional, *, named_only0, named_only1):
  # do some things

def only_takes_named_args(*, named_only0, named_only1):
  # do some things

This is useful when defining functions that may have arguments added and removed over time, explicitly preventing code from relying on the positional order.


Preserve array dimensionality when slicing

When slicing a plane i from a multidimensional array A, use A[i:i+1,...] to preserve the array dimensionality with an empty dimension of size 1.

import numpy as np
A = np.random.random(5,5,5)
i = 0

A[:,i,:].shape # (5,5)
A[:,i:i+1,:].shape # (5,1,5)

Add an empty dimension by indexing

You can add an empty dimension of size 1 to an np.ndarray by passing None to one of the axes while indexing.

A = np.random.random((3,3))

B = A[:, :, None]
print(B.shape) # (3, 3, 1)

C = np.expand_dims(A, -1)
print(C.shape) # (3, 3, 1)

np.all(B == C) # True


Split a column by a text delimiter

Use .str.split

# split by a '-' delimiter
# split is a pd.DataFrame, with each delimited column separated out
split = df.str.split('-', expand=True)

Replicate each row in a DataFrame N times

Use the .values attribute of a DataFrame and np.repeat

N = 3 # times to replicate
newdf = pd.DataFrame(np.repeat(df.values, N, axis=0))
newdf.columns = df.columns

Sort a DataFrame by multiple columns

Use the sort_values method of DataFrames.

df.sort_values(['a', 'b'], ascending=[True, False])


Check if rows are equal to an array-like vector

Given an array-like vector v with same dimensionality as rows in a DataFrame df, check which rows in df are equal to v.

df = pd.DataFrame([[0,1],[2,3],[4,5]], columns=['A', 'B'])
v = np.array([0,1])
(df == v).all(1) # checks for boolean True across columns


Matplotlib / Seaborn

Create editable, uncropped PDF exports

import matplotlib
# ensure text in PDF exports is editable.
matplotlib.rcParams['pdf.fonttype'] = 42
matplotlib.rcParams['ps.fonttype'] = 42
# prevent the PDF from being clipped to the "figsize".
# NOTE: this is different than `plt.tight_layout()`
# despite the similar name.
matplotlib.rcParams['savefig.bbox'] = 'tight'

Rotate Seaborn axis labels

g = sns.barplot(...)
g.set_xticklabels(g.get_xticklabels(), rotation=45)

Plot a line with a continuous color variable

Use a matplotlib.collections LineCollection to plot a set of smaller lines each with a different color, as desired.

StackOverflow Credit

import numpy as np
from matplotlib import pyplot as plt
from matplotlib.collections import LineCollection

x  = np.sin(np.linspace(0, 2*np.pi, 100))
y  = np.cos(np.linspace(0, 2*np.pi, 100))
t = np.linspace(0,1,x.shape[0]) # your "time" variable

# set up a list of (x,y) points
points = np.array([x,y]).transpose().reshape(-1,1,2)
print points.shape  # Out: (len(x),1,2)

# set up a list of segments
segs = np.concatenate([points[:-1],points[1:]],axis=1)
print segs.shape  # Out: ( len(x)-1, 2, 2 )
                  # see what we've done here -- we've mapped our (x,y)
                  # points to an array of segment start/end coordinates.
                  # segs[i,0,:] == segs[i-1,1,:]

# make the collection of segments
lc = LineCollection(segs, cmap=plt.get_cmap('viridis'))
lc.set_array(t) # color the segments by our parameter

# plot the collection
fig, ax = plt.subplots(1,1)
ax.add_collection(lc) # add the collection to the plot
ax.set_xlim(x.min(), x.max()) # line collections don't auto-scale the plot
ax.set_ylim(y.min(), y.max())

Connect two points with a curve

To connect two points in a plot with a curved line, we use scipy.interpolate.CubicSpline to interpolate a continuous series of points along a cubic function between the two points.

from scipy.interpolate import CubicSpline

points = np.array(
    [0, 0],
    [1, 0],

# draw curve connecting start and destinations
xvals = np.linspace(x[0], x[1], 100)
# generate a coordinate at the midpoint to set the middle
# position of our arc
x_coords = [
  points[0, 0],
  points[0,0] + (points[1,0]-points[0,0])/2,

# ensure that the x_coords sequence is increasing
if x_coords[-1] < x_coords[0]:
    x_coords = x_coords[::-1]

# set a midpoint a bit above the two lower points
y = [0, 0.1, 0]

# generate smoothed values
spline = CubicSpline(x_coords, y)
y_smooth = spline(xvals)

# plot the original points
ax.scatter(points[:, 0], points[:, 1])

# plot the smooth curve
ax.plot(xvals, y_smooth, color='lightgray', linestyle='--', alpha=0.5,)
ax.set_ylim(bottom=0.05, top=0.15)
ax.set_xlim(-0.1, 1.1)

Add a label to heatmap colorbars in seaborn

  cbar_kws={'label': 'colorbar title'})

Remove space between subplots

This is useful when plotting a grid of images.

H, W = 2, 2
fig, ax = plt.subplots(H, W)
for i in range(4):
  ax[i//2, i%2].imshow(I)

Remove axis spines from a matplotlib plot

fig, ax = plt.subplots(H, W, figsize=(h, w))
# `.spines` keys are {'left', 'right', 'top', 'bottom'}

Animate an image

from matplotlib import animation, rc

fig, ax = plt.subplots(1, 1, figsize=(10,10))
# remove white frame around image
fig.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=None, hspace=None)

im = ax.imshow(data, animated=True)

def updatefig(idx):
    return im

anim = animation.FuncAnimation(
  fig, # figure with initialized artists
  updatefig, # updating function
  frames=100, # number of iterations, passes `range(0, frames)` to `updatefig`
  interval=1e3/30, # ms between frames, i.e. 1e3/FPS for a FPS argument
  blit=True) # drawing optimization

# if in a Jupyter notebook, the HTML module can display the animation inline
from IPython.display import HTML

Add a row/column color legend to seaborn clustermap


# define some row clusters
row_clusters = get_row_clusters(data) # np.ndarray,

# set up a LUT to assign colors to `row_clusters`
pal = sns.color_palette('tab20')

# make a clustermap
clmap = sns.clustermap(
  row_colors = pal[row_clusters]

for label in np.unique(clusters):,
clmap.ax_col_dendrogram.legend(loc="center", ncol=5, frameon=False)

Add a second set of xticklabels to a seaborn heatmap

fig, ax = plt.subplots(1, 1, ...)


# clone the x-axis
ax2 = ax.twiny()

# clean up the plotting aesthetics introduced by the second axis

for x in ['top', 'bottom', 'right', 'left']:

Genomics Tools

Genomics has its own set of standard tools, and it can be baffling to remember all the useful one-liners each tool offers. Here are some useful snippets to bring some sanity to bioinformatics workflows.


Most next-generation sequencing data generated in the USA is submitted to the Gene Expression Omnibus. Getting access to the raw sequencing reads is a bit trickier than you’d think. For collecting data from GEO, we largely rely on sra-tools.

Download FASTQs associated with a GEO submission

This one-liner will find all the FASTQ files associated with a project in GEO then use fastq-dump to download associated reads as .fastq.gz files.

Biostars Credit

 # set a project ID
esearch -db sra -query $PROJECT_ID  | efetch --format runinfo | cut -d ',' -f 1 | grep SRR | xargs fastq-dump -X 10 --split-files --gzip


I love LaTeX. LaTex does not love me back. Here are some snippets to make our relationship more functional.

Use if/then control flow in a LaTeX build

% defines \newtoggle, \settoggle
\newtoggle{somevar} % set a new boolean variable

% run an if then
  % do thing
  % else, do other thing or blank for nothing

Generate a custom bibtex style

# outputs
#   some_name.dbj - instructions for making a `bst`
#   some_name.bst - compiled `bst`
latex makebst
# to remake a `bst` from the `dbj`
tex some_name.dbj # outputs some_name.bst

Remove numbers or citation labels from reference list

SE Credit

% we can also replace numbers with a common character, like a bullet

Customize figure captions

% remove separator between "Figure XYZ" and caption.
% print the figure number, but no caption
% useful for separating figures and captions in journal proofs
% e.g. "Figure 1", the caption text is suppressed
% use normal caption text, colon figure separator
% e.g. "Figure 1: Caption text here"

Suppress graphics

Journals often want captions and figures separated in a final proof. We can insert captions without graphics by redefining the includegraphics command.


Ensure figures and tables only appear after the insert position

The flafter package prevents any floats (figures, tables) from being presented above the position in the code where they are inserted. This is useful for ensuring a float appears below the relevant section title, or to ensure it doesn’t insert too early in the paper and give away your punchline before you’re ready.


Shrink pdflatex output size

pdflatex tends to render huge PDF outputs. My understanding is that pdflatex is very conservative and applies no compression scheme. Substantial improvements in the file size can be found with loseless or modest lossy compression.

Ghostscript is installed by default on most *nix and does a good job of reducing file size on its own.

gs \
  -sDEVICE=pdfwrite \
  -dCompatibilityLevel=1.5 \
  -dPDFSETTINGS=/printer \
  -dBATCH \
  -sOutputFile=small.pdf big.pdf

The /printer argument will render 300 dpi compressed images that generally look great. If you need more compression, try swapping /ebook in place of printer. The images will still be legible, but you’ll almost certainly notice compression artifacts.

Another approach is to use a third-party tool pdfsizeopt.

After installation, compression is a one-liner.

/path/to/pdfsizeopt big.pdf

pdfsizeopt tends to get a smaller file size with better Ghostscript with the /printer setting, but it takes much longer to execute.

Jacob C. Kimmel

Jacob C. Kimmel

Computational Fellow @ Calico. Interested in aging, genomics, imaging, & machine learning.

rss facebook twitter github youtube mail spotify instagram linkedin google pinterest medium vimeo gscholar