Every number has a story.
A reference for the commands that turn data into a report. Click any stage below to expand it; use the Ask button (top right) to search by question.
About this reference — how it is structured and why
An analysis does not begin with a file. It begins with a question for which numbers might form part of the answer — and a file is one of many intermediate states between that question and the eventual response. The «messy CSV» is not the starting point; it is the point at which the data first becomes visible and uncomfortable.
Before reaching for any command, the analyst should be able to state three things in plain language: (i) what question is being asked, (ii) which population or unit the answer concerns, and (iii) what an answer would look like if one were given the result in finished form. Only then is it productive to ask which tools are required.
The remainder of this reference is organised around the typical journey from such a question to a delivered result. Each stage lists the commands one is likely to encounter, with — for each command — a description of what it does, a token-by-token reading of what the code itself means, the conditions under which it applies, and the situations in which one would reach for it.
1 Locate & Load — bringing the data into Python
Loading is the operation by which a file on disk is read into a Python object that can be queried, transformed and summarised. The choice of loader depends on the file format; the choice of parameters depends on what the file's author did with characters, separators, and missing markers.
How to choose: The file extension is a useful first signal, but not authoritative. Open the file in a plain-text editor first and read the first ten lines. Establish: which character separates fields, whether the first row contains column names, how the file encodes special characters (German umlauts are a frequent indicator), and how missing values are written. Only then call a loader.
2 Inspect — understanding what was loaded
Before any analytical operation, the analyst should be able to describe the loaded object in four respects: its size, the type of each column, the prevalence of missing values, and the presence of duplicate records. These four checks correspond to questions a careful reader would ask before trusting any subsequent claim.
How to choose: Begin with shape and head(); these confirm that the loader produced what was expected. Follow with info() and isna().sum(); these reveal the structural integrity of the table. Use value_counts() and describe() when interest shifts to the distribution of a single column.
3 Clean — addressing irregularities
Cleaning denotes the corrections applied so that the data can be analysed without introducing systematic error. The most common irregularities are inconsistent types, missing values, duplicate records, and inconsistent string representations of the same underlying category. Each requires a deliberate decision; defaults rarely capture clinical intent.
How to choose: Cleaning proceeds in a fixed order: types are corrected first (so that subsequent comparisons work), then missing values (so that they do not propagate silently), then duplicates (so that aggregations are not inflated), then string normalisation (so that categories collapse correctly). Decisions about how to treat missingness are addressed in the Decision Boxes at the end of this document.
4 Transform — deriving variables, filtering, sorting
Transformation reshapes the cleaned table toward the question. New variables are computed from existing ones; rows are filtered to the subpopulation of interest; the order of the table is chosen so that a reader can locate the relevant rows quickly.
How to choose: If a new value is to be computed for every row, use a vectorised expression (column arithmetic, np.where, np.select). If only a subset of rows is of interest, build a boolean mask. If a top-N or bottom-N is required, prefer nlargest / nsmallest over a full sort.
5 Aggregate — summarising by group
Aggregation reduces a long table to a short one by computing a summary statistic within each group. The mental model is split–apply–combine: split the table by one or more key columns, apply a function to each group, combine the results into a single summary.
How to choose: groupby followed by an aggregation produces a long-format summary suitable for further analysis. pivot_table produces a wide-format summary suitable for display. transform is appropriate when the group-level statistic must be aligned back to every original row (for instance, to compute a deviation from group mean).
6 Merge — combining tables and arrays
Most analyses require information from more than one source. Combining sources may take three forms: joining tables by a shared key, stacking tables that share structure, or complementing one source with another to fill in missing values. NumPy arrays are combined in analogous, but simpler, ways.
How to choose: If the two tables share a key column and each row of the first should pick up additional columns from the second, the operation is a merge. If the two tables have the same columns and represent different periods or sources, the operation is a concatenation. If the two tables describe the same entities but each contains gaps the other can fill, the operation is a complement (combine_first).
7 Visualize — graphical summaries
A graphical summary supports the reader in seeing a pattern that a tabular summary cannot. The choice of plot type follows from the question: comparison across categories (bar), evolution over time (line), distribution of a single variable (histogram), relation between two variables (scatter), distribution by group (boxplot).
How to choose: Matplotlib's object-oriented interface (fig, ax = plt.subplots() followed by methods on ax) is the recommended foundation. Seaborn provides briefer syntax for common statistical plots and is appropriate once the foundations are familiar. Every plot intended for an external reader requires, at minimum, a descriptive title, axis labels with units, and — if multiple series are present — a legend.
8 Export — delivering the result
A result that remains in a notebook is not yet delivered. Delivery takes the form of data files for downstream use, image files for inclusion in reports, and the notebook itself rendered for an external reader.
How to choose: Tabular data intended for re-use by humans should be exported as CSV or Excel. Tabular data intended for re-use by other Python programs is better stored as Parquet. Figures intended for slides should be exported as PNG at high resolution; figures intended for printed reports as PDF or SVG.
⊡ Select & Index — addressing rows, columns and cells
Every operation downstream of loading depends on being able to name the rows, columns, or cells it concerns. pandas separates selection by label from selection by position, and treats a boolean condition as a first-class way to address a subpopulation. Confusing these is the most frequent source of beginner errors and of the SettingWithCopyWarning.
How to choose: Use loc for labels and iloc for positions; for a single cell, at / iat. To select a subpopulation, build a boolean mask. To write into a filtered subset, assign through df.loc[mask, col] = value; to work on an independent subset, take an explicit .copy(). Promote a key column with set_index; restore clean numbering — or recover groupby keys — with reset_index.
Σ Statistics — describing and summarising columns
Beyond the pipeline, a recurring need is to describe a single column or the relationship between columns: its centre, its spread, its shape, its extremes, and its distinct values. These reductions answer questions of the form "what is typical", "how variable", and "which is the largest".
How to choose: For a typical value, prefer median over mean when the column is skewed or has outliers. For spread, std accompanies the mean; the interquartile range via quantile accompanies the median. To find which row attains an extreme, use idxmax / idxmin, not max / min. For categorical columns, value_counts and nunique replace the numeric summaries.
Aa Strings — operating on text columns
Text columns frequently carry structure — codes, compound fields, free-text notes — that must be tested, decomposed, or extracted before the data can be analysed. The .str accessor applies string operations element-wise across a Series; without it the methods would apply to the Series object itself, not to each value.
How to choose: To select rows by text, build a mask with str.contains (pattern), str.startswith / str.endswith (fixed affix). To take a field apart, use str.split; to assemble one, str.cat. To lift a structured fragment out of free text, use str.extract for one match or str.findall for all. Always pass na=False when the result will index rows.
⏱ Datetime — working with dates and times
Dates almost always arrive as strings and must first be parsed into datetime64 with pd.to_datetime; only then do the .dt accessor, chronological sorting, and time arithmetic become available. Time-aware aggregation is a distinct operation from ordinary grouping.
How to choose: Parse first with pd.to_datetime (in the Clean stage). Extract components with .dt.year / .dt.month / .dt.weekday / .dt.hour. For a grouping key that sorts correctly across years, prefer .dt.to_period over a formatted string. To summarise a time series at a regular frequency, use resample on a datetime index rather than grouping on a month number.
⇄ Reshape — changing the table's layout
The same data can be held in a long (tidy) layout — one observation per row — or a wide layout — one row per entity with measures spread across columns. Analysis and grouping want long; display and matrices want wide. Reshaping moves between the two; it changes layout, not content.
How to choose: Wide to long: melt (named columns) or stack (column index). Long to wide: pivot when each cell is unique, pivot_table when duplicates must be aggregated, unstack to lift an index level from a grouped result. Use explode to turn list-valued cells into rows, and crosstab for a contingency table of two categoricals.
N NumPy — the array layer beneath pandas
Every pandas column is a NumPy array underneath, and pandas borrows NumPy's vectorised arithmetic. Direct NumPy is needed when constructing data, generating reproducible random values, applying a vectorised conditional, or arranging data into the matrix form a modelling API expects.
How to choose: Build arrays with np.array; build sequences with np.arange (by step) or np.linspace (by count); pre-allocate with np.zeros / np.ones. Use np.where for a vectorised if-else, np.clip to bound values, np.log to tame skew. For randomness, always go through a seeded np.random.default_rng so results are reproducible.
★ Decision boxes — the considered choices
▣ A complete worked example