{ "cells": [ { "cell_type": "markdown", "id": "e99c0f38", "metadata": {}, "source": [ "# ESRA Workshop Demo (Notebook)\n", "\n", "Browse real battery data on ESRA (Argonne National Laboratory's energy storage data platform), filter it, find where a dataset's files live, and download them - all from this notebook.\n", "\n", "Works on Mac, Windows, and Linux.\n", "\n", "## Before you begin\n", "- You need **Python 3.11 or newer**, on Windows, macOS or Linux. **Step 0** shows how to pick it in a few clicks.\n", "- You need a free Globus account (any email works - no institutional login required).\n", "- Globus Connect Personal should already be installed and running on this computer (it's what lets Globus send files here). One-time setup: https://www.globus.org/globus-connect-personal\n", "- Run the cells **one at a time**, top to bottom, with **Shift+Enter** - don't use \"Run All\", since the sign-in cell needs you to interact with your browser before the next cell continues.\n", "- If a cell errors on the very first run, just run it again - the setup cell below installs anything missing into whichever Python environment this notebook is using." ] }, { "cell_type": "markdown", "id": "18e70558", "metadata": {}, "source": [ "## 0. Choose the Python that runs this notebook (once)\n", "\n", "**You need Python 3.11 or newer.** That's the only requirement, and it's the same on Windows, macOS and Linux. The first code cell installs everything else for you.\n", "\n", "### In VS Code: no terminal needed\n", "\n", "1. Click **Select Kernel** (top right of this notebook).\n", "2. Choose **Python Environments**.\n", "3. Choose **+ Create Python Environment → Venv**, then pick **Python 3.11 or newer**. VS Code makes a clean `.venv` folder next to the notebook, used only by this tutorial; delete that folder afterwards to remove it.\n", "4. If VS Code offers to install **ipykernel**, click **Install**.\n", "\n", "Already have a Python 3.11+ environment you use for notebooks (a venv or conda env)? Picking it in step 3 works too.\n", "\n", "That's it. Go to **step 1**.\n", "\n", "| You see | What it means |\n", "|---|---|\n", "| Only Python **3.10 or older** in the list (for example the macOS built-in `/usr/bin/python3`, which is 3.9) | Install a newer one (below), then reopen **Select Kernel** |\n", "| No **Create Python Environment** option | Open the folder that holds this notebook first: **File → Open Folder** |\n", "| The first code cell says the Python is too old | Same fix: pick or create a 3.11+ Python, then run it again |\n", "| *\"requires the ipykernel package\"*, and **Install** fails with *externally-managed-environment* (Homebrew or Linux system Python) | Don't install into that Python. Click **Change Kernel** and create a **Venv** as in step 3 |\n", "\n", "### No Python 3.11 or newer on this computer?\n", "\n", "- **Windows or macOS:** install it from [python.org/downloads](https://www.python.org/downloads/). On Windows, tick **\"Add python.exe to PATH\"** in the installer.\n", "- **Linux:** install `python3.12` with your package manager.\n", "\n", "Then reopen **Select Kernel → Python Environments**.\n", "\n", "
\n", "Prefer a terminal, or using JupyterLab instead of VS Code?\n", "\n", "[uv](https://docs.astral.sh/uv/) creates the environment and downloads Python if needed. Run these in a terminal (macOS/Linux) or PowerShell (Windows), not in this notebook.\n", "\n", "First install uv once, then close and reopen the terminal:\n", "\n", "- macOS / Linux: `curl -LsSf https://astral.sh/uv/install.sh | sh`\n", "- Windows: `powershell -ExecutionPolicy ByPass -c \"irm https://astral.sh/uv/install.ps1 | iex\"`\n", "\n", "| | macOS / Linux | Windows (PowerShell) |\n", "|---|---|---|\n", "| Create the environment | `uv venv ~/esra-demo-env --python 3.12` | `uv venv $HOME\\esra-demo-env --python 3.12` |\n", "| Install what the notebook needs | `uv pip install --python ~/esra-demo-env ipykernel requests pandas globus-cli` | `uv pip install --python $HOME\\esra-demo-env ipykernel requests pandas globus-cli` |\n", "| VS Code: *Enter interpreter path* | `~/esra-demo-env/bin/python` | `%USERPROFILE%\\esra-demo-env\\Scripts\\python.exe` |\n", "| Register it as a named kernel (shows up under *Jupyter Kernel…* in VS Code and in JupyterLab) | `~/esra-demo-env/bin/python -m ipykernel install --user --name esra-demo --display-name \"ESRA demo\"` | `$HOME\\esra-demo-env\\Scripts\\python -m ipykernel install --user --name esra-demo --display-name \"ESRA demo\"` |\n", "| Remove it afterwards | `~/esra-demo-env/bin/jupyter kernelspec remove -y esra-demo` then `rm -rf ~/esra-demo-env` | `$HOME\\esra-demo-env\\Scripts\\jupyter kernelspec remove -y esra-demo` then `Remove-Item -Recurse $HOME\\esra-demo-env` |\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "2d2d1a9c", "metadata": {}, "source": [ "## Settings\n", "Change these to search for something else." ] }, { "cell_type": "code", "execution_count": null, "id": "d1bbb3cd", "metadata": {}, "outputs": [], "source": [ "API = \"https://data.energystoragera.org/api/v1\"\n", "GUEST = \"9ccf9475-0e6a-4922-ae86-dddc119cf4be\" # ESRA's public Globus collection\n", "PREFIX = \"/global/cfs/cdirs/m4845/gsharing\"\n", "\n", "MATERIAL = \"NASICON\"\n", "TECHNIQUE = \"EIS\"\n", "DATASET = \"ESRA-EIS-0010\"\n", "DOWNLOAD_BASE = \"/~/Downloads\"" ] }, { "cell_type": "markdown", "id": "090d767b", "metadata": {}, "source": [ "## 1. Get the download tool ready, and sign in\n", "The first cell installs everything this notebook needs, into whichever Python environment is actually running this notebook right now - so it self-heals if your editor picked an environment that's missing a package. The sign-in cell after it signs you in to Globus - if you're not already signed in, your browser will open; log in, click Allow, and this cell finishes on its own." ] }, { "cell_type": "code", "execution_count": null, "id": "2ee73bf9", "metadata": {}, "outputs": [], "source": [ "import importlib.util\n", "import os\n", "import shutil\n", "import subprocess\n", "import sys\n", "import sysconfig\n", "\n", "print(\"Using Python:\", sys.executable)\n", "if sys.version_info < (3, 11): # noqa: UP036 - this runs in the reader's Python, not the platform's\n", " raise RuntimeError(\n", " f\"This kernel is Python {sys.version.split()[0]}; the notebook needs 3.11 or newer. \"\n", " \"See step 0 to pick or create a newer Python, then select it as the kernel.\"\n", " )\n", "\n", "\n", "def have_module(name):\n", " return importlib.util.find_spec(name) is not None\n", "\n", "\n", "def install(*packages):\n", " # Some environments (e.g. a venv made by \"uv venv\") have no pip at all -\n", " # uv can still install into them directly, without needing pip present.\n", " if shutil.which(\"uv\"):\n", " subprocess.run(\n", " [\"uv\", \"pip\", \"install\", \"--python\", sys.executable, \"-q\", *packages], check=True\n", " )\n", " else:\n", " subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *packages], check=True)\n", "\n", "\n", "needed = [p for p in (\"requests\", \"pandas\") if not have_module(p)]\n", "if not shutil.which(\"globus\"):\n", " needed.append(\"globus-cli\")\n", "\n", "if needed:\n", " print(\"Installing:\", \", \".join(needed))\n", " try:\n", " install(*needed)\n", " except Exception as e:\n", " print(\"Automatic install failed:\", e)\n", " print(\"Please install these yourself, then restart this notebook's kernel:\")\n", " print(\" uv pip install\", *needed, \" (if you use uv)\")\n", " print(\" pip install\", *needed, \" (otherwise)\")\n", " raise\n", "\n", "# Make sure a freshly-installed 'globus' command is found by the ! commands below,\n", "# even if this environment's own script folder isn't already on PATH.\n", "scripts_dir = sysconfig.get_path(\"scripts\")\n", "if scripts_dir not in os.environ.get(\"PATH\", \"\").split(os.pathsep):\n", " os.environ[\"PATH\"] = scripts_dir + os.pathsep + os.environ.get(\"PATH\", \"\")\n", "\n", "!globus version" ] }, { "cell_type": "code", "execution_count": null, "id": "23b686f8", "metadata": { "jupyter": { "source_hidden": true } }, "outputs": [], "source": [ "# Optional, presenters only: set True and run to sign out, then run the next cell to show the sign-in.\n", "SIGN_OUT_FIRST = False\n", "\n", "if SIGN_OUT_FIRST:\n", " subprocess.run([\"globus\", \"logout\", \"--yes\"], check=False)\n", " print(\"Signed out. Run the next cell to sign in again.\")\n", "else:\n", " print(\"Skipped - you stay signed in.\")" ] }, { "cell_type": "code", "execution_count": null, "id": "94e03405", "metadata": {}, "outputs": [], "source": [ "already_in = subprocess.run([\"globus\", \"whoami\"], capture_output=True).returncode == 0\n", "if already_in:\n", " print(\"Already signed in:\")\n", " !globus whoami\n", "else:\n", " print(\"Opening your browser to sign in...\")\n", " !globus login" ] }, { "cell_type": "markdown", "id": "c1de97bf", "metadata": {}, "source": [ "## 2. Browse the catalog\n", "Ask the API what's in the catalog - five results at a time." ] }, { "cell_type": "code", "execution_count": null, "id": "1ff5aa2b", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import requests\n", "\n", "pd.set_option(\"display.max_colwidth\", 60)\n", "\n", "\n", "def browse(**params):\n", " resp = requests.get(f\"{API}/browse\", params=params)\n", " resp.raise_for_status()\n", " data = resp.json()\n", " total = data.get(\"total\", len(data.get(\"files\", [])))\n", " print(f\"{params or '(no filter)'} -> {total} matches\")\n", " return pd.DataFrame(data.get(\"files\", []))[[\"esra_id\", \"type\", \"pi\", \"project\"]]\n", "\n", "\n", "browse(limit=5)" ] }, { "cell_type": "markdown", "id": "46b24815", "metadata": {}, "source": [ "## 3. Filter by technique and material\n", "Narrow it down - by technique, by material, then both together." ] }, { "cell_type": "code", "execution_count": null, "id": "d0ef7299", "metadata": {}, "outputs": [], "source": [ "from IPython.display import display\n", "\n", "display(browse(type=TECHNIQUE, limit=5))\n", "display(browse(material=MATERIAL, limit=5))\n", "display(browse(material=MATERIAL, type=TECHNIQUE, limit=5))" ] }, { "cell_type": "markdown", "id": "0beeb292", "metadata": {}, "source": [ "## 4. Find where the actual data files live\n", "The platform's record for one dataset (`DATASET`) names a storage location on Globus." ] }, { "cell_type": "code", "execution_count": null, "id": "8d30d94d", "metadata": {}, "outputs": [], "source": [ "from urllib.parse import parse_qs, urlparse\n", "\n", "resp = requests.get(f\"{API}/datasets/{DATASET}/jsonld\")\n", "resp.raise_for_status()\n", "record = resp.json()\n", "\n", "dist = record[\"distribution\"]\n", "if isinstance(dist, list):\n", " dist = dist[0]\n", "\n", "query = parse_qs(urlparse(dist[\"contentUrl\"]).query)\n", "origin = query[\"origin_path\"][0]\n", "PATH = origin[len(PREFIX) :] if origin.startswith(PREFIX) else origin\n", "print(\"Files are stored at:\", PATH)" ] }, { "cell_type": "markdown", "id": "9440b6e7", "metadata": {}, "source": [ "## 5. Peek inside before downloading anything\n", "Count the files and total size at the source." ] }, { "cell_type": "code", "execution_count": null, "id": "401d3129", "metadata": {}, "outputs": [], "source": [ "import json\n", "\n", "listing = subprocess.run(\n", " [\n", " \"globus\",\n", " \"ls\",\n", " f\"{GUEST}:{PATH}\",\n", " \"--recursive\",\n", " \"--recursive-depth-limit\",\n", " \"20\",\n", " \"--format\",\n", " \"JSON\",\n", " ],\n", " capture_output=True,\n", " text=True,\n", ")\n", "items = json.loads(listing.stdout).get(\"DATA\", [])\n", "files = [i for i in items if i.get(\"type\") == \"file\"]\n", "size_mb = sum(int(i.get(\"size\") or 0) for i in files) / (1024 * 1024)\n", "print(f\"Files: {len(files)} Total size: {size_mb:.1f} MB\")\n", "\n", "# (using Python to show just the first few lines, instead of the Unix-only 'head' command,\n", "# so this works the same on Windows, Mac, and Linux)\n", "preview = subprocess.run([\"globus\", \"ls\", f\"{GUEST}:{PATH}\"], capture_output=True, text=True)\n", "print(\"\\n\".join(preview.stdout.splitlines()[:5]))" ] }, { "cell_type": "markdown", "id": "d5bc15db", "metadata": {}, "source": [ "## 6. Download the files\n", "Find your own computer's destination, then start the transfer. Globus's own servers move the files - this notebook just asks for it." ] }, { "cell_type": "code", "execution_count": null, "id": "406af052", "metadata": {}, "outputs": [], "source": [ "DEST = !globus endpoint local-id\n", "DEST = DEST[0].strip() if DEST else \"\"\n", "if not DEST:\n", " raise RuntimeError(\n", " \"No Globus Connect Personal collection found on this computer. \"\n", " \"Install and set it up once: https://www.globus.org/globus-connect-personal\"\n", " )\n", "print(\"Destination:\", DEST)\n", "\n", "\n", "def globus_ls(path):\n", " # The raw listing also reports which folder the path really resolved to (\"absolute_path\")\n", " out = subprocess.run(\n", " [\n", " \"globus\",\n", " \"api\",\n", " \"transfer\",\n", " \"GET\",\n", " f\"/operation/endpoint/{DEST}/ls\",\n", " \"-Q\",\n", " f\"path={path}\",\n", " ],\n", " capture_output=True,\n", " text=True,\n", " )\n", " return json.loads(out.stdout) if out.returncode == 0 else None\n", "\n", "\n", "# On Windows, Globus Connect Personal's \"/~/\" is your Documents folder, so \"/~/Downloads\"\n", "# would quietly land in Documents\\Downloads. Use the real Downloads folder when it's reachable.\n", "if DOWNLOAD_BASE == \"/~/Downloads\":\n", " home = (globus_ls(\"/~/\") or {}).get(\"absolute_path\", \"\").rstrip(\"/\")\n", " if home.lower().endswith(\"/documents\"):\n", " downloads = home.rsplit(\"/\", 1)[0] + \"/Downloads\"\n", " if globus_ls(downloads) is not None:\n", " DOWNLOAD_BASE = downloads\n", "print(\"Files will go to:\", f\"{DOWNLOAD_BASE}/{DATASET}/\")\n", "\n", "started = subprocess.run(\n", " [\n", " \"globus\",\n", " \"transfer\",\n", " f\"{GUEST}:{PATH}\",\n", " f\"{DEST}:{DOWNLOAD_BASE}/{DATASET}/\",\n", " \"--recursive\",\n", " \"--label\",\n", " \"ESRA notebook demo\",\n", " \"--jmespath\",\n", " \"task_id\",\n", " \"--format\",\n", " \"UNIX\",\n", " ],\n", " capture_output=True,\n", " text=True,\n", ")\n", "if started.returncode != 0:\n", " raise RuntimeError(\"Globus did not start the transfer:\\n\" + started.stderr)\n", "TASK = started.stdout.strip()\n", "print(\"Started task:\", TASK)" ] }, { "cell_type": "markdown", "id": "25864559", "metadata": {}, "source": [ "## 7. Watch it arrive\n", "Globus copies the files in the background - it would carry on even if you closed this notebook. This cell just **watches**: it counts the files as they land in your download folder and fills the bar, with a clock showing how long the transfer has been running. The first few seconds are Globus setting up; then the files arrive one after another.\n", "\n", "Watching doesn't slow the download - the files go straight from NERSC to your computer, not through this notebook. For big datasets it checks less often, so it stays light; stop it any time with **Interrupt** and the transfer carries on.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "384cb07b", "metadata": {}, "outputs": [], "source": [ "import time\n", "from datetime import UTC, datetime\n", "\n", "from IPython.display import Pretty, display\n", "\n", "\n", "def local_path(globus_path):\n", " \"\"\"Where a Globus path on this computer really is (Globus writes Windows' C:\\\\ as /C/).\n", " The folder may not exist yet, so find the nearest one that does and add the rest.\"\"\"\n", " path, missing = globus_path.rstrip(\"/\"), []\n", " while path:\n", " found = globus_ls(path + \"/\")\n", " real = (found or {}).get(\"absolute_path\", \"\").rstrip(\"/\")\n", " if real:\n", " if sys.platform == \"win32\" and real[:1] == \"/\" and real[2:3] in (\"/\", \"\"):\n", " real = real[1] + \":\\\\\" + real[3:].replace(\"/\", \"\\\\\")\n", " return os.path.join(real, *reversed(missing))\n", " path, _, last = path.rpartition(\"/\")\n", " missing.append(last)\n", " return None\n", "\n", "\n", "def task_info():\n", " shown = subprocess.run(\n", " [\"globus\", \"task\", \"show\", TASK, \"--format\", \"JSON\"], capture_output=True, text=True\n", " )\n", " return json.loads(shown.stdout) if shown.returncode == 0 else {}\n", "\n", "\n", "def size_text(mb):\n", " return f\"{mb / 1024:.1f} GB\" if mb >= 1024 else f\"{mb:.1f} MB\"\n", "\n", "\n", "def clock_text(secs):\n", " hours, rest = divmod(int(secs), 3600)\n", " return f\"{hours}:{rest // 60:02d}:{rest % 60:02d}\" if hours else f\"{rest // 60}:{rest % 60:02d}\"\n", "\n", "\n", "def bar(done, total, width=24):\n", " filled = min(width, round(width * done / total)) if total else 0\n", " return \"█\" * filled + \"░\" * (width - filled)\n", "\n", "\n", "def arrived_locally(folder, since=0.0):\n", " \"\"\"Files in the folder written since the transfer started (older copies don't count).\"\"\"\n", " count = size = 0\n", " for root, _, names in os.walk(folder or \"\"):\n", " for name in names:\n", " try:\n", " info = os.stat(os.path.join(root, name))\n", " except OSError:\n", " continue # a file still being written\n", " if info.st_mtime >= since:\n", " count += 1\n", " size += info.st_size\n", " return count, size / (1024 * 1024)\n", "\n", "\n", "base = local_path(DOWNLOAD_BASE)\n", "folder = os.path.join(base, DATASET) if base else None\n", "total_files, total_mb = len(files), size_mb # from step 5\n", "task = task_info()\n", "started = datetime.fromisoformat(task[\"request_time\"]) if task.get(\"request_time\") else None\n", "since = started.timestamp() - 5 if started else 0.0\n", "\n", "# Stay light on big datasets: count the folder less often when it holds many files\n", "# (twice a second for a few hundred, every 10 s for 10,000+), and ask Globus every\n", "# 3 s for the first minute, then every 10 s.\n", "scan_every = min(10.0, max(0.5, total_files / 1000))\n", "count, mb, last_scan, last_check = 0, 0.0, 0.0, time.time()\n", "progress = display(Pretty(\"Asking Globus...\"), display_id=True)\n", "\n", "try:\n", " while task.get(\"status\", \"ACTIVE\") == \"ACTIVE\":\n", " now = time.time()\n", " elapsed = (datetime.now(UTC) - started).total_seconds() if started else 0\n", " if now - last_check >= (3 if elapsed < 60 else 10):\n", " task, last_check = task_info() or task, now\n", " if now - last_scan >= scan_every:\n", " (count, mb), last_scan = arrived_locally(folder, since), now\n", " if count:\n", " line = f\"{bar(count, total_files)} {count} of {total_files} files · {size_text(mb)} of {size_text(total_mb)}\"\n", " else:\n", " line = f\"{bar(0, total_files)} Globus is setting up the transfer\"\n", " progress.update(Pretty(f\"{line} · {clock_text(elapsed)}\"))\n", " time.sleep(0.5)\n", "except KeyboardInterrupt:\n", " print(\n", " f\"Stopped watching - the transfer carries on. Run this cell again, or: globus task show {TASK}\"\n", " )\n", "\n", "status = task.get(\"status\")\n", "if status == \"SUCCEEDED\":\n", " count, mb = arrived_locally(folder, since)\n", " count = count or task.get(\"files_transferred\") or 0\n", " secs = (datetime.fromisoformat(task[\"completion_time\"]) - started).total_seconds()\n", " progress.update(Pretty(f\"{bar(1, 1)} {count} of {total_files} files · {clock_text(secs)}\"))\n", " print(f\"Done - {count} files ({size_text(total_mb)}) in {clock_text(secs)}.\")\n", "elif status == \"FAILED\":\n", " print(\n", " \"Globus reports the transfer failed:\",\n", " task.get(\"nice_status\") or \"run: globus task show \" + TASK,\n", " )\n", "elif status == \"INACTIVE\":\n", " print(\n", " \"Globus paused the transfer - usually Globus Connect Personal isn't running. Start it; the transfer resumes.\"\n", " )" ] }, { "cell_type": "markdown", "id": "55fcf13b", "metadata": {}, "source": [ "## 8. Find your files\n", "Where on **this computer** the files were saved, and how many arrived. Open that folder in Finder (macOS) or File Explorer (Windows) to use them - they are the original instrument files, exactly as the researchers uploaded them.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "15b1cf73", "metadata": {}, "outputs": [], "source": [ "task = task_info() # is Globus still copying? (e.g. step 7 was stopped early)\n", "if task.get(\"status\") == \"ACTIVE\":\n", " count, mb = arrived_locally(folder, since)\n", " print(\n", " f\"Still arriving: {count} of {total_files} files so far - run this cell again in a moment.\"\n", " )\n", "elif task.get(\"status\") == \"INACTIVE\":\n", " print(\n", " \"Globus paused the transfer - usually Globus Connect Personal isn't running. Start it; the transfer resumes.\"\n", " )\n", "elif folder and os.path.isdir(folder):\n", " count, mb = arrived_locally(folder)\n", " print(f\"Saved on this computer in:\\n {folder}\\n\")\n", " print(f\"{count} files, {size_text(mb)}. First few:\")\n", " print(\"\\n\".join(sorted(os.listdir(folder))[:10]))\n", "else:\n", " print(\"The folder isn't there yet - wait for step 7 to finish, then run this cell again.\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "pygments_lexer": "ipython3" } }, "nbformat": 4, "nbformat_minor": 5 }