diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml new file mode 100644 index 0000000..d0cf4a2 --- /dev/null +++ b/.github/workflows/publish.yml @@ -0,0 +1,51 @@ +name: Publish to GitHub Pages + +on: + push: + branches: + - main + pull_request: + branches: + - main + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: "pages" + cancel-in-progress: false + +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Setup Quarto + uses: quarto-dev/quarto-actions/setup@v2 + with: + version: pre-release + + - name: Render Quarto Project + run: quarto render + + - name: Upload artifact + uses: actions/upload-pages-artifact@v3 + with: + path: _site + + deploy: + if: github.event_name != 'pull_request' + needs: build + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v4 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..96db9cc --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +/_site/ +/.quarto/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..8fa127a --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,155 @@ +# Contributing to Format Landscape + +Thank you for your interest in contributing to the Format Landscape project! This guide will help you understand how to add or update content. + +## Content Organization + +The repository contains Quarto Markdown (`.qmd`) files that are automatically converted to HTML and published to GitHub Pages. + +### Main Pages + +- `index.qmd` - Landing page with overview and navigation +- `serialization-formats.qmd` - General overview of common serialization formats +- `*-physics.qmd` - Domain-specific pages for each physics area + +### Supporting Files + +- `_quarto.yml` - Quarto configuration (navigation, theme, settings) +- `styles.css` - Custom CSS styling +- `.github/workflows/publish.yml` - CI/CD automation +- `README.md` - Repository documentation + +## How to Contribute + +### Adding a New Format or Resource + +1. **Identify the relevant page**: Determine which physics domain or the general serialization page is most appropriate + +2. **Edit the `.qmd` file**: Add your content following the existing structure: + - Use appropriate markdown headers (`##`, `###`) + - Include links to official websites/repositories + - Provide brief, clear descriptions + - Mention key features and use cases + +3. **Follow the existing format**: + ```markdown + ### Format Name + - **Website**: [URL](URL) + - **Description**: Brief description + - **Use Cases**: Where it's used + - **Key Features**: Notable characteristics + ``` + +4. **Test your changes locally** (optional but recommended): + ```bash + quarto preview + ``` + +5. **Submit a pull request**: Once you're satisfied with your changes + +### Adding a New Physics Domain + +If you need to add an entirely new physics domain: + +1. **Create a new `.qmd` file**: e.g., `quantum-computing.qmd` + +2. **Follow the template** from existing domain pages: + - Front matter with title + - Introduction section + - Organized sections for different types of formats/tools + - Related Resources section + - External Links section + +3. **Update `_quarto.yml`**: Add your new page to the navigation menu + +4. **Update `index.qmd`**: Add a link to your new page + +5. **Cross-reference**: Add links from related pages + +## Content Guidelines + +### Quality Standards + +- **Accuracy**: Ensure all information is correct and up-to-date +- **Relevance**: Focus on formats and DSLs actually used in the physics community +- **Clarity**: Write clearly and concisely +- **Links**: Provide working links to official sources when possible +- **Neutral tone**: Maintain an objective, educational tone + +### Markdown Style + +- Use proper markdown headers (`#`, `##`, `###`) +- Use bullet points for lists +- Use **bold** for emphasis on key terms +- Use `code blocks` for technical terms +- Include links using `[text](URL)` format + +### Structure + +Each domain page should include: + +1. **Introduction**: Brief overview of the domain and its data needs +2. **Main sections**: Organized by format type or use case +3. **Best Practices**: Guidelines for using formats in the domain +4. **Related Resources**: Links to other relevant pages +5. **External Links**: Links to organizations, standards bodies, etc. + +## Quarto Markdown Features + +You can use Quarto-specific features: + +- **Code blocks** with syntax highlighting +- **Callout blocks** for notes/warnings +- **Cross-references** between pages +- **Tables** and **figures** + +See [Quarto documentation](https://quarto.org/docs/authoring/markdown-basics.html) for more details. + +## Local Development + +### Prerequisites + +1. Install Quarto: https://quarto.org/docs/get-started/ + +### Building Locally + +```bash +# Preview (with live reload) +quarto preview + +# Render to HTML +quarto render +``` + +The rendered site will be in the `_site` directory. + +### Checking Your Changes + +Before submitting a PR: + +1. Preview your changes locally +2. Check that all links work +3. Verify cross-references to other pages +4. Ensure the navigation works correctly + +## Automated Checks + +When you submit a pull request: + +- GitHub Actions will automatically build the site +- You can preview the build in the Actions tab +- Any build errors will be reported + +## Questions? + +If you have questions about contributing, please: + +1. Check existing issues for similar questions +2. Open a new issue with your question +3. Tag it appropriately (e.g., `question`, `documentation`) + +## License and Attribution + +By contributing, you agree that your contributions will be part of the open educational resource that is Format Landscape. + +Thank you for helping make scientific data formats more accessible! diff --git a/README.md b/README.md index 8cc5c95..a391f41 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,61 @@ # Format-Landscape -This public repository gathers links and resources relevant for the project + +This repository gathers links and resources on **serialization formats** and **domain-specific languages (DSLs)** used in various physics domains. + +## About + +Format Landscape is a collection of Markdown documents that provides an organized overview of data formats, file formats, and domain-specific languages used across different areas of physics research: + +- **Lattice Physics** - Lattice QCD and related simulations +- **Hadron Physics** - Hadron structure and interactions +- **Nuclear Physics** - Nuclear structure and reactions +- **Astrophysics** - Astronomical observations and cosmology +- **Laser Physics** - Laser-plasma interactions and ultrafast phenomena +- **Particle Physics** - High-energy particle physics experiments +- **Neutrino Physics** - Neutrino experiments and simulations + +## Website + +The website is automatically built from the Markdown source files using [Quarto](https://quarto.org/) and deployed to GitHub Pages. + +Visit the live site at: **https://democratizing-models-project.github.io/Format-Landscape/** + +## Local Development + +To build and preview the site locally: + +1. Install Quarto: https://quarto.org/docs/get-started/ +2. Clone this repository +3. Run `quarto preview` in the repository root +4. Open your browser to the provided local URL + +To render the site: +```bash +quarto render +``` + +The output will be in the `_site` directory. + +## Contributing + +Contributions are welcome! To add information about a format or correct existing content: + +1. Fork this repository +2. Edit the relevant `.qmd` files +3. Submit a pull request + +## Structure + +- `index.qmd` - Main landing page +- `serialization-formats.qmd` - Overview of common serialization formats +- `*-physics.qmd` - Domain-specific pages for each physics area +- `_quarto.yml` - Quarto configuration +- `.github/workflows/publish.yml` - CI/CD workflow for automatic deployment + +## License + +This is an open educational resource. Content is provided as-is for the benefit of the scientific community. + +## Part of + +This project is part of the **Democratizing Models Project**, aimed at making scientific software and data more accessible and interoperable. diff --git a/_quarto.yml b/_quarto.yml new file mode 100644 index 0000000..60e1220 --- /dev/null +++ b/_quarto.yml @@ -0,0 +1,41 @@ +project: + type: website + output-dir: _site + +website: + title: "Format Landscape" + navbar: + left: + - href: index.qmd + text: Home + - href: serialization-formats.qmd + text: Serialization Formats + - text: Physics Domains + menu: + - href: lattice-physics.qmd + text: Lattice Physics + - href: hadron-physics.qmd + text: Hadron Physics + - href: nuclear-physics.qmd + text: Nuclear Physics + - href: astro-physics.qmd + text: Astro Physics + - href: laser-physics.qmd + text: Laser Physics + - href: particle-physics.qmd + text: Particle Physics + - href: neutrino-physics.qmd + text: Neutrino Physics + sidebar: false + page-footer: + left: "Format Landscape - A collection of serialization formats and DSLs for physics" + right: "Built with Quarto" + +format: + html: + theme: cosmo + css: styles.css + toc: true + toc-depth: 3 + code-copy: true + code-overflow: wrap diff --git a/astro-physics.qmd b/astro-physics.qmd new file mode 100644 index 0000000..85063f8 --- /dev/null +++ b/astro-physics.qmd @@ -0,0 +1,262 @@ +--- +title: "Astro Physics Formats" +--- + +# Astrophysics: Formats and DSLs + +Astrophysics and cosmology rely on diverse data formats for observations, simulations, and theoretical calculations. This page covers the landscape of formats used across astronomical and cosmological research. + +## Observational Data Formats + +### FITS (Flexible Image Transport System) +- **Website**: [https://fits.gsfc.nasa.gov/](https://fits.gsfc.nasa.gov/) +- **Description**: The primary standard for astronomical data +- **Format Type**: Binary with ASCII headers +- **Use Cases**: + - Telescope images + - Spectroscopic data + - Tables and catalogs + - Multi-dimensional data cubes +- **Key Features**: + - Self-describing headers + - World Coordinate System (WCS) support + - Binary tables and image extensions + - Universal astronomy standard + +### HDF5 in Astronomy +- **Use Cases**: + - Large survey data (LSST, SKA) + - Simulation outputs + - Instrument data +- **Tools**: h5py, pytables +- **Examples**: Some radio astronomy correlators, simulation codes + +### VOTable (Virtual Observatory Table) +- **Website**: [https://www.ivoa.net/documents/VOTable/](https://www.ivoa.net/documents/VOTable/) +- **Format**: XML-based table format +- **Use Cases**: + - Data exchange in Virtual Observatory + - Catalog queries + - Cross-matching services +- **Key Features**: Rich metadata, links to external resources + +## Simulation Data Formats + +### Gadget/Gadget-2/Gadget-4 +- **Description**: N-body/SPH cosmological simulation code +- **Format**: Binary snapshot files +- **Structure**: Header + particle blocks +- **Tools**: pynbody, yt, custom readers +- **Use Cases**: Dark matter simulations, galaxy formation + +### RAMSES +- **Website**: [https://www.ics.uzh.ch/~teyssier/ramses/RAMSES.html](https://www.ics.uzh.ch/~teyssier/ramses/RAMSES.html) +- **Description**: Adaptive mesh refinement code +- **Format**: Binary AMR files +- **Output**: Multiple files per snapshot +- **Tools**: yt, custom readers + +### Enzo +- **Website**: [https://enzo-project.org/](https://enzo-project.org/) +- **Description**: AMR cosmology simulation code +- **Format**: HDF5 +- **Tools**: yt (primary analysis tool) + +### AREPO +- **Description**: Moving-mesh cosmological code +- **Format**: HDF5 (compatible with Gadget) +- **Use Cases**: IllustrisTNG simulations + +### Rockstar Halo Catalogs +- **Repository**: [https://github.com/jwise77/rockstar](https://github.com/jwise77/rockstar) +- **Description**: Halo finder output format +- **Format**: ASCII and binary catalogs +- **Contents**: Halo properties, merger trees + +## Analysis Tools and Formats + +### Astropy +- **Website**: [https://www.astropy.org/](https://www.astropy.org/) +- **Description**: Core Python library for astronomy +- **Supported Formats**: + - FITS (primary) + - ASCII tables (various formats) + - VOTable + - HDF5 +- **Key Modules**: + - astropy.io.fits + - astropy.table + - astropy.wcs + +### yt (Analysis toolkit) +- **Website**: [https://yt-project.org/](https://yt-project.org/) +- **Description**: Analysis and visualization for volumetric data +- **Supported Formats**: + - Gadget, RAMSES, Enzo, AREPO + - Athena, FLASH + - Generic HDF5, FITS + +### TOPCAT (Tool for OPerations on Catalogues And Tables) +- **Website**: [http://www.star.bris.ac.uk/~mbt/topcat/](http://www.star.bris.ac.uk/~mbt/topcat/) +- **Description**: Interactive catalog analysis tool +- **Supported Formats**: FITS, VOTable, CSV, SQL databases + +## Catalog and Survey Formats + +### SDSS (Sloan Digital Sky Survey) +- **Formats**: + - FITS images and spectra + - FITS binary tables (catalogs) + - SQL database (CASJobs) +- **Access**: SciServer, SDSS data access tools + +### Gaia Archive +- **Format**: VOTable, FITS, CSV +- **Access**: TAP (Table Access Protocol) queries +- **Contents**: Astrometry, photometry for billions of stars + +### 2MASS, WISE, etc. +- Standard format: FITS tables +- Distributed via IPAC, VizieR, CDS + +## Coordinate and Reference Systems + +### WCS (World Coordinate System) +- **Standard**: FITS WCS keywords +- **Purpose**: Map pixel coordinates to sky coordinates +- **Implementations**: astropy.wcs, wcslib + +### SkyCoord Formats +- **Systems**: ICRS, FK5, Galactic, etc. +- **Serialization**: String formats (e.g., "12h34m56s +12d34m56s") +- **Tools**: astropy.coordinates + +## Spectroscopic Data + +### FITS Spectra +- 1D spectra: Binary tables or images +- Multi-object spectra: Extensions per object +- Data cubes: 3D FITS (wavelength + 2D spatial) + +### IRAF Formats +- .fits spectra with specific headers +- Legacy but still encountered +- Tools: pyraf, astropy compatibility + +### SpecViz/SpecUtils +- Modern spectroscopic analysis +- Input: FITS, ASCII, various observatory formats +- Output: FITS, ASCII + +## Radio Astronomy Specific + +### Measurement Sets (CASA) +- **Format**: Table-based (casa::Table) +- **Use Cases**: Radio interferometry visibility data +- **Tools**: CASA (Common Astronomy Software Applications) +- **Structure**: Multiple tables for visibilities, antenna info, etc. + +### UVFITS +- **Description**: FITS format for UV (visibility) data +- **Use Cases**: Interferometric data exchange +- **Compatibility**: Multiple radio astronomy packages + +### Miriad/AIPS Formats +- Legacy radio astronomy formats +- AIPS: Custom binary format +- Miriad: Directory-based data sets +- Conversion tools available to/from FITS + +## Gravitational Wave Data + +### LIGO/Virgo Data Formats +- **Primary**: HDF5 (strain data, events) +- **Access**: GWOSC (Gravitational Wave Open Science Center) +- **Tools**: gwpy, PyCBC, LALSuite +- **Contents**: Time series, spectrograms, event parameters + +### Frame Files +- **Description**: LIGO's time-series frame format +- **Format**: Custom binary +- **Tools**: lalframe, gwpy + +## Cosmological Parameter Files + +### CAMB/CLASS +- **Input**: INI files or Python dictionaries +- **Output**: ASCII tables of power spectra +- **Use Cases**: CMB and matter power spectrum calculations + +### CosmoMC +- **Input**: INI parameter files +- **Output**: Chain files (MCMC chains in ASCII) +- **Use Cases**: Cosmological parameter inference + +### Cobaya +- **Input**: YAML configuration files +- **Output**: HDF5 or text chains +- **Use Cases**: Modern cosmological sampling + +## Cosmological Simulation Catalogs + +### Halo Occupation Distribution (HOD) Files +- Typically ASCII or HDF5 +- Contents: Galaxy-halo connection parameters + +### Mock Catalogs +- **Formats**: FITS, HDF5, ASCII +- **Contents**: Galaxy positions, velocities, properties +- **Tools**: nbodykit, Corrfunc + +## Machine Learning in Astronomy + +### LSST Data Products +- **Format**: Primarily HDF5, some FITS +- **Butler**: Data access abstraction layer +- **Schema**: Well-defined data products + +### Neural Network Training Sets +- **Formats**: HDF5, TFRecord, PyTorch tensors +- **Examples**: Galaxy Zoo (FITS + labels), SDSS (cutouts) + +## Domain-Specific Languages + +### SAOImage DS9 Region Files +- **Format**: Text-based region specifications +- **Use Cases**: Defining analysis regions on images +- **Syntax**: Shapes (circle, box, polygon) with coordinates + +### Astronomical Query Languages +- **ADQL** (Astronomical Data Query Language): SQL-like for astronomy +- **TAP** (Table Access Protocol): Web service standard +- **VO Standards**: Various Virtual Observatory protocols + +### Pipeline Definition Languages +- **LSST**: Python-based pipeline tasks +- **Astropy Pipelines**: Python workflows +- **Common Pattern**: Python/YAML for pipeline specification + +## Best Practices + +1. **Use FITS** as the primary format for images and tables +2. **Include WCS** information in all image headers +3. **Document provenance** (processing steps, calibrations) +4. **Archive data** with comprehensive metadata +5. **Use Virtual Observatory standards** for data sharing +6. **Version control** reduction pipelines +7. **Use HDF5** for large simulation outputs +8. **Follow IVOA standards** for interoperability + +## Related Resources + +- [Serialization Formats Overview](serialization-formats.qmd) +- [Particle Physics Formats](particle-physics.qmd) +- [Neutrino Physics Formats](neutrino-physics.qmd) + +## External Links + +- [IVOA (International Virtual Observatory Alliance)](https://www.ivoa.net/) +- [AAS WorldWide Telescope](http://www.worldwidetelescope.org/) +- [FITS Support Office](https://fits.gsfc.nasa.gov/) +- [Astropy Documentation](https://docs.astropy.org/) +- [yt Project](https://yt-project.org/) diff --git a/hadron-physics.qmd b/hadron-physics.qmd new file mode 100644 index 0000000..daf4771 --- /dev/null +++ b/hadron-physics.qmd @@ -0,0 +1,183 @@ +--- +title: "Hadron Physics Formats" +--- + +# Hadron Physics: Formats and DSLs + +Hadron physics studies the structure and interactions of hadrons (particles made of quarks). This field bridges nuclear and particle physics, requiring specialized formats for experimental data, theoretical calculations, and simulations. + +## Experimental Data Formats + +### ROOT (CERN Data Analysis Framework) +- **Website**: [https://root.cern.ch/](https://root.cern.ch/) +- **Description**: Primary format for hadron physics experiments +- **Use Cases**: + - Event data from experiments (JLab, MAMI, ELSA, etc.) + - Histogram storage + - Analysis results +- **Key Features**: TTrees, histograms, fitting capabilities + +### HIPO (High Performance Output) +- **Repository**: [https://github.com/gavalian/hipo](https://github.com/gavalian/hipo) +- **Description**: Binary format developed for CLAS12 at Jefferson Lab +- **Use Cases**: Fast event data storage and streaming +- **Language Support**: C++, Java, Python +- **Key Features**: + - Schema-based binary format + - High-performance I/O + - Compact storage + +### CODA (CEBAF Online Data Acquisition) +- **Description**: Data acquisition format for Jefferson Lab experiments +- **Format Type**: Binary event format +- **Use Cases**: Raw experimental data from detectors +- **Tools**: ET system, CODA components + +## Theoretical Calculation Formats + +### Partial Wave Analysis Formats + +#### AmpTools +- **Repository**: [https://github.com/mashephe/AmpTools](https://github.com/mashephe/AmpTools) +- **Description**: Framework for amplitude analysis +- **Input Format**: Configuration files (plain text) +- **Data Format**: ROOT files for data and Monte Carlo +- **DSL**: Custom configuration syntax for defining amplitudes + +#### SAID (Scattering Analysis Interactive Dial-in) +- **Website**: [https://gwdac.phys.gwu.edu/](https://gwdac.phys.gwu.edu/) +- **Description**: Database and analysis tool for hadron scattering +- **Data Format**: Custom ASCII format for partial waves +- **Output**: Tables of partial wave amplitudes + +#### PWA (Partial Wave Analysis) formats +- Various experiments use custom formats +- Common elements: energy, angles, cross sections, asymmetries +- Often ASCII or CSV for exchange + +### Form Factor Data + +#### World databases +- Various groups maintain databases of form factor measurements +- Formats: ASCII tables, ROOT files, JSON +- Examples: + - Proton/neutron form factors + - Pion/kaon form factors + - Transition form factors + +## Monte Carlo Event Generators + +### PYTHIA Event Format +- **Website**: [https://www.pythia.org/](https://www.pythia.org/) +- **Description**: General-purpose event generator +- **Output Formats**: + - HepMC (event record standard) + - LHEF (Les Houches Event Format) + - Custom ASCII + +### HERWIG +- **Website**: [https://herwig.hepforge.org/](https://herwig.hepforge.org/) +- **Output Formats**: HepMC, LHEF +- **Use Cases**: Hadronization modeling + +### HepMC (High Energy Physics Monte Carlo) +- **Website**: [http://hepmc.web.cern.ch/](http://hepmc.web.cern.ch/) +- **Description**: Standard event record for Monte Carlo generators +- **Format**: C++ object structure, ASCII/binary I/O +- **Version**: HepMC3 is current standard + +## Parton Distribution Functions (PDFs) + +### LHAPDF (Les Houches Accord PDF) +- **Website**: [https://lhapdf.hepforge.org/](https://lhapdf.hepforge.org/) +- **Description**: Standard library for PDF access +- **Format**: Grid files (plain text) +- **Use Cases**: Providing PDF sets to Monte Carlo generators and calculations +- **PDF Sets**: CT, MMHT, NNPDF, etc. + +### PDF Grid Formats +- Plain text grid files +- Interpolation data for x, Q² dependence +- Metadata: PDF name, order, flavors + +## Geant4 Detector Simulation + +### GDML (Geometry Description Markup Language) +- **Website**: [https://gdml.web.cern.ch/GDML/](https://gdml.web.cern.ch/GDML/) +- **Format**: XML-based geometry description +- **Use Cases**: Detector geometry for hadron physics experiments +- **Key Features**: Materials, volumes, placement, visualization + +### Geant4 Output +- **Formats**: ROOT, HDF5, ASCII +- **Contents**: Particle tracks, energy deposits, hits +- **Integration**: Often feeds into reconstruction chains + +## Analysis Frameworks + +### JANA2 (JLab ANAlysis framework) +- **Repository**: [https://github.com/JeffersonLab/JANA2](https://github.com/JeffersonLab/JANA2) +- **Description**: Multi-threaded event processing framework +- **Input**: HIPO, EVIO formats +- **Output**: ROOT, custom formats + +### CLAS12 Analysis Tools +- Based on JANA2 and HIPO +- Configuration: YAML, CCDB (calibration database) +- Output: ROOT, HDF5 + +## Databases and Archives + +### Jefferson Lab Data Archive +- Stores experimental data from Hall A, B, C, D +- Formats: EVIO, HIPO, ROOT +- Access: Tape archives, online catalogs + +### Particle Data Group (PDG) +- **Website**: [https://pdg.lbl.gov/](https://pdg.lbl.gov/) +- **Description**: Repository of particle properties +- **Formats**: + - Online database + - ASCII data files + - Machine-readable formats + +### Durham HepData +- **Website**: [https://www.hepdata.net/](https://www.hepdata.net/) +- **Description**: Repository for experimental results +- **Format**: YAML-based data tables +- **Use Cases**: Publishing and archiving experimental measurements + +## Domain-Specific Languages + +### Reaction Definition Languages +- GlueX: Reaction filter syntax +- CLAS12: Particle combination specifications +- Used for event selection and filtering + +### Amplitude Specification +- AmpTools configuration language +- Defines: waves, resonances, couplings +- Example syntax for amplitude models + +## Best Practices + +1. **Use ROOT** for general experimental data +2. **Use HIPO** for high-rate experiments (if applicable) +3. **Document detector geometry** in GDML +4. **Archive raw data** with full metadata +5. **Use HepData** for publishing results +6. **Follow LHAPDF** standards for PDFs +7. **Use HepMC** for event generator output + +## Related Resources + +- [Particle Physics Formats](particle-physics.qmd) +- [Nuclear Physics Formats](nuclear-physics.qmd) +- [Lattice Physics Formats](lattice-physics.qmd) +- [Serialization Formats Overview](serialization-formats.qmd) + +## External Links + +- [Jefferson Lab Software](https://www.jlab.org/software) +- [GlueX Software](https://halld.jlab.org/software/) +- [CLAS12 Software](https://clasweb.jlab.org/wiki/index.php/CLAS12_Software) diff --git a/index.qmd b/index.qmd new file mode 100644 index 0000000..81880ee --- /dev/null +++ b/index.qmd @@ -0,0 +1,42 @@ +--- +title: "Format Landscape" +--- + +# Welcome to Format Landscape + +This repository gathers links and resources on **serialization formats** and **domain-specific languages (DSLs)** used across various physics domains. + +## Overview + +Modern physics research relies heavily on data exchange and communication between different software tools, simulations, and experiments. Understanding the landscape of serialization formats and domain-specific languages is crucial for: + +- **Data interoperability** between different tools and frameworks +- **Efficient data storage** and transmission +- **Reproducibility** of scientific results +- **Collaboration** across different research groups + +## Physics Domains Covered + +This collection organizes formats and DSLs by physics domain: + +- [**Lattice Physics**](lattice-physics.qmd) - Formats for lattice QCD and related simulations +- [**Hadron Physics**](hadron-physics.qmd) - Formats for hadron structure and interactions +- [**Nuclear Physics**](nuclear-physics.qmd) - Formats for nuclear structure and reactions +- [**Astro Physics**](astro-physics.qmd) - Formats for astrophysics and cosmology +- [**Laser Physics**](laser-physics.qmd) - Formats for laser-plasma interactions +- [**Particle Physics**](particle-physics.qmd) - Formats for high-energy particle physics +- [**Neutrino Physics**](neutrino-physics.qmd) - Formats for neutrino experiments + +## General Resources + +For a comprehensive overview of common serialization formats used across domains, see: + +- [**Serialization Formats**](serialization-formats.qmd) - Common data formats (HDF5, ROOT, JSON, XML, etc.) + +## Contributing + +This is an evolving collection. Contributions are welcome! Please submit issues or pull requests to add new formats, correct information, or suggest improvements. + +## About + +This project is part of the Democratizing Models Project, aimed at making scientific software and data more accessible and interoperable. diff --git a/laser-physics.qmd b/laser-physics.qmd new file mode 100644 index 0000000..55152ee --- /dev/null +++ b/laser-physics.qmd @@ -0,0 +1,280 @@ +--- +title: "Laser Physics Formats" +--- + +# Laser Physics: Formats and DSLs + +Laser physics, including laser-plasma interactions, high-field physics, and ultrafast phenomena, involves complex simulations and experimental diagnostics. This page covers formats and DSLs used in this domain. + +## Particle-in-Cell (PIC) Simulation Formats + +### EPOCH +- **Repository**: [https://github.com/Warwick-Plasma/epoch](https://github.com/Warwick-Plasma/epoch) +- **Description**: Extensible PIC Open Collaboration +- **Input Format**: Fortran-style deck files +- **Output Format**: SDF (Self-Describing Format) +- **Key Features**: + - Custom DSL for physics setup + - Laser pulse specifications + - Species definitions + - Boundary conditions + +### SDF (Self-Describing Format) +- **Description**: Binary format used by EPOCH +- **Features**: + - Self-describing metadata + - Efficient storage of fields and particles +- **Tools**: + - Python readers (sdf module) + - VisIt plugin + - Custom analysis scripts + +### WarpX +- **Repository**: [https://github.com/ECP-WarpX/WarpX](https://github.com/ECP-WarpX/WarpX) +- **Description**: Advanced electromagnetic PIC code +- **Input Format**: Text-based input files (AMReX style) +- **Output Format**: + - HDF5 (via openPMD) + - PlotFiles (AMReX format) +- **Key Features**: Adaptive mesh refinement + +### openPMD (Open Standard for Particle-Mesh Data) +- **Website**: [https://www.openpmd.org/](https://www.openpmd.org/) +- **Description**: Standard for PIC simulation output +- **Format**: HDF5 or ADIOS2 backend +- **Support**: WarpX, PIConGPU, FBPIC +- **Key Features**: + - Metadata conventions + - Particle and field data + - Time series support + - Interoperability between codes + +### PIConGPU +- **Repository**: [https://github.com/ComputationalRadiationPhysics/picongpu](https://github.com/ComputationalRadiationPhysics/picongpu) +- **Description**: GPU-accelerated PIC code +- **Input**: Configuration via C++ templates and .cfg files +- **Output**: openPMD (HDF5), ADIOS +- **DSL**: Template metaprogramming for physics modules + +### OSIRIS +- **Website**: [https://osiris-code.github.io/](https://osiris-code.github.io/) +- **Description**: PIC framework for plasma physics +- **Input Format**: Fortran namelist-style input +- **Output Format**: HDF5 +- **Visualization**: VisXD, custom tools + +### SMILEI +- **Website**: [https://smileipic.github.io/Smilei/](https://smileipic.github.io/Smilei/) +- **Description**: PIC code for plasma physics +- **Input Format**: Python-based input files +- **Output Format**: HDF5 +- **Key Features**: Python DSL for setup + +## Laser Pulse Representation + +### Laser Pulse Parameters +Common parameters stored in simulations: +- Wavelength (λ) +- Intensity (a₀ or I) +- Pulse duration (τ) +- Focal spot size (w₀) +- Polarization +- Temporal profile (Gaussian, flat-top, etc.) +- Spatial profile (Gaussian, super-Gaussian, etc.) + +### Complex Pulse Formats +- **Files**: ASCII, HDF5, or custom binary +- **Contents**: + - Spatiotemporal field profiles + - Phase information + - Spectral content +- **Tools**: Laser design software, FFT analysis + +## Ray Tracing and Propagation + +### Zemax +- **Description**: Commercial optical design software +- **Format**: Proprietary .zmx files +- **Export**: Various formats (ASCII, MATLAB) +- **Use Cases**: Laser beam line design + +### FRED (Photon Engineering) +- **Description**: Optical engineering software +- **Format**: Proprietary +- **Use Cases**: Ray tracing, illumination + +### BeamProp/RSoft +- **Description**: Beam propagation software +- **Format**: Proprietary input/output +- **Use Cases**: Waveguide design, laser propagation + +## Diagnostics and Experimental Data + +### Oscilloscope Data +- **Formats**: + - CSV (common export) + - Binary (manufacturer-specific: Tektronix, LeCroy, etc.) + - HDF5 (some modern scopes) +- **Tools**: PyVISA, manufacturer SDKs + +### Camera/Detector Data +- **Formats**: + - TIFF (16-bit, 32-bit for intensity) + - HDF5 for scientific cameras + - Manufacturer formats (e.g., .spe for Princeton Instruments) +- **Metadata**: Exposure time, gain, timestamp + +### Spectrometer Data +- **Formats**: ASCII, CSV, manufacturer binary +- **Contents**: Wavelength vs. intensity +- **Calibration**: Wavelength calibration data + +### Particle Diagnostics +- **Thomson Parabola**: Image files + analysis +- **Particle Spectrometers**: Energy vs. count data +- **Format**: Often custom ASCII or ROOT + +## Laser Amplifier Modeling + +### MIRO/MIRÓ +- **Description**: Laser amplifier modeling +- **Format**: Custom input files +- **Use Cases**: Amplifier chain design + +### Laser Pulse Modeling Tools +- Various in-house codes +- Input: Pulse parameters, material properties +- Output: Amplified pulse characteristics + +## Plasma Diagnostics Formats + +### Interferometry/Shadowgraphy +- **Data**: Image sequences (TIFF, HDF5) +- **Analysis**: Phase maps, density profiles +- **Output**: Density maps (2D arrays, often HDF5) + +### X-ray Spectroscopy +- **Format**: Similar to general spectroscopy +- **Special**: Energy calibration for X-ray range + +### Proton Radiography +- **Data**: Image stacks +- **Analysis**: Field reconstruction algorithms +- **Output**: E and B field maps + +## Theoretical Calculation Formats + +### Atomic Physics Calculations +- **FLYCHK**: Plasma atomic physics + - Input: ASCII parameter files + - Output: ASCII tables +- **PrismSPECT**: Spectroscopic modeling + - Proprietary formats + +### QED Cascade Simulations +- **Formats**: Often custom for each code +- **Outputs**: Particle phase space, photon spectra +- **Common**: HDF5 for large datasets + +## High-Performance Computing Output + +### ADIOS2 +- **Website**: [https://adios2.readthedocs.io/](https://adios2.readthedocs.io/) +- **Description**: Adaptable I/O System +- **Use Cases**: + - Large-scale parallel I/O + - Streaming data + - In-situ visualization +- **Backend**: Can write to HDF5, binary, etc. +- **Used by**: WarpX, other HPC codes + +### Parallel I/O Patterns +- **MPI-IO**: Low-level parallel file access +- **HDF5 Parallel**: HDF5 with MPI +- **Libraries**: ADIOS2, PnetCDF + +## Visualization Tools and Formats + +### VisIt +- **Website**: [https://visit-dav.github.io/visit-website/](https://visit-dav.github.io/visit-website/) +- **Supported Formats**: + - HDF5, NetCDF + - SDF (EPOCH) + - openPMD + - VTK +- **Use Cases**: Large-scale 3D visualization + +### ParaView +- **Website**: [https://www.paraview.org/](https://www.paraview.org/) +- **Supported Formats**: + - VTK, HDF5, XDMF + - Custom readers via plugins +- **Use Cases**: Interactive 3D visualization + +### VTK (Visualization Toolkit) +- **Format**: XML-based or legacy ASCII/binary +- **Use Cases**: Structured/unstructured grid visualization +- **Converters**: Available for many simulation formats + +## Domain-Specific Languages + +### EPOCH Input Deck Language +``` +begin:laser + boundary = x_min + intensity_w_cm2 = 1.0e18 + lambda = 0.8 * micron + profile = gauss(y,0,4*micron) + ... +end:laser +``` + +### WarpX Input File Syntax +``` +max_step = 1000 +amr.n_cell = 128 128 256 +particles.species_names = electrons ions +electrons.charge = -q_e +... +``` + +### Python-based DSLs +- SMILEI uses Python for input +- Allows programmatic setup +- Access to full Python capabilities + +## Analysis and Post-Processing + +### Custom Python/MATLAB Scripts +- Read HDF5, SDF, or other formats +- Analysis pipelines +- Common libraries: h5py, numpy, scipy, matplotlib + +### Jupyter Notebooks +- Interactive analysis +- Documentation + code + results +- Reproducible workflows + +## Best Practices + +1. **Use openPMD** for PIC simulation output when possible +2. **Store metadata** comprehensively (laser parameters, plasma conditions) +3. **Use HDF5** for large datasets +4. **Version control** input files and analysis scripts +5. **Document** simulation parameters thoroughly +6. **Archive raw data** before processing +7. **Use standard visualization formats** (VTK, HDF5) for sharing +8. **Include units** explicitly in data files + +## Related Resources + +- [Particle Physics Formats](particle-physics.qmd) +- [Serialization Formats Overview](serialization-formats.qmd) + +## External Links + +- [openPMD Standard](https://github.com/openPMD/openPMD-standard) +- [EPOCH Code](https://cfsa-pmw.warwick.ac.uk/EPOCH/epoch) +- [WarpX Documentation](https://warpx.readthedocs.io/) +- [PIConGPU Documentation](https://picongpu.readthedocs.io/) +- [OSIRIS Framework](https://github.com/osiris-code/osiris) diff --git a/lattice-physics.qmd b/lattice-physics.qmd new file mode 100644 index 0000000..d04f090 --- /dev/null +++ b/lattice-physics.qmd @@ -0,0 +1,145 @@ +--- +title: "Lattice Physics Formats" +--- + +# Lattice Physics: Formats and DSLs + +Lattice physics, particularly lattice QCD (Quantum Chromodynamics), involves discretizing space-time on a lattice for numerical simulations. This domain has developed specialized formats for gauge configurations, propagators, and analysis results. + +## Data Formats + +### ILDG (International Lattice Data Grid) Format +- **Website**: [https://www.usqcd.org/ildg/](https://www.usqcd.org/ildg/) +- **Description**: Standard format for storing lattice QCD gauge configurations +- **Format Type**: Binary (often based on LIME) +- **Use Cases**: Exchange of gauge configurations between collaborations +- **Key Features**: + - Metadata standards for lattice parameters + - Checksum verification + - Platform-independent binary format + +### LIME (Lattice QCD Interchange Message Encapsulation) +- **Repository**: [https://github.com/usqcd-software/lime](https://github.com/usqcd-software/lime) +- **Description**: Message passing protocol for lattice QCD +- **Format Type**: Binary with structured records +- **Use Cases**: Wrapping gauge configurations and propagators +- **Key Features**: + - Self-describing binary format + - Support for large files + - Metadata embedding + +### QDP/C and QDP++ Formats +- **QDP/C**: [https://usqcd-software.github.io/qdpxx/](https://usqcd-software.github.io/qdpxx/) +- **Description**: Level 3 data-parallel API for lattice QCD +- **Use Cases**: Portable lattice QCD code +- **I/O**: Typically uses LIME or custom binary formats + +### SciDAC QIO (QCD I/O) +- **Repository**: Part of SciDAC software suite +- **Description**: Parallel I/O library for lattice QCD +- **Format Type**: Binary, supports LIME +- **Key Features**: + - Parallel file I/O + - Checkpointing support + - Checksums and metadata + +## Software Frameworks and DSLs + +### Chroma +- **Website**: [https://jeffersonlab.github.io/chroma/](https://jeffersonlab.github.io/chroma/) +- **Description**: Lattice QCD application built on QDP++ +- **Input Format**: XML configuration files +- **Data Format**: Uses QIO/LIME for gauge configurations +- **DSL**: XML-based parameter specification + +### Grid +- **Repository**: [https://github.com/paboyle/Grid](https://github.com/paboyle/Grid) +- **Description**: Data parallel C++ library for lattice field theory +- **Input Format**: XML, JSON, or command-line parameters +- **Data Format**: Custom binary formats, HDF5 support +- **Key Features**: Performance-portable across different architectures + +### tmLQCD +- **Repository**: [https://github.com/etmc/tmLQCD](https://github.com/etmc/tmLQCD) +- **Description**: Twisted mass lattice QCD simulation suite +- **Input Format**: Custom parameter files +- **Data Format**: LIME format for configurations + +### HiRep +- **Repository**: [https://github.com/claudiopica/HiRep](https://github.com/claudiopica/HiRep) +- **Description**: Lattice gauge theory simulations with higher representations +- **Input Format**: Custom input files +- **Data Format**: Custom binary format + +## Analysis Tools and Formats + +### HDF5 for Lattice Data +- Many modern lattice QCD projects are adopting HDF5 for: + - Correlator data + - Analysis results + - Ensemble metadata +- **Advantages**: Self-describing, hierarchical, widely supported + +### LatAnalyze +- **Repository**: [https://github.com/aportelli/LatAnalyze](https://github.com/aportelli/LatAnalyze) +- **Description**: C++ library for lattice QCD data analysis +- **Input Formats**: ASCII, HDF5, XML +- **Output Formats**: HDF5, ASCII, plots + +### hadron +- **Repository**: [https://github.com/HISKP-LQCD/hadron](https://github.com/HISKP-LQCD/hadron) +- **Description**: R package for lattice QCD analysis +- **Input Formats**: Custom binary, ASCII +- **Use Cases**: Statistical analysis of correlation functions + +## Configuration Metadata Standards + +### Ensemble Metadata +Key metadata typically stored with lattice configurations: +- Lattice dimensions (Nt, Nx, Ny, Nz) +- Gauge action parameters (β, c_sw, etc.) +- Fermion action parameters (κ, masses, etc.) +- Trajectory number +- Plaquette values +- Polyakov loops +- Checksums + +### Naming Conventions +- Different collaborations use different naming schemes +- ILDG promotes standardized metadata +- Typically includes: action, volume, parameters, trajectory + +## Domain-Specific Languages + +### QDP Expression Templates +- C++ expression templates for lattice operations +- Provides domain-specific notation for lattice field operations +- Example: `u[mu] * psi` for gauge field times fermion field + +### Chroma XML +- Declarative specification of: + - Propagator calculations + - Measurement programs + - Analysis workflows +- Allows complex calculations to be specified without coding + +## Best Practices + +1. **Use ILDG/LIME** for gauge configuration exchange +2. **Store metadata** comprehensively with all configurations +3. **Verify checksums** after file transfers +4. **Document ensemble parameters** thoroughly +5. **Use HDF5** for analysis data and correlators +6. **Archive configurations** with full provenance information + +## Related Resources + +- [Serialization Formats Overview](serialization-formats.qmd) +- [Particle Physics Formats](particle-physics.qmd) +- [Nuclear Physics Formats](nuclear-physics.qmd) + +## External Links + +- [USQCD Software](https://www.usqcd.org/software.html) +- [ILDG Working Group](https://www.usqcd.org/ildg/) +- [SciDAC Software](https://www.usqcd.org/scidac.html) diff --git a/neutrino-physics.qmd b/neutrino-physics.qmd new file mode 100644 index 0000000..fbab181 --- /dev/null +++ b/neutrino-physics.qmd @@ -0,0 +1,355 @@ +--- +title: "Neutrino Physics Formats" +--- + +# Neutrino Physics: Formats and DSLs + +Neutrino physics experiments and simulations span a wide range of energy scales and detector technologies, from reactor neutrinos to high-energy atmospheric and accelerator neutrinos. This page covers the data formats and tools specific to neutrino physics. + +## Experimental Data Formats + +### ROOT-based Formats +- **Description**: Most neutrino experiments use ROOT for data storage +- **Experiments**: + - T2K, NOvA, MicroBooNE, DUNE + - Super-Kamiokande, Hyper-Kamiokande + - IceCube (partial) +- **Contents**: + - Raw waveforms + - Reconstructed events + - Calibration data + - Monte Carlo truth + +### I3 (IceCube Format) +- **Description**: Custom format for IceCube experiment +- **Format**: Serialized C++ objects (Boost serialization) +- **File Extension**: .i3 +- **Tools**: IceTray framework +- **Python Support**: icecube Python modules + +### WCSim Output +- **Description**: Water Cherenkov Simulator +- **Format**: ROOT files +- **Contents**: + - PMT hits + - True particle information + - Detector geometry +- **Used by**: Super-K, Hyper-K, WATCHMAN studies + +## Event Generators for Neutrinos + +### GENIE (Generates Events for Neutrino Interaction Experiments) +- **Website**: [https://www.genie-mc.org/](https://www.genie-mc.org/) +- **Description**: Comprehensive neutrino event generator +- **Input Format**: XML configuration files +- **Output Formats**: + - GHEP (GENIE native format in ROOT) + - NUISANCE format + - ROOT TTree +- **Key Features**: + - Cross section models + - Nuclear effects + - Hadronization + +### NuWro +- **Website**: [https://nuwro.github.io/user-guide/](https://nuwro.github.io/user-guide/) +- **Description**: Neutrino event generator +- **Input Format**: Parameter files (params.txt) +- **Output Format**: ROOT files +- **Features**: Lepton-nucleus scattering + +### NEUT +- **Description**: Neutrino event generator (used by T2K, Super-K) +- **Input Format**: Card file configuration +- **Output Format**: ROOT files +- **Status**: Limited public distribution + +### GiBUU +- **Website**: [https://gibuu.hepforge.org/](https://gibuu.hepforge.org/) +- **Description**: Transport model for nuclear reactions +- **Input Format**: Fortran namelists (JobCard) +- **Output Format**: Binary and ASCII +- **Use Cases**: Neutrino-nucleus interactions, heavy-ion collisions + +### NuSTEC Formats +- **Description**: Neutrino Scattering Theory Experiment Collaboration +- **Data Format**: Standardized data tables +- **Use Cases**: Generator validation, cross section archives + +## Detector Simulation + +### Geant4-based Simulations +- **Framework**: Most experiments use Geant4 +- **Input**: + - GDML for geometry + - Macro files for configuration +- **Output**: ROOT files with hits, tracks, energy deposits + +### LArSoft (Liquid Argon Software) +- **Website**: [https://larsoft.org/](https://larsoft.org/) +- **Description**: Common framework for LAr TPC experiments +- **Format**: art ROOT files +- **Experiments**: MicroBooNE, SBND, ICARUS, DUNE +- **Key Features**: + - Modular simulation and reconstruction + - Common data products + - Gallery for lightweight analysis + +### art Framework +- **Description**: Event processing framework (Fermilab) +- **Format**: art ROOT files +- **Features**: + - Data products + - Module system + - Provenance tracking +- **Used by**: NOvA, MicroBooNE, DUNE, Mu2e + +### RAT (Reactor Analysis Tool) +- **Repository**: [https://github.com/rat-pac/rat-pac](https://github.com/rat-pac/rat-pac) +- **Description**: Simulation and analysis for reactor and solar neutrinos +- **Input**: JSON-like configuration +- **Output**: ROOT files +- **Used by**: SNO+, KamLAND studies + +### WCSim +- **Description**: Geant4-based Water Cherenkov simulation +- **Configuration**: Macro files +- **Output**: ROOT with custom hit classes +- **Used by**: Super-K, Hyper-K, T2K + +## Reconstruction Frameworks + +### LArSoft Reconstruction +- **Data Products**: + - Hits (ADC signals) + - Clusters + - Tracks and showers + - Vertices + - Particle IDs +- **Format**: art event data model + +### Pandora +- **Description**: Pattern recognition toolkit +- **Used by**: LAr TPC experiments (via LArPandora) +- **Format**: Internal XML-based event representation +- **Output**: Reconstructed particles in experiment format + +### IceTray (IceCube) +- **Description**: Modular event processing framework +- **Configuration**: Python scripts +- **Data Model**: I3 frames with physics objects +- **Modules**: C++ modules configured via Python + +## Cross Section Data + +### NUISANCE (Neutrino Interaction validation using Sparse Conditioning Events) +- **Repository**: [https://github.com/NUISANCEMC/nuisance](https://github.com/NUISANCEMC/nuisance) +- **Description**: Cross section validation framework +- **Input**: ROOT files from generators +- **Data Format**: XML specification of experiments +- **Use Cases**: + - Generator tuning + - Systematic studies + - Model comparison + +### NuSTEC Database +- **Format**: Various (ROOT, ASCII, HepData YAML) +- **Contents**: Experimental cross section measurements +- **Access**: Web interface, data downloads + +## Oscillation Analysis + +### GLoBES (General Long Baseline Experiment Simulator) +- **Website**: [https://www.mpi-hd.mpg.de/personalhomes/globes/](https://www.mpi-hd.mpg.de/personalhomes/globes/) +- **Input Format**: AEDL (Abstract Experiment Definition Language) +- **Use Cases**: + - Experiment sensitivity studies + - Oscillation parameter fits +- **Output**: ASCII tables, ROOT compatible + +### AEDL (Abstract Experiment Definition Language) +- **Description**: DSL for defining neutrino experiments in GLoBES +- **Syntax**: Custom text-based format +- **Elements**: + - Beam fluxes + - Cross sections + - Detector responses + - Systematics + +### Oscillation Fitting Tools +- Various experiments use custom tools +- Common: MINUIT-based fitters +- Output: ROOT files with fit results, contours + +## Flux Predictions + +### Neutrino Beam Simulation +- **Tools**: + - FLUKA (hadronic interactions) + - Geant4 (beam line and target) + - Custom focusing horn simulations +- **Format**: + - ROOT histograms (energy, angle bins) + - dk2nu format (common flux format) + +### dk2nu Format +- **Description**: Common format for neutrino flux files +- **Format**: ROOT TTree +- **Contents**: + - Parent particle information + - Decay kinematics + - Weight information +- **Used by**: Multiple experiments (T2K, NOvA, DUNE) + +### Atmospheric Neutrino Fluxes +- **Formats**: ASCII tables, ROOT +- **Calculators**: + - FLUKA, CORSIKA + - Honda flux, Bartol flux +- **Contents**: Flux vs. energy, zenith angle + +## Calibration and Conditions Data + +### Database Systems +- **Formats**: + - PostgreSQL, SQLite + - ROOT files + - JSON, XML +- **Contents**: + - PMT calibrations + - Electronics calibrations + - Detector conditions + - Alignment + +### Calibration Constants +- Often stored per run, per channel +- Versioning systems +- Access via framework-specific interfaces + +## Machine Learning in Neutrino Physics + +### Training Data Formats +- **Primary**: ROOT files with TTrees +- **Modern**: HDF5, Parquet for ML pipelines +- **Tools**: + - uproot for ROOT → NumPy/Pandas + - TensorFlow/PyTorch data loaders + +### Model Deployment +- **Formats**: ONNX, SavedModel, TorchScript +- **Integration**: Within reconstruction frameworks +- **Examples**: + - Event classification + - Energy reconstruction + - Particle ID + +## Analysis Frameworks and Tools + +### CAFAna (Common Analysis Format and Algorithms) +- **Used by**: NOvA, DUNE +- **Format**: CAF (Common Analysis Format) ROOT files +- **Features**: + - Standardized variable names + - Spectrum manipulation + - Oscillation fits + +### MaCh3 (MCMC Analysis Code Hierachy 3) +- **Used by**: T2K +- **Description**: MCMC-based fitting framework +- **Configuration**: XML, card files +- **Output**: ROOT files with chains, posteriors + +### VALOR (Likelihood Analysis Framework) +- **Description**: Validation and fitting framework +- **Format**: Configuration files, ROOT I/O + +## Data Preservation and Open Data + +### HEPData +- **Use**: Publishing neutrino cross section measurements +- **Format**: YAML tables + +### Experiment Data Releases +- **Examples**: + - MicroBooNE open data + - IceCube public data +- **Formats**: ROOT, I3, HDF5 +- **Documentation**: README files, data models + +## Specific Experiment Formats + +### Super-Kamiokande +- **Format**: ZEBRA (legacy), ROOT (modern) +- **Access**: Collaboration-internal formats + +### T2K +- **Format**: ROOT files (oa*, nd280*, sk*) +- **Software**: nd280Highland, T2K-SK, custom tools + +### NOvA +- **Format**: art ROOT files, CAF files +- **Tools**: NOvAsoft, CAFAna + +### MicroBooNE/DUNE +- **Format**: art/LArSoft ROOT files +- **Tools**: LArSoft, Pandora, custom modules + +### IceCube +- **Format**: I3 files +- **Tools**: IceTray, custom analysis frameworks + +## Domain-Specific Languages + +### GENIE Configuration XML +```xml + + 0.97428 + true + ... + +``` + +### GLoBES AEDL +``` +/* Define an experiment */ +experiment(#exp_name, flux, cross, detector, smearing); +``` + +### LArSoft FHiCL (Fermilab Hierarchical Configuration Language) +``` +physics: { + producers: { + pandora: @local::microboone_pandora + } + analyzers: { + myanalyzer: { ... } + } +} +``` + +## Best Practices + +1. **Use ROOT** for most neutrino physics data +2. **Document** detector geometry comprehensively +3. **Validate** generators against world data (NUISANCE) +4. **Archive** flux predictions with uncertainties +5. **Use common formats** (dk2nu for flux, GHEP for events) +6. **Version** calibration databases +7. **Preserve** complete analysis chains +8. **Publish** results to HEPData +9. **Use standardized** oscillation parameter conventions + +## Related Resources + +- [Particle Physics Formats](particle-physics.qmd) +- [Nuclear Physics Formats](nuclear-physics.qmd) +- [Hadron Physics Formats](hadron-physics.qmd) +- [Serialization Formats Overview](serialization-formats.qmd) + +## External Links + +- [NuSTEC](https://nustec.fnal.gov/) +- [GENIE Homepage](https://www.genie-mc.org/) +- [LArSoft](https://larsoft.org/) +- [IceCube Software](https://software.icecube.wisc.edu/) +- [Neutrino Platform (CERN)](https://neutrino-platform.web.cern.ch/) diff --git a/nuclear-physics.qmd b/nuclear-physics.qmd new file mode 100644 index 0000000..3702dec --- /dev/null +++ b/nuclear-physics.qmd @@ -0,0 +1,219 @@ +--- +title: "Nuclear Physics Formats" +--- + +# Nuclear Physics: Formats and DSLs + +Nuclear physics involves the study of atomic nuclei, nuclear reactions, and nuclear structure. This field uses specialized formats for experimental data, theoretical calculations, and nuclear data evaluation. + +## Nuclear Data Formats + +### ENDF (Evaluated Nuclear Data File) +- **Website**: [https://www.nndc.bnl.gov/endf/](https://www.nndc.bnl.gov/endf/) +- **Description**: International standard for evaluated nuclear reaction data +- **Format**: Structured ASCII (ENDF-6 format) +- **Use Cases**: + - Cross sections + - Angular distributions + - Energy spectra + - Decay data +- **Key Features**: + - Material-temperature (MT) numbers + - Reaction channel identification + - Error/covariance data + +### ACE (A Compact ENDF) Format +- **Description**: Binary format used by MCNP and other Monte Carlo codes +- **Source**: Processed from ENDF +- **Use Cases**: Fast neutron transport calculations +- **Tools**: NJOY for ENDF to ACE conversion + +### EXFOR (Experimental Nuclear Reaction Data) +- **Website**: [https://www-nds.iaea.org/exfor/](https://www-nds.iaea.org/exfor/) +- **Description**: Database of experimental nuclear reaction data +- **Format**: Structured text format (EXFOR syntax) +- **Contents**: + - Experimental measurements + - Cross sections + - Bibliographic information + - Experimental details + +### ENSDF (Evaluated Nuclear Structure Data File) +- **Website**: [https://www.nndc.bnl.gov/ensdf/](https://www.nndc.bnl.gov/ensdf/) +- **Description**: Database of nuclear structure and decay data +- **Format**: Structured ASCII +- **Contents**: + - Energy levels + - Gamma rays + - Decay schemes + - Half-lives + +## Monte Carlo Transport Codes + +### MCNP (Monte Carlo N-Particle) +- **Website**: [https://mcnp.lanl.gov/](https://mcnp.lanl.gov/) +- **Input Format**: Custom text-based DSL +- **Data Libraries**: ACE format +- **Output**: Binary and ASCII tallies +- **DSL Features**: + - Cell/surface geometry description + - Material specifications + - Source definitions + - Tally specifications + +### Geant4 (Geometry And Tracking) +- **Website**: [https://geant4.web.cern.ch/](https://geant4.web.cern.ch/) +- **Input**: + - GDML (XML geometry) + - Macro files (custom command language) +- **Data Libraries**: G4EMLOW, PhotonEvaporation, etc. +- **Output**: ROOT, HDF5, ASCII + +### FLUKA +- **Website**: [https://fluka.cern/](https://fluka.cern/) +- **Input Format**: Custom card-based input +- **Data Libraries**: Internal nuclear data +- **Output**: Binary and ASCII + +### PHITS (Particle and Heavy Ion Transport code System) +- **Website**: [https://phits.jaea.go.jp/](https://phits.jaea.go.jp/) +- **Input Format**: Card-based input similar to MCNP +- **Output**: Various formats including ROOT + +## Nuclear Structure Codes + +### NuShellX +- **Description**: Shell model calculations +- **Input Format**: Custom interaction files +- **Output**: ASCII energy levels, transitions +- **Interaction Files**: USD, GXPF1, KB3G, etc. + +### OXBASH +- **Description**: Shell model code +- **Input Format**: Custom format for interactions and model spaces +- **Output**: ASCII tables of states and transitions + +### TALYS +- **Website**: [https://tendl.web.psi.ch/tendl_2021/talys.html](https://tendl.web.psi.ch/tendl_2021/talys.html) +- **Description**: Nuclear reaction simulation code +- **Input Format**: Keyword-based input files +- **Output**: ASCII tables, can generate ENDF format + +## Experimental Data Acquisition + +### MIDAS (Maximum Integrated Data Acquisition System) +- **Website**: [https://midas.triumf.ca/](https://midas.triumf.ca/) +- **Description**: Data acquisition system used in many nuclear physics labs +- **Format**: Custom binary event format +- **Output**: Can export to ROOT +- **Features**: Online analysis, slow control integration + +### NSCLDAQ (National Superconducting Cyclotron Laboratory DAQ) +- **Description**: Data acquisition framework +- **Format**: Ring buffer based event format +- **Use Cases**: Rare isotope beam experiments + +### ROOT-based DAQ +- Many experiments use ROOT for data storage +- Event structures defined with ROOT dictionaries +- Enables immediate analysis + +## Gamma-Ray Spectroscopy + +### Radware +- **Description**: Gamma-ray spectroscopy analysis package +- **Formats**: + - .spe (spectrum files) + - .mat (matrix files) + - .lev (level scheme files) +- **Tools**: gf3, levit8r, escl8r + +### GammaVision / Genie +- Commercial gamma spectroscopy software +- Proprietary formats with some export options +- ASCII export for spectra + +### SPE Files +- Simple ASCII or binary format for gamma spectra +- Header with calibration info +- Channel counts + +## Nuclear Reaction Theory + +### Coupled-Channels Codes +- **FRESCO**: [https://www.fresco.org.uk/](https://www.fresco.org.uk/) + - Input: Custom namelist format + - Output: Cross sections, analyzing powers +- **ECIS**: Coupled-channels code + - Custom input format + +### R-Matrix Codes +- **AZURE2**: [https://azure.nd.edu/](https://azure.nd.edu/) + - Input: XML configuration + - Data: ASCII tables +- EDA (R-Matrix code) + - Custom input format + +## Data Processing Libraries + +### NJOY +- **Website**: [https://www.njoy21.io/](https://www.njoy21.io/) +- **Description**: Process ENDF data into usable formats +- **Input**: ENDF-6 format +- **Output**: ACE, MATXS, ACER, etc. +- **DSL**: Card-based input for processing instructions + +### PREPRO +- **Description**: ENDF preprocessing utilities +- **Use Cases**: Format checking, unit conversion, plotting + +### FUDGE (For Updating Data and Generating ENDF) +- **Repository**: [https://github.com/LLNL/fudge](https://github.com/LLNL/fudge) +- **Description**: Python package for nuclear data processing +- **Input/Output**: ENDF, GNDS (next-gen format) + +## Next-Generation Formats + +### GNDS (Generalized Nuclear Data Structure) +- **Description**: XML-based successor to ENDF +- **Features**: + - More flexible than ENDF + - Better uncertainty quantification + - Hierarchical structure +- **Status**: Under development, some adoption + +## Domain-Specific Languages + +### MCNP Input Language +- Geometry: Cell and surface definitions +- Materials: Material cards with isotope fractions +- Source: Complex source specifications +- Tallies: Particle and energy filters + +### ENDF Format Language +- Structured records with MT/MF numbers +- Tabulated data with interpolation schemes +- Covariance matrices + +## Best Practices + +1. **Use ENDF** for evaluated nuclear data +2. **Archive in EXFOR** for experimental data +3. **Document geometries** thoroughly (especially for simulations) +4. **Validate** against benchmarks +5. **Version control** nuclear data libraries +6. **Use standard formats** for data exchange +7. **Preserve raw data** from experiments + +## Related Resources + +- [Particle Physics Formats](particle-physics.qmd) +- [Hadron Physics Formats](hadron-physics.qmd) +- [Serialization Formats Overview](serialization-formats.qmd) + +## External Links + +- [NNDC (National Nuclear Data Center)](https://www.nndc.bnl.gov/) +- [IAEA Nuclear Data Services](https://www-nds.iaea.org/) +- [NEA Data Bank](https://www.oecd-nea.org/jcms/pl_19666/nuclear-data) +- [TENDL (TALYS-based Evaluated Nuclear Data Library)](https://tendl.web.psi.ch/) diff --git a/particle-physics.qmd b/particle-physics.qmd new file mode 100644 index 0000000..70bc9d9 --- /dev/null +++ b/particle-physics.qmd @@ -0,0 +1,306 @@ +--- +title: "Particle Physics Formats" +--- + +# Particle Physics: Formats and DSLs + +High-energy particle physics (HEP) has developed a rich ecosystem of data formats and domain-specific languages for experimental data, simulations, and theoretical calculations. + +## Experimental Data Formats + +### ROOT +- **Website**: [https://root.cern.ch/](https://root.cern.ch/) +- **Description**: The de facto standard for HEP data analysis +- **Format**: Binary with self-describing schemas +- **Key Features**: + - TTrees for event data + - Histograms and statistical tools + - I/O compression + - C++ and Python interfaces +- **Use Cases**: + - LHC experiments (ATLAS, CMS, LHCb, ALICE) + - Belle II, BaBar + - Most modern HEP experiments + +### EDM4hep (Event Data Model for HEP) +- **Repository**: [https://github.com/key4hep/EDM4hep](https://github.com/key4hep/EDM4hep) +- **Description**: Common event data model for future colliders +- **Format**: Based on PODIO +- **Use Cases**: FCC, CEPC, ILC, CLIC studies +- **Key Features**: Generator-based C++ code, ROOT I/O + +### PODIO (Plain Old Data I/O) +- **Repository**: [https://github.com/AIDASoft/podio](https://github.com/AIDASoft/podio) +- **Description**: Event data model toolkit +- **Backend**: ROOT, SIO (Simple I/O) +- **Features**: + - Schema definition in YAML + - Code generation for C++ + - Relations between objects + +### LCIO (Linear Collider I/O) +- **Website**: [https://lcio.desy.de/](https://lcio.desy.de/) +- **Description**: Event data model for linear collider studies +- **Format**: Custom binary or ROOT +- **Language Support**: C++, Java, Python, Fortran +- **Use Cases**: ILC detector studies + +## Monte Carlo Event Generators + +### HepMC (High Energy Physics Monte Carlo) +- **Website**: [http://hepmc.web.cern.ch/](http://hepmc.web.cern.ch/) +- **Description**: Standard event record for MC generators +- **Version**: HepMC3 is current +- **Format**: + - In-memory C++ objects + - ASCII I/O + - ROOT I/O +- **Supported by**: PYTHIA, Herwig, Sherpa, MadGraph, etc. + +### Les Houches Event Format (LHEF) +- **Description**: XML-based format for hard process events +- **Use Cases**: + - Matrix element generators → Parton shower + - Interfacing different generator components +- **Standard**: Les Houches Accord +- **Version**: LHEF 3.0 current + +### PYTHIA +- **Website**: [https://www.pythia.org/](https://www.pythia.org/) +- **Description**: General-purpose event generator +- **Input**: Command-based configuration +- **Output**: HepMC, LHEF, custom formats +- **Configuration**: Key-value pairs or XML + +### Herwig +- **Website**: [https://herwig.hepforge.org/](https://herwig.hepforge.org/) +- **Input**: .in files with ThePEG syntax +- **Output**: HepMC, LHEF + +### MadGraph +- **Website**: [https://launchpad.net/mg5amcnlo](https://launchpad.net/mg5amcnlo) +- **Description**: Matrix element generator +- **Input**: Process definitions in custom syntax +- **Output**: LHEF events, ROOT, HepMC +- **DSL**: Process specification language + +### Sherpa +- **Website**: [https://sherpa-team.gitlab.io/](https://sherpa-team.gitlab.io/) +- **Input**: YAML configuration files +- **Output**: HepMC, ROOT + +## Detector Simulation + +### Geant4 +- **Website**: [https://geant4.web.cern.ch/](https://geant4.web.cern.ch/) +- **Description**: Toolkit for particle physics detector simulation +- **Input Formats**: + - GDML (XML geometry description) + - Macro files (.mac) + - C++ geometry construction +- **Output**: ROOT, HDF5, ASCII +- **Physics Lists**: Modular physics models + +### GDML (Geometry Description Markup Language) +- **Website**: [https://gdml.web.cern.ch/GDML/](https://gdml.web.cern.ch/GDML/) +- **Format**: XML-based +- **Use Cases**: + - Detector geometry for Geant4 + - Visualization + - Interchange between simulation frameworks +- **Elements**: Materials, solids, volumes, placement + +### DD4hep (Detector Description for HEP) +- **Repository**: [https://github.com/AIDASoft/DD4hep](https://github.com/AIDASoft/DD4hep) +- **Description**: Generic detector description toolkit +- **Format**: XML or C++ compact files +- **Backend**: Geant4, ROOT geometry +- **Use Cases**: Future collider detector studies + +## Fast Simulation + +### Delphes +- **Website**: [https://cp3.irmp.ucl.ac.be/projects/delphes](https://cp3.irmp.ucl.ac.be/projects/delphes) +- **Description**: Fast detector simulation framework +- **Input**: + - HepMC, LHEF, ROOT + - Configuration cards (Tcl-based) +- **Output**: ROOT files with reconstructed objects +- **Use Cases**: Phenomenology studies, detector optimization + +### Parametric Simulation +- Various experiments have custom fast simulation +- Configuration: Often text-based parameter files +- Output: Same format as full simulation (ROOT) + +## Analysis Frameworks + +### RDataFrame (ROOT) +- **Description**: High-level interface for data analysis in ROOT +- **Language**: C++, Python +- **Features**: + - Declarative analysis + - Automatic parallelization + - Lazy evaluation + +### Scikit-HEP +- **Website**: [https://scikit-hep.org/](https://scikit-hep.org/) +- **Description**: Python ecosystem for HEP +- **Key Packages**: + - uproot (ROOT I/O) + - awkward (jagged arrays) + - hist (histogramming) + - vector (Lorentz vectors) + - particle (PDG particle data) + +### Coffea (Column Object Framework For Effective Analysis) +- **Repository**: [https://github.com/CoffeaTeam/coffea](https://github.com/CoffeaTeam/coffea) +- **Description**: Python-based analysis framework +- **Backend**: Awkward Array, Dask for scaling +- **Input**: ROOT files via uproot + +## Theoretical Calculation Formats + +### SLHA (SUSY Les Houches Accord) +- **Description**: Format for SUSY model parameters +- **Format**: ASCII with blocks and values +- **Use Cases**: + - Spectrum calculators → Event generators + - Experimental limit interpretation +- **Versions**: SLHA1, SLHA2 + +### UFO (Universal FeynRules Output) +- **Description**: Model format for MC generators +- **Format**: Python modules +- **Use Cases**: BSM physics models in generators +- **Tools**: FeynRules, MadGraph + +### CalcHEP/CompHEP +- **Description**: Symbolic calculation programs +- **Format**: Custom model files +- **Output**: Cross sections, decay widths + +## Parton Distribution Functions + +### LHAPDF (Les Houches Accord PDF) +- **Website**: [https://lhapdf.hepforge.org/](https://lhapdf.hepforge.org/) +- **Description**: Standard library for PDF access +- **Format**: Grid files (text-based) +- **PDF Sets**: CT18, MSHT20, NNPDF4.0, etc. +- **Language Support**: C++, Python, Fortran + +## Reconstruction and Trigger + +### Trigger Configuration +- **Formats**: XML, JSON, Python +- **Examples**: + - ATLAS: XML + - CMS: Python configuration +- **Contents**: Trigger logic, thresholds, prescales + +### Calibration Data +- **Formats**: Database systems (Oracle, SQLite) +- **Contents**: + - Detector conditions + - Calibration constants + - Alignment parameters +- **Access**: Conditions database interfaces + +## Data Preservation + +### REANA (Reusable Analyses) +- **Website**: [http://www.reanahub.io/](http://www.reanahub.io/) +- **Description**: Platform for reproducible research +- **Format**: YAML workflow specification +- **Use Cases**: Preserving analysis workflows + +### HEPData +- **Website**: [https://www.hepdata.net/](https://www.hepdata.net/) +- **Description**: Repository for HEP experimental results +- **Format**: YAML data tables +- **Contents**: + - Cross sections + - Limits + - Distributions from publications + +### INSPIRE-HEP +- **Website**: [https://inspirehep.net/](https://inspirehep.net/) +- **Description**: Digital library for HEP +- **Integration**: Links to HEPData, arXiv, experiments + +## Control and Configuration Languages + +### CMS Configuration (Python) +- **Description**: Python-based configuration framework +- **Features**: + - Object-oriented configuration + - Module composition + - Parameter sets + +### ATLAS JO (Job Options) +- **Language**: Python +- **Use Cases**: + - Athena framework configuration + - Job steering + - Algorithm sequencing + +### Gaudi Framework +- **Description**: Event processing framework +- **Configuration**: Python +- **Used by**: LHCb, ATLAS (Athena) + +## Machine Learning Integration + +### Model Formats +- **ONNX**: [https://onnx.ai/](https://onnx.ai/) + - Interoperable ML model format + - Used in trigger systems, reconstruction +- **SavedModel** (TensorFlow) +- **TorchScript** (PyTorch) + +### Training Data +- **Formats**: HDF5, ROOT, Parquet +- **Tools**: TensorFlow, PyTorch data loaders +- **Preprocessing**: Often custom per experiment + +## Workflow Management + +### PanDA (Production and Distributed Analysis) +- **Used by**: ATLAS +- **Configuration**: JSON, Python +- **Task definition**: Custom formats + +### DIRAC (Distributed Infrastructure with Remote Agent Control) +- **Used by**: LHCb, Belle II, others +- **Configuration**: Python, XML + +### HTCondor +- **Configuration**: ClassAds (custom syntax) +- **Job description**: Submit files + +## Best Practices + +1. **Use ROOT** for event data storage (standard in HEP) +2. **Use HepMC3** for event generator interfaces +3. **Use GDML** for geometry interchange +4. **Document** all Monte Carlo settings +5. **Preserve** analysis code with data (REANA, Git) +6. **Publish** final results to HEPData +7. **Use standard formats** (LHEF, SLHA) for model interchange +8. **Version** data formats and software +9. **Archive** complete analysis chains + +## Related Resources + +- [Hadron Physics Formats](hadron-physics.qmd) +- [Nuclear Physics Formats](nuclear-physics.qmd) +- [Neutrino Physics Formats](neutrino-physics.qmd) +- [Serialization Formats Overview](serialization-formats.qmd) + +## External Links + +- [HEP Software Foundation](https://hepsoftwarefoundation.org/) +- [CERN Computing](https://computing.web.cern.ch/) +- [Scikit-HEP](https://scikit-hep.org/) +- [ROOT Forum](https://root-forum.cern.ch/) +- [HEPData](https://www.hepdata.net/) diff --git a/serialization-formats.qmd b/serialization-formats.qmd new file mode 100644 index 0000000..a427a62 --- /dev/null +++ b/serialization-formats.qmd @@ -0,0 +1,121 @@ +--- +title: "Serialization Formats" +--- + +# Common Serialization Formats in Physics + +This page provides an overview of widely-used serialization formats in physics research and scientific computing. + +## Binary Formats + +### HDF5 (Hierarchical Data Format 5) +- **Website**: [https://www.hdfgroup.org/solutions/hdf5/](https://www.hdfgroup.org/solutions/hdf5/) +- **Description**: A versatile data model, library, and file format for storing and managing data +- **Use Cases**: Large-scale numerical simulations, experimental data storage +- **Language Support**: C, C++, Fortran, Python, Java, MATLAB, Julia +- **Physics Applications**: Climate modeling, particle physics, astrophysics + +### ROOT +- **Website**: [https://root.cern.ch/](https://root.cern.ch/) +- **Description**: A data analysis framework developed at CERN +- **Use Cases**: High-energy physics data analysis and storage +- **Language Support**: C++, Python (PyROOT) +- **Physics Applications**: Particle physics experiments (LHC, Belle II, etc.) + +### NetCDF (Network Common Data Form) +- **Website**: [https://www.unidata.ucar.edu/software/netcdf/](https://www.unidata.ucar.edu/software/netcdf/) +- **Description**: A set of libraries and machine-independent data formats +- **Use Cases**: Array-oriented scientific data +- **Language Support**: C, Fortran, Python, Java, MATLAB +- **Physics Applications**: Climate science, oceanography, atmospheric physics + +### FITS (Flexible Image Transport System) +- **Website**: [https://fits.gsfc.nasa.gov/](https://fits.gsfc.nasa.gov/) +- **Description**: Open standard for astronomical data +- **Use Cases**: Astronomical images and tables +- **Language Support**: C, Fortran, Python (astropy), IDL +- **Physics Applications**: Astronomy, astrophysics + +## Text-Based Formats + +### JSON (JavaScript Object Notation) +- **Website**: [https://www.json.org/](https://www.json.org/) +- **Description**: Lightweight data-interchange format +- **Use Cases**: Configuration files, lightweight data exchange +- **Language Support**: Nearly all programming languages +- **Physics Applications**: Configuration management, metadata storage + +### XML (eXtensible Markup Language) +- **Website**: [https://www.w3.org/XML/](https://www.w3.org/XML/) +- **Description**: Markup language for encoding documents +- **Use Cases**: Complex hierarchical data, metadata +- **Language Support**: Nearly all programming languages +- **Physics Applications**: Experiment metadata, detector descriptions (GDML) + +### YAML (YAML Ain't Markup Language) +- **Website**: [https://yaml.org/](https://yaml.org/) +- **Description**: Human-friendly data serialization standard +- **Use Cases**: Configuration files, parameter files +- **Language Support**: Most modern programming languages +- **Physics Applications**: Simulation parameters, workflow configurations + +### CSV (Comma-Separated Values) +- **Description**: Simple tabular data format +- **Use Cases**: Simple data tables, spreadsheet data +- **Language Support**: Universal support +- **Physics Applications**: Small datasets, experimental measurements + +## Specialized Physics Formats + +### GDML (Geometry Description Markup Language) +- **Website**: [https://gdml.web.cern.ch/GDML/](https://gdml.web.cern.ch/GDML/) +- **Description**: XML-based geometry description for particle physics detectors +- **Use Cases**: Detector geometry in Geant4 simulations +- **Physics Applications**: Detector simulation, particle physics + +### SLHA (SUSY Les Houches Accord) +- **Description**: Standard format for supersymmetry parameters +- **Use Cases**: Exchange of SUSY model parameters between tools +- **Physics Applications**: Beyond Standard Model physics + +### LHE (Les Houches Event format) +- **Description**: Standard format for particle physics event generators +- **Use Cases**: Monte Carlo event generation +- **Physics Applications**: High-energy physics simulations + +## Protocol Buffers and Modern Formats + +### Protocol Buffers +- **Website**: [https://protobuf.dev/](https://protobuf.dev/) +- **Description**: Google's language-neutral, platform-neutral extensible mechanism for serializing structured data +- **Language Support**: C++, Java, Python, Go, and more +- **Physics Applications**: High-performance data exchange + +### Apache Parquet +- **Website**: [https://parquet.apache.org/](https://parquet.apache.org/) +- **Description**: Columnar storage format for Hadoop ecosystem +- **Use Cases**: Big data analytics +- **Physics Applications**: Large-scale data analysis pipelines + +### MessagePack +- **Website**: [https://msgpack.org/](https://msgpack.org/) +- **Description**: Efficient binary serialization format +- **Use Cases**: High-performance data exchange +- **Language Support**: 50+ programming languages + +## Choosing a Format + +When selecting a serialization format for your physics project, consider: + +1. **Data Size**: Binary formats (HDF5, ROOT) for large datasets; text formats (JSON, YAML) for small configs +2. **Performance**: Binary formats are generally faster for I/O operations +3. **Human Readability**: Text formats are easier to inspect and debug +4. **Tool Support**: Consider the tools and languages in your workflow +5. **Community Standards**: Use established formats in your field (e.g., ROOT in HEP, FITS in astronomy) +6. **Longevity**: Choose formats with good long-term support and documentation + +## Related Resources + +- [Lattice Physics Formats](lattice-physics.qmd) +- [Particle Physics Formats](particle-physics.qmd) +- [Astro Physics Formats](astro-physics.qmd) diff --git a/styles.css b/styles.css new file mode 100644 index 0000000..ea57a42 --- /dev/null +++ b/styles.css @@ -0,0 +1,43 @@ +/* Custom styles for Format Landscape */ + +/* Improve link visibility */ +a { + text-decoration: underline; +} + +a:hover { + text-decoration: none; +} + +/* Code blocks */ +pre { + border-left: 3px solid #2780e3; + padding-left: 1em; +} + +/* Make headers stand out */ +h1, h2, h3 { + color: #2c3e50; +} + +/* Improve list spacing */ +ul, ol { + margin-bottom: 1em; +} + +li { + margin-bottom: 0.3em; +} + +/* Table styling */ +table { + margin-top: 1em; + margin-bottom: 1em; +} + +/* Footer styling */ +.page-footer { + margin-top: 2em; + padding-top: 1em; + border-top: 1px solid #ddd; +}