# proMGE version 2 Download Files

All files are tab-separated (`.tsv`).

The canonical merge key across all MGE-level files is **`mge_genome_position`**.

---

## Files

### mge_profile.tsv
One row per non-nested MGE. Core reference table (~17 million rows).

| Column | Type | Description |
|--------|------|-------------|
| id | integer | Row identifier |
| cluster_id | text | MGE100 cluster ID (100%-identity cluster) |
| source | text | `pg3` (isolates) or `spire` (MAGs) |
| genome | text | Host genome assembly identifier |
| speci | text | specI cluster identifier |
| species | text | GTDB species name of host genome |
| mge_genome_position | text | **Merge key** — unique MGE position identifier |
| mge_length | integer | MGE length in base pairs |
| number_of_proteins | integer | Number of protein-coding genes in the MGE |
| phage_genes | integer | Count of phage-annotated machinery genes |
| conjugation_genes | text | Semicolon-separated conjugation/secretion system gene names |
| recombinase | text | Semicolon-separated recombinase gene names |
| mge_category | text | MGE type: `is_tn`, `phage`, `phage_like`, `conjugative_element`, `integron`, `mobility_island`, `cellular` |
| eggnog_ogs | text | Semicolon-separated root-level COG identifiers (eggNOG-mapper) |

---

### mge_nested.tsv
One row per nested MGE (an MGE embedded within another). Merge via `mge_genome_position`.

| Column | Type | Description |
|--------|------|-------------|
| id | integer | Row identifier |
| cluster_id | text | MGE100 cluster identifier |
| source | text | `pg3` or `spire` |
| species | text | GTDB species name |
| genome | text | Host genome assembly identifier |
| speci | text | specI cluster identifier |
| mge_genome_position | text | **Merge key** |
| mge_length | integer | MGE length in base pairs |
| phage_genes | integer | Count of phage machinery genes |
| conjugation_genes | text | Conjugation gene names |
| recombinase | text | Recombinase gene names |
| eggnog_ogs | text | COG identifiers |
| is_tn | integer | Transposons/IS elements within this nested MGE |
| integron | integer | Integrons |
| phage | integer | Phages |
| phage_like | integer | Phage-like elements |
| conjugative_element | integer | Conjugative elements |
| mobility_island | integer | Mobility islands |
| cellular | integer | Cellular elements |

---

### mge100_clusters.tsv
Groups all MGEs into 100%-nucleotide-identity clusters. One row per MGE (member).

To join with mge_profile.tsv: `member_id = 'MGE_' + mge_genome_position`

| Column | Type | Description |
|--------|------|-------------|
| member_id | text | MGE identifier — equal to `'MGE_' + mge_genome_position` |
| representative_id | text | Cluster representative identifier |
| mge100_cluster_id | text | Cluster ID: `proMGE100_1.0_{id}_{mge_cat}_{n_members}_{n_species}` |

---

### mge_antibiotic.tsv
One row per MGE–antibiotic class combination. Nested MGEs excluded. Merge via `mge_genome_position`.

| Column | Type | Description |
|--------|------|-------------|
| id | integer | Row identifier |
| genome | text | Host genome assembly identifier |
| speci | text | specI cluster identifier |
| source | text | `pg3` or `spire` |
| cluster_id | text | MGE100 cluster identifier |
| mge_genome_position | text | **Merge key** |
| mge_length | integer | MGE length in base pairs |
| mge_category | text | MGE type |
| antibiotic | text | Antibiotic resistance class (spaces replaced with `_`) |
| aro | text | Semicolon-separated ARO accession numbers |
| recombinase | text | Recombinase gene names |
| phage_genes | integer | Count of phage machinery genes |
| conjugation_genes | text | Conjugation gene names |
| eggnog_ogs | text | COG identifiers |
| species | text | GTDB species name |
| taxonomy | text | Lineage string: `domain : phylum : class : order : family : genus` |

---

### hgt.tsv
Predicted HGT events: MGE100 clusters present in ≥2 distinct taxonomic families.
Each row is one cross-family genome pair. Join to mge_profile.tsv via
`hgt_genome_position_1` or `hgt_genome_position_2` = `mge_genome_position`.
For taxonomy of HGT species, join on `species` to genomes_taxonomy.tsv (no separate hgt_taxonomy file).

| Column | Type | Description |
|--------|------|-------------|
| id | integer | Row identifier |
| cluster_id | text | Shared MGE100 cluster (present in ≥2 families) |
| recombinase | text | Recombinase genes of the MGE |
| eggnog_ogs | text | COG identifiers |
| mge | text | MGE category |
| hgt_genome_1 | text | Assembly identifier — partner genome 1 |
| hgt_species_1 | text | GTDB species — partner 1 |
| source_genome_1 | text | `pg3` or `spire` — partner 1 |
| hgt_genome_position_1 | text | `mge_genome_position` in partner genome 1 |
| hgt_genome_2 | text | Assembly identifier — partner genome 2 |
| hgt_species_2 | text | GTDB species — partner 2 |
| source_genome_2 | text | `pg3` or `spire` — partner 2 |
| hgt_genome_position_2 | text | `mge_genome_position` in partner genome 2 |

---

### genomes_quality.tsv
Assembly quality and pangenome statistics for genomes carrying at least one MGE, one row per genome.
Note: `source` here is `Isolates` / `MAGs`; in mge_profile.tsv the same genomes appear as `pg3` / `spire`.
Join to mge_profile.tsv: `genome_id = mge_profile.genome` (isolates). For MAGs, `genome_id` is the bin
identifier and does not directly match `mge_profile.genome` (which holds the formatted SPIRE ID).

| Column | Type | Description |
|--------|------|-------------|
| genome_id | text | Genome identifier — assembly accession (isolates) or bin ID (MAGs) |
| speci | text | specI cluster identifier |
| representative | boolean | True if this genome is the specI cluster representative |
| sample_name | text | Sample identifier |
| completeness | float | CheckM completeness (%) |
| contamination | float | CheckM contamination (%) |
| n50 | integer | Assembly N50 in base pairs |
| num_contigs | integer | Number of contigs |
| mean_contig_length | float | Mean contig length in base pairs |
| genome_size | integer | Total genome size in base pairs |
| source | text | `Isolates` or `MAGs` |
| total | integer | Total gene count |
| core | integer | Core gene count |
| acc | integer | Accessory gene count |
| core_prop | float | Proportion of core genes |
| acc_prop | float | Proportion of accessory genes |
| pangenome | boolean | True if included in pangenome analysis |

---

### genomes_taxonomy.tsv
GTDB taxonomy for genomes carrying at least one MGE, one row per genome.
Includes a `representative` flag. Also covers taxonomy for all HGT-participating species.

Merge with mge_profile.tsv: `genome = mge_profile.genome`.

| Column | Type | Description |
|--------|------|-------------|
| genome | text | Assembly identifier — join key to mge_profile.genome |
| source | text | `pg3` (isolates) or `spire` (MAGs) |
| speci | text | specI cluster identifier |
| kingdom | text | GTDB domain |
| phylum | text | GTDB phylum |
| class | text | GTDB class |
| tax_order | text | GTDB order |
| family | text | GTDB family |
| genus | text | GTDB genus |
| species | text | GTDB species |
| representative | boolean | True if this genome is the specI cluster representative |

---

### mge100.gff3
Functional annotations for all genes of non-redundant MGE sequences (MGE100 representatives) as output by eggNOG-mapper. Additionally, information about recombinase, phage structural genes and machinery can be obtained. 
To get functional annotations for not representative MGEs, join with mge100.tsv: `repr_id = 'MGE_' + mge_genome_position`.

---

### mge100.fna
Nucleotide sequences of all non-redundant MGE sequences (MGE100 representatives). To get nucleotide sequences for not representative MGEs, join the header with mge100.tsv: `repr_id = 'MGE_' + mge_genome_position`.
The FASTA header consists of MGE identifier `'MGE_' + mge_genome_position` and the following attributes: 
- `mge=` full MGE category, ';'-separated in case of nested MGEs;
- `genome_type=`: core (COR) or accessory (ACC) recombinase island;
- `mge_type=`: nested or non-nested MGE;
- `size=`: the MGE length in base pairs;
- `n_genes=`: the number of protein-coding genes in the MGE;
- `mgeR=`: ';'-separated list of recombinases;

---

## Cross-table join examples (Python / pandas)

```python
import pandas as pd

mge   = pd.read_csv('mge_profile.tsv',      sep='\t')
amr   = pd.read_csv('mge_antibiotic.tsv',   sep='\t')
clust = pd.read_csv('mge100_clusters.tsv',  sep='\t')
hgt   = pd.read_csv('hgt.tsv',              sep='\t')
tax   = pd.read_csv('genomes_taxonomy.tsv', sep='\t')
qual  = pd.read_csv('genomes_quality.tsv',  sep='\t')

# Annotate MGE profile with AMR data
mge_amr = mge.merge(amr, on='mge_genome_position', how='inner')

# Add full taxonomy to MGE profile
mge_tax = mge.merge(tax, on='genome', how='left')

# Add quality metrics to MGE profile (isolates only — direct join via genome_id)
mge_qual = mge[mge['source'] == 'pg3'].merge(
    qual, left_on='genome', right_on='genome_id', how='left'
)

# Get HGT events with species taxonomy
hgt_tax = hgt.merge(tax.rename(columns=lambda c: c + '_1' if c != 'species' else c)
                       .rename(columns={'species': 'hgt_species_1'}),
                    on='hgt_species_1', how='left')

# Join mge100 clusters: member_id == 'MGE_' + mge_genome_position
clust['mge_genome_position'] = clust['member_id'].str.replace(r'^MGE_', '', regex=True)
mge_clust = mge.merge(clust, on='mge_genome_position', how='left')
```