integration of RE Galaxy

README.md

@@ -74,6 +74,14 @@ pulsar_ssh_key: |
   -----END OPENSSH PRIVATE KEY-----
 ```
 
+## RBD storage
+On production instances, the Galaxy datasets and other mutable data are stored on RBD (RADOS Block Device) using Ceph. Its installation and configuration is covered by this playbook and the following variables help to maintain the process:
+```
+rbd_mount_point: /data
+rbd_data_dir: "{{ rbd_mount_point }}/galaxy"
+rbd_cluster_code: CL4
+```
+
 ## running playbook
 `$ ansible-playbook --limit <YOUR_HOST> galaxy.yml`
 

files/galaxy/config/sanitize_allowlist.txt

@@ -0,0 +1,48 @@
+toolshed.g2.bx.psu.edu/repos/devteam/fastqc
+toolshed.g2.bx.psu.edu/repos/iuc/fastp
+toolshed.g2.bx.psu.edu/repos/iuc/fastqe
+toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse
+toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc
+toolshed.g2.bx.psu.edu/repos/iuc/quast
+toolshed.g2.bx.psu.edu/repos/paulo_lyra_jr/ludwig_experiment
+toolshed.g2.bx.psu.edu/repos/petr-novak/dante/dante
+toolshed.g2.bx.psu.edu/repos/petr-novak/dante/domains_extract
+toolshed.g2.bx.psu.edu/repos/petr-novak/dante/domains_filter
+toolshed.g2.bx.psu.edu/repos/petr-novak/dante/gff_summary
+toolshed.g2.bx.psu.edu/repos/petr-novak/dante/gff_to_tabular
+toolshed.g2.bx.psu.edu/repos/petr-novak/dante_ltr
+toolshed.g2.bx.psu.edu/repos/petr-novak/dante_tir
+toolshed.g2.bx.psu.edu/repos/petr-novak/krona_text/krona-text
+toolshed.g2.bx.psu.edu/repos/petr-novak/long_reads_sampling/long2short
+toolshed.g2.bx.psu.edu/repos/petr-novak/long_reads_sampling/long_reads_sampling
+toolshed.g2.bx.psu.edu/repos/petr-novak/repeat_annotation_pipeline3/annotate_contigs
+toolshed.g2.bx.psu.edu/repos/petr-novak/repeat_annotation_pipeline3/compare_gff
+toolshed.g2.bx.psu.edu/repos/petr-novak/repeat_annotation_pipeline3/extract_re_contigs
+toolshed.g2.bx.psu.edu/repos/petr-novak/repeat_annotation_pipeline3/repeat_annotate
+toolshed.g2.bx.psu.edu/repos/petr-novak/repeat_annotation_pipeline3/summarize_gff_by_attribute
+toolshed.g2.bx.psu.edu/repos/petr-novak/repeatexplorer2_testing/repeatexplorer2
+toolshed.g2.bx.psu.edu/repos/petr-novak/repeatexplorer2_testing/tarean
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/chip_seq_ratio_1
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/cluster_table2krona_format
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/extract_contigs
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/extract_var_files_from_re
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/fasta_affixer
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/fasta_input
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/fasta_interlacer
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/names_affixer
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/paired_fastq_filtering
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/pairScan
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/plot_comparative
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/rename_sequences
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/RMsearch2
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/sampler
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/single_fastq_filtering
+toolshed.g2.bx.psu.edu/repos/petr-novak/re_utils/summarize_annotation
+toolshed.g2.bx.psu.edu/repos/petr-novak/tidecluster/tc_reannotate
+toolshed.g2.bx.psu.edu/repos/petr-novak/tidecluster/tidecluster
+toolshed.g2.bx.psu.edu/repos/petr-novak/various_galaxy_tools/calculate_density
+toolshed.g2.bx.psu.edu/repos/petr-novak/various_galaxy_tools/Extract_features1
+toolshed.g2.bx.psu.edu/repos/petr-novak/various_galaxy_tools/gff2bed1
+toolshed.g2.bx.psu.edu/repos/petr-novak/various_galaxy_tools/gff2tabular
+toolshed.g2.bx.psu.edu/repos/petr-novak/various_galaxy_tools/gff_filter_by_attribute
+toolshed.g2.bx.psu.edu/repos/saskia-hiltemann/krona_text/krona_text

files/galaxy/lib/examples.py

@@ -0,0 +1,14 @@
+import logging
+log = logging.getLogger( __name__ )
+
+def restrict_testing_tools_to_admins( context, tool ):
+    """
+    This tool filter will hide the upload tool from all users except admin
+    users. This can be enabled by renaming this file to examples.py and adding
+    the following to the ``app:main`` section of ``galaxy.ini``:
+
+        tool_filters = examples:restrict_testing_tools_to_admins
+    """
+    if tool.id == "testing_html" or tool.id == "testing_pbs" or tool.id == "testing":
+        return context.trans.user_is_admin
+    return True

files/nginx/warning.html

@@ -0,0 +1,14 @@
+<!DOCTYPE html>
+<html lang="en">
+   <head>
+       <meta charset="UTF-8">
+       <meta name="viewport" content="width=device-width, initial-scale=1.0">
+       <title>Access Denied</title>
+   </head>
+   <body>
+       <h1>Server is down</h1>
+       <p>Please, be patient and try to reach it in couple of minutes.</p>
+<!--       <h1>Scheduled Maintenance</h1>
+       <p>The server is undergoing scheduled maintenance. Server will be down for a couple of hours, running jobs will not be affected.</p> -->
+   </body>
+</html>

files/repeatexplorer-elixir.cerit-sc.cz/README

@@ -0,0 +1,120 @@
+# usegalaxy
+
+Simple galaxy installation steps.
+
+On fresh install of ubuntu:
+```
+sudo apt-get update
+sudo apt-get upgrade -y
+sudo apt update
+sudo apt upgrade -y
+sudo apt install -y software-properties-common
+sudo apt-add-repository --yes --update ppa:ansible/ansible
+sudo apt install -y ansible
+```
+
+Clone the repo and instal required galaxy roles.
+```
+git clone https://github.com/CESNET/usegalaxy.git
+cd usegalaxy
+ansible-galaxy install -p roles -r requirements.yml
+```
+Note that the roles are already present in this repo.
+
+After the instalation, you should configure your [inventory file](https://training.galaxyproject.org/training-material/topics/admin/tutorials/ansible/tutorial.html#inventory-file) ([hosts](https://github.com/CESNET/usegalaxy/blob/main/hosts)) and set up a vault:
+```
+openssl rand -base64 24 > .vault-password.txt
+ansible-vault create group_vars/secret.yml
+```
+The second command opens an editor wherein you should define variables eg.:
+```
+vault_id_secret: "a-long-random-value"
+rabbitmq_users_password:
+  mqadmin: "a-long-random-value"
+  galaxy: "a-long-random-value"
+  galaxy_gpu: "a-long-random-value"
+```
+The value should be a long random value, which can be obtained by command `openssl rand -base64 24`, though you should avoid having a `/` or `\` in your password as galaxy will interpret this as a path.
+
+Make sure you have these ports enabled on your machine:
+```
+80(HTTP)
+443(HTTPS)
+4369(TCP)
+5671(TCP)
+15672(TCP)
+25672(TCP)
+```
+
+Once everything have been installed and configured, the ansible playbook can be run as follows:
+```
+ansible-playbook galaxy.yml
+```
+
+## Usegalaxy tools
+
+Install ephemeris for tool management:
+```
+virtualenv -p python3 ~/ephemeris_venv
+. ~/ephemeris_venv/bin/activate
+pip install ephemeris
+```
+To obtain the list of tools from the [usegalaxy.eu](https://usegalaxy.eu) use:
+```
+get-tool-list -g "https://usegalaxy.eu" -o "eu_tool_list.yaml"
+```
+and then to install them:
+```
+shed-tools install -g https://your-galaxy -a <api-key> -t eu_tool_list.yaml
+```
+The api key can be found in the User -> Preferences -> Manage API Key in Galaxy menu.
+
+...
+
+Later we should use this [role](https://github.com/galaxyproject/ansible-galaxy-tools) to automate this, but there are too many tools to be installed and it is very inpractical for testing purposes. 
+
+## Debian 11
+
+### TUS
+
+On ubuntu < 22.04 and Debian < 12 the GLIBC version is incompatible with the ```galaxyproject.tusd``` GLIBC, therfore playbook ends in error.
+- in `roles/galaxyproject.tusd/defaults/main.yml` you have to set the `tusd_version:` to 'v1.8.0' (tested, possibly higher)
+
+### Ansible
+
+On Debian 11 the default version of `ansible` is 2.10.08 which fails to extract binaries for TUSd.
+- workaround:
+  - because of Puppet, we need to create dedicated sources.list file for ansible
+  - `sudo vim /etc/apt/sources.list.d/ansible_ubuntu_focal_main.list` add this line : `deb http://ppa.launchpad.net/ansible/ansible/ubuntu focal main`
+  - then `sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 93C4A3FD7BB9C367`
+  - followed by `sudo apt-get update` and `sudo apt-get install ansible -y`
+- Note that you might need to `sudo apt remove ansible` `sudo apt autoremove` before you try to install ansible.
+
+### Nginx Upload Module
+
+Our playbook makes use of `nginx-upload-module`, which is not included by default in any nginx-* distribution from `apt`.
+This is a simple recipe for installing nginx with the upload-module.
+
+```
+sudo apt remove *nginx* -y &&
+sudo apt autoremove -y &&
+
+wget http://nginx.org/download/nginx-1.24.0.tar.gz &&
+tar -xzf nginx-1.24.0.tar.gz &&
+cd nginx-1.24.0/ &&
+git clone https://github.com/hongzhidao/nginx-upload-module.git &&
+sudo apt-get install -y libpcre3 libpcre3-dev libssl-dev zlib1g-dev &&
+./configure --user=nginx --group=nginx --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/sbin/nginx --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_sub_module --add-module=nginx-upload-module &&
+make && make install
+```
+
+Note that you might need to `sudo systemctl unmask nginx.service` and remove the `galaxy` and `redirect-ssl` from `/etc/nginx/sites-available and `/etc/nginx/sites-enabled` before running the playbook again.
+
+### Other issues
+
+
+Use `sudo apt install python3-docker` before running the `usegalaxy.rabbitmqserver` role.
+
+For influxdb apt key error: https://www.influxdata.com/blog/linux-package-signing-key-rotation/
+
+For proftpd add "LoadModule mod_tls.c" to roles/galaxyproject.proftpd/templates/tls.conf.j2 just before the TLSEngine 

files/repeatexplorer-elixir.cerit-sc.cz/all_tools.yml

@@ -0,0 +1,128 @@
+# DEMON: This file is a copy from usegalaxy.cz, it wasn't adjusted to the needs of RE yet
+name: Tools
+type: generic
+id: all_tools
+items:
+- sections:
+  - upload_file
+  - get_data
+  - data_source_tools
+  - send_data
+  - collection_operations
+  - expression_tools
+- id: general_text_label
+  text: General Text Tools
+  type: label
+- sections:
+  - text_manipulation
+  - filter_and_sort
+  - join__subtract_and_group
+  - convert
+  - convert_formats
+  - builtin_converters
+- id: genomic_manip_label
+  text: Genomic File Manipulation
+  type: label
+- sections:
+  - fasta_fastq
+  - fastq_quality_control
+  - quality_control
+  - sam_bam
+  - bed
+  - vcf_bcf
+  - nanopore
+  - lift_over
+  - extract_features
+  - operate_on_genomic_intervals
+  - fetch_sequences___alignments
+  - chromosome_conformation
+- id: genomics_analysis_label
+  text: Genomics Analysis
+  type: label
+- sections:
+  - assembly
+  - annotation
+  - mapping
+  - variant_calling
+  - rna_analysis
+  - multiple_alignments
+  - hgv
+  - evolution
+  - regional_variation
+  - biodiversity_data_exploration
+  - genome_diversity
+  - rna_seq
+  - peak_calling
+  - genome_editing
+  - epigenetics
+  - phylogenetics
+  - phenotype_association
+  - single_cell
+  - hca_single_cell
+  - hca-single_cell
+  - single-cell
+- id: stats_viz
+  text: Statistics and Visualization
+  type: label
+- sections:
+  - statistics
+  - machine_learning
+  - plots
+  - interactive_tools
+  - graph_display_data
+- id: genomics_toolkits_label
+  text: Genomics Toolkits
+  type: label
+- sections:
+  - mothur
+  - qiime
+  - qiime_2
+  - picard
+  - deeptools
+  - emboss
+  - ncbi_blast
+  - mimodd
+  - scanpy
+  - hicexplorer
+  - gemini
+  - motif_tools
+  - gatk_tools
+  - rad_seq
+  - sanger_sequencing
+  - dna_metabarcoding
+  - apollo
+- id: domain_tools_label
+  text: Domain Tools
+  type: label
+- sections:
+  - virology
+  - metagenomic_analysis
+  - proteomics
+  - metabolomics
+  - muon_spectroscopy
+  - chemicaltoolbox
+  - astronomy
+  - imaging
+  - climate_analysis
+  - gis_data_handling
+  - spatial_omics
+  - ecoregionalization
+  - graphclust
+  - species_abundance
+  - data_and_metadata_management
+  - obo_ontology_manipulation
+  - ontology
+  - compute_indicators_for_satellite_remote_sensing
+  - compute_indicators_for_turnover_boulders_fields
+  - animal_detection_on_acoustic_recordings
+# - id: other_tools_label
+#   text: Other Tools
+#   type: label
+- sections:
+  - other_tools
+# - id: testing_tools_label
+#   text: Testing Tools
+#   type: label
+- sections:
+  - local_tools
+  - test_tools

files/repeatexplorer-elixir.cerit-sc.cz/pulsar.pub

@@ -0,0 +1 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPJ4pNj4pZBfm/hGSD04ZkuxtCbageNxwSD6EAZ+wyjQ [email protected]