View source for SHORE Overview

= What is SHORE and what is SHORE for? =

SHORE is a data analysis and management application for short DNA/RNA reads produced by the various contemporary sequencing platforms. It is developed at the Max Planck Institute for Developmental Biology, Tübingen, Germany.

SHORE is designed to support different sequencing applications including [[shore consensus|genomic re-sequencing]], [[shore peak|ChIP-Seq]], [[shore count|mRNA-Seq]], [[shore srna|sRNA-Seq]] and [[shore methyl|BS-seq]]. SHORE aims to provide an end-to-end sequencing data analysis pipeline providing modules for all necessary steps starting from raw read data processing up to primary analysis results.

Available modules include [[shore import|quality filtering]] of raw reads, adapter clipping for sRNA-seq, [[shore mapflowcell|mapping of reads]] against a reference genome, [[shore correct4pe|correction for read pair information]] and quantitative analysis of ChIP or transcriptome data.

Several [[shore convert|file format converters]] ensure the compatibility of SHORE with other widely used file formats like fastq, qual, SAM/BAM, BED and GFF. SHORE was developed for applications in ''Arabidopsis thaliana'' but has been successfully used with other genomes, including human, mouse, ''D. melanogaster'', ''C. elegans'', maize and several bacterial genomes.

= The SHORE main program =

SHORE is a command line application consisting of a variety of [[subprograms]] which are invoked through the top-level '''shore''' command. For example, the command [[shore import]] calls the initial read data import and filtering subprogram.

When called without further command line arguments from a terminal, both '''shore''' and its subordinate commands will display a basic help page. For many subprograms, more detailed information is further available through the command '''shore help <command>''', e.g. '''shore help import'''.

=== Help page and command line options ===

The help page displayed by commands is a three-column table providing the respective command line option, the option's default value enclosed in parenthesis as well as a description. For example,
 -a, --application=STRING             (=genomic)         Applications: genomic, mRNA, ChIPseq or sRNA
indicates the option may be specified either as ''-a <value>'' or as ''--application=<value>''. The default value if the option is not specified by the user is ''genomic'', indicated by the middle column; further values accepted as arguments to the option are ''mRNA'', ''ChIPseq'' and ''sRNA''. Special values displayed in the default value column ''(not set)'' and ''(auto)'' indicate the option has no default value, or that the option's default value depends on the specification of further options, respectively.

Many command line options accept a list of arguments, indicated by suffixes '''[,...]''' or '''[:...][,...]''' to the help page's first column:

  -P, --input-paths=STRING[,...]       (not set)          Import shore directories or arbitrary read files (*)
  -x, --reads-fastq=STRING[:...][,...] (not set)          List(s) of fastq files

The ''--input-paths'' option for example may be specified either multiple times, or with a comma-separate list of files:
 shore import -P batch1.fl -P batch2.fl ...

 shore import -P batch1.fl,batch2.fl ...

The suffix '''[:...][,...]''' indicates that the option's argument list is allowed to be 2-dimensional, with the inner grouping indicated by colon-separated values, e.g.:
 shore import -x batch1_1.fq:batch1_2.fq:batch1_3.fq -x batch2_1.fq:batch2_2.fq:batch2_3.fq ...

 shore import -x batch1_1.fq:batch1_2.fq:batch1_3.fq,batch2_1.fq:batch2_2.fq:batch2_3.fq ...

=== Global command line options ===

Several parameters applicable to all SHORE subprograms can be set globally though options to the main program. These main program options must be specified before the respective subordinate command, e.g. '''shore --tmpdir=/large_tmp mapflowcell ...'''. The following options may be set globally:

{|
 |----
 | ''-u'', ''--umask=STRING''              || ''(auto)''          || Force a different file creation mask (octal value)
 |----
 | ''-T'', ''--tmpdir=STRING''             || ''(auto)''          || Temporary directory
 |----
 | ''-Z'', ''--compression=STRING''        || ''(=xz)''           || Set the default compression format (''plain'' or ''gzip'' or ''xz'')
 |----
 | ''-G'', ''--random-access=STRING''      || ''(=fast)''         || Random access granularity for compressed output files. Slower random access allows for better compression: ''fast'' or ''medium'' or ''slow''
 |----
 |''-C'', ''--config=STRING''              || ''(=col-0)''        || Load an alternate set of default values
 |}

The ''--umask'' option may be used to grant other users access to all output files generated by shore.
For the meaning of the three-digit values, see the system manual page of the ''chmod'' command by typing '''man chmod'''.
By default, the option's value is determined by  your system configuration.

Many SHORE commands may create large files with temporary data that are removed after the command completes.
The directory that these [[temporary files]] are created in may be specified explicitly through the ''--tmpdir'' option.

The option ''--compression'' may be used to specify the compression format for output files created by shore. By default, output files are compressed using the [http://tukaani.org/xz/ XZ] file format.
Specifying the value ''gzip'' will use [http://www.gzip.org/ GZIP] compression for created files, resulting in faster write performance, but larger output files. The value ''plain'' may be used to completely disable file compression.

With the ''--random-access'' option, users may adjust the trade-off between random access performance on generated files on the one hand, and compression ratio on the other hand.
By default, SHORE's output files are optimized to allow fast retrieval of data from any offset into the file, e.g. through the query facilities provided by the [[shore sort]] and [[shore 2dex]] utilities.
The cost is however a slight increase in output file size.
Users may adjust for improved compression and worse random access performance using ''--random-access=medium'' or ''--random-access=slow''.

The ''--config'' option allows to specify complete set of option default values, as described in the following.

=== Configuring option default values ===

For convenience, default values for the shore main program and subordinate commands may be pre-configured via a configuration file.
Default values are configured by creating the file
 $HOME/.config/shore/default.cfg

The program or programs that a section of the configuration file refers to must be specified in square brackets;
options are specified by their name as displayed on the help page without the introductory minuses:

 [shore]
 # Use the 'medium' setting for the --random-access option instead of the default 'fast'.
 random-access=medium
 # Create temporary data in the directory '/large_tmp' by default.
 tmpdir=/large_tmp
 
 [shore mapflowcell]
 # Use TAIR10 as the default reference sequence for shore mapflowcell.
 index-file=~/Genomes/SHORE/TAIR10/TAIR10.fa.shore
 # Always generate the statistics plot.
 rplot=yes
 
 [shore consensus,shore qVar]
 # Command line options without a --option variant must be enclosed in single quotes.
 'f'=~/Genomes/SHORE/TAIR10/TAIR10.fa.shore

The ''--config'' option to the SHORE main program allows specification of an alternate set of default values.
E.g.
 shore -C col-0 mapflowcell ...

will attempt to use option default values specified in a file
 $HOME/.config/shore/col-0.cfg

These defaults are valid only throughout the respective run of a subprogram.

= Overview of SHORE’s data structure =
SHORE structures directories and files in several predefined ways. One central directory, referred to as ''IndexFolder'', stores all information about a reference sequence including the indexes (created and used by mapping tools), GC content and low complexity sequence analysis. A second directory, referred to as ''ProjectFolder'', contains all information about a certain sequencing project.
The ''IndexFolder'' is kept separate from the ''ProjectFolder'', because one ''IndexFolder'' can be reused for multiple projects, e.g., different re-sequencing projects (''ProjectFolders'') can refer to the same reference sequence (''IndexFolder'').
The ''IndexFolder'' is populated using ''[[shore preprocess]]''. It will store a fasta file named <original_fasta_file>.shore. This file is used as prefix to find all needed index files for SHORE.
It is not necessary to keep the original fasta file.

The ''ProjectFolder'' contains one or more ''RunFolders'', which are also organized as a predefined directory hierarchy. This is where the actual [[SHORE file formats|read and alignment data]] are stored.
Although many [[SHORE Subprograms|SHORE subprograms]] can also be run stand-alone for single files, this partitioning is required to take full advantage of all SHORE features.
However, this should not be too confusing, as all directories are created automatically by SHORE.

All files in the ''RunFolders'' are ASCII formatted text files and can easily be viewed, moved and manipulated.

It is also possible to use SHORE e.g. just for read mapping and then [[shore convert|convert]] the [[map.list|SHORE alignments]] into a different file format (e.g. SAM for SNP detection with SAMtools).

=== The RunFolder and the read data ===
Directory structure of the ''RunFolder'' for read storage:

 RunFolder/LaneFolder[/SampleFolder]/ReadFolder[/LengthFolder]

 Note: Many [[SHORE subprograms]] accept ''shore directories'' as input; this term is used to refer to any of ''RunFolder'', ''LaneFolder'', ''SampleFolder'' and ''ReadFolder''.

The ''SampleFolder'' and ''LengthFolder'' levels are optional and will be automatically created where required.

{|
| ''RunFolder''/    || A project starts with the sequencing of one or more flowcells. Each flowcell is stored in a separate directory, which is named by the user.
|----
| ''LaneFolder''/   || An Illumina flowcell consists of one to eight lanes, each of them represented by a single directory, named by a single digit. For technologies other than Illumina, this will always be called ''1''. An additional directory called ''filtered'' will be created containing all reads which did not pass the quality filter.
|----
| ''SampleFolder''/ || This directory level will only be created when using bar codes for de-multiplexing multiple samples. Directories are named ''sample_<user_defined_label>/''.
|----
| ''ReadFolder''/   || The ReadFolders separate the read pairs, in case of paired-end or mate-pair data. A directory ''1'' contains the reads from the first sequencing run and in ''2'' the reads from the second, respectively. In ''single'' are reads from the first and from the second run which lost their read partner in the quality filtering process. For a single-end runs, all reads passing the quality filter will be stored in ''single''.
|----
| ''LengthFolder''/ || SHORE can optionally sort the reads according to their length, e.g. when trimming away bad quality sequence or sequencing adapters. These directories are named ''length_<read_length>/''. Created by default for small RNA only.
|}

For example, a directory could be named like:
 
 EAS67_0018_FC12398AAXX/5/sample_phiX/single/length_80