
GITHUB . COM {
}
Detected CMS Systems:
- Wordpress (2 occurrences)
Title:
Multi-core trimming with info file using all available RAM. Β· Issue #524 Β· marcelm/cutadapt
Description:
Hi Marcel, I hope that you are well! Previously, I have used version 1.14 of Cutadapt to trim and generate INFO files which has worked great on a single core. More recently, I have been attempting to implement the multi-core options to i...
Website Age:
17 years and 8 months (reg. 2007-10-09).
Matching Content Categories {π}
- Technology & Computing
- Video & Online Content
- Telecommunications
Content Management System {π}
What CMS is github.com built with?
Github.com operates using WORDPRESS.
Traffic Estimate {π}
What is the average monthly size of github.com audience?
ππ Tremendous Traffic: 10M - 20M visitors per month
Based on our best estimate, this website will receive around 10,000,019 visitors per month in the current month.
However, some sources were not loaded, we suggest to reload the page to get complete results.
check SE Ranking
check Ahrefs
check Similarweb
check Ubersuggest
check Semrush
How Does Github.com Make Money? {πΈ}
Subscription Packages {π³}
We've located a dedicated page on github.com that might include details about subscription plans or recurring payments. We identified it based on the word pricing in one of its internal links. Below, you'll find additional estimates for its monthly recurring revenues.How Much Does Github.com Make? {π°}
Subscription Packages {π³}
Prices on github.com are in US Dollars ($).
They range from $4.00/month to $21.00/month.
We estimate that the site has approximately 4,989,889 paying customers.
The estimated monthly recurring revenue (MRR) is $20,957,532.
The estimated annual recurring revenues (ARR) are $251,490,385.
Wordpress Themes and Plugins {π¨}
What WordPress theme does this site use?
It is strange but we were not able to detect any theme on the page.
What WordPress plugins does this website use?
It is strange but we were not able to detect any plugins on the page.
Keywords {π}
marcelm, cutadapt, ram, info, core, sign, multicore, file, issue, jburgess, files, single, progress, runs, commented, memory, content, navigation, pull, requests, actions, security, trimming, closed, version, worked, great, run, python, install, mamba, command, line, bar, reads, consumption, running, linux, consuming, ill, owner, processes, github, labels, projects, milestone, footer, skip, menu, product,
Topics {βοΈ}
multi-core command line multi-core options marcelm edits owner single core runs fastq --info-file=outfile marcelm closed comment metadata assignees info -o4 infile improve run times generate info files info files produced progress bar displaying progress bar halts single core conda install worked great 32 core machine content cutadapt version 3 info file consuming ~25gb output files run overnight python processes core input fastq cutadapt cutadapt steadily increase machines limit resource consumption htop showed worked fine time wanted handle compression rapid response assigned labels labels projects projects milestone milestone relationships 48gb file 2m reads github memory leak ram consumption outfile fastq progress version 1 python 3
Payment Methods {π}
- Braintree
Questions {β}
- Already have an account?
- Are you running this on Linux or macOS?
- Is it actually the Python processes that consume the memory, or does that come from a subprocess?
Schema {πΊοΈ}
DiscussionForumPosting:
context:https://schema.org
headline:Multi-core trimming with info file using all available RAM.
articleBody:Hi Marcel, I hope that you are well!
Previously, I have used version 1.14 of Cutadapt to trim and generate INFO files which has worked great on a single core. More recently, I have been attempting to implement the multi-core options to improve run times.
* Versions:
+ Cutadapt version 3.4. & Python 3.6.13
* Installation:
+ conda install -c conda-forge mamba
+ mamba install -c bioconda cutadapt
* Command line:
+ `cutadapt -g <adapter> -j 10 -o outfile.fastq --info-file=outfile.info -O4 infile.fastq.gz >> outfile.stats`
Everything seems to be processing normally, I can see the output files being generated and the progress bar displaying how many reads have been processed thus far. However, for every core the RAM consumption seems to steadily increase until reaching my machines limit and the progress bar halts but does not exit with an error. This is running on Linux with a 32 core machine and 252Gb of RAM. Monitoring the resource consumption with htop showed that the 10 cores were consuming ~25Gb of RAM each after leaving it to run overnight and checking on the progress in the morning which had frozen.
The input FASTQ is ~48Gb and the INFO files produced from the single core runs were ~250Gb. I do not remember the single core runs consuming too much RAM but I will check that again. I've performed the same multi-core command line on a subset of 2M reads from the 48Gb file and it worked fine. I'll keep trying to find a solution but in the mean time wanted to post this. Thanks!
Kind regards,
James
author:
url:https://github.com/J-Burgess
type:Person
name:J-Burgess
datePublished:2021-04-15T12:05:20.000Z
interactionStatistic:
type:InteractionCounter
interactionType:https://schema.org/CommentAction
userInteractionCount:3
url:https://github.com/524/cutadapt/issues/524
context:https://schema.org
headline:Multi-core trimming with info file using all available RAM.
articleBody:Hi Marcel, I hope that you are well!
Previously, I have used version 1.14 of Cutadapt to trim and generate INFO files which has worked great on a single core. More recently, I have been attempting to implement the multi-core options to improve run times.
* Versions:
+ Cutadapt version 3.4. & Python 3.6.13
* Installation:
+ conda install -c conda-forge mamba
+ mamba install -c bioconda cutadapt
* Command line:
+ `cutadapt -g <adapter> -j 10 -o outfile.fastq --info-file=outfile.info -O4 infile.fastq.gz >> outfile.stats`
Everything seems to be processing normally, I can see the output files being generated and the progress bar displaying how many reads have been processed thus far. However, for every core the RAM consumption seems to steadily increase until reaching my machines limit and the progress bar halts but does not exit with an error. This is running on Linux with a 32 core machine and 252Gb of RAM. Monitoring the resource consumption with htop showed that the 10 cores were consuming ~25Gb of RAM each after leaving it to run overnight and checking on the progress in the morning which had frozen.
The input FASTQ is ~48Gb and the INFO files produced from the single core runs were ~250Gb. I do not remember the single core runs consuming too much RAM but I will check that again. I've performed the same multi-core command line on a subset of 2M reads from the 48Gb file and it worked fine. I'll keep trying to find a solution but in the mean time wanted to post this. Thanks!
Kind regards,
James
author:
url:https://github.com/J-Burgess
type:Person
name:J-Burgess
datePublished:2021-04-15T12:05:20.000Z
interactionStatistic:
type:InteractionCounter
interactionType:https://schema.org/CommentAction
userInteractionCount:3
url:https://github.com/524/cutadapt/issues/524
Person:
url:https://github.com/J-Burgess
name:J-Burgess
url:https://github.com/J-Burgess
name:J-Burgess
InteractionCounter:
interactionType:https://schema.org/CommentAction
userInteractionCount:3
interactionType:https://schema.org/CommentAction
userInteractionCount:3
External Links {π}(2)
Analytics and Tracking {π}
- Site Verification - Google
Libraries {π}
- Clipboard.js
- D3.js
- Lodash
Emails and Hosting {βοΈ}
Mail Servers:
- aspmx.l.google.com
- alt1.aspmx.l.google.com
- alt2.aspmx.l.google.com
- alt3.aspmx.l.google.com
- alt4.aspmx.l.google.com
Name Servers:
- dns1.p08.nsone.net
- dns2.p08.nsone.net
- dns3.p08.nsone.net
- dns4.p08.nsone.net
- ns-1283.awsdns-32.org
- ns-1707.awsdns-21.co.uk
- ns-421.awsdns-52.com
- ns-520.awsdns-01.net