In recent time I've been asked for pointers on analysing complex healthcare data. This is a difficult issue. Healthcare analytics / health informatics / medical informatics / etc. range over a wide area, driven a wide variety of interests and outcomes, overlapping hugely in some areas and not at all in others. The …
While mentoring some juniors, I started to think about the rules of thumb for analysing data that I've built up over the years. While I'm certainly not the world's greatest data scientist (or it's greatest bioinformatician, statistician, biomedical scientist, etc.), it seems worthwhile trying to capture them here. these are …
Last updated 2020/4/4. Originally published on LinkedIn
I won't be updating this list further as the intention was to point people to the right places to help in a crisis.
Should you have an potentially useful information set or data source, the best course of action is you …
For those who don’t know, EvolDir is a worldwide mailing list on evolutionary biology that has been running since approximately forever. Everyone who works even vaguely in the area of evolution subscribes to it. Every day It typically carries several posts on conferences, book announcements, funding opportunities and job …
You might have come across confusing statements like "octopuses have arms, squids have tentacles" and wondered what's the difference. I did and here's the (unsatisfactory) answer.
Cephalopods (squid, octopus, nautilus and a number of other aquatic creatures, which are a class of mollusc) have a number of muscular "limbs". Traditionally …
It's a common cry amongst working bioinformaticians that they're unappreciated, undervalued and generally "get no respect". While people love to complain, from discussions with peers and colleagues, the same stories come up again and again:
This is based upon my reply to a question on reddit concerning experiences with using Amazon Web Services (including ELastic Computing, Glacier, etc.) for research.
I was part of a pan-European research consortium that used it for our shared computing infrastructure: databases, a few web apps and web sites, mailers …
Refer back to the previous article for the background and some introduction to REDCap.
Note
These notes are based upon REDCap version 6.4.4. Different versions may have fixed or adjusted some of this behaviour.
To add a user, you need an email address to send a …
Maintain employable, valuable skills. Never evince any technical skills. Never casually offer to help a colleague with computer problems, etc. First you'll end up being known for that. Second, people will keep asking you to encrypt their hard-drive, build a database, etc. Third, technical matters are low status "working class …
Some years ago I wrote a popular piece Words a bioinformatician doesn't want to hear. Sadly, a lot of practitioners found it all too real. This time, I'm concentrating on the data end of things.
"We really need a database / webapp, so I can query / analyse / visualise this data!"
[database …
Some years ago, I got into an argument with someone abut the relative merits of Bayesian versus Maximum Likelihood in phylogenetics. They asserted the two were basically the same or would come to the same answers. I countered that while they would often agree, they were measuring different things …
For those not in the know, REDCap is a platform for creating and editing databases through the web. And by and large, it works fine. It saves a lot of development effort. It provides good reporting tools for users. It's secure and robust. But there are some things to be …
If you're doing reproducbility in R Studio, you're probably using knitr. And if you're using knitr, you're probably using Markdown. Unfortunately, due to the lack of a standard for Markdown (and the subsequent proliferation of various flavours and extensions), it's sometimes not clear what syntax is available to you. Consequently …
You have a bunch of related CSV files.
Maybe they're the result of a raw database dump. Maybe they've been generated in some other way: experimental results, various public data sets, whatever. But the important thing is that you need to make a database from them. Perhaps because …
(This first appeared on biocodershub.net courtesy of Rad, and has since popped up on coderscrowd.com. It enjoyed some moments of viral popularity, with many aggrieved practitioners chipping in on the comments of the article. Following the resurrection of my website, it's a good opportunity to bring this piece …
Prompted by a recent tweet asking what people used for storing and managing their data, I wrote down my own hard-won lessons on the topic. In rough order of preference and data complexity:
Or markdown / asciidoc. The advantages of this being:
No matter how many drafts you go through, the number of helpful suggestions make by your co-authors will approach but never quite reach zero.
Distinguished or influential co-authors have a tendency to invite, introduce or insist upon …
(Originally published on BiocodersHub.)
If you’re doing bioinformatics in Python, you’re probably using Biopython. Actually, Biopython is a good reason for using Python. But it can be formidable to newcomers: there’s a lot there and there’s not a huge amount of learning material. This then is …
knitr is a useful R package/tool for documenting analysis. Basically, it allows the embedding of R code "chunks" within a simple text document. This document can then be "knitted", which means that the R code is interpreted and reinserted in the document along with the results of that code …
It's all there in the documentation, but sometimes it's hard to find. This document gives you another place to look.
So how do I ...
Curiously, the identity of admin users is hardcoded into the Galaxy configuration file. (Which makes it secure, I guess, but separate from the …
Problems with building qjoin and getting it to read stockholm files.
Quickjoin / qjoin is an excellent commandline program for rapid construction of neighbour-joining trees. However, while using it recently, I had a few problems getting it to read Stockholm files, the most accessible of the formats it can use.
The …
Galaxy toolsheds
Relatively painless tool-sharing
This is a more recent innovation in Galaxy, which can make it a somewhat confused one: the concept of the toolshed has changed over its lifetime, the documentation is incomplete, and there's a slightly strange emphasis in the documentation that exists. So …
Toolsheds …
(Originally published on BiocodersHub)
Following several lengthy and passionate discussions in different venues on what language to use for teaching bioinformatics, I've started cutting and pasting my reply. And here it is.
You'll get a lot of different opinions on this because:
(Previously published on BiocodersHub.)
Previously I'd spoken about how Biopython represents sequence data with the Seq class. But there is also the SeqRecord class:
A simple script to check and purge sequence files of possible problems.
Some times you need sequences that are unambiguous (i.e. only 'ACGT', lacking gaps) whether it's because of the limitations or assumptions of tools (like omegaMap) or just because you want to know where SNPs or sequencing ambiguities …
The need had arisen for the tips of a large phylogeny to be labelled in a systematic way. Rather than "point and click" within Dendroscope, this script takes a .den/dendro file and colors the tips according to a "color description" file. This is a simple csv file with taxa …
In BioRuby, alignments are equipped with several methods for obtaining consensus sequences. Unfortunately, these have terse descriptions which point you at the BioPerl documentation, with the added bonus of not quite working like the BioPerl equivalents.
First, let's create a very simple alignment, where everything agrees except the last sequence …
Odds and ends and the surprising.
If you are serving the installation with a proxy redirect (e.g. the galaxy server is running on port 7070 but is being redirect by Apache to appear at port 80 on /galaxy), while you can access Galaxy at both addresses, login will …
Some (more) notes about the venerable Bayesian reconstruction program.
When attempting to execute a Nexus file, MrBayes kept spitting back this cryptic error upon loading:
Executing file "c_vp1_nuc_seqs.nxs" [...] Reading data block Allocated matrix [...] Data is Dna Gap character matches matching or missing characters …
More things I done learned about Galaxy tool development
A pot-pouri of titbits that are probably documented somewhere, but weren't obvious to me.
The sudden death of Ross Crozier on the 12th of November was heralded largely by a slow ripple of email, phone calls and Facebook messages across the globe. I found out from an email that started with a short but singularly complete sentence:
Terrible news.
It is sobering to think …
You could spend all day just keeping up with developments in next-generation sequencing. Companies announce new and revolutionary technologies seemingly every month, promising to do more, better and for less. Yet at the same time, it’s difficult to hack your way through the marketing tallk and get hard figures …
Sequence logos are a common way of representing SNPs and diversity in groups of sequences. This script automates the task. It's a bit rough around the edges and serves mainly as a base for further hacking.
Usage is:
drawlogo.rb [options] FILE1 [FILE2 ...]
where options are:
-h, --help Display this …
A largely self-explanatory script. This will "shrink" an alignment, deleting all sites that don't contain a polymorphism in some member sequence. A little bit of script candy as well, this takes any number of files and saves the results in a new file named according to a definable schema:
#!/usr …
jsPhyloSVG (or JPS from this point) is a nifty Javascript library for displaying phylogenetic trees in your browser. It can:
The Ordinance Survey is a UK-peculiar geospatial format, ubiquitous via street atlases, hiking charts and (yes) farming and epidemiological maps. It is explained in great detail is several places, but here's a quick overview:
The OS grid is a set of 25 squares, 500 kilometers a side, arranged 5-by-5 that …
If this happens to you when trying to compile MrBayes:
% make gcc -DUNIX_VERSION -DUSE_READLINE -O3 -Wall -c -o mb.o mb.c gcc -DUNIX_VERSION -DUSE_READLINE -O3 -Wall -c -o mcmc.o mcmc.c gcc -DUNIX_VERSION -DUSE_READLINE -O3 -Wall -c -o bayes.o bayes.c bayes.c:45:31: readline/readline …
This just wraps the BioRuby fetch functionality in a friendly commandline interface. In brief, it can accept accession ids on the commandline or from a piped file (one accession per line) and save the corresponding sequences from the db. Sequences may be downloaded via the bioruby or EBI servers. The …
This presents one way to create an optimized production Galaxy instance. Variations are certainly possible and some of the choices presented are/were dictated by local culture. Certain settings may be more suitable for production or development environments. Nonetheless, this presents a start-to-stop process for installation and setup.
Note: this …
For when you have to do lots to a big tree.
Previously, I showed how Dendroscope files can be easily manipulated with brute-force regex, so you can right scripts to color a mass of nodes, rather than having to format them one-by-one in the GUI. However, more complex manipulations require …