Skip to content

HCP Pipelines Improvements (ICAFIX, PostFix, RestingStateStats, TaskfMRIAnalysis, londitudinal/cross-sectional logic) #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 82 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
ce2c6db
first commit to github.umn.edu of bids apps hcppipeline app
tjhendrickson Mar 28, 2018
56f01cb
made changes to run.py in order to account for my BIDS infrastructure
tjhendrickson Mar 29, 2018
8f46b5b
last commit prior to removing branch
tjhendrickson Apr 3, 2018
53d615b
Added IntendedFor.py to create IntendedFor field if it does not exist…
tjhendrickson Apr 3, 2018
3b23b48
Merge branch 'dev' of github.umn.edu:hendr522/HCPPipelines into extend
tjhendrickson Apr 3, 2018
4f98e77
added IntendedFor scrit to run.py so it will take care of adding Inte…
tjhendrickson Apr 3, 2018
8ac3dad
changed run.py for Dockerfile implementation
tjhendrickson Apr 3, 2018
6c61788
added intendedfor script to run.py to add intendedfor field if it doe…
tjhendrickson Apr 6, 2018
bbab867
changed set_fieldmap with return_list=True to allow for multiple fiel…
tjhendrickson Apr 10, 2018
b91946f
merged changes from extend to allow for a list of fieldmaps
tjhendrickson Apr 10, 2018
4696caa
made changes to IntendedFor.py for user determined fudge factor betwe…
tjhendrickson Apr 10, 2018
197e040
added /bin/bash -c to entrypoint to fix bit set problem with udocker
tjhendrickson Apr 12, 2018
937bc62
fixed conflicts between local and remote branch
tjhendrickson Apr 12, 2018
35e86d3
remove entrypoint command so udocker will run
tjhendrickson Apr 12, 2018
ae79bb6
remove entrypoint command so udocker will run
tjhendrickson Apr 12, 2018
8b5e288
merged conflicts with master
tjhendrickson Apr 12, 2018
3f0c1ea
added diffusion pieces from jokedurnez fork commit 2c44cc0f6b
tjhendrickson Apr 12, 2018
6378f36
Added function to run ICA-FIX processing stream
tjhendrickson Apr 12, 2018
dc33ab0
install FIX along with dependencies to Dockerfile
Apr 13, 2018
5fbf323
added FIX along with dependencies such as R to dockerfile in order to…
tjhendrickson Apr 13, 2018
d52473d
added install for pcre for R install
tjhendrickson Apr 13, 2018
f47ac58
made several changes to download and incorporate FIX into container
tjhendrickson Apr 26, 2018
16861b9
created a TODO to determine input_file variable for ICAFIX
tjhendrickson Apr 26, 2018
fdc426f
ensure clean working tree
tjhendrickson Apr 27, 2018
ab32ac1
added v0.1 to master for versioning
tjhendrickson Apr 27, 2018
1cc0c75
added if-else statement to run pipelines when session labels are invo…
tjhendrickson May 1, 2018
5118fd3
changes made to run.py to ensure session label conditional would work
tjhendrickson May 1, 2018
eaead66
Merge branch 'extend' into master
tjhendrickson May 2, 2018
07d8b30
Merge pull request #2 from hendr522/master
tjhendrickson May 2, 2018
8b06f71
Finished writing ICAFIX portion of pipeline, can be tested
tjhendrickson May 2, 2018
1460c75
merged branch extend with master
tjhendrickson May 2, 2018
41efa1c
made changes within run.py so data outputted is owned by original use…
tjhendrickson May 3, 2018
47199c3
change how subject ID and how subject path is determined for session …
tjhendrickson May 3, 2018
64a75ba
added reptime variable so ICAFIX will work
tjhendrickson May 4, 2018
a189963
highpass variable was not being found, needed to change scope
tjhendrickson May 4, 2018
7bd1b56
changed if zooms and reptime statment
tjhendrickson May 4, 2018
cc17817
changed scope offmri processing call
tjhendrickson May 4, 2018
2275f9b
changed reptime to float
tjhendrickson May 4, 2018
2179add
made sure that output folders will be created if they don't exist alr…
tjhendrickson May 4, 2018
a16f9b3
changed DockerFile for singularity compatiability and updated version
tjhendrickson May 23, 2018
3c6f2ef
adding fsl_sub capability into image
May 29, 2018
0f83447
modified fsl_sub so queue names are generated
tjhendrickson May 29, 2018
36193db
added fsl_sub to DockerFile
tjhendrickson May 29, 2018
bf9ea23
changed environment variable FSL_FIX from opt/fix to /opt/fix so path…
tjhendrickson May 29, 2018
f396d16
changed R versions so FIX would run properly
tjhendrickson May 29, 2018
fbf253a
changed R versions so FIX would run properly
tjhendrickson May 29, 2018
31dbef0
ensured that dependencies were met to run PostFix (i.e. MCR version v81)
tjhendrickson May 29, 2018
d338bc8
Finished PostFix call within run.py. Execution testing needed. Began …
May 30, 2018
7bc3a05
remove SetUpHCPPipeline.sh from run.py
May 30, 2018
2890771
removed unneeded files and commented out sections in progress within …
tjhendrickson May 31, 2018
f27050f
changes to DockerFile so environment is set up correctly
tjhendrickson May 31, 2018
2c50592
made changes to run.py and DockerFile to include RestingStateStats
tjhendrickson Jun 1, 2018
3af4730
Delete mri_convert
tjhendrickson Jun 1, 2018
25ee198
Delete mri_convert.bin
tjhendrickson Jun 1, 2018
9401eed
Delete sources.sh
tjhendrickson Jun 1, 2018
dd20804
Delete SetUpHCPPipeline.sh
tjhendrickson Jun 1, 2018
4292ebc
Delete MCR_installer_input.txt
tjhendrickson Jun 1, 2018
873f47e
changed identation of posData within run.py
tjhendrickson Jun 1, 2018
cf17e6a
Merge branch 'master' of https://github.umn.edu/hendr522/HCPPipelines
tjhendrickson Jun 1, 2018
782df47
made changes to README.md to reflect current capabilities of release
tjhendrickson Jun 1, 2018
86e10e2
Update README.md
tjhendrickson Jun 2, 2018
943afa0
Update README.md
tjhendrickson Jun 2, 2018
856e95c
Update run.py
tjhendrickson Jun 2, 2018
b939175
playing around with R packages to meet FIX dependencies
tjhendrickson Jun 4, 2018
dd17549
finished testing of ICAFIX, RestingStateStats, and PostFix. Additiona…
tjhendrickson Jun 5, 2018
5401cc6
moved IntendedFor.py back to top level directory so it would run prop…
tjhendrickson Jun 6, 2018
8b5c071
removed pdb.set_trace() commands from run.py
tjhendrickson Jun 6, 2018
9ac1544
made a small change to run.py, for resting state processing use fmrin…
tjhendrickson Jun 6, 2018
a8675c3
finish out session label vs subject label if else condition. Addition…
tjhendrickson Jun 7, 2018
35cbd7d
session label logic was incorrect resulting in failed processing for …
tjhendrickson Jun 7, 2018
ad5fc48
created TaskfMRIAnalysis directory within container
Jun 11, 2018
156b66a
incremental progress on TaskfMRI implementation
tjhendrickson Jun 11, 2018
ce466ce
Reverted TaskfMRIAnalysis pipeline to v3.17.0
tjhendrickson Jun 12, 2018
23a4e6f
added generate_level1_fsf.sh to docker image
tjhendrickson Jun 12, 2018
2048ab4
place function and resting state underneath for loop
tjhendrickson Jun 14, 2018
0d40451
making sure odin repo is up to date
tjhendrickson Jun 15, 2018
2e3a4c9
backed out of user input section within intendedfor script
tjhendrickson Jun 18, 2018
3b1b3dc
added necessary features to generatefsf and TaskfMRIAnalysis. Still h…
tjhendrickson Jun 18, 2018
e67b1cd
added Singularity usage to README.md
tjhendrickson Jun 21, 2018
cec30da
resolved merge conflicts
tjhendrickson Jun 21, 2018
5a0b02f
removed TaskfMRIAnalysis and .idea
tjhendrickson Jun 21, 2018
6b0502d
correct merge conflict with Dockerfile
tjhendrickson Jun 21, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"/.idea"
Binary file added 360CortSurf_19Vol_parcel.dlabel.nii
Binary file not shown.
219 changes: 122 additions & 97 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,36 @@
# Use Ubuntu 14.04 LTS
FROM ubuntu:trusty-20170119

# Configure environment
ENV OS Linux
ENV FS_OVERRIDE 0
ENV FIX_VERTEX_AREA=
ENV SUBJECTS_DIR /opt/freesurfer/subjects
ENV FSF_OUTPUT_FORMAT nii.gz
ENV MNI_DIR /opt/freesurfer/mni
ENV LOCAL_DIR /opt/freesurfer/local
ENV FREESURFER_HOME /opt/freesurfer
ENV FSFAST_HOME /opt/freesurfer/fsfast
ENV MINC_BIN_DIR /opt/freesurfer/mni/bin
ENV MINC_LIB_DIR /opt/freesurfer/mni/lib
ENV MNI_DATAPATH /opt/freesurfer/mni/data
ENV FMRI_ANALYSIS_DIR /opt/freesurfer/fsfast
ENV PERL5LIB /opt/freesurfer/mni/lib/perl5/5.8.5
ENV MNI_PERL5LIB /opt/freesurfer/mni/lib/perl5/5.8.5
ENV PATH /opt/freesurfer/bin:/opt/freesurfer/fsfast/bin:/opt/freesurfer/tktools:/opt/freesurfer/mni/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH
ENV PYTHONPATH=""
ENV FSL_FIXDIR /opt/fix
ENV FSLDIR=/usr/share/fsl/5.0
ENV FSL_DIR="${FSLDIR}"
ENV FSLOUTPUTTYPE=NIFTI_GZ
ENV PATH=/usr/lib/fsl/5.0:$PATH
ENV FSLMULTIFILEQUIT=TRUE
ENV POSSUMDIR=/usr/share/fsl/5.0
ENV LD_LIBRARY_PATH=/usr/lib/fsl/5.0
ENV FSLTCLSH=/usr/bin/tclsh
ENV FSLWISH=/usr/bin/wish
ENV FSLOUTPUTTYPE=NIFTI_GZ
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was there a reason for moving this here?


# Download FreeSurfer
RUN apt-get -y update \
&& apt-get install -y wget && \
Expand All @@ -20,115 +50,100 @@ RUN apt-get -y update \
--exclude='freesurfer/lib/qt' && \
apt-get install -y tcsh bc tar libgomp1 perl-modules curl

# Set up the environment
ENV OS=Linux \
FS_OVERRIDE=0 \
FIX_VERTEX_AREA= \
SUBJECTS_DIR=/opt/freesurfer/subjects \
FSF_OUTPUT_FORMAT=nii.gz \
MNI_DIR=/opt/freesurfer/mni \
LOCAL_DIR=/opt/freesurfer/local \
FREESURFER_HOME=/opt/freesurfer \
FSFAST_HOME=/opt/freesurfer/fsfast \
MINC_BIN_DIR=/opt/freesurfer/mni/bin \
MINC_LIB_DIR=/opt/freesurfer/mni/lib \
MNI_DATAPATH=/opt/freesurfer/mni/data \
FMRI_ANALYSIS_DIR=/opt/freesurfer/fsfast \
PERL5LIB=/opt/freesurfer/mni/lib/perl5/5.8.5 \
MNI_PERL5LIB=/opt/freesurfer/mni/lib/perl5/5.8.5 \
PATH=/opt/freesurfer/bin:/opt/freesurfer/fsfast/bin:/opt/freesurfer/tktools:/opt/freesurfer/mni/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH


# https://bugs.debian.org/830696 (apt uses gpgv by default in newer releases, rather than gpg)
RUN set -x \
&& apt-get update \
&& { \
which gpg \
|| apt-get install -y --no-install-recommends gnupg \
; } \
# Ubuntu includes "gnupg" (not "gnupg2", but still 2.x), but not dirmngr, and gnupg 2.x requires dirmngr
# so, if we're not running gnupg 1.x, explicitly install dirmngr too
&& { \
gpg --version | grep -q '^gpg (GnuPG) 1\.' \
|| apt-get install -y --no-install-recommends dirmngr \
; } \
&& rm -rf /var/lib/apt/lists/*

# apt-key is a bit finicky during "docker build" with gnupg 2.x, so install the repo key the same way debian-archive-keyring does (/etc/apt/trusted.gpg.d)
# this makes "apt-key list" output prettier too!
RUN set -x \
&& export GNUPGHOME="$(mktemp -d)" \
&& gpg --keyserver ha.pool.sks-keyservers.net --recv-keys DD95CC430502E37EF840ACEEA5D32F012649A5A9 \
&& gpg --export DD95CC430502E37EF840ACEEA5D32F012649A5A9 > /etc/apt/trusted.gpg.d/neurodebian.gpg \
&& rm -rf "$GNUPGHOME" \
&& apt-key list | grep neurodebian

RUN { \
echo 'deb http://neuro.debian.net/debian trusty main'; \
echo 'deb http://neuro.debian.net/debian data main'; \
echo '#deb-src http://neuro.debian.net/debian-devel trusty main'; \
} > /etc/apt/sources.list.d/neurodebian.sources.list
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you want to remove this? It's an important workaround for flaky key server.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obviously I did not understand the purpose of this. Feel free to leave it in!


RUN sed -i -e 's,main *$,main contrib non-free,g' /etc/apt/sources.list.d/neurodebian.sources.list; grep -q 'deb .* multiverse$' /etc/apt/sources.list || sed -i -e 's,universe *$,universe multiverse,g' /etc/apt/sources.list

# Install FSL 5.0.9
RUN apt-get update && \
apt-get install -y --no-install-recommends curl && \
curl -sSL http://neuro.debian.net/lists/trusty.us-ca.full >> /etc/apt/sources.list.d/neurodebian.sources.list && \
apt-key adv --recv-keys --keyserver hkp://pgp.mit.edu:80 0xA5D32F012649A5A9 && \
apt-get update && \
apt-get install -y fsl-core=5.0.9-4~nd14.04+1
curl -sSL http://neuro.debian.net/lists/trusty.us-ca.full >> /etc/apt/sources.list.d/neurodebian.sources.list
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment above.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comment above.


# Configure environment
ENV FSLDIR=/usr/share/fsl/5.0
ENV FSL_DIR="${FSLDIR}" \
FSLOUTPUTTYPE=NIFTI_GZ \
PATH=/usr/lib/fsl/5.0:$PATH \
FSLMULTIFILEQUIT=TRUE \
POSSUMDIR=/usr/share/fsl/5.0 \
LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH \
FSLTCLSH=/usr/bin/tclsh \
FSLWISH=/usr/bin/wish \
FSLOUTPUTTYPE=NIFTI_GZ
RUN apt-key adv --recv-keys --keyserver hkp://pgp.mit.edu:80 0xA5D32F012649A5A9

RUN echo "cHJpbnRmICJrcnp5c3p0b2YuZ29yZ29sZXdza2lAZ21haWwuY29tXG41MTcyXG4gKkN2dW12RVYzelRmZ1xuRlM1Si8yYzFhZ2c0RVxuIiA+IC9vcHQvZnJlZXN1cmZlci9saWNlbnNlLnR4dAo=" | base64 -d | sh
RUN apt-get update && \
apt-get install -y fsl-core=5.0.9-4~nd14.04+1

RUN apt-get build-dep -y gridengine && apt-get update -y

# Install Connectome Workbench
RUN apt-get update && apt-get -y install connectome-workbench=1.2.3-1~nd14.04+1

ENV CARET7DIR=/usr/bin

# Install HCP Pipelines and MSM binaries
WORKDIR /opt
RUN apt-get -y update \
# Install HCP Pipelines
RUN apt-get update \
&& apt-get install -y --no-install-recommends python-numpy && \
wget -q https://github.com/Washington-University/Pipelines/archive/v3.17.0.tar.gz -O pipelines.tar.gz && \
tar zxf pipelines.tar.gz && \
mv /opt/Pipelines-* /opt/HCP-Pipelines && \
rm pipelines.tar.gz && \
wget -q https://www.doc.ic.ac.uk/~ecr05/MSM_HOCR_v1/MSM_HOCR_v1-download.tgz -O MSMs.tar.gz && \
tar zxf MSMs.tar.gz && \
rm MSMs.tar.gz && \
mv MSM_HOCR_v1/Ubuntu /opt/HCP-Pipelines/MSMBinaries && \
rm -rf MSM_HOCR_v1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for not installing MSM?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No reason, let's go with your version.


WORKDIR /

wget https://github.com/Washington-University/Pipelines/archive/v3.17.0.tar.gz -O pipelines.tar.gz && \
cd /opt/ && \
mkdir /opt/HCP-Pipelines && \
tar zxf /pipelines.tar.gz -C /opt/HCP-Pipelines --strip-components=1 && \
rm /pipelines.tar.gz

# Install FIX
RUN apt-get update && apt-get install -y build-essential libpcre3 libpcre3-dev fort77 xorg-dev libbz2-dev liblzma-dev libblas-dev gfortran gcc-multilib gobjc++ libreadline-dev bzip2 libcurl4-gnutls-dev default-jdk gdebi

RUN cd /opt && \
wget http://www.fmrib.ox.ac.uk/~steve/ftp/fix.tar.gz && \
tar zxvf fix.tar.gz && \
rm fix.tar.gz
RUN mv /opt/fix* /opt/fix

RUN cd /opt && \
wget https://cloud.r-project.org/bin/linux/ubuntu/trusty/r-base-core_3.4.4-1trusty0_amd64.deb && \
wget https://cloud.r-project.org/bin/linux/ubuntu/trusty/r-base-dev_3.4.4-1trusty0_all.deb && \
gdebi -n r-base-core_3.4.4-1trusty0_amd64.deb && \
gdebi -n r-base-dev_3.4.4-1trusty0_all.deb && \
apt-get install -y libssl-dev

RUN R --vanilla -e "install.packages('coin', repos='http://cran.us.r-project.org', dependencies=TRUE)" -e "install.packages('strucchange', repos='http://cran.us.r-project.org', dependencies=TRUE)" -e "install.packages('https://cran.r-project.org/src/contrib/Archive/party/party_1.0-25.tar.gz', repos=NULL, type='source')" -e "install.packages('https://cran.r-project.org/src/contrib/Archive/kernlab/kernlab_0.9-24.tar.gz', repos=NULL, type='source')" -e "install.packages('ROCR', repos='http://cran.us.r-project.org', dependencies=TRUE)" -e "install.packages('https://cran.r-project.org/src/contrib/Archive/e1071/e1071_1.6-7.tar.gz', repos=NULL, type='source')" -e "install.packages('https://cran.r-project.org/src/contrib/Archive/randomForest/randomForest_4.6-12.tar.gz', repos=NULL, type='source')"

RUN mkdir /tmp/v83 && \
cp /opt/fix*/compiled/Linux/x86_64/MCRInstaller.zip /tmp/v83 && \
cd /tmp/v83 && \
unzip MCRInstaller.zip

COPY modified_files/MCR_installer_input_v83.txt /tmp/v83/MCR_installer_input.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate on the changes required to this file? It might be clearer to apply them via diff instead of replacing a file.

RUN cd /tmp/v83 && ./install -mode silent -inputFile MCR_installer_input.txt

# Ensure Dependencies for PostFix are met
RUN apt-get update
RUN mkdir /tmp/v81 && \
cd /tmp/v81 && \
wget http://ssd.mathworks.com/supportfiles/MCR_Runtime/R2013a/MCR_R2013a_glnxa64_installer.zip && \
unzip MCR_R2013a_glnxa64_installer.zip
COPY modified_files/MCR_installer_input_v81.txt /tmp/v81/MCR_installer_input.txt
RUN cd /tmp/v81 && ./install -mode silent -inputFile MCR_installer_input.txt

#Create necessary environment variables
RUN echo "cHJpbnRmICJrcnp5c3p0b2YuZ29yZ29sZXdza2lAZ21haWwuY29tXG41MTcyXG4gKkN2dW12RVYzelRmZ1xuRlM1Si8yYzFhZ2c0RVxuIiA+IC9vcHQvZnJlZXN1cmZlci9saWNlbnNlLnR4dAo=" | base64 -d | sh
ENV CARET7DIR=/usr/bin
ENV HCPPIPEDIR=/opt/HCP-Pipelines
ENV HCPPIPEDIR_Templates=${HCPPIPEDIR}/global/templates \
HCPPIPEDIR_Bin=${HCPPIPEDIR}/global/binaries \
HCPPIPEDIR_Config=${HCPPIPEDIR}/global/config \
HCPPIPEDIR_PreFS=${HCPPIPEDIR}/PreFreeSurfer/scripts \
HCPPIPEDIR_FS=${HCPPIPEDIR}/FreeSurfer/scripts \
HCPPIPEDIR_PostFS=${HCPPIPEDIR}/PostFreeSurfer/scripts \
HCPPIPEDIR_fMRISurf=${HCPPIPEDIR}/fMRISurface/scripts \
HCPPIPEDIR_fMRIVol=${HCPPIPEDIR}/fMRIVolume/scripts \
HCPPIPEDIR_tfMRI=${HCPPIPEDIR}/tfMRI/scripts \
HCPPIPEDIR_dMRI=${HCPPIPEDIR}/DiffusionPreprocessing/scripts \
HCPPIPEDIR_dMRITract=${HCPPIPEDIR}/DiffusionTractography/scripts \
HCPPIPEDIR_Global=${HCPPIPEDIR}/global/scripts \
HCPPIPEDIR_tfMRIAnalysis=${HCPPIPEDIR}/TaskfMRIAnalysis/scripts \
MSMBin=${HCPPIPEDIR}/MSMBinaries
ENV HCPPIPEDIR_Templates=${HCPPIPEDIR}/global/templates
ENV HCPPIPEDIR_Bin=${HCPPIPEDIR}/global/binaries
ENV HCPPIPEDIR_Config=${HCPPIPEDIR}/global/config
ENV HCPPIPEDIR_PreFS=${HCPPIPEDIR}/PreFreeSurfer/scripts
ENV HCPPIPEDIR_FS=${HCPPIPEDIR}/FreeSurfer/scripts
ENV HCPPIPEDIR_PostFS=${HCPPIPEDIR}/PostFreeSurfer/scripts
ENV HCPPIPEDIR_fMRISurf=${HCPPIPEDIR}/fMRISurface/scripts
ENV HCPPIPEDIR_fMRIVol=${HCPPIPEDIR}/fMRIVolume/scripts
ENV HCPPIPEDIR_tfMRI=${HCPPIPEDIR}/tfMRI/scripts
ENV HCPPIPEDIR_dMRI=${HCPPIPEDIR}/DiffusionPreprocessing/scripts
ENV HCPPIPEDIR_dMRITract=${HCPPIPEDIR}/DiffusionTractography/scripts
ENV HCPPIPEDIR_Global=${HCPPIPEDIR}/global/scripts
ENV HCPPIPEDIR_tfMRIAnalysis=${HCPPIPEDIR}/TaskfMRIAnalysis/scripts
ENV MSMBin=${HCPPIPEDIR}/MSMBinaries

# Install python Dependencies
RUN apt-get update && apt-get install -y --no-install-recommends python-pip python-six python-nibabel python-setuptools
RUN pip install pybids==0.5.1
RUN pip install --upgrade pybids


#make /bids_dir and /output_dir
RUN mkdir /bids_dir && \
mkdir /output_dir
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary?


#create 360CortSurf_19Vol_parcel.dlabel.nii
RUN cd / && \
wget https://github.umn.edu/hendr522/HCPPipelines/blob/master/modified_files/360CortSurf_19Vol_parcel.dlabel.nii

## Install the validator
RUN apt-get update && \
Expand All @@ -144,7 +159,17 @@ RUN pip install pybids==0.5.1
ENV PYTHONPATH=""

COPY run.py /run.py
RUN chmod +x /run.py
RUN chmod 555 /run.py

COPY version /version
COPY IntendedFor.py /IntendedFor.py
COPY modified_files/fsl_sub /usr/lib/fsl/5.0/fsl_sub
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What changes are required to this file? It would be better to apply them via diff

COPY modified_files/settings.sh /opt/fix/settings.sh
COPY modified_files/360CortSurf_19Vol_parcel.dlabel.nii /360CortSurf_19Vol_parcel.dlabel.nii
COPY modified_files/PostFix.sh /opt/HCP-Pipelines/PostFix/PostFix.sh
COPY modified_files/run_prepareICAs.sh /opt/HCP-Pipelines/PostFix/Compiled_prepareICAs/distrib/run_prepareICAs.sh
COPY modified_files/RestingStateStats.sh /opt/HCP-Pipelines/RestingStateStats/RestingStateStats.sh
COPY modified_files/run_RestingStateStats.sh /opt/HCP-Pipelines/RestingStateStats/Compiled_RestingStateStats/distrib/run_RestingStateStats.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to avoid departing from the original HCP Pipelines as much as possible. Why those files need to be replaced?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally I absolutely agree with you, however, I was required to make mods to the RestingStateStats and PostFix scripts because the paths to the matlab compiler runtime are hardcoded to paths that fall within WashU's server systems.



ENTRYPOINT ["/run.py"]
67 changes: 67 additions & 0 deletions IntendedFor.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@

#!/usr/bin/python
import json
import os
from glob import glob
import numpy

"""
This script is designed to determine which field maps apply to discrete fMRI scans
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks very similar to get_fieldmap implemented in pybids https://github.com/INCF/pybids/tree/master/bids/grabbids#get-a-fieldmap-files-intended-for-a-given-map. Maybe we should use that one?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. It has been a while since I looked at the get_fieldmap function implemented in pybids so perhaps it has changed, but doesn't the get_fieldmap function just retrieve the fieldmap in which the fmriname is within the "IntendedFor" field? I was under the impression that the "IntendedFor" field does not automatically generate through the BIDS conversion, thus it has to be added manually.

What am I missing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. Maybe I misunderstood what your function does. If it's part of a conversion process this repo might not be the best place for it. Maybe it would be better to put it in pybids?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see why you may have been confused. The comments I provide I very unclear. Obviously I created them when I did not understand what the script was doing myself, and forgot to come through and change them. It is added the "IntendedFor" fields to the fmap json sidecars, so yes I suppose it has more to do with the BIDS conversion. I can propose incorporation to pybids, but perhaps as a catch-all let's leave IntendedFor..py as is for now.


Author - Timothy J Hendrickson
"""

def setup(subject_path):
if "ses" in os.listdir(subject_path)[0]:
for item in (os.listdir(subject_path)):
session_path = subject_path + '/'+ item
IntendedFor(session_path)
IntendedFor(subject_path)

def IntendedFor(data_path):
for fmap in sorted(glob(data_path+'/fmap/*.json')):
with open(fmap, 'r') as f:
fmap_json = json.load(f)
f.close()
if "IntendedFor" in fmap_json:
del fmap_json["IntendedFor"]
shim_fmap = fmap_json["ShimSetting"]
patient_pos_fmap = fmap_json["ImageOrientationPatientDICOM"]
func_list = []
for func in sorted(glob(data_path+'/func/*bold.json')):
with open(func, 'r') as g:
func_json = json.load(g)
shim_func = func_json["ShimSetting"]
patient_pos_func = func_json["ImageOrientationPatientDICOM"]
g.close()
if shim_fmap == shim_func:
func_nii = glob(data_path+'/func/' +func.split('/')[-1].split('.')[0]+".nii*")[0]
if "ses" in data_path:
func_nii = "/".join(func_nii.split("/")[-3:])
func_list.append(func_nii)
else:
func_nii = "/".join(func_nii.split("/")[-2:])
func_list.append(func_nii)
#elif patient_pos_fmap == patient_pos_func:
# shim_func_np = numpy.array(shim_func)
# shim_fmap_np = numpy.array(shim_fmap)
# diff_shims = shim_fmap_np - shim_func_np
# print("difference between fieldmap and functional scan is " + str(sum(abs(diff_shims))))
# user_input = raw_input("Do you accept this difference? [y/n]: ")
# if user_input == 'y':
# func_nii = glob(data_path + '/func/' + func.split('/')[-1].split('.')[0] + ".nii*")[0]
# if "ses" in data_path:
# func_nii = "/".join(func_nii.split("/")[-3:])
# func_list.append(func_nii)
# else:
# func_nii = "/".join(func_nii.split("/")[-2:])
# func_list.append(func_nii)
# else:
# print(fmap + " does not have match yet...")
entry = {"IntendedFor": func_list}

fmap_json.update(entry)
with open(fmap, 'w') as f:
json.dump(fmap_json, f)
f.close()

Loading