dynamic installation and import of packages based on each component

[hosseinfani]

We need to make the pkgmgr to support installation with version, but for without version, it first checks the requirements.txt to find a version, otherwise simply with no version. This way, the pipeline does not need to specify the lib versions in multiple locations.

[gabrielrueda]

Hi @hosseinfani

I started working on this task and had a few questions.

1. For the following line in the requirements.txt:

# torch_geometric==2.6.1 torch_cluster==1.6.3 torch_sparse==0.6.18 torch_scatter==2.1.2 -f https://data.pyg.org/whl/torch-{Gnn.torch.__version__}.html

a. Since these 3 packages are in one line, does that mean that these 3 packages should be installed in conjunction, or can i just assume that these 3 packages can be installed separately.

b. For the option of -f https://data.pyg.org/whl/torch-{Gnn.torch.__version__}.html would I assume that these options would apply for all the packages in the line or just the last line. In the normal requirements.txt syntax, it seems that this applies to all packages in the line.

2. For deciding which lines to parse and extract package versions, I'm looking at lines with single # starting the line. Would this be the right approach or would I need to consider other lines.

For example:
#pytrec-eval-terrier==0.5.2 -> I would extract the package version
######reranking==0.3.6 -> I would ignore this line.

[hosseinfani]

Hi @gabrielrueda
thank you.

1.a. should be installed in conjuction
1.b. all packages
2. I would suggest add #@ or #$ or any weird combination of # with a char that never happens for those lines that we processed them, whether for package version or where they have been used. For other lines, as long as they are just #, it is a comment line.

[hosseinfani]

Hi @gabrielrueda
Also, can you change the code for pkgmgr such that if there is a call to a package,

• if the package is available, (does not raise an exception), then check the version of that package.
• if the version is not equal to the one that is in requirement.txt, reinstall or upgrade to the version in requirement.txt.

The reason for this change is that

• I ask package A, it is not there, it tries to install it, but it may uninstall an already installed package for a compatible version.
• e.g., when I want to import onmt, it should install opennmt. But opennmt searches for torch, uninstall it, then re-install it for 2.0.1. So, it changes our torch version.
• I want to restore the torch version by the next call to a package of torch. That is, the torch version in requirement.txt is 2.4.1, after opennmt becomes 2.0.1, by the first recall to torch, I want to restore-upgrade to 2.4.1.

[gabrielrueda]

hi @hosseinfani

Thanks, I will keep that mind when I start to implement the pkgmgr. As of now, I have update below, where I looked the parsing the requirements.txt file for packages and their versions.

Update

• the idea is that at start of the program a dictionary of the packages based on requirements.txt is created
  • each element will be: {pkg_name : [line_in_requirements.txt, version_num]}
  • this dictionary will stay stored in pkgmgr.py
• then when we want to install a package...
  • we find the package in the dictionary -> use it's value in the pip install command
• as an alternative, we could avoid storing the dictionary in the memory and instead have the pkgmgr search in requirements.txt every time a package install is inititated

Filters

• every line for dynamic package retrieval will start with #$
• a line with another # in it will be assumed as comment (anything to the right of it will be deleted)
• e.g. scikit-learn==1.2.2 # scikit learn -> scikit-learn==1.2.2

Example

Here I made a sample requiements.txt that I will use to show how it works...

hydra-core==1.3.2
numpy==1.24.4 #cannot go above for gensim

# on-demand package installations
#$ scikit-learn==1.2.2 # scikit learn 
## ...
#$ pandas==2.0.0
## ...
#$ torch==2.2.2 --index-url https://download.pytorch.org/whl/cu124
## ...
#$ torch_geometric==2.6.1 torch_cluster==1.6.3 torch_sparse==0.6.18 -f https://data.pyg.org/whl/torch-{Gnn.torch.__version__}.html

####### OpenNMT-py==3.3 

This below is what the final dictionary from the requiements.txt file would look like

{
	'scikit-learn': ['scikit-learn==1.2.2', 1.2.2]
	'pandas': ['pandas==2.0.0', 2.0.0],
	'torch': ['torch==2.2.2 --index-url https://download.pytorch.org/whl/cu124', 2.2.2],
	'torch_geometric': ['torch_geometric==2.6.1 torch_cluster==1.6.3 torch_sparse==0.6.18 -f https://data.pyg.org/whl/torch-{Gnn.torch.__version__}.html', 2.6.1], 
	'torch_cluster': ['torch_geometric==2.6.1 torch_cluster==1.6.3 torch_sparse==0.6.18 -f https://data.pyg.org/whl/torch-{Gnn.torch.__version__}.html', 1.6.3], 
	'torch_sparse': ['torch_geometric==2.6.1 torch_cluster==1.6.3 torch_sparse==0.6.18 -f https://data.pyg.org/whl/torch-{Gnn.torch.__version__}.html', 0.6.18]
}

• now suppose we want to dynamically install torch
  • then the following command will run: pip install torch==2.2.2 --index-url https://download.pytorch.org/whl/cu124
    • if already installed, then the version numbers can be compared (version number is 2.2.2)

Package name extraction

• the package name is extracted by a regular expression
• here is the method that basically does the extraction and stores a key-value pair for each package found in a given line
  • the outputs of these methods can then be merged into one dictionary

# Assume that only lines starting with "#$" are called
def extract_package_name(line):
    line = line[2:] # Remove the "#$"
    line = line.split("#")[0]  # Remove comments
    line = line.strip() # Remove leading and trailing whitespace


    package_name = "([-A-Za-z0-9_\.]+)"
    comp = "(==|!=|<=|>=|<|>|~=|===)"
    ver_num = "([0-9]+[0-9\.\*]*)"

    out = {}

    for pkg in re.findall(f"{package_name}[\s]*{comp}[\s]*{ver_num}", line):
        out[pkg[0]] = [line, pkg[2]]

    return out # {pkg_name_a: [line, ver_num_a], pkg_name_b: [line, ver_num_a], ...}