make Docker w. linux base image and anaconda env build on Mac M1 processor and linux intel processor

Job ID: 34237483

Budget: $30 – $250 USD

You need a Mac machine with M1 processor and also another machine w. an intel processor running either linux or mac. You need to use it, install docker on it and you will also need 2 terminal windows. Open the first terminal window and build the following docker file (save this as Dockerfile.packagebuilder):

FROM continuumio/miniconda3:4.12.0
# See https://hub.docker.com/r/continuumio/miniconda3/tags

WORKDIR /opt/wkdir
ENV CCHANS="-c zbl -c defaults -c conda-forge"
RUN DEBIAN_FRONTEND=noninteractive \
apt-get update --fix-missing && apt-get upgrade -y && \
apt-get install --no-install-recommends -y \
build-essential libpq-dev netcat wget

RUN conda update --all --yes
RUN conda install -y python=3.9
RUN conda install -c conda-forge mamba grayskull
RUN conda install anaconda-client
RUN conda install conda-build
RUN conda update --all --yes
RUN conda clean -tipy

Build this image using: docker build -t condabuilder -f Dockerfile.packagebuilder .
Make a directory somewhere on your machine called conda_builds and then run the container like this:
docker run --rm -it -v /path/to/conda_builds:/build condabuilder bash and remember to set the path correctly.

Now run:
anaconda login
where username and password i will give you

Now, open a second terminal window, and in this one, you need to make one more dockerfile called Dockerfile.envbuilder with the following content:

FROM continuumio/miniconda3:4.12.0
# See https://hub.docker.com/r/continuumio/miniconda3/tags

WORKDIR /opt/wkdir
RUN DEBIAN_FRONTEND=noninteractive \
apt-get update --fix-missing && apt-get upgrade -y && \
apt-get install --no-install-recommends -y \
build-essential libpq-dev netcat graphviz-dev \
graphviz openssh-server rsync util-linux wget

ENV CODEHOME="/opt/wkdir"
ENV CCHANS="-c zbl"
RUN conda config --set channel_priority strict
RUN conda update --all --yes
RUN conda install -y python=3.9
RUN conda install -c conda-forge mamba
COPY requirements.txt ${CODEHOME}/requirements.txt
RUN mamba install -y --file ${CODEHOME}/requirements.txt ${CCHANS}
RUN conda clean -tipy

You will need this file (requirements.txt) with this content (YOU ARE NOT ALLOWED TO ALTER THIS FILE):

Django==3.2.14
gunicorn==20.0.4
django-extensions
djangorestframework
django-redis
django-filter
djangorestframework-csv
psycopg2
bokeh
docker-py
PyYAML
pandas
ipython
redis
tabulate
celery
toolz
numpy
scipy
nodejs==14.18.3

Build this image using: docker build -f Dockerfile.envbuilder -t envbuilder .
You will get errors during this build - and it is your job to solve these errors. Errors are caused by missing packages. Now you go and build these packages and upload it to the anaconda channel called zbl using the first terminal made above. When done, try to rerun the build described here to see if error is gone.... if gone move on to fix the next error/missing package etc.... until this image can build 100%. When it builds 100% on linux M1 chip, make sure it builds 100% on linux intel chip...all via docker as explained!!

This is how you use your terminal 1 to actually build a package.
In order to build a package, lets say a package for django 3.2.14:
In the docker container you opened above using this command:
docker run --rm -it -v /path/to/conda_builds:/build condabuilder bash you first run thiscommand:
mkdir -p /build/recipes, then
cd /build/recipes.
Then you specify which package you want to create, for example
grayskull pypi django=3.2.14
When completed, make sure output dir exists:
mkdir -p /build/output
now build the package:
conda-build --python 3.9 --output-folder /build/output django
This will create a build file like: /build/output/linux-aarch64/django-3.2.14-py39_0.tar.bz2

Once built, you need to upload it using:
anaconda upload /build/output/linux-aarch64/django-3.2.14-py39_0.tar.bz2

Note the dir name where it got built: linux-aarch64. this means linux on M1 processor. If located in nooarch subdir - indicate that it runs on all architectures etc
Related categories: Python Linux Software Architecture Bash Scripting