How to create a Devcontainer for your Python project
0 likes128 views
The document provides a guide on creating a devcontainer for a Python project, outlining the requirements for specific Java, Python, and PySpark versions. It explains how to set up the devcontainer structure, including necessary files like devcontainer.json and Dockerfile, and details the benefits such as reproducibility and faster onboarding. The document also covers extending the devcontainer with custom user settings, accessing Spark UI, and using CI/CD pipelines with the devcontainer.
](
https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?
url=https://github.com/godatadriven/python-devcontainer-template
)](https://image.slidesharecdn.com/howtocreateadevcontainerforyourpythonprojectslides-221031085901-584b5df6/85/How-to-create-a-Devcontainer-for-your-Python-project-17-320.jpg)
![Passing custom VSCode settings
"customizations": {
"vscode": {
"extensions": [
"ms-python.python"
],
"settings": {
"python.testing.pytestArgs": [
"."
],
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true,
"python.formatting.provider": "black",
"python.linting.mypyEnabled": true,
"python.linting.enabled": true
}
}
}](https://image.slidesharecdn.com/howtocreateadevcontainerforyourpythonprojectslides-221031085901-584b5df6/85/How-to-create-a-Devcontainer-for-your-Python-project-24-320.jpg)
![Accessing Spark UI
Sinceweareusingpyspark,itwouldbenicetobeabletoaccessSparkUI.
"portsAttributes": {
"4040": {
"label": "SparkUI",
"onAutoForward": "notify"
}
},
"forwardPorts": [
4040
]](https://image.slidesharecdn.com/howtocreateadevcontainerforyourpythonprojectslides-221031085901-584b5df6/85/How-to-create-a-Devcontainer-for-your-Python-project-25-320.jpg)
![First, devcontainer.json :
{
"build": {
"dockerfile": "Dockerfile",
"context": ".."
},
"remoteUser": "nonroot",
"customizations": {
"vscode": {
"extensions": [
"ms-python.python"
],
"settings": {
"python.testing.pytestArgs": [
"."
],
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true,
"python.formatting.provider": "black",
"python.linting.mypyEnabled": true,
"python.linting.enabled": true
}
}
},
...
}](https://image.slidesharecdn.com/howtocreateadevcontainerforyourpythonprojectslides-221031085901-584b5df6/85/How-to-create-a-Devcontainer-for-your-Python-project-35-320.jpg)
!["portsAttributes": {
"4040": {
"label": "SparkUI",
"onAutoForward": "notify"
}
},
"forwardPorts": [
4040
]](https://image.slidesharecdn.com/howtocreateadevcontainerforyourpythonprojectslides-221031085901-584b5df6/85/How-to-create-a-Devcontainer-for-your-Python-project-36-320.jpg)
How to create a Devcontainer for your Python project
- 1. How to create a Devcontainer for your Python project đł
- 4. The use case Youareassignedtosetupanewrepoforateam.Therequirementsareasfollows: Soweneedtoalignon: đ AspecificJavaversion đ AspecificPythonversion đ Aspecific pyspark version âotherwisewedonotenjoytheguaranteeswewantinproductioncode
- 6. Devcontainers to the rescue â! Dockerhelpsuscreateaformaldefinitionofourenvironment.Devcontainersallowyouto connectyoureditor(IDE)tothatcontainer. đ NotethatthisdoesmeanrunningDockerimagesonyourlaptop(performance requirement). Devcontainerscanhelpus: đ Reproducibledevelopmentenvironment âĄď¸ Fasterprojectsetupâfasteronboarding đ¨âđŠâđ§âđŚ Betteralignmentbetweenteammembers âą Forcedtokeepyourdevenvironmentup-to-date&reproducible âsavesyourteamtimegoingintoproductionlater
- 7. đˇđťââď¸Let's build a Devcontainer!
- 8. Letâssaywehaveareallysimpleprojectthatlookslikethis: $ tree . . âââ README.md âââ requirements.txt âââ requirements-dev.txt âââ sales_analysis.py âââ test_sales_analysis.py
- 9. The .devcontainer folder YourDevcontainerspecwillliveinsidethe .devcontainer folder. Therewillbetwomainfiles: devcontainer.json Dockerfile
- 10. Createanewfilecalled devcontainer.json : { "build": { "dockerfile": "Dockerfile", "context": ".." } }
- 11. Sohowdoesthis Dockerfile looklike? FROM python:3.10 # Install Java RUN apt update && apt install -y sudo && sudo apt install default-jdk -y ## Pip dependencies # Upgrade pip RUN pip install --upgrade pip # Install production dependencies COPY requirements.txt /tmp/requirements.txt RUN pip install -r /tmp/requirements.txt && rm /tmp/requirements.txt # Install development dependencies COPY requirements-dev.txt /tmp/requirements-dev.txt RUN pip install -r /tmp/requirements-dev.txt && rm /tmp/requirements-dev.txt
- 12. Opening the Devcontainer The .devcontainer folderinplace,nowitâstimetoopenourDevcontainer.
- 13. Openupthecommandpallete( CMD + Shift + P )andselectâDevContainers:Reopen inContainerâ:
- 16. What is happening under the hood đ BesidesstartingtheDockerimageandattachingtheterminaltoit,VSCodeisdoinga couplemorethings: 1. isbeinginstalledonyourDevcontainer. VSCodeServerisinstalledasaserviceinthecontaineritselfsoyourVSCode installationcancommunicatewiththecontainer. Forexample,installandrunextensions. VSCodeServer 2.Configiscopiedover. Configlike ~/.gitconfig and ~/.ssh/known_hosts arecopiedovertotheir respectivelocationsinthecontainer. 3.Filesystemmounts. VSCodeautomaticallytakescareofmounting: ThefolderyouarerunningtheDevcontainerfrom. YourVSCodeworkspacefolder.
- 17. Opening your Devcontainer with the click of a button YourentireprojectsetupisnowencapsulatedintheDevcontainer.Soactuallywecanadd aMarkdownbuttontoopenuptheDevcontainer: JustmodifytheGitHubURLafter url= â. [  ]( https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume? url=https://github.com/godatadriven/python-devcontainer-template )
- 18. Thisrendersthefollowingbutton: Remote - Containers Remote - Containers Open Open
- 22. Installing a non-root user Ifyou pip install anewpackage,youwillseethefollowingmessage:
- 23. Solet'sgoaheadandcreateauserforthisscenario. # Add non-root user ARG USERNAME=nonroot RUN groupadd --gid 1000 $USERNAME && useradd --uid 1000 --gid 1000 -m $USERNAME ## Make sure to reflect new user in PATH ENV PATH="/home/${USERNAME}/.local/bin:${PATH}" USER $USERNAME Addthefollowingpropertyto devcontainer.json : "remoteUser": "nonroot" That'sgreat!Whenwenowstartthecontainerweshouldconnectastheuser nonroot .
- 24. Passing custom VSCode settings "customizations": { "vscode": { "extensions": [ "ms-python.python" ], "settings": { "python.testing.pytestArgs": [ "." ], "python.testing.unittestEnabled": false, "python.testing.pytestEnabled": true, "python.formatting.provider": "black", "python.linting.mypyEnabled": true, "python.linting.enabled": true } } }
- 25. Accessing Spark UI Sinceweareusingpyspark,itwouldbenicetobeabletoaccessSparkUI. "portsAttributes": { "4040": { "label": "SparkUI", "onAutoForward": "notify" } }, "forwardPorts": [ 4040 ]
- 28. â¨
- 29. Running our CI in the Devcontainer Therearetwobasicoptions: 1.BuildtheDockerimagewithintheCI/CDpipeline 2.Prebuildingtheimage Let'sseeaboutoptionnumber(1).
- 30. 1. Build the Docker image within the CI/CD pipeline Luckily,aGitHubActionwasalreadysetupforustodoexactlythis: devcontainers/ci
- 31. Tonowbuild,pushandrunacommandintheDevcontainerisaseasyas: name: Python app on: ... jobs: build: runs-on: ubuntu-latest steps: - name: Checkout (GitHub) uses: actions/checkout@v3 - name: Login to GitHub Container Registry uses: docker/login-action@v2 with: registry: ghcr.io username: ${{ github.repository_owner }} password: ${{ secrets.GITHUB_TOKEN }} - name: Build and run dev container task uses: devcontainers/ci@v0.2 with: imageName: ghcr.io/${{ github.repository }}/devcontainer runCmd: pytest .
- 33. Awesome!
- 34. The final Devcontainer definition WebuiltthefollowingDevcontainerdefinitions.
- 35. First, devcontainer.json : { "build": { "dockerfile": "Dockerfile", "context": ".." }, "remoteUser": "nonroot", "customizations": { "vscode": { "extensions": [ "ms-python.python" ], "settings": { "python.testing.pytestArgs": [ "." ], "python.testing.unittestEnabled": false, "python.testing.pytestEnabled": true, "python.formatting.provider": "black", "python.linting.mypyEnabled": true, "python.linting.enabled": true } } }, ... }
- 36. "portsAttributes": { "4040": { "label": "SparkUI", "onAutoForward": "notify" } }, "forwardPorts": [ 4040 ]
- 37. Andour Dockerfile : FROM python:3.10 # Install Java RUN apt update && apt install -y sudo && sudo apt install default-jdk -y # Add non-root user ARG USERNAME=nonroot RUN groupadd --gid 1000 $USERNAME && useradd --uid 1000 --gid 1000 -m $USERNAME ## Make sure to reflect new user in PATH ENV PATH="/home/${USERNAME}/.local/bin:${PATH}" USER $USERNAME ## Pip dependencies # Upgrade pip RUN pip install --upgrade pip # Install production dependencies COPY --chown=nonroot:1000 requirements.txt /tmp/requirements.txt RUN pip install -r /tmp/requirements.txt && rm /tmp/requirements.txt # Install development dependencies COPY --chown=nonroot:1000 requirements-dev.txt /tmp/requirements- dev.txt RUN pip install -r /tmp/requirements-dev.txt && rm /tmp/requirements-dev.txt
- 39. Going further đŽ Devcontainerfeatures Devcontainertemplates đĄ Protip:mountyourAWS/GCP/Azurecredentials ...andmuchmore: Mountingdirectories
- 40. Awesome resources .RunyourCIinyourDevcontainers.Builtonthe . .TheofficialDevcontainerspecification. .Acollectionofready-to-useDevcontainerimages. .Moreexplanations&instructionsforaddinga non-rootusertoyour Dockerfile and devcontainer.json . .Arepopointingtoyetevenmoreawesomeresources. devcontainers/ci DevcontainerCLI https://containers.dev/ devcontainers/images Addanon-rootusertoacontainer Pre-buildingdevcontainerimages awesome-devcontainers
- 41. Concluding Devcontainershaveprovenusefulto: đ Reproducibledevelopmentenvironment âĄď¸ Fasterprojectsetupâfasteronboarding đ¨âđŠâđ§âđŚ Betteralignmentbetweenteammembers âą Forcedtokeepyourdevenvironmentup-to-date&reproducible âsavesyourteamtimegoingintoproductionlater NowonlyVSCode,but takingshape. openspecification