Update Services to Utilize .env File #37

Open
opened 2024-09-25 11:47:55 -03:00 by MasonStooksbury · 17 comments
MasonStooksbury commented 2024-09-25 11:47:55 -03:00 (Migrated from gitlab.com)

After some discussion on this issue, it was decided to modify all of the services within the docker-compose.yaml to utilize the .env file. This isn't a small change and will require some testing


SUB TASKS:

[1] This requires modifying all the services to switch from using "environment: STUFF" to "env_file: .env" in their properties

[2] We also need to update the docker-compose.example.yaml file so that the install.sh script works correctly

[3] Update any relevant documentation by updating it in the yams.media repo

[4] Update the .env.example file to include all of the newly added properties

[5] Update the install.sh script. Lines ~230-250 have a bunch of sed operations that probably need to be modified since some of those properties will be moved

After some discussion on [this issue](https://gitlab.com/rogs/yams/-/issues/36), it was decided to modify all of the services within the `docker-compose.yaml` to utilize the `.env` file. This isn't a small change and will require some testing <br> SUB TASKS: [1] This requires modifying all the services to switch from using "environment: STUFF" to "env_file: .env" in their properties [2] We also need to update the `docker-compose.example.yaml` file so that the `install.sh` script works correctly [3] Update any relevant documentation by updating it in the [yams.media](https://gitlab.com/rogs/yams.media) repo [4] Update the `.env.example` file to include all of the newly added properties [5] Update the `install.sh` script. Lines ~230-250 have a bunch of `sed` operations that probably need to be modified since some of those properties will be moved
MasonStooksbury commented 2024-09-25 11:49:03 -03:00 (Migrated from gitlab.com)

@rogs It's a little bigger than I initially thought. It may be worth breaking out into its own branch so that we can keep things separate and have less to test

Thoughts?

@rogs It's a little bigger than I initially thought. It may be worth breaking out into its own branch so that we can keep things separate and have less to test Thoughts?
MasonStooksbury commented 2024-09-25 11:49:11 -03:00 (Migrated from gitlab.com)

changed the description

changed the description
MasonStooksbury commented 2024-09-25 11:49:21 -03:00 (Migrated from gitlab.com)

changed the description

changed the description
MasonStooksbury commented 2024-09-25 11:49:43 -03:00 (Migrated from gitlab.com)

changed the description

changed the description
MasonStooksbury commented 2024-09-25 11:50:09 -03:00 (Migrated from gitlab.com)

changed the description

changed the description
rogs commented 2024-09-25 12:08:02 -03:00 (Migrated from gitlab.com)

Everything looks good! It should be good to go 😀

Now, I just need a free weekend to work on this hehe. Thanks!

Everything looks good! It should be good to go :grinning: Now, I just need a free weekend to work on this hehe. Thanks!
rogs commented 2024-09-25 12:08:09 -03:00 (Migrated from gitlab.com)

assigned to @rogs

assigned to @rogs
MasonStooksbury commented 2024-09-25 12:16:51 -03:00 (Migrated from gitlab.com)

I feel that lol

Did you wanna roll this into the changes you're already making or do it as a separate branch?

I feel that lol Did you wanna roll this into the changes you're already making or do it as a separate branch?
kragil commented 2024-09-25 13:02:58 -03:00 (Migrated from gitlab.com)

Hello guys, great to see so much progress in such a short time. Amazing!

Just to understand this completely: So this would mean that you have to edit the .env.example to a .env with all your settings for all enviro settings and then no update would change them??

Hello guys, great to see so much progress in such a short time. Amazing! Just to understand this completely: So this would mean that you have to edit the .env.example to a .env with all your settings for all enviro settings and then no update would change them??
MasonStooksbury commented 2024-09-25 13:17:15 -03:00 (Migrated from gitlab.com)

Not necessarily. Though your question brings up a good point I hadn't considered. I'll address that after this; but first I'll explain the proposed change and you'll have to let me know if I actually answered your question

Right now, most services in the docker-compose.yaml file have an "environment" section. A lot of them have variables that reference things in your .env file (e.g. "INSTALL_DIRECTORY", "VPN_USER", etc).

This change would make it so that every service no longer has an "environment" section but rather has an "env_file" section that is pointed at .env. Meaning that your .env file would get a bit longer, but you'd be able to configure everything environment related from just that file.

-------- Now, on to why this is potentially annoying and possibly a bad idea

It's annoying in that if anyone were to want these changes there's really only 3 ways to do so once it was merged in:

[1] Type it all yourself (then you don't need to pull any new changes, you can just make it how you want

[2] Download the new docker-compose.yaml and example .env file, migrate any docker-compose changes you have, and make your .env file look like the new example

[3] We'd need to create some sort of migration script that would automate some of the above for you

It's possibly a bad idea because now every container has the same context as every other container. Where as right now, it only has whatever it pulls from the .env file directly. If we just fist the .env file into every container, now they all know about the variables for everything else.

While highly unlikely, if someone were able to hack into part of your network or system and expose the environment, they would know about ALL of the variables for every other container you have running via YAMS. Whereas right now, they would only get a few things for that specific container. Again, the chances are extremely low, but not zero

@rogs What are your thoughts on all of this? Given the cost-to-benefit ratio, I'm not sure people gain a whole lot from us doing this.

Not necessarily. Though your question brings up a good point I hadn't considered. I'll address that after this; but first I'll explain the proposed change and you'll have to let me know if I actually answered your question Right now, most services in the `docker-compose.yaml` file have an "environment" section. A lot of them have variables that reference things in your `.env` file (e.g. "INSTALL_DIRECTORY", "VPN_USER", etc). This change would make it so that every service no longer has an "environment" section but rather has an "env_file" section that is pointed at `.env`. Meaning that your `.env` file would get a bit longer, but you'd be able to configure everything environment related from just that file. -------- Now, on to why this is potentially annoying and possibly a bad idea It's annoying in that if anyone were to want these changes there's really only 3 ways to do so once it was merged in: [1] Type it all yourself (then you don't need to pull any new changes, you can just make it how you want [2] Download the new `docker-compose.yaml` and example `.env` file, migrate any docker-compose changes you have, and make your `.env` file look like the new example [3] We'd need to create some sort of migration script that would automate some of the above for you It's possibly a bad idea because now every container has the same context as every other container. Where as right now, it only has whatever it pulls from the `.env` file directly. If we just fist the `.env` file into every container, now they all know about the variables for everything else. While highly unlikely, if someone were able to hack into part of your network or system and expose the environment, they would know about ALL of the variables for every other container you have running via YAMS. Whereas right now, they would only get a few things for that specific container. Again, the chances are extremely low, but not zero @rogs What are your thoughts on all of this? Given the cost-to-benefit ratio, I'm not sure people gain a whole lot from us doing this.
rogs commented 2024-09-25 14:29:31 -03:00 (Migrated from gitlab.com)

Same PR would be best, since it's going to touch a lot of what this ticket needs 👍

Same PR would be best, since it's going to touch a lot of what this ticket needs :thumbsup:
rogs commented 2024-09-25 14:32:50 -03:00 (Migrated from gitlab.com)

I wouldn't recommend anyone that's running YAMS right now to upgrade to this new version because a lot will change. I would strongly recommend clean installs from this point on.

I wouldn't recommend anyone that's running YAMS right now to upgrade to this new version because a lot will change. I would strongly recommend clean installs from this point on.
MasonStooksbury commented 2024-09-25 15:01:19 -03:00 (Migrated from gitlab.com)

Makes sense.

What about from a security standpoint? Again, low risk but not zero. Any worries there?

Makes sense. What about from a security standpoint? Again, low risk but not zero. Any worries there?
kragil commented 2024-09-25 15:11:18 -03:00 (Migrated from gitlab.com)

Thanks for the explanation! I guess I would do a clean install then. But maybe do a few more things to really make the update worth peoples while. How about the possibility to disable certain containers like lidarr for all the spotify victims etc.

Thanks for the explanation! I guess I would do a clean install then. But maybe do a few more things to really make the update worth peoples while. How about the possibility to disable certain containers like lidarr for all the spotify victims etc.
rogs commented 2024-09-25 15:13:05 -03:00 (Migrated from gitlab.com)

Not really. Since it's a test file, it will be as vulnerable as any other env variable in your system. If we really want more security, we should use something like authinfo.gpg, but that's very overkill

Not really. Since it's a test file, it will be as vulnerable as any other env variable in your system. If we really want more security, we should use something like [authinfo.gpg](https://www.emacswiki.org/emacs/GnusAuthinfo), but that's very overkill
rogs commented 2024-09-25 15:14:41 -03:00 (Migrated from gitlab.com)

@kragil, this update is basically going to be a full rewrite for YAMS, so it might be worth to upgrade

@kragil, this update is basically going to be a full rewrite for YAMS, so it _might_ be worth to upgrade
kragil commented 2024-09-25 15:15:49 -03:00 (Migrated from gitlab.com)

I'm also not worried about the security. I would like to see YAMS as a really lean simple solution to run and manage docker containers that deal with homelab stuff. And I would like to be able to make the containers run in a certain timed order, because some depend on the VPN being connected for example.

PS: Thanks for everything Roger, really cool project!

I'm also not worried about the security. I would like to see YAMS as a really lean simple solution to run and manage docker containers that deal with homelab stuff. And I would like to be able to make the containers run in a certain timed order, because some depend on the VPN being connected for example. PS: Thanks for everything Roger, really cool project!
rogs self-assigned this 2025-05-11 19:22:40 -03:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rogs/yams#37
No description provided.