86
Reddit DevOps. #devops Thanks @reddit2telegram and @r_channels
Any experiences or opinions on using Sentry.io vs Azure App Insights
It doesn't appear anyone has written a comparison article so far. I'm interested to know if people have used Azure App Insights and Sentry.io for error tracking and how they compare. We're happily using Sentry.io for error capture from an app running on AWS, looking to build a totally separate app in Azure and I can see that Application Insights can do exceptions/failures management too (along with a lot of other stuff).
https://redd.it/kqzfgk
@r_devops
Tool: Product Score Card
My team manages several LOBs, each having their own Products (I define a Product as not only the Application, but all the ancillary stuff required to make it Operationalized - backups, replication, defined as IaC using Terraform, having Azure DevOps Pipeline for deployment, etc.)
One thing I would like to do, is be able to choose certain metrics that we could use to measure to and be able to give a "Score" of a Product".
For example, lets say we have Products that have Services using Azure KeyVault (the fact that the Service is using a KeyVault could be scored a 1/1 for that metric), but say we are introducing a new "feature" where all KeyVault's should be using Private Endpoint. Now all of our legacy Services would be getting a 0/1 for that new metric until they have been switched to using Private Endpoint.
Also, we have a centralized repository for our Azure DevOps Pipelines base templates, which are versioned and referenced by various Services. So if a Service is using the latest version they get a high score, but as we release newer versions of the pipeline and they continue to use the older version, their score starts to decline.
I was curious if there was a tool for doing something such as this? Bonus points if it can be automated to doing certain checks on the specific metric.
I had began tinkering with developing something myself, but, as with tools like Terraform, it would be nice if there was already a product with a set standard.
https://redd.it/kqf5nw
@r_devops
How do you set up a virtual machine with a box located on a Google Drive using Google Drive File Stream?
​
Could not open the medium 'F:\My Drive\VM\macOS Catalina Virtual Disk Image.vmdk'.
VMDK: error writing extent header in 'F:\My Drive\VM\macOS Catalina Virtual Disk Image.vmdk' (VERRNOMEMORY).
VD: error VERRNOMEMORY opening image file 'F:\My Drive\VM\macOS Catalina Virtual Disk Image.vmdk' (VERRNOMEMORY).
I was trying to use a virtual hard disk file in order to create a virtual machine, but it gave me this error. Is it possible to make it work?
https://redd.it/kqp3jb
@r_devops
Hosting options: Gatsby+Flask
Hi everyone!
I'm working on a Gatsby project with a Flask backend and I'm starting to think about the viability of this combination in terms of hosting options before it's "too late" to reconsider.
Heroku was the first option that came to mind and they do indeed seem to support Flask.
Even so, do you reckon it will be a smooth experience deploying a Gatsby+Flask website to Heroku?
Looking forward to hearing your insights!
P.S. The reason I've chosen Flask is twofold: learning something new and not having to reimplement the working Python web scraper I already have in JavaScript.
https://redd.it/kqqwm7
@r_devops
Building an uptime dashboard for a distributed system
We have a product for which we would like to create a dashboard to show
the historic uptime and display any service outages or issues.
​
Our service is a Rails app that is running on a handful of servers and depends on
other components like database, cache, S3 storage and some other supporting services
both internal and vendor provided
​
Currently we are running health checks on rails app only and providing their results
in an html page. Since the system is constructed using many cooperating parts the
current html page many times says that the system is up and available but some customers
can be experiencing issues in production. It happens fairly frequently like a few times a week.
​
We would like to build a better solution beyond polling port 80 alone and provide more
details like which component of the system is having issues and how customers are being
impacted like whether system is degraded and operating with limited functionality or
completely down.
​
The system is running in AWS and monitoring individual components like EC2, EBS volumes,
RDS instances, Kinesis etc is very low level only engineers can consume that type of info.
​
We like a dashboard that can be consumed by customers and executives who might not know
or care about all the components/services that the final product is built with but are very
interested in uptime.
​
An open source solution is highly preferred and we are willing to invest engineering effort to put it together if there is nothing out there already.
​
https://redd.it/kql7hf
@r_devops
"apps", so that there's no weird dependency issues, and you can easily reason about how each works. If you need to re-use code between multiple "apps", you put it in the next directory.
​
# lib/
This directory is the same as your standard Unix "lib/" directory: libraries of code. No default configuration at all. It's intended only to be re-usable components that are used in "apps".
So this is where you put Terraform sub-modules, and anything else you need to keep DRY. You can depend on other lib/ directories from a lib/ directory, but keep it to a bare minimum. It's better to link to multiple lib/ folders from your app/ . This helps reduce dependency conflicts and makes it easier to reason about components.
​
# bin/
Nothing fancy here. Shell scripts used as part of your CI/CD process, or wrappers to run tools with the above hierarchy. You can also keep them in app/ for more simplicity.
https://redd.it/kqfl5e
@r_devops
Official Salary Sharing thread for devops :: Jan 2021
Crediting this thread from /r/cscareerquestions that gets posted monthly December Salary Sharing Thread for Experienced Devs
I like to keep up to date with the current state of salaries/compensation across the world. Feel free to share your information below.
This thread is aimed at anyone from entry > Sr level DevOps/SRE/Infra engineers.
Please only post an offer if you're including hard numbers, but feel free to use a throwaway account if you're concerned about anonymity. You can also generalize some of your answers (e.g. "Biotech company" or "Hideously Overvalued Unicorn"), or add fields if you feel something is particularly relevant.
Education:
Prior Experience:
$Internship
$RealJob
Company/Industry:
Title:
Tenure length:
Location:
Salary:
Relocation/Signing Bonus:
Stock and/or recurring bonuses:
Total comp:
Note that you only really need to include the relocation/signing bonus into the total comp if it was a recent thing. Also, while the primary purpose of these threads is obviously to share compensation info, discussion is also encouraged.
The format here is slightly unusual, so please make sure to post under the appropriate top-level thread, which are: US High/Medium/Low CoL, Western Europe, Eastern Europe, Latin America, Aus/NZ, Canada, Asia, or Other.
If you don't work in the US, you can ignore the rest of this post. To determine cost of living buckets, I used this site: http://www.bestplaces.net/
If the principal city of your metro is not in the reference list below, go to bestplaces, type in the name of the principal city (or city where you work in if there's no such thing), and then click "Cost of Living" in the left sidebar. The buckets are based on the Overall number: Low: < 100, Medium: >= 100, < 150, High: >= 150. (last updated Dec. 2019)
High CoL: NYC, LA, DC, SF Bay Area, Seattle, Boston, San Diego
Medium CoL: Orlando, Tampa, Philadelphia, Dallas, Phoenix, Chicago, Miami, Atlanta, Riverside, Minneapolis, Denver, Portland, Sacramento, Las Vegas, Austin, Raleigh
Low CoL: Houston, Detroit, St. Louis, Baltimore, Charlotte, San Antonio, Pittsburgh, Cincinnati, Kansas City
https://redd.it/kqo29h
@r_devops
WebMap : A Python tool used to automate the execution of the following tools : Nmap , Nikto and Dirsearch but also to automate the report generation during a Web Penetration Testing
I am very excited to showcase my new python script : WebMap
https://github.com/Anteste/WebMap
All started from a small idea: how to automate a Web Penetration Testing .
I used to run every tool in his own terminal and it was taking a lot of time, but now with this tool you can execute all of them just with a simple command : ./webmap.py
This project is free and Open Source so use it as you want and if you have any suggestion you can submit a pull request 😉
https://redd.it/kqk645
@r_devops
Slack is down on the first work day of 2021
Looks like we're going straight into Episode 1 of the new season of 2020
https://redd.it/kqc3ga
@r_devops
I am a jr. web developer looking to move to Devops. Career Advice
I'm currently working as a jr. web developer mostly using html and css not so good with javascript or typescript. On my team I build the required webpages using html, css and my other team members will do the functionality part of the webpages using javascript. I am looking to move to devops as it involves much to little of programming and also I know how important devops is as part of a software development lifecycle so I am not worried about future in devops.
I wanted to move to devops because I am not so good with programming but can handle identifying and fixing bugs and devops mostly involves working using tools (this is what I heard from couple of my friends in devops role) correct if my understanding is wrong on this.
As most of the members here are on devops, it would be helpful for me if you could give me some insights on this.
https://redd.it/kqdgnk
@r_devops
Deployment Strategies Every DevOps Should Know
Hey guys i wrote an article today and I am interested if I've missed out on anything or if I could improve it since as you can see my articles are more of a reminder for myself, that's why they are not monetized. At least not yet. Let me know what do you think, cheers!
Article
https://redd.it/kqariv
@r_devops
Release Dash - Dashboard for Visualising Commits in Pipelines
I've put together a simple dashboard for seeing what commits/changes are waiting to be released in your pipelines - https://github.com/lobsterdore/release-dash.
This dashboard is useful for teams that have multiple environments with manual gates in between, offering a quick overview of the pipelines of all registered services, I find it useful for preventing a buildup of changes and getting developers to stick to small releases.
The dashboard needs a Github PAT to read repos, each repo needs a YAML config file so the dashboard can figure out what tags needs to be diffed to construct the changelogs. Images are available on Docker Hub (https://hub.docker.com/r/lobsterdore/release-dash) and I've knocked a simple Helm Chart for pushing to Kubes (https://github.com/lobsterdore/release-dash-helm).
All feedback and suggestions are welcome, this dashboard is obviously not intended for lucky people that are releasing straight to production.
https://redd.it/kqb8tl
@r_devops
Dev2Prod
Hey y'all!
I'm a self taught developer. I'm a college student. I've never been into an IT company and worked on anything from scratch.
I've developed a web app. I've written it on MERN stack and with some technologies like Redis, RabbitMQ, Firebase, Algolia and other stuffs. I've eight different services which talk among themselves using RabbitMQ. And Redis to store my active auth tokens in it.
I need some advice from you guys on DevOps. What is the process that happens from development to production? All my service aren't in a docker. I just have repositories for each and every services and my frontend. I am the only developer who developed this web app. So I didn't feel anything hard which coding. I've no CI/CD pipeline or any other sort of it.
Now I need to deploy my product in Heroku. What should I do now? Should I manually deploy every single service or is there any other way to do it? I will migrate from Heroku to AWS in a year mebbe. Just because of financial limitations, I use Heroku for now.
Also now it is in development environment. Should I stage it and then to production or dev env to prod env?
I need some advices from you people on how an IT industry develops a software from scratch to production release.
I'm going to release my app in publicly for everyone to use.
What are the checks should I look before it gets into production and how should I release my app in a production environment?
https://redd.it/kq8tvv
@r_devops
How does one calculate toil and measure toil reduction?
There's lots of talk of toil reduction but, how exactly do we know that we are reducing toil? Is there a quantifiable way of knowing which items to tackle to reduce x% of toil?
https://redd.it/kq1nm2
@r_devops
Does anyone feel that there’s so many CI/CD tools that it’s impossible to keep up with?
I’m still very early on in my career but there are soooo many technologies that comes out on the daily. Being a contractor, i have to learn new tools almost every job. “Hey have you heard of StrumCI? No, we use Jenkins but are moving onto Drone?” I love it too because there’s something for everyone.
https://redd.it/kpuhkd
@r_devops
How To Drastically Optimize Your Software Team’s Workflow
Optimizing your software team’s workflow hinges on the tools you use, an analysis of where you are, and being agile in your forward progression. Read more here.
https://www.codemotion.com/magazine/dev-hub/devops-engineer/optimize-software-workflow/
https://redd.it/kqxw2j
@r_devops
Do I need to configure ssl certs on nginx itself if nginx is inside ec2 instance, which in turn is on loadbalancer which can only be accessed using https?
note: nginx routes traffic to the app inside the same ec2 instance.
I have the ssl certs applied to the application load balancer
https://redd.it/kqjcxx
@r_devops
Solution for cloud assets inventory management.
Hi,
I am looking for solution that will help me manage the inventory of our cloud assets in Azure / AWS. R&D, QA, Solution architects, sales all open assets for POC \\ development \\ testing etc... some forget to terminate the resource after its not needed any more. i tried using tagging but its not clear enough, not centralized enough and I find it hard to enforce users who open new assets to follow the tagging policy (although i see now that AWS does provide such a thing), I thought of using a shared excel sheet where the user would fill the details, but i am sure there is a better solution
I to be able to know who is the owner of the asset
what is for? RnD / POC / Testing etc...
creation date
end date - does it have a date that it could be shutdown or stopped.
get alerts on assets the need to be closed
etc...
Thanks!
https://redd.it/kqu9sb
@r_devops
Help:Automating installation of PostgreSQL & keycloak in Ubuntu & Windows
I am on my first job.My first task is this:Manager is asking me to automate installation & configuration of PostgreSQL & keycloak in Ubuntu & Windows.He asks for a design document.I am completely new to devops and job environment.I don't know where to start.I need to finish this today :(.
What tools are available for this types of tasks?
Should I write script (batch/shell) or should I use some tools?
Please help me get started!
Any link,advice will be very helpful! Thanks
--Noob devops guy
https://redd.it/kqrc70
@r_devops
Cloudformation templates
I am using Cloudformation templates to build a Ec2 instance(https://pastebin.com/MYuc0UU1). Its very time confusing to stop and restart the Ec2 image afer I make a change to the .yml file. This is all for a AMI image creation.
​
Is there a way to spin up my ec2 instance and execute the .yml file to ease my testing?
https://redd.it/kqli0l
@r_devops
A structure for infrastructure repos
After working over many iterations of how to organize Infrastructure as Code in repositories, I've landed on the following general-purpose structure.
It took a lot of experimenting with different forms (many of them more complex and "fancy") to finally end on this. It's as simple as I could make it while also making it easy to reason about and solve some general problems of organizing content.
I've used it with many tools on very large-scale infrastructure, in monorepos and individual repos. I give Terraform as an example, but you can use it for deploying/maintaining all kinds of things.
# env/
This is almost exactly the same as a Unix system's etc/ directory. The difference is env/ describes environments. Know how the 12 Factor App says your configuration should be stored in "the environment"? That's this.
It's also hierarchical. Each directory is intended to "inherit" configuration from a parent directory.
The general structure I follow is env/{environment-label}/{region-label}/ .The {environment-label} typically contains the vendor, product name, and account name.Each directory is composed of json config files.
Example:terraform plan \ -var-file env/aws-myproduct-nonprod/terraform.tfvars.json \ -var-file env/aws-myproduct-nonprod/us-east-2/terraform.tfvars.json
If you're deploying changes to just us-east-2, you can still inherit the variables that apply to all of nonprod. Your configuration is DRY, but you didn't need to do any "templating" shenanigans.
For regionless infrastructure, I use env/aws-myproduct-nonprod/all/{iam,route53,acm}/.You want to deploy regionless stuff separate from region-specific.
The default config file in a directory may be terraform.tfvars.json, and if needed provide overrides like override.auto.tfvars.json . Finally you have a root-module-name.tfvars.json file.In this way you have configuration for each of account-alias, nonprod, us-east-2, and root-module.
This way you can separate and re-use configuration at each level by just passing several -var-file options to Terraform.
To deploy all this, I recommend cd ing into a specific env/ directory and running a generic deployment command, like make plan-terraform or something. Your Makefile has the relative paths back to each config file. It would preserve those paths (like using readlink -f) , then change to one of the app/ directories below, and run terraform plan , like so:
frontend_conf=$(shell readlink -f webserver.tfvars.json) region_conf=$(shell readlink -f terraform.tfvars.json) account_conf=$(shell readlink -f ../terraform.tfvars.json) plan: cd ../../../app/tf-web-frontend/ && \ terraform plan \ -var-file $(account_conf) \ -var-file $(region_conf) \ -var-file $(frontend_conf)
The point behind this is so anybody can deploy anything without needing to actually know anything about how the deploy works. You just change to a directory and run make plan. (Ideally from a Docker container with pinned versions of tools for this repo)
Also, it's important not to reference configuration across hierarchies. Only reference configuration from your current hierarchical tree/level or below. Otherwise you get into dependency issues across environments/regions/accounts. If you have to reference something in some other environment, use something like a Terraform remote state data source.
# app/
This folder is used to store "apps". Think of them as a complete "application", like a Python module, or C program. They are intended to be "run" the same way you would "run" any other application. They have default configuration (that is overridden by whatever's passed in from env/), they take options/arguments/input, and produce output.
Basically this is where a Terraform root module would go - but not a sub-module (see below). All kinds of things can be "apps": Packer configs, Makefiles, Shell scripts, etc. Each directory should be its own complete component.
And no "apps" should depend upon other
DevOps Adoption
Has anyone here followed the Three ways from the DevOps Handbook by Gene Kim?
https://redd.it/kqlwx2
@r_devops
What's your thought about AWS having downtime today again?
Seems like AWS had the second in a row downtime https://downdetector.com/status/aws-amazon-web-services/
Seems like like Notion, Slack, Zoom was affected by that. Seems like solutions like CAST AI https://resources.cast.ai/blog/when-aws-sinks-why-sink-with-it or not independent but still even Anthos might be a solution
https://redd.it/kqi7md
@r_devops
IaaS providers in Europe
Hi All,
​
​
I'm currently looking for an IaaS provider which should be based in Europe and have European datacenters.
Would prefer not dedicated servers but at least a public cloud/VM-based solution.
The goal is for a small one-man startup launching a SaaS product.
So, based on the above, immediately out of scope are: Azure, Google Cloud, AWS, DigitalOcean, Vultr, Linode. As I said: European.
What I experienced myself until now:
\- OVH: have tried their public cloud last year - absolutely terrible with regard to stability and uptime.
\- Hetzner: tried a dedicated server there in the past, not impressed.
Others I've found until now:
\- Scaleway: looks as bad as OVH, would rather stay away from them.
\- Exoscale: looks ok I guess, can't really find a lot of info on their quality.
\- Cloudsigma: looks ok I guess, can't really find a lot of info on their quality.
\- Ionos: seems like a lot of negative info on them online?
\- Leaseweb: also looks like a lot of negative info on them online?
​
So, my 2 questions:
\- of those listed above (not OVH, not Hetzner, not Scaleway): anyone here has any positive experiences with them that they would care to share?
\- are there any other relevant European providers I missed and some here have good experience with?
​
Thanks!
https://redd.it/kqa4bc
@r_devops
why is my package manager missing newer versions?
So I do DevOps at a small company. Part of our build pipeline scans our docker images for vulnerabilities. When it does find them, more often than not I have to go into the Dockerfile and tell apk/apt/yum to upgrade to the version where it was fixed.
Why doesn't apk update && apk upgrade do this for me? My assumption is that some package versions are marked as stable or something so it will only upgrade so far, but I'm curious to know what the actual answer is.
https://redd.it/kqdv6f
@r_devops
Create a Quick and Easy Prometheus Exporter
Create a quick and easy Prometheus exporter is a blog post I recently wrote. Prometheus is a really powerful platform, but it does require a strict metrics format for consumption.
This blog post shows how to quickly and easily write a Prometheus exporter. Let me know what you think!
This also might be helpful for those implementing AWS' new managed Prometheus service.
https://redd.it/kqak3r
@r_devops
If your team does ML, what is your "MLOps" stack?
I'm getting more interested in/involved in machine learning, but the DevOps ecosystem around ML feels... rough, to say the least.
I'm looking for anyone with experience running ML in production. What does your MLOps stack look like? What platforms have you found that you love/hated?
https://redd.it/kqabgb
@r_devops
Need advice on microservices and database access.
I currently have microservices connecting to one database. i have every service connect to the DB with its own role with specific permissions. For example, the auth service has a DB role which has only SELECT access to the "users" table I'm using PostgreSQL btw.
Questions: Is there a better/easier way of doing this? Am I going the right direction here?
I ask because this might be overkill. On the flip side, some may think I'm not going far enough and that I need a different schema for each service.
I've read that to do ms properly, each service should have its own DB. I think that's impractical and a maintenance nightmare. At least for my project it is. So, i'm not going that route. But, I need to somehow isolate activities between the services regarding the DB.
I'm speaking only for production environments.
https://redd.it/kq67ix
@r_devops
Turn existing setup into code
I have an existing setup that I want to turn into code. I need some advice regarding tools and strategy.
What I have:
* Infrastructure
* A couple of linodes
* A bunch of linode block storage volumes
* A bunch of Backblaze buckets (S3-compatible)
* A bunch of somains and a whole lot of DNS resords
I run:
* A few Wordpress sites
* A few other PHP-based applications
* A couple of Ruby on Rails applications
* A MySQL server
* A PostgreSQL server
* Nginx for webservers
* A rather complex mail server (Postfix, Dovecot, OpenDKIM, rspamd, virtual domains, etc.)
* Backups with Duplicity
What I want to achieve:
* I'd like to turn this setup into code so that I could nuke all (or part) of it and recreate with a single (or at least very few) command.
* I'd like to be able to move things around relatively easily. Ideally, tools would be able to move data around, too. E.g. if I move MySQL db from one linode to another it would make sure MySQL server is set up there and would move data from the old server to the new one, and maybe remove MySQL server from the old linode if nothing uses it there any more. Is this even a thing?
* I'm fine with changes that would make making changes easier.
* Ideally, I don't want to add much complexity/cost on top the existing setup (apart from my time). That sort of disqualifies configuration servers, turning everything into containers or moving to AWS, etc.
I have very little exposure to DevOps tools but I'm willing to learn. The question is what to learn.
https://redd.it/kpia73
@r_devops
Need guidance on approach for CI/CD for ECS + Docker
Hi All,
A friend and I (software developers) started building out an app on the side and decided to go with angular/node stack with AWS infrastructure. I'm thinking of using Docker with ECS to containerize and host the application. I'm looking for a free / open source build / deploy tool that will pair well with this stack. Ideally, the flow would be once master is merged, the tools recognize the merge and deploy a new container within ECS. I may be bastardizing the flow but I've just started looking into docker so I'm not super familiar with it. Any feedback would be appreciated.
https://redd.it/kpoagd
@r_devops