-
Reddit SystemAdmin. Thanks @reddit2telegram and @r_channels.
Certificates rant
So, yeah, I'm admin, have been since 2000, but I do dba work mostly, so no experience in certificates. Now I have to replace the expiring certificate for the mail server. What a pain in the ....
Please provide a CRS. WHAT? Ok it's an application for a certificate. Looked up a documentation how to do it, but it wouldn't work. The properties window of the domain simply won't open. Ok, use the tool of the certification website. Then nothing happens. Support: OK, you need to validate it via mails we sent to your mailbox(es). Which ones? Ok, here they are, tried to validate them: lots of error messages, damn it. Ok, we sent several, you don't need all of those. WHAT? Now pu 'em into place on your mail server and firewall.
How I miss writing some SQL scripts.
https://redd.it/1po0vov
@r_systemadmin
Chrome AI is taking ~4GB per user on our RDS servers
We just discovered that Chrome’s AI features are using around 4GB of disk space per user on our RDS servers.We were wondering why our RDS disk space had been decreasing so quickly lately. So we ran a quick TreeSize scan and came across this strange Google folder.
I’ll point you to this post where we learn that it’s yet another AI-related issue ! https://www.reddit.com/r/chrome/comments/1jslb22/optguideondevicemodel\_folder\_taking\_up\_3gb\_have/?tl=fr
https://redd.it/1pnzuqb
@r_systemadmin
The return of 8GB RAM laptops (RAM mayhem) - Good luck with your Service Desk
As everyone already probably know, RAM situation is only getting worse. This means that in the near future a lot of companies will be relying on entry-level workstations (laptops) featuring the absolute minimum amount of RAM. Many of us are aware what happens once you run Windows 11 with Office applications, Outlook and a browser with bunch of opened tabs .
The reason why I'm posting this is that if this becomes a reality many Service Desks will be full of complains how everything is slow and tech support have no clue how to resolve the situation.
https://wccftech.com/you-might-soon-see-8gb-laptops-everywhere/
Good luck to everyone related to Service Desk responsibilities.
https://redd.it/1pnvte5
@r_systemadmin
Tired of working in IT
I’m just really tired of working in IT, been doing it for 11 years now. Exhusted and just struggling and feeling like giving up.
https://redd.it/1pnkq7w
@r_systemadmin
Microsoft, if you're going to send us powershell commands, at least check them for accuracy first.
Just got an email from MS about the retirement of Activesync 16.0 and below in march. Nice that microsoft included an exchangeonline powershell string to quickly assess which devices might be impacted.
Except the string / query doesnt work because its not written properly.
I was able to fix the glaring issues quickly without any help from AI.
Original string sent to us my microsoft. Am I crazy?:
Get-MobileDevice | Where-Object {($_.ClientType -eq 'EAS' -or $_.ClientType -match 'ActiveSync') -and $_.ClientVersion -and (version$_.ClientVersion -lt version'16.1')} | Sort-Object UserDisplayName | Select-Object UserDisplayName, UserPrincipalName, DeviceId, DeviceModel
Fixed:
Get-MobileDevice | Where-Object {($_.ClientType -eq 'EAS' -or $_.ClientType -match 'ActiveSync') -and $_.ClientVersion -lt '16.1'} | Sort-Object UserDisplayName | Select-Object UserDisplayName, UserPrincipalName, DeviceId, DeviceModel
https://redd.it/1pniamu
@r_systemadmin
Told to purchase AI licensing because the board members want it.
Company created an "AI" taskforce which includes myself. Was told to find how employees are using AI and come up with a gameplan. After inquiring with employees I find that they're only using it to edit documents and don't need any upgraded licensing with it. Propose guidelines etc. and inform management upgraded licensing isn't needed and would cost ~10K/month if purchased. Apparently the board members really want to see us using AI and am told $10K/month is worth it to keep them happy.
Not my money but we're still in the start up phase and blowing through cash. I wonder how much money is wasted on things company wide because the board wants to see it regardless of if it's necessary.
https://redd.it/1pne8bf
@r_systemadmin
Notepad++ fixes flaw that let attackers push malicious update files
Didn't see this posted here but a lot of people use N++, so I thought it worth mentioning. I believe they had another malware issue a few years ago.
https://www.bleepingcomputer.com/news/security/notepad-plus-plus-fixes-flaw-that-let-attackers-push-malicious-update-files/
https://redd.it/1pn8vro
@r_systemadmin
Bought RAM in October to dodge price spikes… now I have to return it because “year-end optics”
Back in late October, I saw leaks on X/Twitter about upcoming RAM price hikes. So I did the smart thing: ordered extra RAM for workstations and laptops, delivery scheduled for December. Prices were great back then.
Fast forward to now: prices have tripled in some cases. My order arrives, I’m feeling good for saving the company a good amount of money.
Then accounting steps in:
>“We can’t spend anything in December, it makes the year-end numbers look bad.”
So now I’m sending back perfectly good, dirty cheap, already delivered RAM because optics. And if we reorder next year? We’ll pay 2–3× more. Brilliant.
Just some galaxy-brain financial engineering I’ll never understand, i guess?
Not my money, not my stress. No rant. I’ll just drink my tea (black with milk) and move on. Luckily, I bought some RAM for myself too.
Now I’m heading into vacation — wishing everyone a stress-free time and happy holidays!
https://redd.it/1pn5q0i
@r_systemadmin
Is it just me, or are we spending more time reverse-engineering how our own systems work than securing them?
The deeper I dig into our environment, the more it feels like half the job now is figuring out what our systems are really doing - not what the docs say they do, or what teams think they do.
Data moving between services nobody remembers, SaaS connectors doing silent jobs, internal automations with no clear owner…
Lately it feels like the real challenge isn’t new threats, it’s understanding the system-of-systems we’ve accidentally built.
Anyone else dealing with this?
https://redd.it/1pn3wx1
@r_systemadmin
i feel like chatgpt is shrinking my skills
Before when I had to run that one basic task/command/scripting thing I didn't fully remember I would have to either: google it, dabble thru man pages/help commands, get grilled on an IRC server/stack overflow by some elitist. And then burn that shit into my memory.
Now I just chatgpt it, ezpz no grilling. But also if I have to write an entire script that I KNOW how to write it correctly(given enough time and patience) I'll just hand it off to chatgpt.
https://redd.it/1paya84
@r_systemadmin
Task Scheduler Status
I'm trying to add/fix a custom task I had for Task Scheduler. A problem arose before where the task itself was not appearing in Event Viewer. In the limited searching of answers, I ended up deleting the task through File Explorer (C:/Windows/System32/Tasks/<task>), and deleted the associated registry keys in TaskCache/Tree and TaskCache/Task.
So the problem of Task Scheduler complaining about the task is over, but when I create a new task with the same exact name as the original (let's say "Backup Data"), it will then create, but not appear in Event Viewer, and looking through schtasks in CMD, it says the Status is N/A, which is probably why Get-ScheduledTasks in powershell complains about a parameter being incorrect.
How do I fix this issue? Any help is appreciated!
EDIT: Some additional info, looking at Event Viewer, this is something that came up with creating the task:
Task registered task "\\Backup Data" , but not all specified triggers will start the task. User Action: Ensure all the task triggers are valid as configured. Additional Data: Error Value: 2147942583.
https://redd.it/1pax1gv
@r_systemadmin
How are you actually managing container vulnerability chaos at scale?
Our security team just dumped a report showing 500+ critical CVEs across our container fleet and wants everything patched immediately. Half are in base OS packages we don't even use, others are in dependencies 3 layers deep.
Currently running Trivy in CI but it's basically crying wolf on everything. Devs are getting frustrated with blocked builds over theoretical vulns while actual exploitable stuff gets lost in the noise.
Looking for real-world approaches that have worked for you:
How do you prioritize what actually needs fixing vs noise?
Any tools that give exploit context or EPSS scoring?
Automation workflows that don't break dev velocity?
Base image strategies that reduce your attack surface from the start?
Any advice would be appreciated.
https://redd.it/1pass85
@r_systemadmin
What's your process for technical vendor evaluations?
I'm leading a platform evaluation for my team and trying to improve our process. Currently we're looking at [category\] tools and I'm finding it takes way longer than it should.
Our current approach:
\- Download spec sheets/docs from each vendor
\- Manually pull key specs into a spreadsheet
\- Try to normalize different terminology
\- Takes 4-6 hours minimum
What does your evaluation process look like? Any frameworks or approaches that have worked well? Especially curious how larger teams handle this.
https://redd.it/1pamzx4
@r_systemadmin
Just caused my first massive outage
Hello everyone,
I just got promoted to a new role 3 weeks ago. I made a new deployment to one of our tools and everything just crashed it caused an outage for around 12 hours!
There was nothing wrong with my deployment, and I was following the process word by word. The system just crashed all of a sudden and we believe it was a firmware bug.
I am still worried and scared of doing any more changes, I still haven't heard anything from management. I am filling some big shoes in the new role and my manager told me that when I started they believed in me being able to learn quickly so I got promoted. Everyone has been really supportive but I am still scared worried that I might not have a job next week...
How do you deal with those feelings? My manager told me numerous times that it's ok as we didn't do anything wrong, but I am still worried, how should I deal with the paranoia, or should I really be worried?
https://redd.it/1pan2u4
@r_systemadmin
Event ID 500, Desktop Window Manager uses too much VRAM. (I hope I'm in the right place, forgive me if not)
Have a wonderful day, everyone!
I'll just say this upfront, and I don't mean it disrespectfully, but I really need someone who does system integration or someone who's deeper into this stuff.
I've got result ID 500 in the Event Viewer under Diagnostics Performance:
"Video memory are over utilized and there is trashing happening. Reducing number of running programms and open Windows may help resolve this"
The task category is Desktop Window Manager Monitoring. So, that's where the error must be.
I have the current drivers.
Or rather, the newest one doesn't work because Adrenalin won't start, so I'm using the October version.
I've also tried almost all the settings that are supposed to fix the DWM (Desktop Window Manager) problem, but it's possible I'm missing something.
What's weird is that sometimes I'm not doing anything on the desktop, and my graphics card is using 12-14GB of VRAM even though I'm not doing anything. I've also made a few logs with HW info, and when it's running normally, like with YouTube or Twitch, it uses about 3-4GB of VRAM.
I have the feeling that the graphics card is also being throttled because I often have a pretty weird render distance in games. But when I look at the clock speeds, they're normally up at 3100mhz (9070xt Nitro+)
I've also reinstalled Windows multiple times. Still get event 500 every time.
I really need help, I've spent almost 100 hours on this and can't get rid of it.
https://redd.it/1pai4c9
@r_systemadmin
How do you secure multi tenant Kubernetes clusters with minimal images?
We run multiple tenants on the same cluster. Using minimal images reduces vulnerabilities, but I'm concerned about isolation between tenants. What patterns or tools do you use to maintain security and prevent lateral movement?
https://redd.it/1pnze13
@r_systemadmin
Weirdest queries at Service Desk
What are some of the weirdest queries that you encountered working at the service desk?
I had a suicidal man come to the desk. I left my station to be his friend for the day. I did enough to make him feel better. Seeing him smile was a relief, and he thanked me for spending the day with him.
Team leader was an ass about it though.
https://redd.it/1pnwz37
@r_systemadmin
DNS entries for gateways, vlans and management ports?
Edit: Resolved. I think he may be asking for reverse lookups but is a little confused. I'll still have to work out some way to resolve internal management IPs without exposing them to clients though.
TLDR; Anyone ever heard of giving DNS entries to gateways and unreachable management ports.
I have a cyber security guy insisting that he needs DNS entries to be added for all sorts of strange things. This is a windows AD environment for reference.
Off the top of my head he wants a DNS entry that represents the default gateway of each vlan and a corresponding dns entry for the management port of the network hardware. Except, the network management ports exist in a vlan that is unreachable from the regular network the DNS exists in. Additionally, he has asked for DNS entries for storage devices that are the backend for our VM environment, which are also unreachable from the regular network. You'd need a jump box.
According to him he needs this information for reports that come out of a security scanning server. The security server actually can talk to all those vlans to interrogate the systems so it seems like he is using DNS as a sort of labeling system. The security server will then have the DNS entry in the report to show what the IP represents... for some reason.
If you can't tell this is for unspecified government work. I have never see DNS used this way, am I crazy for pushing back on this? It seems really weird to give a gateway its own name in DNS.
https://redd.it/1pnmn78
@r_systemadmin
Robin and OfficeSpace pricing
I’m trying to compare Robin vs. OfficeSpace for hot desking and room booking and just want a general idea of pricing but I’m struggling to find info on their pricing. I’m not looking for an exact quote because I know that would require a sales call and I’m more at a research stage. Just trying to understand if these tools are more budget friendly or enterprise so I can compare them and move on.
If anyone knows ballpark pricing for either one, I’d really appreciate it. Open to other tools too if they’re more upfront about costs and I can take some notes right away..
https://redd.it/1pnhe3g
@r_systemadmin
You guys ever just not contact vendor support because you're tired of their terrible troubleshooting?
I've literally set up an email template when I work with a particular vendor because they ask for tons of the same details every single time.
I'm tired, boss. I'll just work through the issue this time because I don't have the energy to deal with the email chain back and forth.
https://redd.it/1png22c
@r_systemadmin
Server Room Cooling Systems
For those of you familiar with the planning for your data room/server room: Do you add your AC Units to the UPS circuits? How do you protect your AC units from power fluctuation and outages before the generator comes on?
https://redd.it/1pn80x5
@r_systemadmin
December is like a year in 30 days
Every vendor: we need to roll out new breaking features now, did you make those urgent changes yet?
Contracts: all renewing now
Employees: Hey remember that important ticket I stopped responding to in May? It needs to be completed by next week.
Management: we need a POC for a new system, can you bang it out next week?
HR: You have 20 PTO days you're losing at the end of the year...
Anyone else really hate December? All I want to do is clean up my desk, wrap up projects and reset for next year, but it never happens. Every year its just literally more everything in the 3 usable weeks of December.
https://redd.it/1pn9bna
@r_systemadmin
best helpdesk software for a tiny it team that is barely keeping it together
so i just got promoted to lead support at our tiny company and suddenly i am the person everyone comes to when slack or email explodes. we dont have anything set up for tickets or tracking issues right now. its all just replies in slack threads and sometimes i forget things and then someone reminds me a week later. its chaos.
i know helpdesk software is supposed to help with that but there are sooo many options and i literally have no idea where to start. we are like 10 people total, and support tickets are not crazy huge volume yet but it feels like it might hit us soon. i dont want something that feels like too much overhead or that i need a phd to understand.
for folks using helpdesk tools what do you actually like about yours? is there stuff you never use or features that seemed cool but ended up annoying? also how steep was the learning curve for your team? did your customers notice a change once you switched?
i also worry about setup time since i have to do this between answering real support questions. how long did it take you to get everything up and running? any tips to make that easier? thanks in advance
https://redd.it/1pn3omb
@r_systemadmin
Domain controller upgrade
Hi, I currently have a few domain controllers running on Windows Server 2016. I want to upgrade them to Windows Server 2022 using new hardware and then retire the old servers. All of the domain controllers are in the same domain and within a single forest. What would be a reasonable cost for an MSP to handle this upgrade?
https://redd.it/1pb16vp
@r_systemadmin
Our country is down
Our TLD (.vu) has gone offline. That's the country of Vanuatu.
Apparently GoDaddy is the registrar for .vu. As much as people crap on them, I wouldn't look there first for the cause. I would guess that whoever pays the bill for .vu, forgot to do so. That can't be quite right. According to digwebinterface.com, there are a handful of .vu domains that have records still, but most only return an SOA. So maybe someone at Godaddy did fat finger it, and deleted most .vu domains? I don't care. I just want it working again.
Contacting GoDaddy support is comedy gold. Can't get past level 1. They won't escalate. They can't get it into their heads the scope of this thing.
* Me: The entire .vu TLD is unavailable. Godaddy is the .vu TLD registrar.
* GoDaddy: To assist you further, we will need to check your account and website. I have sent a one-time code to the registered email address on your account for the validation process. Can you please help me with that code?
* Me: Can't do that since .vu is down our ********.vu email and web sites are also down.
* GoDaddy: I see, but we haven't received reports of similar errors from our other customers using this extension. To assist you further, we will need to check your account and website. For that first, we need to validate your account.
* Me: (Sigh)
Anyway, all you guys who think you've blown it because you took down the corporate DHCP server, give yourselves a break. This is next-level.
https://redd.it/1paytf4
@r_systemadmin
Little advice for a guy recently laid off, looking to update skills
Hey guys, like it says, laid off from a job I was sr admin and responsible for sccm, Citrix, DR/Backuos using Commvault. I have 25 years experience in everything from Cisco to all Windows stuff. As a guy in his 50’s I decided to go for a few certs while I had the time. (Not a lot of hiring in Q4)
I’ve started SSCP as a mid level security cert, was doing CCSP but I don’t have the year of actual cloud security. In addition I’m going after AWS and Azure certs. If there was an AI cert for agentic or generative AI I’d be interested in that.
Does sound like a solid plan?
https://redd.it/1pawr1f
@r_systemadmin
Riverbird RMM
Hey everyone,
Do any of you from Riverbird use the RMM and use it for monitoring and RMM? Would you like to hear your experiences?
We want to use it as an MSP for our customers and replace ATERA.
https://redd.it/1paqfcv
@r_systemadmin
Windows Event Collector freezing - suggestions?
Hi, and thanks in advance:
I was brought to a Windows Event Collector server, getting events from 2.5K endpoints. It is set to send fowarded events to c:/default-really??, and to rewrite itself after 20MB of data processed. Splunk Universal Forwarder is installed on the server to ingest stuff to Splunk.
Event logs on the server have nothing really useful (Com service (in Korean?) failed to start...) and the forwarded-log-file states last updated about 10min after the last event in the log.
I have not had a chance to see the server running after reboot to check resource use, and apparently after being rebooted - it runs 2-3 days before freezing the Windows Event Collector service so badly it cannot be stopped from the services menu.
The only ting I can think of (after glancing at it), is perhaps an interaction between Splunk UF, and the forwarded log getting full.
If anyone has suggestions: Thanks.
If not, Hope you had a good weekend.
Semi Ninja Edit
The Forwarded Event log states that there are ~2650 endpoints reporting, and the registry has under 3K hives in it.
https://redd.it/1pap4gq
@r_systemadmin
Power of VSCode Editor
TIL you can open an entire folder of scripts in VSCode and do a quick Replace of a search string for all scripts in that folder. I’m sure many of you already knew about this, but it sure saved me a few hours of work.
https://redd.it/1paio0g
@r_systemadmin
Switching from LDAP to LDAPS — how bad is the migration?
Our cybersecurity team just told us to disable LDAP and move to LDAPS. Anyone else dealing with this?
https://redd.it/1pagz4t
@r_systemadmin