"Microsoft hasn't really been helpful in trying to track this, either. I've sent over logs and information, but they haven't really followed this up. They seem more interested in closing the case.”
That's the Microsoft way: ignore the bug report for a month or two then close the case for "inactivity".
It's sometimes the issue with relying on metrics and stuff and being purely quantitative. A lot of us of have worked at companies where it's been like this. To deal with volume they need to rely on numbers to gauge so you tell the workers they'll be ranked on closed cases.
With these updated routing tables, a lot of people were unable to make calls, as we didn't have a correct state
You're relying on windows for critical infrastructure? Are you nuts?
Linux can also die in weird ways...
It's just that Windows is more prone to some issues.
Indeed, nothing is perfect, but closed source stuff doesn't provide a lot of recourse. If you have a linux expert in your team, they can investigate and if need be even dig into the code of linux itself to find the core issue. Microsoft doesn't provide anything even remotely similar.
How many dev teams have a kernel dev on them?
Don't need one. If you can read C/C++ you can read the kernel code. And in most cases, you won't have to, as the problem is probably in a component in the distro. Those are written in python, ruby, or bash, which are all much more readable than C/C++.
No such luck on windows
I worked at a small company without a kernel dev and we periodically looked into the code to solve problems. I don't know how much we upstreamed, but we relied on Linux so it was either the or try to get someone on the mailing list to care.
It's really not that hard to look through the kernel source, it's pretty well written and documented. It's a lot harder to be a kernel developer writing new code, but finding bugs and contributing fixes isn't that bad.
You don't have to be a kernel dev for that. Reading source code is much easier than writing it. I myself have even read the code that handles the battery management drivers, and it's mostly self documenting, even though I'm bad at C and it's pointers, and also have never yet written a kernel driver.
The US navy ran on windows xp for so long that they paid Microsoft to continue maintaining it after EOL.
"feature"
This is the best summary I could come up with:
A few months ago, an engineer in a data center in Norway encountered some perplexing errors that caused a Windows server to suddenly reset its system clock to 55 days in the future.
“With these updated routing tables, a lot of people were unable to make calls, as we didn't have a correct state!” the engineer, who asked to be identified only by his first name, Simen, wrote in an email.
Simen had experienced a similar error last August when a machine running Windows Server 2019 reset its clock to January 2023 and then changed it back a short time later.
Windows systems with clocks set to the wrong time can cause disastrous errors when they can’t properly parse timestamps in digital certificates or they execute jobs too early, too late, or out of the prescribed order.
The mechanism, Microsoft engineers wrote, “helped us to break the cyclical dependency between client system time and security keys, including SSL certificates.”
Simen and Ken, who both asked to be identified only by their first names because they weren’t authorized by their employers to speak on the record, soon found that engineers and administrators had been reporting the same time resets since 2016.
The original article contains 701 words, the summary contains 200 words. Saved 71%. I'm a bot and I'm open source!
Ignoring tickets and than closing it for inactivity is how big companies ignore their own fuck-ups.
I've read the documentation on that feature, and still don't get over it. How can anyone with knowledge of computers be so dumb to even consider such an idea, lest implement it?
This feature is just a BIG flag waving "AbUsE mE!"
My guess, since Microsoft has trouble replicating this effect and it's mostly related to internal servers, is that there is some kind of middlebox that messes with TLS traffic (one of those plug&play "HTTPS firewalls" or whatever they're called). Because of potential interception, every device on the network has to trust these boxes completely, potentially landing their certs in the "super duper secure" key store rather than the third party CA key store they should be in.
It's also possible one of Microsoft's servers has gotten the time wrong or has a bug in its TLS library, but I think MS would've caught that already.
STS is actually a pretty good idea because NTP attacks are pretty reliable in practice. All you need is a computer on a shitty ISP that doesn't filter outbound traffic right (allowing spoofing of reliable NTP servers) and spamming NTP packets with fake origins, pretending to be the servers for the real NTP pool. This way you can slowly force the clock to drift (limited by the standard maximum NTP offset each time) and get your outdated credentials to work again, get your old certificates and signatures to work again, and do other kinds of time-related attacks.
NTP can't be done over TLS the same way DNS can be protected because TLS has certificates with limited validity and you don't know the time you should use to validate them. Using TLS traffic from known-good servers is pretty smart, since every TLS handshake already sends back and forth the current system time anyway.
If you read the article it's explained that some SSL implementations put random data in the time field (OpenSSL was given as an example). Microsoft knows about this and so needs a certain number of closely matching timestamps to be confident about the new time to change the system time. However, if you get particularly unlucky with a string of random timestamps that match, you end up with a random time.
Yes, it’s a dog shit implementation to rely 3rd parties to make guarantees about their data that they never agreed to.
Linux and MacOS handle this just fine. Why blame SSL when you’re the one using it wrong?
And most NTP clients already handle this by not changing the time automatically if it would be too much of a jump. Microsoft is trying to fix what's not broken.
I was under the impression that this system mostly used Microsoft services, but if they take all TLS handshakes as input then that does make more sense. The fix also seems rather obvious (only use Microsoft connections with certificate pinning for time source input, since Windows will talk to Microsoft every minute anyway).
Even if openssl/boringssl was spec compliant as Microsoft assumed (hasn't been for almost a decade of course) you'd still want to make sure the data is all close enough just in case you hit a few misconfigured servers. Otherwise you'd get a viral effect where badly configured servers start infecting each other as they communicate.
Fun fact: Apparently M$ laid off their QA team for Windows so if you're wondering why updates break so much, that's why.
Makes sense. I had to trouble shoot my WiFi card for like an hour yesterday to get it to work after a Windows Update killed it for a bit.
Well, now you know why.
And when they laid off their QA team with the testing lab of thousands of unique computers, they replaced it with VMs and AI. Because VMs are a totally good way to troubleshoot very specific bugs. The AI part is used to supposedly figure out when you're "idle" so what Windows can update.
Imagine needing AI to update a computer lmao
“The false assumption is that most SSL implementations return the server time,” Simen said. “This was probably true in a Microsoft-only ecosystem back when they implemented it, but at that time [when STS was introduced], OpenSSL was already sending random data instead.”
This is so amazing, NTP is too insecure, so we relied on random data from random servers instead
Ok, but can we talk about the absolute doorknob who made that graphic? The clock has two different mounting points for the hands. Come on, now.
That's the gag, I think. The clock is fundamentally broken.
Last problem I had was our server serving 2FA decided it was not on the same timezone, so when I tried to connect with my Authenticator code, it says "check your time on your cellphone". I had to call IT...
Companies still using windows are causing problems
Sounds like an interesting idea. Pity MS can't be bothered to iron out the issues with it.
This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.
Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.
Rules:
1: All Lemmy rules apply
2: Do not post low effort posts
3: NEVER post naziped*gore stuff
4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.
5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)
6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist
7: crypto related posts, unless essential, are disallowed