Perfect, now you just have to wrap your program inside a debugger in production!
We test AND develop in production. Get on my level.
There’s a name for that: DEVELOPMESTUCTION
One of our customers does that. It happened multiple times already that one dev fixed an issue in production, and the next regular deployment overwrote everything.
But fortunately, it’s just critical infrastructure and nothing important.
When I left my last job they were using the zip file method for version control and one creative developer managed to link two versions of libc at the same time.
Software is so useful that the standard for utility is extremely low.
Aren’t those almost always race condition bugs? The debugger slows execution, so the bug won’t appear when debugging.
I had one years ago with internet explorer that ended up being because “console.log” was not defined in that browser unless you had the console window open. That was fun to troublshoot.
Turned out that the bug ocurred randomly. The first tries I just had the “luck” that it only happened when the breakpoints were on.
Fixed it by now btw.bug ocurred randomly.
Fixed it by now btw.
someone’s not sharing the actual root cause.
I’m new to Go and wanted to copy some text-data from a stream into the outputstream of the HTTP response. I was copying the data to and from a []byte with a single Read() and Write() call and expexted everything to be copied as the buffer is always the size of the while data. Turns out Read() sometimes fills the whole buffer and sometimes don’t.
Now I’m using io.Copy().Note that this isn’t specific to Go. Reading from stream-like data, be it TCP connections, files or whatever always comes with the risk that not all data is present in the local buffer yet. The vast majority of read operations returns the number of bytes that could be read and you should call them in a loop. Same of write operations actually, if you’re writing to a stream-like object as the write buffers may be smaller than what you’re trying to write.
Ah yes… several years ago now I was working on a tool called Toxiproxy that (among other things) could slice up the stream chunks into many random small pieces before forwarding them along. It turned out to be very useful for testing applications for this kind of bug.
I’ve run into the same problem with an API server I wrote in rust. I noticed this bug 5 minutes before a demo and panicked, but fixed it with a 1 second sleep. Eventually, I implemented a more permanent fix by changing the simplistic io calls to ones better designed for streams
The actual recommended solution is to just read in a loop until you have everything.
I had a bug like that today . A system showed 404, but about 50% of the time. Turns out I had two vhosts with the same name, and it hit them roughly evenly 😃
Had a similar thing at work not long ago.
A newly deployed version of a component in our system was only partially working, and the failures seemed to be random. It’s a distributed system, so the error could be in many places. After reading the logs for a while I realized that only some messages were coming through (via a message queue) to this component, which made no sense. The old version (on a different server) had been stopped, I had verified it myself days earlier.
Turns out that the server with the old version had been rebooted in the meantime, therefore the old component had started running again, and was listening to the same message queue! So it was fairly random which one actually received each message in the queue 😂
Problem solved by stopping the old container again and removing it completely so it wouldn’t start again at the next boot.
sometimes it’s also bugs caused by optimizations.
And that’s where Release with debug symbols comes in. Definitely harder to track down what’s going on when it skips 10 lines of code in one step though. Usually my code ends up the other way though, because debug mode has extra assertions to catch things like uninitialized memory or access-after-free (I think specifically MSVC sets memory to
0xcdcdcdcdon free in debug mode).
I always thought it was Cary Elwes.
He worked for the gaming site/podcast “Giantbomb” years ago. Pretty sure the image macro is pulled from one of their podcast videos.
this happens with so many scripts I’ve tried to debug with strace because strace requires to run as root or sudo which elevates the niceness of process which prevents certain errors from occuring when the script is run with root permissions and so it runs flawlessly without bugs and you sit wondering wtf
Ya, fuck legacy aggrid
Haha, heisenbugs, always a fun time.
More seriously, I’d be surprised if this wasn’t a classic race condition
It wasn’t :D
See my comments below.Well, technically it was a race condition. Just one between two different programs.
One of the worst words in the English language is “intermittent.”
Just run your prod env in debug mode! Problem solved.
Lol my workplace ships Angular in debug mode. Don’t worry though, the whole page kills itself if a dubious third-party library detects the console is open. Very secure and not brittle at all!
Please send helpNow I’m curious how this detection would work.
I’ve seen some that activate an insane number of breakpoints, so that the page freezes when the dev tools open. Although Firefox let’s you disable breaking on breakpoints all together, so it only really stops those that don’t know what they’re doing.
Blink-blink-blink. Blink. Blink. Blink. Blink-blink-blink.
No, I don’t have something in my eyes, I swear I’m fine looks nervously at boss.
Hang tight help is on the way.
You can imagine how many node projects there are running in production with
npm run. I have encountered js/ts/node devs that don’t even know that you should like, build your project, withnpm buildand then ship and serve the bundle.I just died a little inside. Thank you.
i have absolutely seen multiple projects on github that specifically tell you to do “npm run” as part of deploying it.
The term is Heisenbug
Behold, https://rr-project.org/
I once had a bug in a C# program I wrote. It made a HTTP request and if the user agent was left to default (whatever that was), the server just gave back an empty string as a reply. I took way to long until I understood what was going on and I kept chasing async, thinking I had messed it up some how.
I found the solution, we’re running debug builds in prod from now on
Heisenbug. Nasty buggers, especially in my domain: Embedded Engineering. When you are in the debugger, the whole processor is stopped, missing tons of data coming in, missing interrupts, getting network timeouts, etc. More often than not, resuming makes no sense, and you have to get straight to reboot.
“You don’t debug embedded” ~my brother, who’s been working in embedded for almost 15 years
That’s why I work with an extraordinary diligence to avoid making errors from the very start. Debugging is only a measure of last resort.
Sound like a critical race condition or bad memory access (this latter only in languages with pointers).
Since it’s HTTP(S) and judging by the average developer experience in the domain of multi-threading I’ve seen even for people doing stuff that naturally tends to involve multiple threads (such as networked access by multiple simultaneous clients), my bet is the former.
PS: Yeah, I know it’s a joke, but I made the serious point anyways because it might be useful for somebody.
This is why we shouldn’t ban Critical Race Theory.
Yeah! Nobody uses CRT monitors anymore.
Lazy load exception anyone?
