Flip side of the coin, I had a sysadmin who wouldn’t increase the tmp size from 1gb because ‘I don’t need more than that recommended size’. I deploy tons of etl jobs, and they download gbs of files for processing to this globally known temp storage. I got it changed for one server successfully after much back and forth, but the other one I just overrode it in my config files for every script.
Yeah python does lack in such things. Half a decade ago, I setup an ml model for tableau using python, and things were fine until one day it just wouldn’t finish anymore. Turns out the model got bigger and python filled out the ram and the swap trying to load the whole model in memory.
During the pandemic I had some unoccupied python graduates I wanted to teach data engineering to.
Initially I had them implement REST wrappers around Apache OpenNLP and SpaCy and then compare the results of random data sets (project Gutenberg, sharepoint, etc..).
I ended up stealing a grad data scientist because we couldn't find a difference (while there was a difference in confidence, the actual matches were identical).
SpaCy required 1vCPU and 12GiB of RAM to produce the same result as OpenNLP that was running on 0.5 vCPU and 4.5 GiB of RAM.
2 grads were assigned a Spring Boot/Camel/OpenNLP stack and 2 a Spacy/Flask application. It took both groups 4 weeks to get a working result.
The team slowly acquired lockdown staff so I introduced Minio/RabbitMQ/Nifi/Hadoop/Express/React and then different file types (not raw UTF-8, but what about doc, pdf, etc..) for NLP pipelines. They built a fairly complex NLP processing system with a data exploration UI.
I figured I had a group to help me figure out Python best approach in the space, but Python limitations just lead to stuff like needing a Kubernetes volume to host data.
Conversely none of the data scientists we acquired were willing to code in anything but Python.
I tried arguing in my company of the time there was a huge unsolved bit of market there (e.g. MLOP's)
Alas unless you can show profit on the first customer no business would invest. Which is why I am trying to start a business.
oh god i felt this one. Devs too busy, incompetent or just plain lazy to figure out why their code is so slow, so just have ops throw more CPU and memory at it to brute force performance. Then ops gets to try to explain to management why we are spending $500k per month to AWS to support 50 concurrent users.
The sad thing: Throwing hardware at a problem was actually cheaper for a long time. You could buy that $1500 CPU and put it in your dedicated server, or spend 40 developer hours at $100 a pop. Obviously I’m talking about after the easy software side optimizations have already been put in (no amount of hardware will save you if you use the wrong data structures).
Nowadays you pay $500 a month for 4 measly CPU cores in Azure. Or “less than 1 core” for an SQL Server.
Obviously you have a lot more scalability and reliability in the cloud. But for $500 a month each we had a 16 core, 512 GB RAM machine in the datacenter (4 of them). That kind of hardware on AWS or Azure would bankrupt most companies in a year.
Well, having been on the other side, sometimes the Dev is also trying to fight the good fight whilst having to use some crap 3rd party system/library that’s imposed from above because somebody at the C-suite level after suitably dinned and wined (and who knows what more, including implied or even explicit promises for the future of their career) signed a massive agreement with one of the big corporate software providers so now those of us at the coalface have to justify to money spent on that contract by using every POS from said big corporate software provider.
I mean, I might be exagerating the overtly corrupt nature of the deal (in my experience its more a mix of CTO incompetence - or being pretty much powerless at the C-Suite level because his is not the core business, hence overriden - and the high-level management trading favours using company money and more for personal rather than corporate reasons) but even competent devs that know their thing can’t really do much when they have to use a bug-riddled POS massive framework from some vendor that doesn’t even have proper support, for “corporate reasons”.
I got somebody at the C-suite level fired after I presented evidence of him wining and dining with a shit supplier (actually being buddy buddy and literally dining with him on a weekly basis), also for not knowing the consequences of his decisions and also for him bring unable to keep his hands off employees below him (me included).
Within 3 months there were 5 severe complaints against him with the CEO and humans resources.
The company had whistleblower protections but obviously fired me for my troubles as well anyway.
I don’t care, the fucker was evil and the company honestly too and I’m happy I’m gone there.
It sucked in the moment, but now I’m more than fine with it. I see the company for what it is now, quite evil and a detriment to society. I’m happy I’m gone there.
I know of a certain big company that has a bug report UI in one of their main products that literally goes nowhere, it used to go to a table in the db, then they removed the table since it was not really used and they wanted to get more storage, so someone quick fixed the bug report to go to a Google sheet
Nobody really checks that sheet and it is not automated or used for anything at all, the person that created that sheet was also deprovisioned some time ago.
Also since many things have changed the only thing that is pulled to the sheet is the first field to specify the category of the bug, all the descriptions, files, photos, logs and more granular items are just not going anywhere.
When this was reported the only reply from management was “QA will check it later” this was 2 years ago so…yeah
It’s a legal complaint. Someone is going to get fined, likely thousands of dollars, if the complaint is substantiated. I strongly suspect a human will be reading the whole thing more than once, before proceeding to gather much more info.
programmer_humor
Newest
This magazine is from a federated server and may be incomplete. Browse more on the original instance.